LoginSignup
1
0
AUTOSAR CountdownAdvent Calendar 2022Day 15
@kaizen_nagoya(Dr. Kiyoshi Ogawa)

dockerで機械学習(32) with anaconda(32)「Hands-On Deep Learning with TensorFlow 」By Dan Van Boxel

Last updated at Posted at 2018-10-23

1.すぐに利用したい方へ(as soon as)

Hands-On Deep Learning with TensorFlow
By Dan Van Boxel

cat32.gif

docker

dockerを導入し、Windows, Macではdockerを起動しておいてください。
Windowsでは、BiosでIntel Virtualizationをenableにしないとdockerが起動しない場合があります。
また、セキュリティの警告などが出ることがあります。

docker pull and run

$ docker pull kaizenjapan/anaconda-nikhil

$  docker run -it -p 8888:8888 kaizenjapan/anaconda-nikhil /bin/bash

以下のshell sessionでは
(base) root@f19e2f06eabb:/#は入力促進記号(comman prompt)です。実際には数字の部分が違うかもしれません。この行の#の右側を入力してください。
それ以外の行は出力です。出力にエラー、違いがあれば、コメント欄などでご連絡くださると幸いです。
それぞれの章のフォルダに移動します。

dockerの中と、dockerを起動したOSのシェルとが表示が似ている場合には、どちらで捜査しているか間違えることがあります。dockerの入力促進記号(comman prompt)に気をつけてください。

ファイル共有または複写

dockerとdockerを起動したOSでは、ファイル共有をするか、ファイル複写するかして、生成したファイルをブラウザ等表示させてください。参考文献欄にやり方のURLを記載しています。

複写の場合は、dockerを起動したOS側コマンドを実行しました。お使いのdockerの番号で置き換えてください。複写したファイルをブラウザで表示し内容確認しました。

logistic.py

(base) root@be53aac2e6d7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter01/Logistic regression model building# python logistic.py 
  File "logistic.py", line 3
    %autoindent
    ^
SyntaxError: invalid syntax

Magic functions
http://ipython.org/ipython-doc/dev/interactive/tutorial.html?highlight=startup%20files
"IPython has a set of predefined ‘magic functions’ that you can call with a command line style syntax. There are two kinds of magics, line-oriented and cell-oriented. Line magics are prefixed with the % character and work much like OS command-line calls: they get as an argument the rest of the line, where arguments are passed without parentheses or quotes."

dqn.py

(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/fdl_examples/chapter9# python dqn.py
Traceback (most recent call last):
  File "dqn.py", line 4, in <module>
    import gym
ModuleNotFoundError: No module named 'gym'
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/fdl_examples/chapter9# pip install gym
Collecting gym
  Downloading https://files.pythonhosted.org/packages/c3/44/3a63e8b87f642db49ac81239620e68df8cfae223dcfda4f8508aec88d204/gym-0.10.8.tar.gz (1.5MB)
    100% |████████████████████████████████| 1.5MB 787kB/s 
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (from gym) (1.1.0)
Requirement already satisfied: numpy>=1.10.4 in /opt/conda/lib/python3.6/site-packages (from gym) (1.15.1)
Requirement already satisfied: requests>=2.0 in /opt/conda/lib/python3.6/site-packages (from gym) (2.19.1)
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from gym) (1.11.0)
Collecting pyglet>=1.2.0 (from gym)
  Downloading https://files.pythonhosted.org/packages/1c/fc/dad5eaaab68f0c21e2f906a94ddb98175662cc5a654eee404d59554ce0fa/pyglet-1.3.2-py2.py3-none-any.whl (1.0MB)
    100% |████████████████████████████████| 1.0MB 16.8MB/s 
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (3.0.4)
Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (2.7)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (2018.8.24)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (1.23)
Collecting future (from pyglet>=1.2.0->gym)
  Downloading https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz (824kB)
    100% |████████████████████████████████| 829kB 7.8MB/s 
Building wheels for collected packages: gym, future
  Running setup.py bdist_wheel for gym ... done
  Stored in directory: /root/.cache/pip/wheels/ea/ec/dd/33bcc8801d345f0b640fced8a0864a7c8474828564bc5ccf70
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /root/.cache/pip/wheels/bf/c9/a3/c538d90ef17cf7823fa51fc701a7a7a910a80f6a405bf15b1a
Successfully built gym future
Installing collected packages: future, pyglet, gym
Successfully installed future-0.16.0 gym-0.10.8 pyglet-1.3.2
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/fdl_examples/chapter9# python dqn.py
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/atari/atari_env.py", line 9, in <module>
    import atari_py
ModuleNotFoundError: No module named 'atari_py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dqn.py", line 371, in <module>
    main('')
  File "dqn.py", line 288, in main
    env = gym.make('Breakout-v4')
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/registration.py", line 167, in make
    return registry.make(id)
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/registration.py", line 119, in make
    env = spec.make()
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/registration.py", line 85, in make
    cls = load(self._entry_point)
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/registration.py", line 14, in load
    result = entry_point.load(False)
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2325, in load
    return self.resolve()
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2331, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/atari/__init__.py", line 1, in <module>
    from gym.envs.atari.atari_env import AtariEnv
  File "/opt/conda/lib/python3.6/site-packages/gym/envs/atari/atari_env.py", line 11, in <module>
    raise error.DependencyNotInstalled("{}. (HINT: you can install Atari dependencies by running 'pip install gym[atari]'.)".format(e))
gym.error.DependencyNotInstalled: No module named 'atari_py'. (HINT: you can install Atari dependencies by running 'pip install gym[atari]'.)
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/fdl_examples/chapter9# pip install gym[atari]
Requirement already satisfied: gym[atari] in /opt/conda/lib/python3.6/site-packages (0.10.8)
Requirement already satisfied: pyglet>=1.2.0 in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (1.3.2)
Requirement already satisfied: numpy>=1.10.4 in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (1.15.1)
Requirement already satisfied: requests>=2.0 in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (2.19.1)
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (1.11.0)
Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (1.1.0)
Collecting atari-py>=0.1.4; extra == "atari" (from gym[atari])
  Downloading https://files.pythonhosted.org/packages/16/cc/aabbe1580df0f9b22851e210d5e35e3faa305d10a8783997fb8e79239dbc/atari_py-0.1.6-py3-none-manylinux1_x86_64.whl (1.7MB)
    100% |████████████████████████████████| 1.7MB 5.6MB/s 
Collecting PyOpenGL; extra == "atari" (from gym[atari])
  Downloading https://files.pythonhosted.org/packages/9c/1d/4544708aaa89f26c97cc09450bb333a23724a320923e74d73e028b3560f9/PyOpenGL-3.1.0.tar.gz (1.2MB)
    100% |████████████████████████████████| 1.2MB 5.8MB/s 
Requirement already satisfied: Pillow; extra == "atari" in /opt/conda/lib/python3.6/site-packages (from gym[atari]) (5.2.0)
Requirement already satisfied: future in /opt/conda/lib/python3.6/site-packages (from pyglet>=1.2.0->gym[atari]) (0.16.0)
Requirement already satisfied: idna<2.8,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[atari]) (2.7)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[atari]) (1.23)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[atari]) (2018.8.24)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[atari]) (3.0.4)
Building wheels for collected packages: PyOpenGL
  Running setup.py bdist_wheel for PyOpenGL ... done
  Stored in directory: /root/.cache/pip/wheels/6c/00/7f/1dd736f380848720ad79a1a1de5272e0d3f79c15a42968fb58
Successfully built PyOpenGL
Installing collected packages: atari-py, PyOpenGL
Successfully installed PyOpenGL-3.1.0 atari-py-0.1.6
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/fdl_examples/chapter9# python dqn.py
2018-10-23 02:35:55.318767: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-10-23 02:35:55.320642: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:From dqn.py:93: calling argmax (from tensorflow.python.ops.math_ops) with dimension is deprecated and will be removed in a future version.
Instructions for updating:
Use the `axis` argument instead
  0%|                                                                                                                    | 0/15000 [00:00<?, ?it/s]Reward Stats (min, max, median, mean):  -1.0 -1.0 -1.0 -1.0
Global Stats (ep_percent, global_step_counter):  0.0 181
  0%|▎                                                                                                        | 49/15000 [01:12<6:20:03,  1.53s/it]

word2vec_fast.py

(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/archive# python word2vec_fast.py 
Traceback (most recent call last):
  File "word2vec_fast.py", line 2, in <module>
    import cPickle as pickle
ModuleNotFoundError: No module named 'cPickle'
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/archive# pip install cPickle
Collecting cPickle
  Could not find a version that satisfies the requirement cPickle (from versions: )
No matching distribution found for cPickle
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/archive# conda install cPicle
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - cpicle

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

lstm.py

base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/archive# python lstm.py
  File "lstm.py", line 25
    print x.get_shape().as_list()
          ^
SyntaxError: invalid syntax
(base) root@be53aac2e6d7:/Fundamentals-of-Deep-Learning-Book/archive# python2 lstm.py
Traceback (most recent call last):
  File "lstm.py", line 2, in <module>
    import numpy as np
ImportError: No module named numpy

simple.py

(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter01/Simple Computations# python simple.py
2018-10-23 10:21:27.139548: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-10-23 10:21:27.143109: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
NN is:
[[ 7. 10.]
 [15. 22.]]
M2 is:
[[1. 4.]]
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py:189: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
W is:
0
WARNING:tensorflow:Variable += will be deprecated. Use variable.assign_add if you want assignment to the variable value or 'x = x + y' if you want a new python Tensor object.
W after adding a:
1
W after adding a again:
2
E as defined:
4
E and d:
[4, 2]
E with custom d=4:
6

basic_nn.py

(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow# cd Chapter02
(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter02# ls
Basic neural networks		      Single hidden layer explained  The multiple hidden layer model
Results of the multiple hidden layer  Single hidden layer model
(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter02# ls -al
total 28
drwxr-xr-x 7 root root 4096 Oct 23 10:19 .
drwxr-xr-x 8 root root 4096 Oct 23 10:19 ..
drwxr-xr-x 2 root root 4096 Oct 23 10:19 Basic neural networks
drwxr-xr-x 2 root root 4096 Oct 23 10:19 Results of the multiple hidden layer
drwxr-xr-x 2 root root 4096 Oct 23 10:19 Single hidden layer explained
drwxr-xr-x 2 root root 4096 Oct 23 10:19 Single hidden layer model
drwxr-xr-x 2 root root 4096 Oct 23 10:19 The multiple hidden layer model
(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter02# cd Basic\ neural\ networks/
(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter02/Basic neural networks# ls
basic_nn.py
(base) root@a221771835f7:/Hands-On-Deep-Learning-with-TensorFlow/Chapter02/Basic neural networks# python basic_nn.py 
2018-10-23 10:23:09.910562: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-10-23 10:23:09.912816: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
[-0.69652796 -1.2178961  -0.65935516 -0.48184317 -1.0316856 ]
[0.4851512  1.4832709  0.43474925 0.23217285 1.0643752 ]
[2.4217906 2.927363  2.899702  3.3005424 2.4061952]
[0.8845072  1.0741019  1.064608   1.1940868  0.87804675]
[ 0.3501876  -0.7230851  -0.43722454 -0.04946229  0.30844414]
[0.58666307 0.32671398 0.3924025  0.48763695 0.5765054 ]
[0.31030515 0.31542578 0.30129102 0.24439405 0.32239842]
[-0.04251816 -0.23547153 -0.1934177  -0.10224681 -0.17500149]
[0.56654954 0.5199779  0.5269422  0.5354771  0.5367827 ]

jupyter notebook

# jupyter notebook --ip=0.0.0.0 --allow-root
py31.png

2. dockerを自力で構築する方へ

ここから下は、上記のpullしていただいたdockerをどういう方針で、どういう手順で作ったかを記録します。
上記のdockerを利用する上での参考資料です。本の続きを実行する上では必要ありません。
自力でdocker/anacondaを構築する場合の手順になります。
dockerfileを作る方法ではありません。ごめんなさい。

docker

ubuntu, debianなどのLinuxを、linux, windows, mac osから共通に利用できる仕組み。
利用するOSの設定を変更せずに利用できるのがよい。
同じ仕様で、大量の人が利用することができる。
ソフトウェアの開発元が公式に対応しているものと、利用者が便利に仕立てたものの両方が利用可能である。今回は、公式に配布しているものを、自分で仕立てて、他の人にも利用できるようにする。

python

DeepLearningの実習をPhthonで行って来た。
pythonを使う理由は、多くの機械学習の仕組みがpythonで利用できることと、Rなどの統計解析の仕組みもpythonから容易に利用できることがある。

anaconda

pythonには、2と３という版の違いと、配布方法の違いなどがある。
Anacondaでpython3をこの１年半利用してきた。
Anacondaを利用した理由は、統計解析のライブラリと、JupyterNotebookが初めから入っているからである。

docker公式配布

ubuntu, debianなどのOSの公式配布,gcc, anacondaなどの言語の公式配布などがある。
これらを利用し、docker-hubに登録することにより、公式配布の質の確認と、変更権を含む幅広い情報の共有ができる。dockerが公式配布するものではなく、それぞれのソフト提供者の公式配布という意味。

docker pull

docker公式配布の利用は、URLからpullすることで実現する。

docker Anaconda

anacondaが公式配布しているものを利用。

$  docker pull kaizenjapan/anaconda-keras
Using default tag: latest
latest: Pulling from continuumio/anaconda3
Digest: sha256:e07b9ca98ac1eeb1179dbf0e0bbcebd87701f8654878d6d8ce164d71746964d1
Status: Image is up to date for continuumio/anaconda3:latest

$ docker run -it -p 8888:8888 continuumio/anaconda3 /bin/bash

実際にはkeras, tensorflow を利用していた他のpushをpull

##apt

(base) root@d8857ae56e69:/# apt update; -y upgrade

(base) root@d8857ae56e69:/# apt install -y procps vim  apt-utils  sudo

ソース git

(base) root@f19e2f06eabb:/# git clone https://github.com/PacktPublishing/Hands-On-Deep-Learning-with-TensorFlow

conda

# conda update --prefix /opt/conda anaconda
Solving environment: done

# conda install theano

pip

(base) root@f19e2f06eabb:/d# pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 2.0MB/s 
distributed 1.21.8 requires msgpack, which is not installed.
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0

#docker hubへの登録

$ docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                    NAMES
caef766a99ff        continuumio/anaconda3   "/usr/bin/tini -- /b…"   10 hours ago        Up 10 hours         0.0.0.0:8888->8888/tcp   sleepy_bassi

$ docker commit caef766a99ff kaizenjapan/anaconda-nikhil

$ docker push kaizenjapan/anaconda-nikhil

参考資料(reference)

なぜdockerで機械学習するか 書籍・ソース一覧作成中 (目標100)
https://qiita.com/kaizen_nagoya/items/ddd12477544bf5ba85e2

dockerで機械学習(1) with anaconda(1)「ゼロから作るDeep Learning - Pythonで学ぶディープラーニングの理論と実装」斎藤 康毅 著
https://qiita.com/kaizen_nagoya/items/a7e94ef6dca128d035ab

dockerで機械学習(2)with anaconda(2)「ゼロから作るDeep Learning2自然言語処理編」斎藤 康毅 著
https://qiita.com/kaizen_nagoya/items/3b80dfc76933cea522c6

dockerで機械学習(3)with anaconda(3)「直感Deep Learning」Antonio Gulli、Sujit Pal　第1章,第2章
https://qiita.com/kaizen_nagoya/items/483ae708c71c88419c32

dockerで機械学習(71) 環境構築(1) docker どっかーら、どーやってもエラーばっかり。
https://qiita.com/kaizen_nagoya/items/690d806a4760d9b9e040

dockerで機械学習(72) 環境構築(2) Docker for Windows
https://qiita.com/kaizen_nagoya/items/c4daa5cf52e9f0c2c002

dockerで機械学習(73) 環境構築(3) docker/linux/macos bash スクリプト, ms-dos batchファイル
https://qiita.com/kaizen_nagoya/items/3f7b39110b7f303a5558

dockerで機械学習(74) 環境構築(4) R 難関いくつ？
https://qiita.com/kaizen_nagoya/items/5fb44773bc38574bcf1c

dockerで機械学習(75)環境構築(5)docker関連ファイルの管理
https://qiita.com/kaizen_nagoya/items/4f03df9a42c923087b5d

OpenCVをPythonで動かそうとしてlibGL.soが無いって言われたけど解決した。
https://qiita.com/toshitanian/items/5da24c0c0bd473d514c8

サーバサイドにおけるmatplotlibによる作図Tips
https://qiita.com/TomokIshii/items/3a26ee4453f535a69e9e

Dockerでホストとコンテナ間でのファイルコピー
https://qiita.com/gologo13/items/7e4e404af80377b48fd5

Docker for Mac でファイル共有を利用する
https://qiita.com/seijimomoto/items/1992d68de8baa7e29bb5

「名古屋のIoTは名古屋のOSで」Dockerをどっかーらどうやって使えばいいんでしょう。TOPPERS/FMP on RaspberryPi with Macintosh編 ５つの関門
https://qiita.com/kaizen_nagoya/items/9c46c6da8ceb64d2d7af

64bitCPUへの道 and/or 64歳の決意
https://qiita.com/kaizen_nagoya/items/cfb5ffa24ded23ab3f60

ゼロから作るDeepLearning２自然言語処理編　読書会の進め方（例）
https://qiita.com/kaizen_nagoya/items/025eb3f701b36209302e

Ubuntu 16.04 LTS で NVIDIA Docker を使ってみる
https://blog.amedama.jp/entry/2017/04/03/235901

文書履歴(document history)

ver. 0.10 初稿 20181023

最後までおよみいただきありがとうございました。

いいね　💚、フォローをお願いします。

Thank you very much for reading to the last sentence.

Please press the like icon 💚　and follow me for your happy life.

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0