1.すぐにプログラムを動かしたい方へ(as soon as you want)
「仕事ではじめる機械学習」有賀康顕、中山心太、西林孝 著
https://www.oreilly.co.jp/books/9784873118215/
<この項は書きかけです。順次追記します。>
docker
dockerは複数人で同じ設定で作業、実験、実習ができるとても便利な道具です。
それまで利用していた環境とは全く別に作ります。
それまでどのような設定をしていても、dockerが起動し、ネットがつながれば大丈夫です。ここでは、本で紹介しているdockerの資料に直接基づかず、すぐに利用できる環境を提供し、またご自身で新たに導入したソフトを含めて、保存しておく方法を紹介します。
docker起動
dockerを導入し、Windows, Macではdockerを起動しておいてください。
Windowsでは、BiosでIntel Virtualizationをenableにしないとdockerが起動しない場合があります。
また、セキュリティの警告などが出ることがあります。
システム管理者での作業が必要になります。
docker pull and run
第1章までやった段階
$ docker pull kaizenjapan/anaconda-ml
$ docker run -it -p 8888:8888 kaizenjapan/anaconda-ml /bin/bash
dockerの中と、dockerを起動したOSのシェルとが表示が似ている場合には、どちらで操作しているか間違えることがあります。dockerの入力促進記号(comman prompt)は、docker idの数字が何桁かついています。
第1章
以下のshell sessionでは
(base) root@f2854307a1c3:/#
は入力促進記号(comman prompt)です。実際には数字の部分が違うかもしれません。この行の#の右側を入力してください。
それ以外の行は出力です。出力にエラー、違いがあれば、コメント欄などでご連絡くださると幸いです。
それぞれの章のフォルダに移動します。
docker
(base) root@f2854307a1c3:/# ls
bin boot deep-learning-with-keras-ja dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
(base) root@f2854307a1c3:/# cd deep-learning-with-keras-ja/
(base) root@f2854307a1c3:/deep-learning-with-keras-ja# ls
README.md ch01 ch02 ch03 ch04 ch05 ch06 ch07 ch08 deep-learning-with-keras-ja.png
(base) root@f2854307a1c3:/ml-at-work# cd chap08
(base) root@f2854307a1c3:/ml-at-work/chap08# ls
kickstarter_clawler.py kickstarter_result_to_csv.py kickstarter_slide.pptx
(base) root@f2854307a1c3:/ml-at-work/chap08# jupyter notebook --ip=0.0.0.0 --allow-root
dockerを起動したOSのブラウザで localhost:8888 を表示してください。
kickstarter_clawler.py
(base) root@f2854307a1c3:/ml-at-work/chap08# python kickstarter_clawler.py
16 progress 12 / 35789 0.03 %
16 progress 24 / 35789 0.07 %
16 progress 36 / 35789 0.1 %
16 progress 48 / 35789 0.13 %
(中略)
kickstarter_result_to_csv.py
(base) root@f2854307a1c3:/ml-at-work/chap08# python kickstarter_result_to_csv.py
2. dockerを自力で構築する方へ
anaconda/keras/tensorflow方針(docker deploy policy)
ここから下は、上記のpullしていただいたdockerをどういう方針で、どういう手順で作ったかを記録します。
上記のdockerを利用する上での参考資料です。１章の続きをする上では必要ありません。
自力でdocker/anaconda/keras/tensorflowを構築する場合の手順になります。
dockerfileを作る方法ではありません。ごめんなさい。
docker
ubuntu, debianなどのLinuxを、linux, windows, mac osから共通に利用できる仕組み。
利用するOSの設定を変更せずに利用できるのがよい。
同じ仕様で、大量の人が利用することができる。
ソフトウェアの開発元が公式に対応しているものと、利用者が便利に仕立てたものの両方が利用可能である。今回は、公式に配布しているものを、自分で仕立てて、他の人にも利用できるようにする。
python
DeepLearningの実習をPhthonで行って来た。
pythonを使う理由は、多くの機械学習の仕組みがpythonで利用できることと、Rなどの統計解析の仕組みもpythonから容易に利用できることがある。
anaconda
pythonには、2と３という版の違いと、配布方法の違いなどがある。
Anacondaでpython3をこの１年半利用してきた。
Anacondaを利用した理由は、統計解析のライブラリと、JupyterNotebookが初めから入っているからである。
docker公式配布
ubuntu, debianなどのOSの公式配布,gcc, anacondaなどの言語の公式配布などがある。
これらを利用し、docker-hubに登録することにより、公式配布の質の確認と、変更権を含む幅広い情報の共有ができる。dockerが公式配布するものではなく、それぞれのソフト提供者の公式配布という意味。
docker pull
docker公式配布の利用は、URLからpullすることで実現する。
docker Anaconda
anacondaが公式配布しているものを利用。
$ docker pull continuumio/anaconda3
Using default tag: latest
latest: Pulling from continuumio/anaconda3
cc1a78bfd46b: Pull complete
314b82d3c9fe: Pull complete
adebea299011: Pull complete
f7baff790e81: Pull complete
Digest: sha256:e07b9ca98ac1eeb1179dbf0e0bbcebd87701f8654878d6d8ce164d71746964d1
Status: Downloaded newer image for continuumio/anaconda3:latest
docker run
実行はdocker runである。今回はブラウザで閲覧するため-pの設定を行う。
$ docker run -it -p 8888:8888 -p 6006:6006 continuumio/anaconda3 /bin/bash
(base) root@0da2c87a513f:/#
apt
ubuntu, debianはDebian系の道具類の配布の仕組みが利用できる。
apt-getのよいところは、debianというカーネルの開発者が大勢集まっており、基本機能の整合性を厳密に取っているところにある。
通信規約の改良にあたって、Linuxのカーネルに手を入れる必要があったときに、ソースをaptで導入し、patch適用後コンパイルしてもエラーなくコンパイルできて以来、linuxといえばaptが利用できるDebian系を愛用している。
Raspberry PIで有名なRaspbianもDebian系で、aptが利用できる。
そのため、dockerで作業した状況をRaspbianで再現することはとても容易である。
(base) root@0da2c87a513f:/# apt update
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [549 kB]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages.diff/Index [5164 B]
Get:7 http://deb.debian.org/debian stretch Release.gpg [2434 B]
Get:8 http://deb.debian.org/debian stretch-updates/main amd64 Packages 2018-07-20-2027.50.pdiff [1134 B]
Get:9 http://deb.debian.org/debian stretch-updates/main amd64 Packages 2018-07-31-2010.17.pdiff [1388 B]
Get:9 http://deb.debian.org/debian stretch-updates/main amd64 Packages 2018-07-31-2010.17.pdiff [1388 B]
Get:10 http://deb.debian.org/debian stretch/main amd64 Packages [9500 kB]
Fetched 10.4 MB in 27s (380 kB/s)
Reading package lists... Done
conda install
pythonには、pipというpythonのライブラリ等を導入する仕組みがある。
今回は、condaというanaconda専用のライブラリ導入の仕組みを利用してみる。
pipではなくcondaを利用した理由は、本に書いてあるように作業したらエラーでうまくいかなかったことがあったから。
condaによるanacondaの更新は下記。
# conda update --prefix /opt/conda anaconda
git
本にあるURLからgitで取得する。
(base) root@b789c278e622:/# git clone https://github.com/oreilly-japan/ml-at-work.git
pip install
(base) root@f2854307a1c3:/ml-at-work/chap08# pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 3.1MB/s
distributed 1.21.8 requires msgpack, which is not installed.
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0
(base) root@f2854307a1c3:/ml-at-work# pip install -r requirements.txt -c constraints.txt
Collecting Cython==0.27.3 (from -c constraints.txt (line 3))
Downloading https://files.pythonhosted.org/packages/e9/91/46cb3f4c73f1e96faa517f96e9d12de5b8c97d404c7ab71553da0e58c980/Cython-0.27.3-cp36-cp36m-manylinux1_x86_64.whl (3.1MB)
100% |████████████████████████████████| 3.1MB 3.1MB/s
Collecting fastFM==0.2.11 (from -c constraints.txt (line 6))
Downloading https://files.pythonhosted.org/packages/f5/15/fdbb9b9455efa48ffb07b9880a1e567e0c7a7de0acc4aa7f1c5ba9ce4f2c/fastFM-0.2.11-cp36-cp36m-manylinux1_x86_64.whl (483kB)
100% |████████████████████████████████| 491kB 3.5MB/s
Requirement already satisfied: jupyter==1.0.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 15)) (1.0.0)
Requirement already satisfied: jupyter-console==5.2.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 17)) (5.2.0)
Collecting matplotlib==2.1.0 (from -c constraints.txt (line 20))
Downloading https://files.pythonhosted.org/packages/b2/9c/fcc9cfbf2454d93be66a615657cda4184954b4b67b9fc07c8511ff152b8f/matplotlib-2.1.0-cp36-cp36m-manylinux1_x86_64.whl (15.0MB)
100% |████████████████████████████████| 15.0MB 1.3MB/s
Requirement already satisfied: nbconvert==5.3.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 22)) (5.3.1)
Requirement already satisfied: nbformat==4.4.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 23)) (4.4.0)
Collecting notebook==5.1.0 (from -c constraints.txt (line 24))
Downloading https://files.pythonhosted.org/packages/24/ef/2332b8bec9bf74e08caed9b2705f89c7c5dfb4d7f62533d08b153ce7964c/notebook-5.1.0-py2.py3-none-any.whl (8.0MB)
100% |████████████████████████████████| 8.0MB 2.3MB/s
Collecting numpy==1.13.3 (from -c constraints.txt (line 25))
Downloading https://files.pythonhosted.org/packages/57/a7/e3e6bd9d595125e1abbe162e323fd2d06f6f6683185294b79cd2cdb190d5/numpy-1.13.3-cp36-cp36m-manylinux1_x86_64.whl (17.0MB)
100% |████████████████████████████████| 17.0MB 1.4MB/s
Collecting pandas==0.20.3 (from -c constraints.txt (line 26))
Downloading https://files.pythonhosted.org/packages/fe/6f/5733658857dffb998afa2120027171c263384ada0487a969e5ecd5bf9ac9/pandas-0.20.3-cp36-cp36m-manylinux1_x86_64.whl (24.5MB)
100% |████████████████████████████████| 24.5MB 1.5MB/s
Requirement already satisfied: pandocfilters==1.4.2 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 27)) (1.4.2)
Requirement already satisfied: prompt-toolkit==1.0.15 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 32)) (1.0.15)
Requirement already satisfied: Pygments==2.2.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 34)) (2.2.0)
Requirement already satisfied: pyparsing==2.2.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 35)) (2.2.0)
Collecting python-dateutil==2.6.1 (from -c constraints.txt (line 36))
Downloading https://files.pythonhosted.org/packages/4b/0d/7ed381ab4fe80b8ebf34411d14f253e1cf3e56e2820ffa1d8844b23859a2/python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
100% |████████████████████████████████| 194kB 3.0MB/s
Collecting pytz==2017.2 (from -c constraints.txt (line 37))
Downloading https://files.pythonhosted.org/packages/55/62/e7cd0e15b76062d298413f14bb4ec3cd8568a22d274427f9c3c7286969f4/pytz-2017.2-py2.py3-none-any.whl (484kB)
100% |████████████████████████████████| 491kB 509kB/s
Requirement already satisfied: qtconsole==4.3.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 39)) (4.3.1)
Requirement already satisfied: scikit-learn==0.19.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 40)) (0.19.1)
Collecting scipy==1.0.0 (from -c constraints.txt (line 41))
Downloading https://files.pythonhosted.org/packages/d8/5e/caa01ba7be11600b6a9d39265440d7b3be3d69206da887c42bef049521f2/scipy-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (50.0MB)
100% |████████████████████████████████| 50.0MB 671kB/s
Requirement already satisfied: six==1.11.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 43)) (1.11.0)
Collecting statsmodels==0.8.0 (from -c constraints.txt (line 44))
Downloading https://files.pythonhosted.org/packages/0d/e9/70d80b48c8c52a8de3ec7cd50e2aa2b1f3cf3f95e42b15fdcb59bd7189f3/statsmodels-0.8.0-cp36-cp36m-manylinux1_x86_64.whl (6.3MB)
100% |████████████████████████████████| 6.3MB 2.4MB/s
Collecting terminado==0.6 (from -c constraints.txt (line 45))
Downloading https://files.pythonhosted.org/packages/58/59/aabe84fce2f45da10165435cec204d982863e176f6849a4a4fe2652a20a8/terminado-0.6.tar.gz
Requirement already satisfied: testpath==0.3.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 46)) (0.3.1)
Collecting tornado==4.5.2 (from -c constraints.txt (line 47))
Downloading https://files.pythonhosted.org/packages/fa/14/52e2072197dd0e63589e875ebf5984c91a027121262aa08f71a49b958359/tornado-4.5.2.tar.gz (483kB)
100% |████████████████████████████████| 491kB 9.2MB/s
Requirement already satisfied: traitlets==4.3.2 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 48)) (4.3.2)
Requirement already satisfied: wcwidth==0.1.7 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 49)) (0.1.7)
Collecting ipykernel==4.6.1 (from -c constraints.txt (line 8))
Downloading https://files.pythonhosted.org/packages/18/c3/76775a650cae2e3d9c033b26153583e61282692d9a3af12a3022d8f0cefa/ipykernel-4.6.1-py3-none-any.whl (104kB)
100% |████████████████████████████████| 112kB 3.3MB/s
Collecting ipywidgets==7.0.1 (from -c constraints.txt (line 11))
Downloading https://files.pythonhosted.org/packages/50/5a/72371d118bf0533034c109436c21124e03c5c67bfd1651e938a6e7103d85/ipywidgets-7.0.1-py2.py3-none-any.whl (66kB)
100% |████████████████████████████████| 71kB 3.3MB/s
Collecting jupyter-client==5.1.0 (from -c constraints.txt (line 16))
Downloading https://files.pythonhosted.org/packages/28/3d/ad981e404f81b44a6e068b53d240414c940cebb92841ca92307dfb67c169/jupyter_client-5.1.0-py2.py3-none-any.whl (84kB)
100% |████████████████████████████████| 92kB 3.3MB/s
Collecting ipython==6.2.1 (from -c constraints.txt (line 9))
Downloading https://files.pythonhosted.org/packages/e1/87/294b718125085559b56453be87d90777863173470167e5f1d5de20b9eea3/ipython-6.2.1-py3-none-any.whl (745kB)
100% |████████████████████████████████| 747kB 7.5MB/s
Requirement already satisfied: cycler==0.10.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 2)) (0.10.0)
Collecting mistune==0.7.4 (from -c constraints.txt (line 21))
Downloading https://files.pythonhosted.org/packages/7b/ab/e71dd1ca31addcd0268c54859eaf75414a10fbc48c79078f7c3066e6ed0d/mistune-0.7.4-py2.py3-none-any.whl
Collecting Jinja2==2.9.6 (from -c constraints.txt (line 13))
Downloading https://files.pythonhosted.org/packages/5e/73/10c45b82a88ed6b7751bd40da31eeefd7b362e07b56a99aa6e56655a0794/Jinja2-2.9.6-py2.py3-none-any.whl (340kB)
100% |████████████████████████████████| 348kB 3.0MB/s
Collecting jupyter-core==4.3.0 (from -c constraints.txt (line 18))
Downloading https://files.pythonhosted.org/packages/48/41/e64f3f38fdb4f7c98814ce93be8b9261068bf0d2ec3550a5ffb492316196/jupyter_core-4.3.0-py2.py3-none-any.whl (76kB)
100% |████████████████████████████████| 81kB 3.6MB/s
Requirement already satisfied: entrypoints==0.2.3 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 5)) (0.2.3)
Collecting bleach==2.1.1 (from -c constraints.txt (line 1))
Downloading https://files.pythonhosted.org/packages/bb/c9/c99cef21591ea872879b5ac104b794093d5354ce2b06846305bc6367b6ad/bleach-2.1.1-py2.py3-none-any.whl
Requirement already satisfied: ipython-genutils==0.2.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 10)) (0.2.0)
Requirement already satisfied: jsonschema==2.6.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 14)) (2.6.0)
Collecting patsy==0.4.1 (from -c constraints.txt (line 29))
Downloading https://files.pythonhosted.org/packages/02/8a/255f80a7f939006ec479275fde6301feedf3fdd9ecee782bb64987b84de8/patsy-0.4.1-py2.py3-none-any.whl (233kB)
100% |████████████████████████████████| 235kB 2.8MB/s
Requirement already satisfied: ptyprocess==0.5.2 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 33)) (0.5.2)
Collecting decorator==4.1.2 (from -c constraints.txt (line 4))
Downloading https://files.pythonhosted.org/packages/a1/4e/c42167ba5c3192bed633726d39d7896cc55d4fa3ec4a1fb60cd3a53fc4c7/decorator-4.1.2-py2.py3-none-any.whl
Collecting widgetsnbextension==3.0.3 (from -c constraints.txt (line 51))
Downloading https://files.pythonhosted.org/packages/73/33/1b7d783be4c1bba7fefb1f2e3fca9efb37439648d8f9e71d95d685e56273/widgetsnbextension-3.0.3-py2.py3-none-any.whl (2.5MB)
100% |████████████████████████████████| 2.5MB 2.8MB/s
Collecting pyzmq==16.0.2 (from -c constraints.txt (line 38))
Downloading https://files.pythonhosted.org/packages/83/1d/fe441c9b3bafc84ef5ec6d2a754d784b1b298722fd0fbd4be29822405452/pyzmq-16.0.2-cp36-cp36m-manylinux1_x86_64.whl (3.0MB)
100% |████████████████████████████████| 3.1MB 7.5MB/s
Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.6/site-packages (from ipython==6.2.1->-c constraints.txt (line 9)) (39.1.0)
Requirement already satisfied: simplegeneric==0.8.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 42)) (0.8.1)
Collecting pexpect==4.2.1 (from -c constraints.txt (line 30))
Downloading https://files.pythonhosted.org/packages/5b/16/4859a0376be8b87bf3920b1f6e63b8a3c0ee42488babee07c87ca9316e03/pexpect-4.2.1-py2.py3-none-any.whl (55kB)
100% |████████████████████████████████| 61kB 432kB/s
Collecting jedi==0.11.0 (from -c constraints.txt (line 12))
Downloading https://files.pythonhosted.org/packages/fb/a8/25978660f20b05b982b54dde61bdd515bc2683f84c906ad7fb8eb1aca517/jedi-0.11.0-py2.py3-none-any.whl (146kB)
100% |████████████████████████████████| 153kB 635kB/s
Requirement already satisfied: pickleshare==0.7.4 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 31)) (0.7.4)
Requirement already satisfied: MarkupSafe==1.0 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 19)) (1.0)
Collecting html5lib==1.0b10 (from -c constraints.txt (line 7))
Downloading https://files.pythonhosted.org/packages/2f/74/7793ca2d36f676b740efc04b7ba887c610119beb5841d1805cb3515616cb/html5lib-1.0b10-py2.py3-none-any.whl (112kB)
100% |████████████████████████████████| 112kB 619kB/s
Collecting parso==0.1.0 (from -c constraints.txt (line 28))
Downloading https://files.pythonhosted.org/packages/fc/af/32b2a0d8b2e5d5c82f6f29752a243255beb4b281617e9581b3d7784aec7a/parso-0.1.0-py2.py3-none-any.whl (89kB)
100% |████████████████████████████████| 92kB 586kB/s
Requirement already satisfied: webencodings==0.5.1 in /opt/conda/lib/python3.6/site-packages (from -c constraints.txt (line 50)) (0.5.1)
Building wheels for collected packages: terminado, tornado
Running setup.py bdist_wheel for terminado ... done
Stored in directory: /root/.cache/pip/wheels/5b/04/58/844654a8da3bb183aa18673c959eedbb992314261049d7e5e1
Running setup.py bdist_wheel for tornado ... done
Stored in directory: /root/.cache/pip/wheels/a5/59/09/79aad6522a5811b546e94d55c1535702dcad35880a09b03471
Successfully built terminado tornado
distributed 1.21.8 requires msgpack, which is not installed.
flask 1.0.2 has requirement Jinja2>=2.10, but you'll have jinja2 2.9.6 which is incompatible.
Installing collected packages: html5lib, bleach, Cython, decorator, numpy, scipy, fastFM, pexpect, parso, jedi, ipython, pyzmq, jupyter-core, python-dateutil, jupyter-client, tornado, ipykernel, Jinja2, terminado, notebook, widgetsnbextension, ipywidgets, pytz, matplotlib, mistune, pandas, patsy, statsmodels
Found existing installation: html5lib 1.0.1
Uninstalling html5lib-1.0.1:
Successfully uninstalled html5lib-1.0.1
Found existing installation: bleach 2.1.3
Uninstalling bleach-2.1.3:
Successfully uninstalled bleach-2.1.3
Found existing installation: Cython 0.28.2
Uninstalling Cython-0.28.2:
Successfully uninstalled Cython-0.28.2
Found existing installation: decorator 4.3.0
Uninstalling decorator-4.3.0:
Successfully uninstalled decorator-4.3.0
Found existing installation: numpy 1.14.3
Uninstalling numpy-1.14.3:
Successfully uninstalled numpy-1.14.3
Found existing installation: scipy 1.1.0
Uninstalling scipy-1.1.0:
Successfully uninstalled scipy-1.1.0
Found existing installation: pexpect 4.5.0
Uninstalling pexpect-4.5.0:
Successfully uninstalled pexpect-4.5.0
Found existing installation: parso 0.2.0
Uninstalling parso-0.2.0:
Successfully uninstalled parso-0.2.0
Found existing installation: jedi 0.12.0
Uninstalling jedi-0.12.0:
Successfully uninstalled jedi-0.12.0
Found existing installation: ipython 6.4.0
Uninstalling ipython-6.4.0:
Successfully uninstalled ipython-6.4.0
Found existing installation: pyzmq 17.0.0
Uninstalling pyzmq-17.0.0:
Successfully uninstalled pyzmq-17.0.0
Found existing installation: jupyter-core 4.4.0
Uninstalling jupyter-core-4.4.0:
Successfully uninstalled jupyter-core-4.4.0
Found existing installation: python-dateutil 2.7.3
Uninstalling python-dateutil-2.7.3:
Successfully uninstalled python-dateutil-2.7.3
Found existing installation: jupyter-client 5.2.3
Uninstalling jupyter-client-5.2.3:
Successfully uninstalled jupyter-client-5.2.3
Found existing installation: tornado 5.0.2
Uninstalling tornado-5.0.2:
Successfully uninstalled tornado-5.0.2
Found existing installation: ipykernel 4.8.2
Uninstalling ipykernel-4.8.2:
Successfully uninstalled ipykernel-4.8.2
Found existing installation: Jinja2 2.10
Uninstalling Jinja2-2.10:
Successfully uninstalled Jinja2-2.10
Found existing installation: terminado 0.8.1
Cannot uninstall 'terminado'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
3. docker hub 登録
ここからは、新たにソフトを導入したdockerを自分のhubに登録する方法です。
ご自身で何かソフトウェアを導入されたら、ぜひhubに登録することをお勧めします。
docker hubへのID登録が必要になります。
続きの作業を誰かに依頼したり、エラーがでてわからなくなったときに、対処方法を問い合わせるのにも役立ちます。
kaizenjapanは小川清のIDです。ご自身のIDで読み替えて、ご登録ください。
docker push
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b789c278e622 continuumio/anaconda3 "/usr/bin/tini -- /b…" 21 hours ago Up 21 hours 0.0.0.0:8888->8888/tcp peaceful_noice
$ docker commit b789c278e622 kaizenjapan/anaconda-ml
$ docker push kaizenjapan/anaconda-ml:latest
The push refers to repository [docker.io/kaizenjapan/anaconda-keras]
95456260f6d1: Pushed
6410333f34cf: Pushed
cf342e34eca3: Pushing [==================> ] 1.21GB/3.27GB
cea95006e36a: Pushed
0f3a12fef684: Pushed
4. 参考資料(reference)
機械学習を半自動化するauto-sklearnの環境構築（Mac&Docker）
https://qiita.com/inoue0426/items/ffd7f4235dcfde88942b
dockerで機械学習(1) with anaconda(1)「ゼロから作るDeep Learning - Pythonで学ぶディープラーニングの理論と実装」斎藤 康毅 著
https://qiita.com/kaizen_nagoya/items/a7e94ef6dca128d035ab
dockerで機械学習(2)with anaconda(2)「ゼロから作るDeep Learning2自然言語処理編」斎藤 康毅 著
https://qiita.com/kaizen_nagoya/items/3b80dfc76933cea522c6
dockerで機械学習(3)with anaconda(3)「直感Deep Learning」Antonio Gulli、Sujit Pal 第1章,第2章
https://qiita.com/kaizen_nagoya/items/483ae708c71c88419c32
dockerで機械学習(71) 環境構築(1) docker どっかーら、どーやってもエラーばっかり。
https://qiita.com/kaizen_nagoya/items/690d806a4760d9b9e040
dockerで機械学習(72) 環境構築(2) Docker for Windows
https://qiita.com/kaizen_nagoya/items/c4daa5cf52e9f0c2c002
dockerで機械学習(73) 環境構築(3) docker/linux/macos bash スクリプト, ms-dos batchファイル
https://qiita.com/kaizen_nagoya/items/3f7b39110b7f303a5558
dockerで機械学習(74) 環境構築(4) R 難関いくつ？
https://qiita.com/kaizen_nagoya/items/5fb44773bc38574bcf1c
dockerで機械学習(75)環境構築(5)docker関連ファイルの管理
https://qiita.com/kaizen_nagoya/items/4f03df9a42c923087b5d
「名古屋のIoTは名古屋のOSで」Dockerをどっかーらどうやって使えばいいんでしょう。TOPPERS/FMP on RaspberryPi with Macintosh編 ５つの関門
https://qiita.com/kaizen_nagoya/items/9c46c6da8ceb64d2d7af
64bitCPUへの道 and/or 64歳の決意
https://qiita.com/kaizen_nagoya/items/cfb5ffa24ded23ab3f60
ゼロから作るDeepLearning２自然言語処理編 読書会の進め方（例）
https://qiita.com/kaizen_nagoya/items/025eb3f701b36209302e
Docker for windows
BIOSでInte Virtualization をenableにしていないと動作しないことを書いていない記事が多い。なぜかは調査中。
今更Docker for Windowsをインストールしたのでその備忘録
https://qiita.com/toro_ponz/items/d66d5571c4646ad33279
Docker for Windowsで起動時に「Docker for Windows - Access denied」と表示される場合の対処法
https://qiita.com/toro_ponz/items/d75706a3039f00ba1205
Windows 10 Hyper-Vユーザに送るDockerの解説
https://qiita.com/banban525/items/48aec05671c3c77d454a
Docker for WindowsでDockerを学ぶ (バージョンCE 17.06.2)
https://qiita.com/rubytomato@github/items/eec2118e89ee9bd8d17a
Docker for Windows をインストールする
https://qiita.com/centipede/items/f8d77b66343ef5096eee
Windows 10(Surface)にDocker for Windowsをインストール
https://qiita.com/chakimar/items/868298096ebf9186d690
後日談
# pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 1.7MB/s
twisted 18.7.0 requires PyHamcrest>=1.9.0, which is not installed.
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-18.1
/# cd ml-at-work/
(base) root@1f725dc9d543:/ml-at-work# pip install -r requirements.txt -c constraints.txt
Collecting Cython==0.27.3 (from -c constraints.txt (line 3))
Downloading https://files.pythonhosted.org/packages/ee/2a/c4d2cdd19c84c32d978d18e9355d1ba9982a383de87d0fcb5928553d37f4/Cython-0.27.3.tar.gz (1.8MB)
100% |████████████████████████████████| 1.8MB 1.8MB/s
Collecting fastFM==0.2.11 (from -c constraints.txt (line 6))
Could not find a version that satisfies the requirement fastFM==0.2.11 (from -c constraints.txt (line 6)) (from versions: 0.2.3, 0.2.5, 0.2.6, 0.2.9, 0.2.10)
No matching distribution found for fastFM==0.2.11 (from -c constraints.txt (line 6))
(base) root@1f725dc9d543:/ml-at-work# pip install -r requirements.txt
Requirement already satisfied: matplotlib in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (2.2.3)
Requirement already satisfied: Cython in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (0.28.5)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (1.15.1)
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.19.2)
Collecting fastFM (from -r requirements.txt (line 5))
Downloading https://files.pythonhosted.org/packages/41/31/7fb81ab6b11bd35f085eb9b4d0e2e48158056e4a639a1e67057519259512/fastFM-0.2.10.tar.gz (1.6MB)
100% |████████████████████████████████| 1.6MB 755kB/s
Requirement already satisfied: jupyter in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.0.0)
Requirement already satisfied: notebook in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.6.0)
Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (0.23.4)
Requirement already satisfied: statsmodels in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (0.9.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (2.7.3)
Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (2018.5)
Requirement already satisfied: six>=1.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (1.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from fastFM->-r requirements.txt (line 5)) (1.1.0)
Requirement already satisfied: nbconvert in /opt/conda/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 6)) (5.4.0)
Requirement already satisfied: ipykernel in /opt/conda/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 6)) (4.9.0)
Requirement already satisfied: jupyter-console in /opt/conda/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 6)) (5.2.0)
Requirement already satisfied: ipywidgets in /opt/conda/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 6)) (7.4.1)
Requirement already satisfied: qtconsole in /opt/conda/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 6)) (4.4.1)
Requirement already satisfied: jupyter-core>=4.4.0 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (4.4.0)
Requirement already satisfied: Send2Trash in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (1.5.0)
Requirement already satisfied: jupyter-client>=5.2.0 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (5.2.3)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (0.3.1)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (2.10)
Requirement already satisfied: terminado>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (0.8.1)
Requirement already satisfied: tornado>=4 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (5.1)
Requirement already satisfied: pyzmq>=17 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (17.1.2)
Requirement already satisfied: nbformat in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (4.4.0)
Requirement already satisfied: traitlets>=4.2.1 in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (4.3.2)
Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.7/site-packages (from notebook->-r requirements.txt (line 7)) (0.2.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->-r requirements.txt (line 1)) (40.2.0)
Requirement already satisfied: mistune>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (0.8.3)
Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (2.2.0)
Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (0.2.3)
Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (2.1.4)
Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (1.4.2)
Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (0.3.1)
Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 6)) (0.5.0)
Requirement already satisfied: ipython>=4.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel->jupyter->-r requirements.txt (line 6)) (6.5.0)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from jupyter-console->jupyter->-r requirements.txt (line 6)) (1.0.15)
Requirement already satisfied: widgetsnbextension~=3.4.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets->jupyter->-r requirements.txt (line 6)) (3.4.1)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2->notebook->-r requirements.txt (line 7)) (1.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.7/site-packages (from nbformat->notebook->-r requirements.txt (line 7)) (2.6.0)
Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from traitlets>=4.2.1->notebook->-r requirements.txt (line 7)) (4.3.0)
Requirement already satisfied: html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert->jupyter->-r requirements.txt (line 6)) (1.0.1)
Requirement already satisfied: simplegeneric>0.8 in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.8.1)
Requirement already satisfied: jedi>=0.10 in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.12.1)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.7.4)
Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.1.0)
Requirement already satisfied: pexpect; sys_platform != "win32" in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (4.6.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit<2.0.0,>=1.0.0->jupyter-console->jupyter->-r requirements.txt (line 6)) (0.1.7)
Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre->bleach->nbconvert->jupyter->-r requirements.txt (line 6)) (0.5.1)
Requirement already satisfied: parso>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.10->ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.3.1)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0->ipykernel->jupyter->-r requirements.txt (line 6)) (0.6.0)
Building wheels for collected packages: fastFM
Running setup.py bdist_wheel for fastFM ... error
Complete output from command /opt/conda/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-l8x_0ss5/fastFM/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-jtkzcf0u --python-tag cp37:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/fastFM
copying fastFM/als.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/base.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/utils.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/validation.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/mcmc.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/sgd.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/__init__.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/bpr.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/datasets.py -> build/lib.linux-x86_64-3.7/fastFM
running build_ext
skipping 'fastFM/ffm.c' Cython extension (up-to-date)
building 'ffm' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/fastFM
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -IfastFM/ -IfastFM-core/include/ -IfastFM-core/externals/CXSparse/Include/ -I/opt/conda/lib/python3.7/site-packages/numpy/core/include -I/opt/conda/include/python3.7m -c fastFM/ffm.c -o build/temp.linux-x86_64-3.7/fastFM/ffm.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for fastFM
Running setup.py clean for fastFM
Failed to build fastFM
Installing collected packages: fastFM
Running setup.py install for fastFM ... error
Complete output from command /opt/conda/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-l8x_0ss5/fastFM/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-g7xwdofi/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/fastFM
copying fastFM/als.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/base.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/utils.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/validation.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/mcmc.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/sgd.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/__init__.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/bpr.py -> build/lib.linux-x86_64-3.7/fastFM
copying fastFM/datasets.py -> build/lib.linux-x86_64-3.7/fastFM
running build_ext
skipping 'fastFM/ffm.c' Cython extension (up-to-date)
building 'ffm' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/fastFM
gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -IfastFM/ -IfastFM-core/include/ -IfastFM-core/externals/CXSparse/Include/ -I/opt/conda/lib/python3.7/site-packages/numpy/core/include -I/opt/conda/include/python3.7m -c fastFM/ffm.c -o build/temp.linux-x86_64-3.7/fastFM/ffm.o
unable to execute 'gcc': No such file or directory
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/opt/conda/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-l8x_0ss5/fastFM/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-g7xwdofi/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-l8x_0ss5/fastFM/
＜この記事は個人の過去の経験に基づく個人の感想です。現在所属する組織、業務とは関係がありません。＞
#6. 文書履歴(document history)
ver. 0.10 初稿 20180928
ver. 0.11 参考文献追記 20181019
最後までおよみいただきありがとうございました。
いいね 💚、フォローをお願いします。
Thank you very much for reading to the last sentence.
Please press the like icon 💚 and follow me for your happy life.