GCE × CentOS8 で Jupyter 環境を構築する
タイトルのままですが、CentOS8 が使えたので、勉強がてら触ってみました。
基本的には Docker および docker-compose を使用してコンテナの Jupyter 環境を構築します。
docker のインストール
どんな方法でもいいので、GCP 上でインスタンスを立ててください。
その際に CentOS8 のイメージにするのを忘れずに。
SSH でコンソールに繋いだところからはじめます。
まずは yum update の代わり。
[user@jupyter ~]$ sudo su -
[root@jupyter ~]# dnf -y update
CentOS-8 - AppStream 1.5 MB/s | 5.6 MB 00:03
CentOS-8 - Base 2.0 MB/s | 5.3 MB 00:02
CentOS-8 - Extras 1.5 kB/s | 2.1 kB 00:01
Google Cloud SDK 5.9 MB/s | 27 MB 00:04
Google Compute Engine 1.6 kB/s | 5.7 kB 00:03
Dependencies resolved.
===================================================================================================================
Package Arch Version Repository Size
===================================================================================================================
Upgrading:
google-cloud-sdk noarch 266.0.0-1 google-cloud-sdk 35 M
Transaction Summary
===================================================================================================================
Upgrade 1 Package
Total download size: 35 M
Downloading Packages:
8ad4cecdf8d21424d0e49bca00836de86cb608625384064f5d6a0c73eae2df16-google-cloud-sdk- 12 MB/s | 35 MB 00:02
-------------------------------------------------------------------------------------------------------------------
Total 12 MB/s | 35 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : google-cloud-sdk-266.0.0-1.noarch 1/2
Running scriptlet: google-cloud-sdk-266.0.0-1.noarch 1/2
Running scriptlet: google-cloud-sdk-265.0.0-1.noarch 2/2
Cleanup : google-cloud-sdk-265.0.0-1.noarch 2/2
Running scriptlet: google-cloud-sdk-265.0.0-1.noarch 2/2
Verifying : google-cloud-sdk-266.0.0-1.noarch 1/2
Verifying : google-cloud-sdk-265.0.0-1.noarch 2/2
Upgraded:
google-cloud-sdk-266.0.0-1.noarch
Complete!
installの前準備
[root@jupyter ~]# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@jupyter ~]# dnf repolist
Docker CE Stable - x86_64 3.0 kB/s | 3.5 kB 00:01
Google Cloud SDK 195 B/s | 454 B 00:02
Google Compute Engine 199 B/s | 454 B 00:02
repo id repo name status
AppStream CentOS-8 - AppStream 4,928
BaseOS CentOS-8 - Base 2,713
docker-ce-stable Docker CE Stable - x86_64 53
extras CentOS-8 - Extras 3
google-cloud-sdk Google Cloud SDK 948
google-compute-engine Google Compute Engine 6
なぜかそのままinstallができないので
別バージョンを指定
[root@jupyter ~]# dnf -y install wget
Last metadata expiration check: 0:02:03 ago on Fri 11 Oct 2019 01:56:03 AM UTC.
Dependencies resolved.
===================================================================================================================
Package Arch Version Repository Size
===================================================================================================================
Installing:
wget x86_64 1.19.5-7.el8_0.1 AppStream 734 k
Transaction Summary
===================================================================================================================
Install 1 Package
Total download size: 734 k
Installed size: 2.8 M
Downloading Packages:
wget-1.19.5-7.el8_0.1.x86_64.rpm 608 kB/s | 734 kB 00:01
---
Total 318 kB/s | 734 kB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : wget-1.19.5-7.el8_0.1.x86_64 1/1
Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 1/1
Verifying : wget-1.19.5-7.el8_0.1.x86_64 1/1
Installed:
wget-1.19.5-7.el8_0.1.x86_64
Complete!
[root@jupyter ~]# wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.2-3.3.el7.x86_64.rpm
--2019-10-11 01:58:53-- https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.2-3.3.
el7.x86_64.rpm
Resolving download.docker.com (download.docker.com)... 13.249.87.50, 13.249.87.47, 13.249.87.20, ...
Connecting to download.docker.com (download.docker.com)|13.249.87.50|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23159364 (22M) [binary/octet-stream]
Saving to: ‘containerd.io-1.2.2-3.3.el7.x86_64.rpm’
containerd.io-1.2.2-3.3.el7. 100%[=============================================>] 22.09M 86.2MB/s in 0.3s
2019-10-11 01:58:53 (86.2 MB/s) - ‘containerd.io-1.2.2-3.3.el7.x86_64.rpm’ saved [23159364/23159364][root@jupyter ~]#
[root@jupyter ~]# dnf remove containerd.io
No match for argument: containerd.io
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[root@jupyter ~]# dnf install containerd.io-1.2.2-3.3.el7.x86_64.rpm
Last metadata expiration check: 0:01:55 ago on Fri 11 Oct 2019 02:26:59 AM UTC.
Dependencies resolved.
=============================================================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================================================
Installing:
containerd.io x86_64 1.2.2-3.3.el7 @commandline 22 M
# Transaction Summary
Install 1 Package
Total size: 22 M
Installed size: 88 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : containerd.io-1.2.2-3.3.el7.x86_64 1/1
Running scriptlet: containerd.io-1.2.2-3.3.el7.x86_64 1/1
Verifying : containerd.io-1.2.2-3.3.el7.x86_64 1/1
Installed:
containerd.io-1.2.2-3.3.el7.x86_64
Complete!
[root@jupyter ~]# dnf install -y docker-ce docker-ce-cli
Last metadata expiration check: 0:04:01 ago on Fri 11 Oct 2019 01:56:03 AM UTC.
Dependencies resolved.
===================================================================================================================
Package Arch Version Repository Size
===================================================================================================================
Installing:
docker-ce x86_64 3:19.03.3-3.el7 docker-ce-stable 24 M
docker-ce-cli x86_64 1:19.03.3-3.el7 docker-ce-stable 39 M
Installing dependencies:
container-selinux noarch 2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7 AppStream 43 k
libcgroup x86_64 0.41-19.el8 BaseOS 70 k
policycoreutils-python-utils noarch 2.8-16.1.el8 BaseOS 228 k
Enabling module streams:
container-tools rhel8
Transaction Summary
===================================================================================================================
Install 5 Packages
Total download size: 64 M
Installed size: 273 M
Downloading Packages:
(1/5): libcgroup-0.41-19.el8.x86_64.rpm 62 kB/s | 70 kB 00:01
(2/5): container-selinux-2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch.rpm 39 kB/s | 43 kB 00:01
(3/5): policycoreutils-python-utils-2.8-16.1.el8.noarch.rpm 187 kB/s | 228 kB 00:01
(4/5): docker-ce-19.03.3-3.el7.x86_64.rpm 41 MB/s | 24 MB 00:00
(5/5): docker-ce-cli-19.03.3-3.el7.x86_64.rpm 45 MB/s | 39 MB 00:00
---
Total 6.4 MB/s | 64 MB 00:10
warning: /var/cache/dnf/docker-ce-stable-091d8a9c23201250/packages/docker-ce-19.03.3-3.el7.x86_64.rpm: Header V4 RS
A/SHA512 Signature, key ID 621e9f35: NOKEY
Docker CE Stable - x86_64 1.4 kB/s | 1.6 kB 00:01
Importing GPG key 0x621E9F35:
Userid : "Docker Release (CE rpm) <docker@docker.com>"
Fingerprint: 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35
From : https://download.docker.com/linux/centos/gpg
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : docker-ce-cli-1:19.03.3-3.el7.x86_64 1/5
Running scriptlet: docker-ce-cli-1:19.03.3-3.el7.x86_64 1/5
Installing : policycoreutils-python-utils-2.8-16.1.el8.noarch 2/5
Installing : container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 3/5
Running scriptlet: container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 3/5
Running scriptlet: libcgroup-0.41-19.el8.x86_64 4/5
Installing : libcgroup-0.41-19.el8.x86_64 4/5
Running scriptlet: libcgroup-0.41-19.el8.x86_64 4/5
Installing : docker-ce-3:19.03.3-3.el7.x86_64 5/5
Running scriptlet: docker-ce-3:19.03.3-3.el7.x86_64 5/5
Verifying : container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 1/5
Verifying : libcgroup-0.41-19.el8.x86_64 2/5
Verifying : policycoreutils-python-utils-2.8-16.1.el8.noarch 3/5
Verifying : docker-ce-3:19.03.3-3.el7.x86_64 4/5
Verifying : docker-ce-cli-1:19.03.3-3.el7.x86_64 5/5
Installed:
docker-ce-3:19.03.3-3.el7.x86_64 docker-ce-cli-1:19.03.3-3.el7.x86_64
container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch libcgroup-0.41-19.el8.x86_64
policycoreutils-python-utils-2.8-16.1.el8.noarch
Complete!
[root@jupyter ~]# docker --version
Docker version 19.03.3, build a872fc2f86
[root@jupyter ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
入りましたね。
自動起動設定をしておく
[root@jupyter ~]# systemctl start docker
[root@jupyter ~]# systemctl enable docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
docker-composeは入っていない状態。
[root@jupyter ~]# docker-compose
-bash: docker-compose: command not found
dnfもない。
[root@jupyter ~]# dnf -y install docker-compose
Last metadata expiration check: 0:07:38 ago on Fri 11 Oct 2019 01:56:03 AM UTC.
No match for argument: docker-compose
Error: Unable to find a match
公式の通りにいれる
[root@jupyter ~]# curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 617 0 617 0 0 2924 0 --:--:-- --:--:-- --:--:-- 2910
100 15.4M 100 15.4M 0 0 20.9M 0 --:--:-- --:--:-- --:--:-- 20.9M
[root@jupyter ~]# chmod +x /usr/local/bin/docker-compose
[root@jupyter ~]# ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
[root@jupyter ~]# docker-compose --version
docker-compose version 1.24.1, build 4667896b
うまく入った。
Jupyterコンテナの準備
Dockerの準備に比べて、やることは少ないです。
juyterのDockerfileを設置したいディレクトリに移動し、
下記のリポジトリをクローンします。
dnf -y install git
git clone https://github.com/hiroyannnn/jupyter.git
cloneするのが嫌な場合は、リポジトリ内のファイルをviなどで作成しても良いです。
cloneした場合、jupyterというディレクトリができるので、cdします。
cd jupyter
configファイルにパスフレーズを入れているので、
opensslなどでパスフレーズを作ってください。
sha1:d4a8b1853fdf:(...省略...)fa181ec59716
みたいなやつです。
# create your key
# and modify config file.
いれる場所はここ
jupyter_notebook_config.py
c.NotebookApp.password = ''
Dockerを起動します。
docker-compose up
コンソールにログが出力されるので、
起動したっぽいログがでたら、
パブリックIP:8888 でアクセスできるはず。
暗号化したパスフレーズを入力してログインしてください。
データはdataディレクトリに入ります。
ついでに、ホスト側のdataは、
GCEの追加diskをマウントしておくと、データ量が増えた時も調整しやすいです。
参考
https://qiita.com/dora_2562/items/24691d3bec4c99c1d794
https://qiita.com/y4m3/items/c2703d4e131e05084b7b
余談
ちなみに一回下記でコマンドミスして
# 本来はこっち
# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# 間違えてliで改行が入っちゃった
dnf config-manager --add-repo https://download.docker.com/li
repolistが失敗するようになりましたが
[root@jupyter ~]# dnf repolist
CentOS-8 - AppStream 1.8 kB/s | 4.3 kB 00:02
CentOS-8 - Base 1.6 kB/s | 3.8 kB 00:02
CentOS-8 - Extras 1.1 kB/s | 1.5 kB 00:01
created by dnf config-manager from https://download.docker.com/li 456 B/s | 550 B 00:01
Error: Failed to synchronize cache for repo 'download.docker.com_li'
そのファイルを消しちゃえばリカバリ可能でした。
[root@jupyter ~]# ls -l /etc/yum.repos.d/
total 56
-rw-r--r--. 1 root root 731 Aug 14 06:42 CentOS-AppStream.repo
-rw-r--r--. 1 root root 712 Aug 14 06:42 CentOS-Base.repo
-rw-r--r--. 1 root root 798 Aug 14 06:42 CentOS-centosplus.repo
-rw-r--r--. 1 root root 1320 Aug 14 06:42 CentOS-CR.repo
-rw-r--r--. 1 root root 668 Aug 14 06:42 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 756 Aug 14 06:42 CentOS-Extras.repo
-rw-r--r--. 1 root root 338 Aug 14 06:42 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 928 Aug 14 06:42 CentOS-Media.repo
-rw-r--r--. 1 root root 736 Aug 14 06:42 CentOS-PowerTools.repo
-rw-r--r--. 1 root root 1382 Aug 14 06:42 CentOS-Sources.repo
-rw-r--r--. 1 root root 74 Aug 14 06:42 CentOS-Vault.repo
-rw-r--r--. 1 root root 2424 Oct 11 01:43 docker-ce.repo
-rw-r--r--. 1 root root 145 Oct 11 01:43 download.docker.com_li.repo
-rw-r--r--. 1 root root 591 Oct 2 22:49 google-cloud.repo
[root@jupyter ~]# rm /etc/yum.repos.d/download.docker.com_li.repo
rm: remove regular file '/etc/yum.repos.d/download.docker.com_li.repo'? y
[root@jupyter ~]# dnf repolist
Docker CE Stable - x86_64 3.0 kB/s | 3.5 kB 00:01
Google Cloud SDK 195 B/s | 454 B 00:02
Google Compute Engine 199 B/s | 454 B 00:02
repo id repo name status
AppStream CentOS-8 - AppStream 4,928
BaseOS CentOS-8 - Base 2,713
docker-ce-stable Docker CE Stable - x86_64 53
extras CentOS-8 - Extras 3
google-cloud-sdk Google Cloud SDK 948
google-compute-engine Google Compute Engine 6
[root@jupyter ~]#