[Python3 / caffe] python3でcaffeを動かすためのdocker imageをbuildする

1. 環境

bash

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
$ docker --version
Docker version 20.10.17, build 100c701
$ docker compose version
Docker Compose version v2.6.0

	version
Host machine	Ubuntu 20.04
Docker	20.10.17
docker compose	v2.6.0

です。

2. 頼りにした記事

このあたりのDockerfileを元にimageを作成してみる。

3. Let's try !

ディレクトリ構成

app_root
    ├──docker
    │     └── caffe_py3
    │           └── Dockerfile
    └── docker-compose.yml

とりあえず実行環境はCPU用のイメージにしておく。

3-1. 初期状態（動かない）

3-1-1. ソースコード

Dockerfile

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    git \
    wget \
    libatlas-base-dev \
    libboost-all-dev \
    libgflags-dev \
    libgoogle-glog-dev \
    libhdf5-serial-dev \
    libleveldb-dev \
    liblmdb-dev \
    libopencv-dev \
    libprotobuf-dev \
    libsnappy-dev \
    protobuf-compiler \
    python3-dev \
    python3-numpy \
    python3-pip \
    python3-setuptools \
    && rm -rf /var/lib/apt/lists/*

ENV CAFFE_ROOT=/opt/caffe
WORKDIR $CAFFE_ROOT

# FIXME: use ARG instead of ENV once DockerHub supports this
# https://github.com/docker/hub-feedback/issues/460
ENV CLONE_TAG=1.0

RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/BVLC/caffe.git . && \
    pip3 install --upgrade pip 

RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done

RUN pip3 install scipy protobuf python-dateutil numpy --upgrade

RUN mkdir build && cd build && \
    cmake -DCPU_ONLY=1 -Dpython_version=3 .. && \
    make -j"$(nproc)"

ENV PYCAFFE_ROOT $CAFFE_ROOT/python
ENV PYTHONPATH $PYCAFFE_ROOT:$PYTHONPATH
ENV PATH $CAFFE_ROOT/build/tools:$PYCAFFE_ROOT:$PATH
RUN echo "$CAFFE_ROOT/build/lib" >> /etc/ld.so.conf.d/caffe.conf && ldconfig

WORKDIR /workspace

docker-compose.yml

version: '3.7'
services:
  caffe:
    container_name: caffe
    build:
      context: ./docker/caffe_py3
    command: >
      bash -c "echo 'Jupyter with caffe is starting ...'"

3-1-2. 動作確認（buildに失敗）

$ docker compose build
[+] Building 4.9s (9/13)                                                                            
 => [internal] load build definition from Dockerfile                                           0.1s
 => => transferring dockerfile: 32B                                                            0.0s
 => [internal] load .dockerignore                                                              0.1s
 => => transferring context: 2B                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:16.04                                2.6s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                  0.0s
 => [1/9] FROM docker.io/library/ubuntu:16.04@sha256:91bd29a464fdabfcf44e29e1f2a5f213c6dfa750  0.0s
 => CACHED [2/9] RUN apt-get update && apt-get install -y --no-install-recommends     build-e  0.0s
 => CACHED [3/9] WORKDIR /opt/caffe                                                            0.0s
 => CACHED [4/9] RUN git clone -b 1.0 --depth 1 https://github.com/BVLC/caffe.git . &&     pi  0.0s
 => ERROR [5/9] RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $r  2.3s
------                                                                                              
 > [5/9] RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done:     
#0 1.133 Traceback (most recent call last):                                                         
#0 1.133   File "/usr/local/bin/pip3", line 7, in <module>                                          
#0 1.133     from pip._internal.cli.main import main                                                
#0 1.133   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57         
#0 1.133     sys.stderr.write(f"ERROR: {exc}")
#0 1.133                                    ^
#0 1.133 SyntaxError: invalid syntax
#0 1.185 Traceback (most recent call last):
#0 1.185   File "/usr/local/bin/pip3", line 7, in <module>
#0 1.185     from pip._internal.cli.main import main
#0 1.185   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 1.185     sys.stderr.write(f"ERROR: {exc}")
#0 1.185                                    ^
#0 1.185 SyntaxError: invalid syntax
#0 1.247 Traceback (most recent call last):
#0 1.247   File "/usr/local/bin/pip3", line 7, in <module>
#0 1.247     from pip._internal.cli.main import main
#0 1.247   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 1.247     sys.stderr.write(f"ERROR: {exc}")
#0 1.247                                    ^

...略

#0 2.075 SyntaxError: invalid syntax
#0 2.144 Traceback (most recent call last):
#0 2.144   File "/usr/local/bin/pip3", line 7, in <module>
#0 2.144     from pip._internal.cli.main import main
#0 2.144   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 2.144     sys.stderr.write(f"ERROR: {exc}")
#0 2.144                                    ^
#0 2.144 SyntaxError: invalid syntax
#0 2.209 Traceback (most recent call last):
#0 2.209   File "/usr/local/bin/pip3", line 7, in <module>
#0 2.209     from pip._internal.cli.main import main
#0 2.209   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 2.209     sys.stderr.write(f"ERROR: {exc}")
#0 2.209                                    ^
#0 2.209 SyntaxError: invalid syntax
------
failed to solve: executor failed running [/bin/sh -c cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done]: exit code: 1

え、ソースコードそのままコピーしたのにSyntaxErrorだと？
さすがにそうは簡単にいかないらしい。。。

3-2. 対策

3-2-1. ソースコード

試行錯誤の結果、以下のようにDockerfileを修正することで、python3上でcaffeを動かすことに成功しました。

FROM ubuntu:18.04

RUN apt-get update \
    # For avoiding the following issue https://www.yamamanx.com/ubuntu-docker-build-timezone-stop/
    && apt-get install -y tzdata
RUN apt-get install -y --no-install-recommends \
    build-essential \
    # To avoid `Problem with the SSL CA cert (path? access rights?)` on git clone
    ca-certificates \
    cmake \
    curl \
    git \
    wget \
    libatlas-base-dev \
    libboost-all-dev \
    libgflags-dev \
    libgoogle-glog-dev \
    libhdf5-serial-dev \
    libleveldb-dev \
    liblmdb-dev \
    libopencv-dev \
    libprotobuf-dev \
    libsnappy-dev \
    protobuf-compiler \
    # https://doitu.info/blog/448768ab50ca208f7648018a6a4ea9f1ef7b1311
    # python3.7-dev \
    python3-numpy \
    python3-pip \
    python3-setuptools \
    python-scipy \
    && rm -rf /var/lib/apt/lists/*

ENV CAFFE_ROOT=/opt/caffe
WORKDIR $CAFFE_ROOT

# FIXME: use ARG instead of ENV once DockerHub supports this
# https://github.com/docker/hub-feedback/issues/460
ENV CLONE_TAG=1.0

RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/BVLC/caffe.git . \
    # NOTE: SyntaxError: invalid syntax (sys.stderr.write(f"ERROR: {exc}")) の回避のため `==20.3` を指定
    #     https://zetton86.github.io/blog/20210422-remove-pip-error/
    && pip3 install --upgrade pip

RUN cd python && for req in $(cat requirements.txt) pydot 'python-dateutil>2'; do pip3 install $req; done && cd ..

# RUN pip3 install scipy protobuf python-dateutil numpy --upgrade

RUN mkdir build && cd build && \
    cmake -DCPU_ONLY=1 -Dpython_version=3 .. && \
    make -j"$(nproc)"

ENV PYCAFFE_ROOT $CAFFE_ROOT/python
ENV PYTHONPATH $PYCAFFE_ROOT:$PYTHONPATH
ENV PATH $CAFFE_ROOT/build/tools:$PYCAFFE_ROOT:$PATH
RUN echo "$CAFFE_ROOT/build/lib" >> /etc/ld.so.conf.d/caffe.conf && ldconfig

WORKDIR /workspace

これでimage buildとimport caffeは成功する。

3-2-2. 動作確認

結果は以下の通り。

$ docker compose build
$ docker compose run caffe bash
# python3 --version
Python 3.6.9
# python3
Python 3.6.9 (default, Jun 29 2022, 11:45:57) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> dir(caffe)
['AdaDeltaSolver', 'AdaGradSolver', 'AdamSolver', 'Classifier', 'Detector', 'Layer', 'NCCL', 'NesterovSolver', 'Net', 'NetSpec', 'RMSPropSolver', 'SGDSolver', 'TEST', 'TRAIN', 'Timer', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_caffe', 'classifier', 'detector', 'get_solver', 'has_nccl', 'init_log', 'io', 'layer_type_list', 'layers', 'log', 'net_spec', 'params', 'proto', 'pycaffe', 'set_device', 'set_mode_cpu', 'set_mode_gpu', 'set_multiprocess', 'set_random_seed', 'set_solver_count', 'set_solver_rank', 'solver_count', 'solver_rank', 'to_proto']
>>> caffe.set_mode_cpu()

うし。

ちょっと、これだけだと本当に動くか不安なので、１個モデル作った動かしてみないとね。