Go to Qiita Advent Calendar 2024 Top
LoginSignup
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@siruku6

[Python3 / caffe] python3でcaffeを動かすためのdocker imageをbuildする

Posted at

1. 環境

bash
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
$ docker --version
Docker version 20.10.17, build 100c701
$ docker compose version
Docker Compose version v2.6.0
version
Host machine Ubuntu 20.04
Docker 20.10.17
docker compose v2.6.0

です。

2. 頼りにした記事

このあたりのDockerfileを元にimageを作成してみる。

3. Let's try !

ディレクトリ構成
app_root
    ├──docker
    │     └── caffe_py3
    │           └── Dockerfile
    └── docker-compose.yml

とりあえず実行環境はCPU用のイメージにしておく。

3-1. 初期状態（動かない）

3-1-1. ソースコード

Dockerfile
FROM ubuntu:16.04

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    git \
    wget \
    libatlas-base-dev \
    libboost-all-dev \
    libgflags-dev \
    libgoogle-glog-dev \
    libhdf5-serial-dev \
    libleveldb-dev \
    liblmdb-dev \
    libopencv-dev \
    libprotobuf-dev \
    libsnappy-dev \
    protobuf-compiler \
    python3-dev \
    python3-numpy \
    python3-pip \
    python3-setuptools \
    && rm -rf /var/lib/apt/lists/*

ENV CAFFE_ROOT=/opt/caffe
WORKDIR $CAFFE_ROOT

# FIXME: use ARG instead of ENV once DockerHub supports this
# https://github.com/docker/hub-feedback/issues/460
ENV CLONE_TAG=1.0

RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/BVLC/caffe.git . && \
    pip3 install --upgrade pip 

RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done

RUN pip3 install scipy protobuf python-dateutil numpy --upgrade

RUN mkdir build && cd build && \
    cmake -DCPU_ONLY=1 -Dpython_version=3 .. && \
    make -j"$(nproc)"

ENV PYCAFFE_ROOT $CAFFE_ROOT/python
ENV PYTHONPATH $PYCAFFE_ROOT:$PYTHONPATH
ENV PATH $CAFFE_ROOT/build/tools:$PYCAFFE_ROOT:$PATH
RUN echo "$CAFFE_ROOT/build/lib" >> /etc/ld.so.conf.d/caffe.conf && ldconfig

WORKDIR /workspace
docker-compose.yml
version: '3.7'
services:
  caffe:
    container_name: caffe
    build:
      context: ./docker/caffe_py3
    command: >
      bash -c "echo 'Jupyter with caffe is starting ...'"

3-1-2. 動作確認（buildに失敗）

$ docker compose build
[+] Building 4.9s (9/13)                                                                            
 => [internal] load build definition from Dockerfile                                           0.1s
 => => transferring dockerfile: 32B                                                            0.0s
 => [internal] load .dockerignore                                                              0.1s
 => => transferring context: 2B                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:16.04                                2.6s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                  0.0s
 => [1/9] FROM docker.io/library/ubuntu:16.04@sha256:91bd29a464fdabfcf44e29e1f2a5f213c6dfa750  0.0s
 => CACHED [2/9] RUN apt-get update && apt-get install -y --no-install-recommends     build-e  0.0s
 => CACHED [3/9] WORKDIR /opt/caffe                                                            0.0s
 => CACHED [4/9] RUN git clone -b 1.0 --depth 1 https://github.com/BVLC/caffe.git . &&     pi  0.0s
 => ERROR [5/9] RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $r  2.3s
------                                                                                              
 > [5/9] RUN cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done:     
#0 1.133 Traceback (most recent call last):                                                         
#0 1.133   File "/usr/local/bin/pip3", line 7, in <module>                                          
#0 1.133     from pip._internal.cli.main import main                                                
#0 1.133   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57         
#0 1.133     sys.stderr.write(f"ERROR: {exc}")
#0 1.133                                    ^
#0 1.133 SyntaxError: invalid syntax
#0 1.185 Traceback (most recent call last):
#0 1.185   File "/usr/local/bin/pip3", line 7, in <module>
#0 1.185     from pip._internal.cli.main import main
#0 1.185   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 1.185     sys.stderr.write(f"ERROR: {exc}")
#0 1.185                                    ^
#0 1.185 SyntaxError: invalid syntax
#0 1.247 Traceback (most recent call last):
#0 1.247   File "/usr/local/bin/pip3", line 7, in <module>
#0 1.247     from pip._internal.cli.main import main
#0 1.247   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 1.247     sys.stderr.write(f"ERROR: {exc}")
#0 1.247                                    ^

...略

#0 2.075 SyntaxError: invalid syntax
#0 2.144 Traceback (most recent call last):
#0 2.144   File "/usr/local/bin/pip3", line 7, in <module>
#0 2.144     from pip._internal.cli.main import main
#0 2.144   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 2.144     sys.stderr.write(f"ERROR: {exc}")
#0 2.144                                    ^
#0 2.144 SyntaxError: invalid syntax
#0 2.209 Traceback (most recent call last):
#0 2.209   File "/usr/local/bin/pip3", line 7, in <module>
#0 2.209     from pip._internal.cli.main import main
#0 2.209   File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 57
#0 2.209     sys.stderr.write(f"ERROR: {exc}")
#0 2.209                                    ^
#0 2.209 SyntaxError: invalid syntax
------
failed to solve: executor failed running [/bin/sh -c cd python && for req in $(cat requirements.txt) pydot; do pip3 install $req; done]: exit code: 1

え、ソースコードそのままコピーしたのにSyntaxErrorだと？
さすがにそうは簡単にいかないらしい。。。

3-2. 対策

3-2-1. ソースコード

試行錯誤の結果、以下のようにDockerfileを修正することで、python3上でcaffeを動かすことに成功しました。

FROM ubuntu:18.04

RUN apt-get update \
    # For avoiding the following issue https://www.yamamanx.com/ubuntu-docker-build-timezone-stop/
    && apt-get install -y tzdata
RUN apt-get install -y --no-install-recommends \
    build-essential \
    # To avoid `Problem with the SSL CA cert (path? access rights?)` on git clone
    ca-certificates \
    cmake \
    curl \
    git \
    wget \
    libatlas-base-dev \
    libboost-all-dev \
    libgflags-dev \
    libgoogle-glog-dev \
    libhdf5-serial-dev \
    libleveldb-dev \
    liblmdb-dev \
    libopencv-dev \
    libprotobuf-dev \
    libsnappy-dev \
    protobuf-compiler \
    # https://doitu.info/blog/448768ab50ca208f7648018a6a4ea9f1ef7b1311
    # python3.7-dev \
    python3-numpy \
    python3-pip \
    python3-setuptools \
    python-scipy \
    && rm -rf /var/lib/apt/lists/*

ENV CAFFE_ROOT=/opt/caffe
WORKDIR $CAFFE_ROOT

# FIXME: use ARG instead of ENV once DockerHub supports this
# https://github.com/docker/hub-feedback/issues/460
ENV CLONE_TAG=1.0

RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/BVLC/caffe.git . \
    # NOTE: SyntaxError: invalid syntax (sys.stderr.write(f"ERROR: {exc}")) の回避のため `==20.3` を指定
    #     https://zetton86.github.io/blog/20210422-remove-pip-error/
    && pip3 install --upgrade pip

RUN cd python && for req in $(cat requirements.txt) pydot 'python-dateutil>2'; do pip3 install $req; done && cd ..

# RUN pip3 install scipy protobuf python-dateutil numpy --upgrade

RUN mkdir build && cd build && \
    cmake -DCPU_ONLY=1 -Dpython_version=3 .. && \
    make -j"$(nproc)"

ENV PYCAFFE_ROOT $CAFFE_ROOT/python
ENV PYTHONPATH $PYCAFFE_ROOT:$PYTHONPATH
ENV PATH $CAFFE_ROOT/build/tools:$PYCAFFE_ROOT:$PATH
RUN echo "$CAFFE_ROOT/build/lib" >> /etc/ld.so.conf.d/caffe.conf && ldconfig

WORKDIR /workspace

これでimage buildとimport caffeは成功する。

3-2-2. 動作確認

結果は以下の通り。

$ docker compose build
$ docker compose run caffe bash
# python3 --version
Python 3.6.9
# python3
Python 3.6.9 (default, Jun 29 2022, 11:45:57) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> dir(caffe)
['AdaDeltaSolver', 'AdaGradSolver', 'AdamSolver', 'Classifier', 'Detector', 'Layer', 'NCCL', 'NesterovSolver', 'Net', 'NetSpec', 'RMSPropSolver', 'SGDSolver', 'TEST', 'TRAIN', 'Timer', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_caffe', 'classifier', 'detector', 'get_solver', 'has_nccl', 'init_log', 'io', 'layer_type_list', 'layers', 'log', 'net_spec', 'params', 'proto', 'pycaffe', 'set_device', 'set_mode_cpu', 'set_mode_gpu', 'set_multiprocess', 'set_random_seed', 'set_solver_count', 'set_solver_rank', 'solver_count', 'solver_rank', 'to_proto']
>>> caffe.set_mode_cpu()

うし。

ちょっと、これだけだと本当に動くか不安なので、１個モデル作った動かしてみないとね。

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?