概要
TensorFlow をソースからコンパイルする. 最新の CUDA, cuDNN に対応したり, AVX がサポートされていない CPU で動作させたり, 最適化のオプションを追加したりするためにはソースからコンパイルする必要がある.
以下を参考にすすめていく.
Ubuntu のインストールから開発環境のセットアップまでは以下ページを参照.
レポジトリ clone
1.10.1 の場合は r1.10 を指定する.
git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout r1.10
Bazel インストール
以下を参考に.
sudo apt install openjdk-8-jdk
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
sudo apt update && sudo apt install bazel
依存パッケージインストール
以下を参考に.
sudo apt install python3-numpy python3-dev python3-pip python3-wheel
configure
以下を参考に.
あらかじめ以下のページで自分の GPU の Compute Capability を調べておく. GeForce GTX 1050 の場合は 6.1.
configure を実行.
./configure
以下の 4 箇所だけ入力して, それ以外はすべて Enter.
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3
Please input the desired Python library path to use. Default is [/usr/local/lib/python3.5/dist-packages]
/usr/lib/python3/dist-packages
Do you wish to build TensorFlow with CUDA support? [y/N]: Y
Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3
実際のログはこちら.
./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.16.1 installed.
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3
Found possible Python library paths:
/usr/local/lib/python3.5/dist-packages
/usr/lib/python3/dist-packages
Please input the desired Python library path to use. Default is [/usr/local/lib/python3.5/dist-packages]
/usr/lib/python3/dist-packages
Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]:
jemalloc as malloc support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]:
Google Cloud Platform support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Hadoop File System support? [Y/n]:
Hadoop File System support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Amazon AWS Platform support? [Y/n]:
Amazon AWS Platform support will be enabled for TensorFlow.
Do you wish to build TensorFlow with Apache Kafka Platform support? [Y/n]:
Apache Kafka Platform support will be enabled for TensorFlow.
Do you wish to build TensorFlow with XLA JIT support? [y/N]:
No XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with GDR support? [y/N]:
No GDR support will be enabled for TensorFlow.
Do you wish to build TensorFlow with VERBS support? [y/N]:
No VERBS support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]:
No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: Y
CUDA support will be enabled for TensorFlow.
Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]:
Please specify the location where CUDA 9.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]:
Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Do you wish to build TensorFlow with TensorRT support? [y/N]:
No TensorRT support will be enabled for TensorFlow.
Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 6.1]
Do you want to use clang as CUDA compiler? [y/N]:
nvcc will be used as CUDA compiler.
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Do you wish to build TensorFlow with MPI support? [y/N]:
No MPI support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
--config=mkl # Build with MKL support.
--config=monolithic # Config for mostly static monolithic build.
Configuration finished
pip パッケージビルド
以下を参考に.
経験的にはここで一旦再起動してマシンを軽くしておいたほうがよい.
リソース制限をせずに bazel build すると途中でスワップを食い潰してフリーズした(実メモリ 6GB, スワップ 4GB, コア数 8 のマシン). なので, 以下ページを参考に --local_resources 2048,8,1.0 で, メモリ(MB) 2048, コア数 8, I/O capabilities 1.0 に制限した. メモリは cat /proc/meminfo, コア数は cat /proc/cpuinfo で調べられる. I/O capabilities はよくわからないのでデフォルトの 1.0.
ビルドする. 1 時間程度かかる.
time bazel build --local_resources=2048,8,1.0 --verbose_failures --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
もし上手く行ったら --copt=-mfpmath=both, --copt=-msse4.2, --copt=-mavx, --copt=-mavx2, --copt=-mfma, --copt=-O3 などの CPU の最適化オプションを追加してみる. どのオプションに対応しているかは cat /proc/cpuinfo で確認.
time bazel build --local_resources 2048,8,1.0 --verbose_failures --config=opt --config=cuda --copt=-mfpmath=both --copt=-msse4.2 --copt=-O3 //tensorflow/tools/pip_package:build_pip_package
--project_name tensorflow_gpu_cuda_9.0_cudnn_7.0.5 などとするとファイル名とパッケージ名を変えられる.
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg --project_name tensorflow_gpu_cuda_9.0_cudnn_7.0.5
venv で仮想環境を作りインストールしてテストしてみる.
python3 -m venv venv
source venv/bin/activate
pip list
pip install wheel
pip install /tmp/tensorflow_pkg/tensorflow_gpu_cuda_9.0_cudnn_7.0.5-1.10.1-cp35-cp35m-linux_x86_64.whl
pip list
python -c "import tensorflow as tf; print(tf.__version__)"
バージョンが表示されれば OK.
以上