最速の numpy がほしい。

環境

pyenv & virtualenv
Python 3.7.2
macOS Mojave

Python のインストール

本題ではないが、Mojave ではそもそも Python のインストールすら通らないので、一応メモ書き。

brew で zlib と sqlite (sqlite3) をあらかじめ入れておく。

$ CPPFLAGS="-I/usr/local/opt/zlib/include -I/usr/local/opt/sqlite/include" LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/sqlite/lib" PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.2

enable-shared をつけないと ignoring file libpython3.7m.a, file was built for archive which is not the architecture being linked (x86_64): libpython3.7m.a で怒られる。

CPPFLAGS と LDFLAGS は、zlib と sqlite3 のため。
sqlite3 はなくてもいいが、zlib は必須なのでないと動かない。

virtualenv はオマケなので、好きな環境を構築してください。

$ pyenv virtualenv 3.7.2 v372

MKL のインストール

Intel のウェブサイトから、ダウンロードしてインストール。
登録が必要（なように感じるが、ダミーのメールアドレスでもダウンロードはできると思う）。

パスは /opt/intel のままでいいと思う。

numpy のインストール

~/.numpy-site.cfg

~/.numpy-site.cfg にコンフィグのファイルを作る。ありがちなやつ。

numpy-site.cfg

[mkl]
library_dirs = /opt/intel/mkl/lib
include_dirs = /opt/intel/mkl/include
libraries = mkl_rt

GitHub に公式のサンプルがある。

~/.config/pip/pip.conf

こっちは pip でインストールするときのコンフィグ。
インストールするときに、コンパイル済みのバイナリを使わせず、強制的にビルドさせる。

pip.conf

[install]
no-binary = numpy,scipy

[wheel]
no-binary = numpy,scipy

インストール

$ pip install numpy

pip.conf を書いていない場合には、次のように強制的にビルドさせる。

$ pip install --no-binary :all: numpy

テスト

$ python
Python 3.7.2 (default, Jan 19 2019, 00:39:46)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

とりあえず import numpy してみる。


>>> import numpy
Traceback (most recent call last):
  File "/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
  File "/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/overrides.py", line 9, in <module>
    from numpy.core._multiarray_umath import add_docstring, ndarray
ImportError: dlopen(/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/__init__.py", line 47, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /Users/ishotihadus/.pyenv/versions/v372/bin/python),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: dlopen(/Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /Users/ishotihadus/.pyenv/versions/v372/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-darwin.so
  Reason: image not found

はい。

MKL のライブラリを @rpath から探しているのだが、その @rpath に /opt/intel/mkl/lib が含まれていないために起きているらしい。

これは LD_LIBRARY_PATH とかを指定しても動かない。

そこで、エラーが出た so ファイルに、強引に @rpath を追加する。

$ install_name_tool -add_rpath /opt/intel/mkl/lib _multiarray_umath.cpython-37m-darwin.so

こうすると、ちゃんと numpy が読み込める。

>>> import numpy
>>> numpy.show_config()
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']

なお、_multiarray_umath.cpython-37m-darwin.so 以外は mkl を使っていないので、add_rpath しなくても問題ない。

$  objdump -dylibs-used -macho *.so
_dummy.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_multiarray_tests.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_multiarray_umath.cpython-37m-darwin.so:
    @rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_operand_flag_tests.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_rational_tests.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_struct_ufunc_tests.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
_umath_tests.cpython-37m-darwin.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

scipy のインストール

こちらは普通にインストールできる。

$ pip install scipy

>>> import scipy
>>> scipy.show_config()
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']

まとめ

ぜってぇバグだろこれ。

追記

site.cfg にオプションを追加しても rpath を追加できる。