これは何？

2022-04-17時点:

MacBook Pro (M1 Pro), MontereyでOpenVINO ARM plugin が動くよ、しかも MYRIAD X (Neural Compute Stick 2)にも負けてないよ、というお話。（やっと動いたし、意外に速かったので嬉しくなって書きたくなりました）

ベンチマーク

benchmark_appのFP32での実行結果です。M1ではFP16は現状では未対応のようです。

$ build-OpenVINO-2022.1/openvino/bin/arm64/Release/benchmark_app -d CPU -m FP32/yolo-v3-tf.xml
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading Inference Engine
[ INFO ] OpenVINO: OpenVINO Runtime version ......... 2022.1.0
[ INFO ] Build ........... custom_HEAD_cdb9bec7210f8c24fde3e416c7ada820faaaa23e
[ INFO ] 
[ INFO ] Device info: 
[ INFO ] CPU
[ INFO ] openvino_arm_cpu_plugin version ......... 2022.1.0
[ INFO ] Build ........... custom_HEAD_cdb9bec7210f8c24fde3e416c7ada820faaaa23e
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] -nstreams default value is determined automatically for CPU device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 4/11] Reading network files
[ INFO ] Loading network files
[ INFO ] Read network took 84.57 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ WARNING ] input_1: layout is not set explicitly, so it is defaulted to NCHW. It is STRONGLY recommended to set layout manually to avoid further issues.
[Step 6/11] Configuring input of the model
[ INFO ] Network batch size: 1
Network inputs:
    input_1 (node: input_1) : u8 / [N,C,H,W]
Network outputs:
    conv2d_58/Conv2D/YoloRegion (node: conv2d_58/Conv2D/YoloRegion) : f32 / [...]
    conv2d_66/Conv2D/YoloRegion (node: conv2d_66/Conv2D/YoloRegion) : f32 / [...]
    conv2d_74/Conv2D/YoloRegion (node: conv2d_74/Conv2D/YoloRegion) : f32 / [...]
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 15.82 ms
[Step 8/11] Setting optimal runtime parameters
[ INFO ] Device: CPU
[ INFO ]   { NETWORK_NAME , yolo-v3-tf }
[ INFO ]   { OPTIMAL_NUMBER_OF_INFER_REQUESTS , 5 }
[ INFO ]   { NUM_STREAMS , 5 }
[ INFO ]   { INFERENCE_NUM_THREADS , 0 }
[Step 9/11] Creating infer requests and preparing input blobs with data
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] input_1  ([N,C,H,W], u8, {1, 3, 416, 416}, static):	random (image is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 5 inference requests, limits: 60000 ms duration)
[ INFO ] BENCHMARK IS IN INFERENCE ONLY MODE.
[ INFO ] Input blobs will be filled once before performance measurements.
[ INFO ] First inference took 767.14 ms

[Step 11/11] Dumping statistics report
[ INFO ] Count:      340 iterations
[ INFO ] Duration:   60389.01 ms
[ INFO ] Latency: 
[ INFO ] 	Median:     783.28 ms
[ INFO ] 	Average:    885.71 ms
[ INFO ] 	Min:        306.94 ms
[ INFO ] 	Max:        4426.69 ms
[ INFO ] Throughput: 5.63 FPS

ビルド

macOS Monterey 12.3.1
- Xcode 13.3
Homebrew
- NCS2を動かす場合は libusb が必要。
Miniforge

以下の手順でビルドします。

OpenCV

git clone https://github.com/kose/build-OpenVINO.git
cd build-OpenVINO
git checkout 2022.1
git submodule update --init --recursive --depth 1


cd opencv
mkdir build
cd build
../../macOS/opencv.sh
sudo make install

cd /opt/intel
sudo ln -s openvino_2022.1 openvino_2022

OpenVINO

以下のページに従いパッチを当てます。

https://github.com/openvinotoolkit/openvino_contrib/wiki/How-to-build-ARM-CPU-plugin#building-for-macos

cd openvino_contrib/modules/arm_plugin/thirdparty/ComputeLibrary && \
wget "https://review.mlplatform.org/changes/ml%2FComputeLibrary~6706/revisions/4/patch?zip" -O patch.zip && \
unzip patch.zip && \
git apply 48f2615.diff

cmake/dependencies.cmakeがarm64に対応していないので、とりあえずfaital-errorを回避します

diff --git a/cmake/dependencies.cmake b/cmake/dependencies.cmake
index e2f941fc..c931867b 100644
--- a/cmake/dependencies.cmake
+++ b/cmake/dependencies.cmake
@@ -154,7 +154,7 @@ if(THREADING STREQUAL "TBB" OR THREADING STREQUAL "TBB_AUTO")
                 ENVIRONMENT "TBBROOT"
                 SHA256 "ad9cf52e657660058aa6c6844914bc0fc66241fec89a392d8b79a7ff69c3c7f6")
     else()
-        message(FATAL_ERROR "TBB is not available on current platform")
+        message("M1 env: TBB is not available on current platform")
     endif()
 
     update_deps_cache(TBBROOT "${TBB}" "Path to TBB root folder")

export INTEL_OPENVINO_DIR=/opt/intel/openvino_2022
export OpenCV_DIR=$INTEL_OPENVINO_DIR/extras/opencv/lib/cmake/opencv4
export PYTHONPATH=$INTEL_OPENVINO_DIR/extras/opencv/lib/python3.9/site-packages

cd openvino
mkdir build
cd build
../../macOS/openvino_with-arm-plugin.sh
sudo make install

re-build OpenCV

Inference Engine対応OpenCVをビルドします。

export INTEL_OPENVINO_DIR=/opt/intel/openvino_2022
source $INTEL_OPENVINO_DIR/setupvars.sh

re-build OpenCV

でも残念ながら PyPi openvino-dev がねぇ。

pip install openvino-dev

は残念ながら M1 Mac (ARM)では失敗します。私は仕方がないのでIntel Mac miniでモデルのダンロード・コンバートをしています。

dockerで以下のようにx86_64環境を作っても失敗します。（Docker x86_64をqemuではなくRosseta2で動かせばなんとかなりそうだけど面倒なので試してないです）

docker buildx build --platform linux/amd64 ....

Reference