Raspberry Pi+NCS2+OpenVINOで物体認識サンプルコードを動かすまでの話

はじめに

Intel Neural Compute Stick2（通称NCS2）を手に入れたのでさっそく遊んでみるぜー

環境

ハード	Raspberry Pi 3 ModelB
OS	Raspbian Desktop (Buster)
OpenVino	l_openvino_toolkit_runtime_raspbian_p_2019.2.242.tgz (ただし、OpenCVは後で入れ替えた)
言語	python 3.7

ちなみに、Windows側でも動かしてみたが普通にCPUのほうがNCS2より早かったし、NCS2の良さは低消費電力なところなので、ラズパイで使うのがいいんじゃないかしら。
あと、Windowsだとインストールであれこれたくさん入れないといけなくてめんどかった。
なので今回の記事はラズパイに絞る。

セットアップ

パッケージのインストール

cmakeが必要

root@k8s01:~# apt-get install cmake

OpenVinoのインストール

基本的に以下の手順に従ってインストール。
https://docs.openvinotoolkit.org/latest/_docs_install_guides_installing_openvino_raspbian.html

まずはOpenVINOひろって展開。
拾ってくるファイルはここらへんにある。
https://download.01.org/opencv/2019/openvinotoolkit/
9/23にR3が出たみたいだが、よくわからんからR2を使う。

root@k8s01:~# wget https://download.01.org/opencv/2019/openvinotoolkit/R2/l_openvino_toolkit_runtime_raspbian_p_2019.2.242.tgz
root@k8s01:~# tar zxvf l_openvino_toolkit_runtime_raspbian_p_2019.2.242.tgz 
root@k8s01:~# sudo mkdir -p /opt/intel/openvino
root@k8s01:~# sudo tar -xf l_openvino_toolkit_raspbi_p_2019.1.144.tgz --strip 1 -C /opt/intel/openvino

環境変数読み込み。
INSTALLDIRというところを書き換えてやる必要がある。

root@k8s01:~# sudo sed -i "s|<INSTALLDIR>|/opt/intel/openvino|" /opt/intel/openvino/bin/setupvars.sh
root@k8s01:~# source /opt/intel/openvino/bin/setupvars.sh
[setupvars.sh] OpenVINO environment initialized
root@k8s01:~# echo "source /opt/intel/openvino/bin/setupvars.sh" >> ~/.bashrc

NCS2を使うためのモジュール読み込み

root@k8s01:~# sh /opt/intel/openvino/install_dependencies/install_NCS_udev_rules.sh
Update udev rules so that the toolkit can communicate with your neural compute stick
[install_NCS_udev_rules.sh] udev rules installed

動作確認

NCS2をラズパイに刺して、サンプルのベンチマークソフトを動かす。
サンプルは
/opt/intel/openvino/deployment_tools/inference_engine/samples/
にいろいろある。
ここのなかからbenchmark_appというのを選んで、下記のような感じで実行する。

python3 benchmark_app.py -i ~/openvino_model/president-reagan-62x62.png -m ~/openvino_model/squeezenet1.1.xml -d MYRIAD

-d MYRIAD
っていうところがNCS使うよっていうやつ。

テストに使ったpresident-reagan-62x62.pngという画像ファイルは下記から取得。
https://software.intel.com/sites/default/files/managed/f5/63/president_reagan-62x62.png
モデルについては下記記事参照
https://qiita.com/ammo0613/items/d9463d11741f1bc5d2e2

実行結果は下記のような感じ。

root@k8s01:/opt/intel/openvino/deployment_tools/inference_engine/samples/python_samples/benchmark_app# python3 benchmark_app.py -i ~/openvino_model/president-reagan-62x62.png -m ~/openvino_model/squeezenet1.1.xml -d MYRIAD
[Step 1/11] Parsing and validating input arguments
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
         2.0.custom_releases/2019/R2_f5827d4773ebbe727c9acac5f007f7d94dd4be4e
[ INFO ] Device is MYRIAD
         MYRIAD
         myriadPlugin............ version 2.0
         Build................... 27579

[Step 3/11] Read the Intermediate Representation of the network
[Step 4/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1, precision FP16
[Step 5/11] Configuring input of the model
[Step 6/11] Setting device configuration
[Step 7/11] Loading the model to the device
[Info   ][VPU][MyriadPlugin] Device #0 MYRIAD-X (USB protocol) allocated
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ INFO ] Network input 'data' precision U8, dimensions (NCHW): 1 3 227 227
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 1 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 2 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 3 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asyncronously, 4 inference requests, limits: 60000 ms duration)
[Step 11/11] Dumping statistics report
Count:      8200 iterations
Duration:   60059.26 ms
Latency:    26.4373 ms
Throughput: 136.53 FPS

物体認識サンプルコードで遊ぶまで

動画の顔認識してみるまで。

サンプルコードの場所

pythonサンプルコードは下記の場所にある。
/opt/intel/openvino/deployment_tools/inference_engine/samples/python_samples
ここにあるobject_detection_demo_ssd_async.py が物体認識のサンプルコードで、これを使って動画上の顔認識をやってみる。

学習済みモデルの取得

OpenVINOに使える学習済みモデルは下記にたくさんある。
https://download.01.org/openvinotoolkit/2018_R4/open_model_zoo/
このなかから顔画像のやつとして下記ファイルを取得する。

• https://download.01.org/openvinotoolkit/2018_R4/open_model_zoo/face-detection-adas-0001/FP16/face-detection-adas-0001.xml
• https://download.01.org/openvinotoolkit/2018_R4/open_model_zoo/face-detection-adas-0001/FP16/face-detection-adas-0001.bin

注意すべき点として、FP32だとMYRIADさんが動かないので、FP16を選ぶ必要がある。

動画ファイルの取得

USBカメラで自分の顔認識試すのが恥ずかしいのと、繰り返し実験するには動画ファイルよいと思ったので、それっぽいファイルを下記から取得。
https://github.com/intel-iot-devkit/sample-videos
今回は
face-demographics-walking-and-pause.mp4
を使用した。

いざ実験

さあどうだ。

pi@k8s01:~$ python3 object_detection_demo_ssd_async.py -i face-demographics-walking-and-pause.mp4 -d MYRIAD -m ./face-detection-adas-0001.xml
[ INFO ] Initializing plugin for MYRIAD device...
[ INFO ] Reading IR...
[ INFO ] Loading IR to the plugin...
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_gstreamer.cpp (701) open OpenCV | GStreamer warning: Error opening file: face-demographics-walking-and-pause.mp4 (
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
VIDIOC_REQBUFS: Inappropriate ioctl for device
[ INFO ] Starting inference in async mode...
[ INFO ] To switch between sync and async modes press Tab button
[ INFO ] To stop the demo execution press Esc button
To close the application, press 'CTRL+C' or any key with focus on the output window

なんかエラー出とる。詳細確認のためにデバッグ出力する。

pi@k8s01:~ $ export OPENCV_LOG_LEVEL=DEBUG
pi@k8s01:~ $ export OPENCV_VIDEOIO_DEBUG=1
pi@k8s01:~ $ 
pi@k8s01:~ $ python3 object_detection_demo_ssd_async.py -i face-demographics-walking-and-pause.mp4 -d MYRIAD -m ./face-detection-adas-0001.xml 
[ INFO ] Initializing plugin for MYRIAD device...
[ INFO ] Reading IR...
[ INFO ] Loading IR to the plugin...
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/videoio_registry.cpp (187) VideoBackendRegistry VIDEOIO: Enabled backends(6, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950)
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(FFMPEG): trying capture filename='face-demographics-walking-and-pause.mp4' ...
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (329) getPluginCandidates VideoIO pluigin (FFMPEG): glob is 'libopencv_videoio_ffmpeg*.so', 1 location(s)
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (336) getPluginCandidates     - /opt/intel/openvino/opencv/lib: 1
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (340) getPluginCandidates Found 1 plugin(s) for FFMPEG
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (172) libraryLoad load /opt/intel/openvino/opencv/lib/libopencv_videoio_ffmpeg.so => FAILED
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (128) open VIDEOIO(FFMPEG): backend is not available (plugin is missing, or can't be loaded due dependencies or it is not compatible)
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(GSTREAMER): trying capture filename='face-demographics-walking-and-pause.mp4' ...
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (329) getPluginCandidates VideoIO pluigin (GSTREAMER): glob is 'libopencv_videoio_gstreamer*.so', 1 location(s)
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (336) getPluginCandidates     - /opt/intel/openvino/opencv/lib: 1
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (340) getPluginCandidates Found 1 plugin(s) for GSTREAMER
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (172) libraryLoad load /opt/intel/openvino/opencv/lib/libopencv_videoio_gstreamer.so => OK
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (220) PluginBackend Video I/O: loaded plugin 'GStreamer OpenCV Video I/O plugin'
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_gstreamer.cpp (701) open OpenCV | GStreamer warning: Error opening file: face-demographics-walking-and-pause.mp4 (
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (112) open VIDEOIO(GSTREAMER): can't create capture
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(INTEL_MFX): trying capture filename='face-demographics-walking-and-pause.mp4' ...
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (329) getPluginCandidates VideoIO pluigin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (336) getPluginCandidates     - /opt/intel/openvino/opencv/lib: 0
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (340) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (128) open VIDEOIO(INTEL_MFX): backend is not available (plugin is missing, or can't be loaded due dependencies or it is not compatible)
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(V4L2): trying capture filename='face-demographics-walking-and-pause.mp4' ...
VIDIOC_REQBUFS: Inappropriate ioctl for device
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (112) open VIDEOIO(V4L2): can't create capture
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(CV_IMAGES): trying capture filename='face-demographics-walking-and-pause.mp4' ...
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: test.mp%01d @ 4
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap_images.cpp (313) open CAP_IMAGES: Stop scanning. Can't read image file: face-demographics-walking-and-pause.mp4
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (104) open VIDEOIO(CV_IMAGES): created, isOpened=0
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (92) open VIDEOIO(CV_MJPEG): trying capture filename='face-demographics-walking-and-pause.mp4' ...
[ WARN:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/cap.cpp (112) open VIDEOIO(CV_MJPEG): can't create capture
[ INFO ] Starting inference in async mode...
[ INFO ] To switch between sync and async modes press Tab button
[ INFO ] To stop the demo execution press Esc button
To close the application, press 'CTRL+C' or any key with focus on the output window
[ INFO:0] global /home/jenkins/workspace/OpenCV/OpenVINO/build/opencv/modules/videoio/src/backend_plugin.cpp (178) libraryRelease unload /opt/intel/openvino/opencv/lib/libopencv_videoio_gstreamer.so

どうやら下記の読み込みに失敗しているということらしい。
/opt/intel/openvino/opencv/lib/libopencv_videoio_ffmpeg.so
いろいろいじってみたが、結論としてopencvのところ、
/opt/intel/openvino/opencv/
を下記のOpenVINO R1のやつにごっそり入れ替えることで解決。

うまく動いた

やったぜ

学習モデルを車のやつに入れ替えれば自動車の動画像認識とかもできる。
汎用的でなかなか便利。
NCS2やInference Engineで何ができるかこれから勉強する。