はじめに
3D Sensor Advent Calendar 2019の3日目の記事です。
今日は、PythonでRealSenseD400シリーズに入門しましょう。（というか、備忘録）
センサーについてはこちらがよくまとまっていますのでご参考まで。
【記事】RealSense D415とD435の比較と選び方
【記事】RealSense D415とD435 および D435iとT265の比較と選び方
開発環境
・Windows 10
・Anaconda 2019.10
・Python 3.6.9
・opencv-python 4.1.1.26
・opencv-contrib-python 4.1.1.26
・pyrealsense2 2.30.0.1184
・open3d-python 0.7.0.0
・RealSenseD435（D415／D435i）
RealSense SDK のインストール
Intel® RealSense™ SDK 2.0 (build 2.29.0) の Intel.RealSense.SDK.exe を実行し、インストールしてください。
Anaconda のインストール
1．Anaconda3-2019.10-Windows-x86_64.exe を実行し、インストールする
2．Anaconda Promptを開き、仮想環境を作成する
(base) C:\Users\xxx> conda create -n py36 python=3.6 anaconda
3．仮想環境を立ち上げる
(base) C:\Users\xxx> conda activate py36
(py36) C:\Users\xxx>
各ライブラリのインストール
(py36) C:\Users\xxx> pip install open3d-python
(py36) C:\Users\xxx> pip install pyrealsense2
(py36) C:\Users\xxx> pip install opencv-python
(py36) C:\Users\xxx> pip install opencv-contrib-python
※アップデートする場合、pip installの後に-Uオプションをつける
サンプルコードのダウンロード
・OpenCV4.1.1 のサンプルはこちら
・OpenCV Contrib 4.1.1 のサンプルはこちら
・RealSense のサンプルはこちら
・Open3Dのサンプルはこちら
RealSenseのサンプルコードをやってみる
ダウンロードした librealsense-2.29.0\wrappers\python\examples へ移動
(py36) C:\Users\xxx> cd librealsense-2.29.0\wrappers\python\examples
align-depth2color.py
RGBD画像の取得、表示するプログラム
(py36) C:\Users\xxx> python align-depth2color.py
Depth Scale is: 0.0010000000474974513
https://t.co/Mjp7ZeY6yr #RealSense #D400Series #Python pic.twitter.com/YbHnHQjv6D
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
qキーで閉じる
export_ply_example.py
1フレーム読み取り、3D点群を.plyとして保存するプログラム
(py36) C:\Users\xxx>python export_ply_example.py
Saving to 1.ply...
Done
examplesフォルダ以下に1.plyが生成されているので、MeshLab2016.12等で開いて確認してみましょう。
export_ply_example.py #RealSense #D400Series #Python pic.twitter.com/Yww18O36gr
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
frame_queue_example.py
slower processing functionがどうとか
(py36) C:\Users\xxx> python frame_queue_example.py
Slow callback
1
2
3
・・・
Slower callback + queue
1
2
3
・・・
138
139
140
Slower callback + keeping queue
__init__(): incompatible constructor arguments. The following argument types are supported:
1. pyrealsense2.pyrealsense2.frame_queue(arg0: int)
2. pyrealsense2.pyrealsense2.frame_queue()
Invoked with: 50; kwargs: keep_frames=True
opencv_pointcloud_viewer.py
OpenCVで3D点群を表示するプログラム
(py36) C:\Users\xxx> python opencv_pointcloud_viewer.py
opencv_pointcloud_viewer.py #RealSense #D400Series #Python pic.twitter.com/ldRRXeeIZm
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
qキーで閉じる
opencv_viewer_example.py
OpenCVでDepth画像、Color画像を表示するプログラム
(py36) C:\Users\xxx> python opencv_viewer_example.py
opencv_viewer_example.py #RealSense #D400Series #Python pic.twitter.com/7kH4F3Gu3L
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
コマンドプロンプトからCtrl+Cで閉じる
cv2.waitKey(1)のところを
if cv2.waitKey(1) & 0xFF == ord('q'):
break
に書き換えると、qキーで閉じれる
pybackend_example_1_general.py
(py36) C:\Users\xxx> python pybackend_example_1_general.py
Traceback (most recent call last):
File "pybackend_example_1_general.py", line 9, in <module>
import pybackend2 as rs
ModuleNotFoundError: No module named 'pybackend2'
pip install できなそうだからやめとく
ModuleNotFoundError: No module named 'pybackend2' #3204
pyglet_pointcloud_viewer.py
pygletを用いて3D点群を表示するプログラム
(py36) C:\Users\xxx> pip install pyglet
(py36) C:\Users\xxx> pyglet_pointcloud_viewer.py
Traceback (most recent call last):
File "pyglet_pointcloud_viewer.py", line 151, in <module>
fps_display = pyglet.clock.ClockDisplay()
AttributeError: module 'pyglet.clock' has no attribute 'ClockDisplay'
Python2っぽい
python-rs400-advanced-mode-example.py
D400シリーズのパラメータを取得するプログラム
(py36) C:\Users\xxx> python python-rs400-advanced-mode-example.py
Found device that supports advanced mode: Intel RealSense D435
Advanced mode is enabled
Depth Control:
minusDecrement: 10, deepSeaMedianThreshold: 500, scoreThreshA: 1, scoreThreshB: 2047, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 325, deepSeaNeighborThreshold: 7, lrAgreeThreshold: 24
RSM:
rsmBypass: 0, diffThresh: 4, sloRauDiffThresh: 1, removeThresh: 63
RAU Support Vector Control:
minWest: 1, minEast: 1, minWEsum: 3, minNorth: 1, minSouth: 1, minNSsum: 3, uShrink: 3, vShrink: 1
Color Control:
disableSADColor: 0, disableRAUColor: 0, disableSLORightColor: 0, disableSLOLeftColor: 0, disableSADNormalize: 0
RAU Thresholds Control:
rauDiffThresholdRed: 51, rauDiffThresholdGreen: 51, rauDiffThresholdBlue: 51
SLO Color Thresholds Control:
diffThresholdRed: 72, diffThresholdGreen: 72, diffThresholdBlue: 72
SLO Penalty Control:
sloK1Penalty: 60, sloK2Penalty: 342, sloK1PenaltyMod1: 105, sloK2PenaltyMod1: 190, sloK1PenaltyMod2: 70, sloK2PenaltyMod2: 130
HDAD:
lambdaCensus: 26, lambdaAD: 800, ignoreSAD: 0
Color Correction:
colorCorrection1: 0.461914, colorCorrection2: 0.540039, colorCorrection3: 0.540039, colorCorrection4: 0.208008, colorCorrection5: -0.332031, colorCorrection6: -0.212891, colorCorrection7: -0.212891, colorCorrection8: 0.68457, colorCorrection9: 0.930664, colorCorrection10: -0.553711, colorCorrection11: -0.553711, colorCorrection12: 0.0458984
Depth Table:
depthUnits: 1000, depthClampMin: 0, depthClampMax: 65536, disparityMode: 0, disparityShift: 0
Auto Exposure Control:
Mean Intensity Set Point: 1536
Census:
uDiameter: 9, vDiameter: 9
Depth Control Min Values:
minusDecrement: 0, deepSeaMedianThreshold: 0, scoreThreshA: 0, scoreThreshB: 0, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 0, deepSeaNeighborThreshold: 0, lrAgreeThreshold: 0
Depth Control Max Values:
minusDecrement: 255, deepSeaMedianThreshold: 1023, scoreThreshA: 1023, scoreThreshB: 4095, textureDifferenceThreshold: 4095, textureCountThreshold: 1023, deepSeaSecondPeakThreshold: 1023, deepSeaNeighborThreshold: 1023, lrAgreeThreshold: 2047
After Setting new value, Depth Control:
minusDecrement: 10, deepSeaMedianThreshold: 500, scoreThreshA: 511, scoreThreshB: 2047, textureDifferenceThreshold: 0, textureCountThreshold: 0, deepSeaSecondPeakThreshold: 325, deepSeaNeighborThreshold: 7, lrAgreeThreshold: 24
Controls as JSON:
{
"aux-param-autoexposure-setpoint": "1536",
"aux-param-colorcorrection1": "0.461914",
"aux-param-colorcorrection10": "-0.553711",
"aux-param-colorcorrection11": "-0.553711",
"aux-param-colorcorrection12": "0.0458984",
"aux-param-colorcorrection2": "0.540039",
"aux-param-colorcorrection3": "0.540039",
"aux-param-colorcorrection4": "0.208008",
"aux-param-colorcorrection5": "-0.332031",
"aux-param-colorcorrection6": "-0.212891",
"aux-param-colorcorrection7": "-0.212891",
"aux-param-colorcorrection8": "0.68457",
"aux-param-colorcorrection9": "0.930664",
"aux-param-depthclampmax": "65536",
"aux-param-depthclampmin": "0",
"aux-param-disparityshift": "0",
"controls-autoexposure-auto": "True",
"controls-autoexposure-manual": "8500",
"controls-color-autoexposure-auto": "True",
"controls-color-autoexposure-manual": "156",
"controls-color-backlight-compensation": "0",
"controls-color-brightness": "0",
"controls-color-contrast": "50",
"controls-color-gain": "64",
"controls-color-gamma": "300",
"controls-color-hue": "0",
"controls-color-power-line-frequency": "3",
"controls-color-saturation": "64",
"controls-color-sharpness": "50",
"controls-color-white-balance-auto": "True",
"controls-color-white-balance-manual": "4600",
"controls-depth-gain": "16",
"controls-laserpower": "150",
"controls-laserstate": "on",
"ignoreSAD": "0",
"param-amplitude-factor": "0",
"param-autoexposure-setpoint": "1536",
"param-censusenablereg-udiameter": "9",
"param-censusenablereg-vdiameter": "9",
"param-censususize": "9",
"param-censusvsize": "9",
"param-depthclampmax": "65536",
"param-depthclampmin": "0",
"param-depthunits": "1000",
"param-disableraucolor": "0",
"param-disablesadcolor": "0",
"param-disablesadnormalize": "0",
"param-disablesloleftcolor": "0",
"param-disableslorightcolor": "0",
"param-disparitymode": "0",
"param-disparityshift": "0",
"param-lambdaad": "800",
"param-lambdacensus": "26",
"param-leftrightthreshold": "24",
"param-maxscorethreshb": "2047",
"param-medianthreshold": "500",
"param-minscorethresha": "511",
"param-neighborthresh": "7",
"param-raumine": "1",
"param-rauminn": "1",
"param-rauminnssum": "3",
"param-raumins": "1",
"param-rauminw": "1",
"param-rauminwesum": "3",
"param-regioncolorthresholdb": "0.0499022",
"param-regioncolorthresholdg": "0.0499022",
"param-regioncolorthresholdr": "0.0499022",
"param-regionshrinku": "3",
"param-regionshrinkv": "1",
"param-robbinsmonrodecrement": "10",
"param-robbinsmonroincrement": "10",
"param-rsmdiffthreshold": "4",
"param-rsmrauslodiffthreshold": "1",
"param-rsmremovethreshold": "0.375",
"param-scanlineedgetaub": "72",
"param-scanlineedgetaug": "72",
"param-scanlineedgetaur": "72",
"param-scanlinep1": "60",
"param-scanlinep1onediscon": "105",
"param-scanlinep1twodiscon": "70",
"param-scanlinep2": "342",
"param-scanlinep2onediscon": "190",
"param-scanlinep2twodiscon": "130",
"param-secondpeakdelta": "325",
"param-texturecountthresh": "0",
"param-texturedifferencethresh": "0",
"param-usersm": "1",
"param-zunits": "1000"
}
パラメータがいっぱい
python-tutorial-1-depth.py
Depthを取得して文字で表現するプログラム
(py36) C:\Users\xxx> python python-tutorial-1-depth.py
https://t.co/G93Lf8gYd8 #RealSense #D400Series #Python pic.twitter.com/nPELNS3bUN
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
read_bag_example.py
ROSでよく使う.bagファイルを読み込み、表示するプログラム
１．Interl RealSense Viewerを用いて、Recordボタンを押し録画。
２．.bagファイルができているので、それを読み込む。
(py36) C:\Users\xxx> python read_bag_example.py -i 20191203_143057.bag
read_bag_example.py #RealSense #D400Series #Python pic.twitter.com/6FinAbjS6d
— 藤本賢志(ガチ本)@MAリーグ2019決勝 (@sotongshi) December 3, 2019
ESCキーで閉じる
FPS-解像度の対応表
解像度に対する最大FPS
| Resolution | Depth | RGB |
|---|---|---|
| 320, 180 | 60 | |
| 320, 240 | 60 | |
| 424, 240 | 90 | 60 |
| 480, 270 | 90 | |
| 640, 360 | 90 | 60 |
| 640, 480 | 90 | 60 |
| 848, 480 | 90 | 60 |
| 960, 540 | 60 | |
| 1280, 720 | 30 | 30 |
| 1920, 1080 | 30 |
FPSに対する最大解像度
| FPS | Depth | RGB |
|---|---|---|
| 6 | 1280, 720 | 1920, 1080 |
| 15 | 1280, 720 | 1920, 1080 |
| 30 | 1280, 720 | 1920, 1080 |
| 60 | 848, 480 | 960, 540 |
| 90 | 848, 480 |
まとめ
RealSenseのサンプルをひととおり動かしてみました。open3dやopencvは今後紹介したいと思います。
明日は @UnaNancyOwen さんによる「みんな大好きOpenNI2でAzure Kinectを動かしてみる」です。お楽しみに！
Intel RealSense Viewer のフィルター項目一覧
Stereo Module
- Emitter Enabled
- Enable Auto Exposure Set ROI
- Controls
- Exposure
- Gain
- Laser Power
- Asic Temperature
- Error Polling Enabled
- Projector Temperature
- Output Trigger Enabled
- Depth Units
- Stereo Baseline
- Inter Cam Sync Mode
- Emitter On Off
- Global Time Enabled
- Advanced Controls
- Depth Control
- DS Second Peak Threshold
- DS Neighbor Threshold
- DS Median Threshold
- Estimate Median Increment
- Estimate Median Decrement
- Score Minimum Threshold
- Score Maximum Threshold
- DS LR Threshold
- Texture Count Threshold
- Texture Difference Threshold
- Rsm
- RSM Bypass
- Disparity Difference Threshold
- SLO RAU Difference Threshold
- Remove Threshold
- Rau Support Vector Control
- Min West
- Min East
- Min WE Sum
- Min North
- Min South
- Min NS Sum
- U Shrink
- V Shrink
- Color Control
- Disable SAD Color
- Disable RAU Color
- Disable SLO Right Color
- Disable SLO Left Color
- Disable SAD Normalize
- Rau Color Thresholds Control
- Diff Threshold Red
- Diff Threshold Green
- Diff Threshold Blue
- SLO Color Thresholds Control
- Diff Threshold Red
- Diff Threshold Green
- Diff Threshold Blue
- SLO Penalty Control
- K1 Penalty
- K2 Penalty
- K1 Penalty Mod1
- K1 Penalty Mod2
- K2 Penalty Mod1
- K2 Penalty Mod2
- HDAD
- Ignore SAD
- AD Lambda
- Census Lambda
- Color Correction
- Color Correction 1
- Color Correction 2
- Color Correction 3
- Color Correction 4
- Color Correction 5
- Color Correction 6
- Color Correction 7
- Color Correction 8
- Color Correction 9
- Color Correction 10
- Color Correction 11
- Color Correction 12
- Depth Table
- Depth Units
- Depth Clamp Min
- Depth Clamp Max
- Disparity Mode
- Disparity Shift
- AE Control
- Mean Intensity Set Point
- Census Enable Reg
- u-Diameter
- v-Diameter
- Disparity Modulation
- A Factor
- Depth Control
- Depth Visualization
- Visual Preset
- Dynamic, Fixed, Near, Far
- Color Scheme
- Jet, Classic, White to Black, Black to White, Bio, Cold, Warm
- Histogram Equalization Enabled
- Min Distance
- Max Distance
- Visual Preset
- Post-Processing
- Decimation Filter
- Filter Magnitude
- Threshold Filter
- Min Distance
- Max Distance
- Depth to Disparity
- Spatial Filter
- Filter Magnitude
- Filter Smooth Alpha
- Filter Delta
- Holes filling mode
- Disabled, 2-pixel radius, 4-pixel radius, 8-pixel radius, 16-pixel radius, Unlimited
- Temporal Filter
- Filter Smooth Alpha
- Filter Smooth Delta
- Persistency mode
- Disabled, Valid in 8/8, Valid in 2/, Valid in 2/8, Valid in 1/la, Valid in 1/la, Valid in 2/
- Hole Filling Filter
- Hole Filling mode
- Fill from Left, Farest from a, Nearest from
- Hole Filling mode
- Disparity to Depth
- Decimation Filter
RGB Camera
- Color
- YUYV, BGR8, RGBA8, BGRA8, Y16, RGB8, RAW16
- Enable Auto Exposure
- Controls
- Backlight Compensation
- Brightness
- Contrast
- Exposure
- Gain
- Gamma
- Hue
- Saturation
- Sharpness
- White Balance
- Enable Auto White Balance
- Power Line Frequency
- Auto Exposure Priority
- Global Time Enabled
- Post-Processing
- Decimation Filter
- Filter Magnitude
- Decimation Filter