はじめに
Script TOPを使ってMediaPipeをやりまーす
開発環境
- TouchDesigner.2021.13610
- Windows 10
- Anaconda
- OpenCV 4.5.2
- Python 3.7.2
導入
1．Dialog->Textport and DATsからバージョンチェック
2．anacondaで同じ環境を作成
conda info -e
conda create -n td python=3.7.2
conda activate td
pip install opencv-python
pip install opencv-contrib-python
pip install mediapipe
3．Edit->Preferencesの設定
- Python 64-bit Module Pathに作成したAnacondaの場所を指定（例：C:\Users\good_\anaconda3\envs\td\Lib\site-packages）
- Increment Filename on SaveをOff
4．TouchDesignerの再起動して、mediapipeがインポートできていればOK
5．Video Device Input（TOP）、Resolution（TOP）、Script（TOP）を配置します
6．Script（TOP）の中身
# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
return
# called whenever custom pulse parameter is pushed
def onPulse(par):
return
def onCook(scriptOp):
input_image = scriptOp.inputs[0].numpyArray()
image = input_image * 255
image = image.astype('uint8')
image_height, image_width = image.shape[:2]
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = hands.process(image)
draw_image = image.copy()
if results.multi_hand_landmarks is not None:
for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
for index, landmark in enumerate(hand_landmarks.landmark):
if landmark.visibility < 0 or landmark.presence < 0:
continue
landmark_x = min(int(landmark.x * image_width), image_width - 1)
landmark_y = min(int(landmark.y * image_height), image_height - 1)
cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)
draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
scriptOp.copyNumpyArray(draw_image)
return
カスタムパラメータで新しいページを作成しようとしましたが、なぜか反映されませんでした。。。
def onSetupParameters(scriptOp):
page = scriptOp.appendCustomPage('Custom')
p = page.appendInt('MaxNumHands', label='Max Num Hands')
p[0].default = 2
p = page.appendFloat('MinDetectionConfidence', label='Min Detection Confidence')
p[0].default = 0.5
p = page.appendFloat('MinTrackingConfidence', label='Min Tracking Confidence')
p[0].default = 0.5
return
def onCook(scriptOp):
MaxNumHands = scriptOp.par.MaxNumHands.eval()
MinDetectionConfidence = scriptOp.par.MinDetectionConfidence.eval()
MinTrackingConfidence = scriptOp.par.MinTrackingConfidence.eval()
Hands#mediapipe #TouchDesigner #kuma_td pic.twitter.com/8A8Eddtt5N
— がちもとさん@熊本 (@sotongshi) June 7, 2021
7．hand_landmarksのTable（DAT）を追加し、Script（TOP）を下記のように編集します。これで手のランドマークの値がTableに表示されます。
# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)
op_hand_landmarks = op('hand_landmarks')
# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
return
# called whenever custom pulse parameter is pushed
def onPulse(par):
return
def onCook(scriptOp):
input_image = scriptOp.inputs[0].numpyArray()
image = input_image * 255
image = image.astype('uint8')
image_height, image_width = image.shape[:2]
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = hands.process(image)
draw_image = image.copy()
op_hand_landmarks.clear()
op_hand_landmarks.appendRow(['x', 'y', 'z'])
if results.multi_hand_landmarks is not None:
for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
for index, landmark in enumerate(hand_landmarks.landmark):
if landmark.visibility < 0 or landmark.presence < 0:
continue
landmark_x = min(int(landmark.x * image_width), image_width - 1)
landmark_y = min(int(landmark.y * image_height), image_height - 1)
cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)
op_hand_landmarks.appendRow([landmark.x, landmark.y, landmark.z])
draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
scriptOp.copyNumpyArray(draw_image)
return
8．Shpere（SOP）、Transform（SOP）、Geometry（COMP）を配置します。
9．GeometryのInstancingをONにし、Translate OPに作成したTable（DAT）を割り当てます。（Table（DAT）をGeometryへD&Dして、Translate OPを選択、Translate X, Y, ZにそれぞれTableのラベルx,y,zを割り当て）
10．Transform（SOP）でUniform Scaleの値を調整します。
手の位置をInstancingする#mediapipe #TouchDesigner #kuma_td pic.twitter.com/FEQSjpKVin
— がちもとさん@熊本 (@sotongshi) June 7, 2021
お疲れ様でした。
Script TOPを使ったサンプルがあるので参考まで
