Script TOPでMediaPipeやーる（TouchDesigner、Python3.7.2、OpenCV4.5.2、Windows10）

はじめに

Script TOPを使ってMediaPipeをやりまーす

開発環境

• TouchDesigner.2021.13610
• Windows 10
• Anaconda
• OpenCV 4.5.2
• Python 3.7.2

導入

1．Dialog->Textport and DATsからバージョンチェック

2．anacondaで同じ環境を作成

conda info -e
conda create -n td python=3.7.2
conda activate td
pip install opencv-python
pip install opencv-contrib-python
pip install mediapipe

3．Edit->Preferencesの設定

• Python 64-bit Module Pathに作成したAnacondaの場所を指定（例：C:\Users\good_\anaconda3\envs\td\Lib\site-packages）
• Increment Filename on SaveをOff

4．TouchDesignerの再起動して、mediapipeがインポートできていればOK

5．Video Device Input（TOP）、Resolution（TOP）、Script（TOP）を配置します

6．Script（TOP）の中身

# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input_image = scriptOp.inputs[0].numpyArray()
	image = input_image * 255
	image = image.astype('uint8')
	image_height, image_width = image.shape[:2]

	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	results = hands.process(image)

	draw_image = image.copy()
	if results.multi_hand_landmarks is not None:
		for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
			for index, landmark in enumerate(hand_landmarks.landmark):
				if landmark.visibility < 0 or landmark.presence < 0:
					continue
				landmark_x = min(int(landmark.x * image_width), image_width - 1)
				landmark_y = min(int(landmark.y * image_height), image_height - 1)
				cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)

	draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
	scriptOp.copyNumpyArray(draw_image)
	return

カスタムパラメータで新しいページを作成しようとしましたが、なぜか反映されませんでした。。。

def onSetupParameters(scriptOp):
	page = scriptOp.appendCustomPage('Custom')
	p = page.appendInt('MaxNumHands', label='Max Num Hands')
	p[0].default = 2
	p = page.appendFloat('MinDetectionConfidence', label='Min Detection Confidence')
	p[0].default = 0.5
	p = page.appendFloat('MinTrackingConfidence', label='Min Tracking Confidence')
	p[0].default = 0.5
	return


def onCook(scriptOp):
	MaxNumHands = scriptOp.par.MaxNumHands.eval()
	MinDetectionConfidence = scriptOp.par.MinDetectionConfidence.eval()
	MinTrackingConfidence = scriptOp.par.MinTrackingConfidence.eval()

Hands#mediapipe #TouchDesigner #kuma_td pic.twitter.com/8A8Eddtt5N

— がちもとさん@熊本 (@sotongshi) June 7, 2021

7．hand_landmarksのTable（DAT）を追加し、Script（TOP）を下記のように編集します。これで手のランドマークの値がTableに表示されます。

# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)

op_hand_landmarks = op('hand_landmarks')

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input_image = scriptOp.inputs[0].numpyArray()
	image = input_image * 255
	image = image.astype('uint8')
	image_height, image_width = image.shape[:2]

	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	results = hands.process(image)

	draw_image = image.copy()
	op_hand_landmarks.clear()
	op_hand_landmarks.appendRow(['x', 'y', 'z'])
	if results.multi_hand_landmarks is not None:
		for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
			for index, landmark in enumerate(hand_landmarks.landmark):
				if landmark.visibility < 0 or landmark.presence < 0:
					continue
				landmark_x = min(int(landmark.x * image_width), image_width - 1)
				landmark_y = min(int(landmark.y * image_height), image_height - 1)
				cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)
				op_hand_landmarks.appendRow([landmark.x, landmark.y, landmark.z])

	draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
	scriptOp.copyNumpyArray(draw_image)
	return

8．Shpere（SOP）、Transform（SOP）、Geometry（COMP）を配置します。

9．GeometryのInstancingをONにし、Translate OPに作成したTable（DAT）を割り当てます。（Table（DAT）をGeometryへD&Dして、Translate OPを選択、Translate X, Y, ZにそれぞれTableのラベルx,y,zを割り当て）

10．Transform（SOP）でUniform Scaleの値を調整します。

手の位置をInstancingする#mediapipe #TouchDesigner #kuma_td pic.twitter.com/FEQSjpKVin

— がちもとさん@熊本 (@sotongshi) June 7, 2021

お疲れ様でした。

Script TOPを使ったサンプルがあるので参考まで