デモ

目次

1. v3d-webの紹介
 2. v3d-webのインストール
 3. Unity連携

1. v3d-webの紹介

v3d-webとは、デモサイトを見ていただければわかるとおりに, Webカメラにかかわらず、すさまじく精度が高い webカメラだけのモーションキャプチャソフトです。

ただ、インストールがTrapだらけなので、非常に使うまでが大変でした（笑）

参考に, モーションキャプチャの方法として、

Webカメラ
光学式センサ
トラッカー

などが存在しますが、Webカメラが一番手軽で、精度が低いです。ただ、上半身に限り、大きな動作を伴わなければ使用性は十分です。

2. v3d-webのインストール

まず, npmがないと始まらないので Node.js をインストールします。

次に v3d-webのレポジトリをclone します。

clone https://github.com/phantom-software-AZ/v3d-web.git

README.mdに従い

npm install　--force
npm run build && tsc

を行います。

ここでは, v3d-webに必要なパッケージのインストールと、v3d-webのbuild + TypeScriptのコンパイルを行っています。

注意1

ちなみに、デモサイトでは React で作られているため、

create-react-app app-name --type typescript

として、Reactアプリを作成し、

npm install v3d-web

としても、普通にバグります。

注意2

The following asset(s) exceed the recommended size limit と warningが出るので、
webpack.configに

 performance: {
    hints: false,
    maxEntrypointSize: 512000,
    maxAssetSize: 512000
}

と設定します. (https://stackoverflow.com/questions/49348365/webpack-4-size-exceeds-the-recommended-limit-244-kib)

注意3

tscは内部または外部コマンドとして認識されません

と出たら

npm install typescript -g

として、typescriptをinstallしてください

注意4

Field 'browser' doesn't contain a valid alias configration

と出るので、srcフォルダ内にindex-text.ts

index-text.ts

/*
Copyright (C) 2021  The v3d Authors.

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, version 3.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.

    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
    
    export * from "./v3d-web";
    export type {CloneableQuaternionMap} from "./helper/quaternion";
    export type {HolisticOptions} from "./mediapipe";
    
    import {V3DWeb} from "./v3d-web";
    
    
    window.onload = async (e) => {
        const vrmFile = 'Maya__Mini.vrm';
        try {
            var v3DWeb = new V3DWeb(vrmFile);
        } catch (e: any) {
            console.error(e);
        }
    
    };
    
    export {};

を作成してください。

デモ

3. Unityとの連携

このままでも十分なのですが、Blenderなどで改造した vrmファイルを突っ込むとバグります。また、パーティクルなどの高度な演出などもできません。

なので、Unityとwebsocketで連携させて、いい感じにします。

システムの構成としてはこんな感じで、pythonのサーバーとv3d-webをwebsocket通信でつなぎ、boneのデータをやり取りします。

pythonのサーバは受け取ったboneのデータをUnityに送り付けます。

3.1 v3d-webの設定

pose-processing.tsについて設定します。

private websocket = new WebSocket('ws://127.0.0.1:9001/');

であらかじめconnectionをpropatyとして設定します。

次に, boneのデータを送る関数を設定します。

 public sendBoneRotations(){
        const items : any[] = [];
        for (const [k, v] of Object.entries(this._boneRotations)){
            items.push([k, v.x, v.y, v.z, v.w]);

        }
        if (this.websocket.OPEN){
            this.websocket.send(JSON.stringify(items));
        }else if(this.websocket.CLOSED){
            console.log("Websocket failed...");            
            this.websocket = new WebSocket('ws://127.0.0.1:9001/');
            this.websocket.send(JSON.stringify(items));
        }

        }

あとは、この関数を process関数内に置きます。

        this.filterBoneRotations(lockBones);

         // here is new code!
        this.sendBoneRotations();
        // Push to main
        this.pushBoneRotationBuffer();

以上により,　websocketの9001番ポートにwebsocketからのデータが送られます。

注意点としては、connectionが爆発するので、接続前に必ず切ってください。

3.2 pythonサーバ

import argparse
import json
from pythonosc import udp_client, osc_message_builder, osc_bundle_builder
from SimpleWebSocketServer import WebSocket, SimpleWebSocketServer


class OscBridge(WebSocket):
    ''' Websocket to OSC bridge '''

    def __init__(self, server, sock, address):
        super(OscBridge, self).__init__(server, sock, address)

        self.oscClient = udp_client.UDPClient('127.0.0.1', 54321)

    def parseMsg(self, address, msg):
        messages = []
        if isinstance(msg, dict):
            [messages.extend(self.parseMsg(address + '/' + k, v))
             for k, v in msg.items()]

        elif isinstance(msg, list):
            if isinstance(msg[0], dict) or isinstance(msg[0], list):
                [messages.extend(self.parseMsg(address, m)) for m in msg]
            else:
                messages.append(self.createOsc(address, msg))
        else:
            messages.append(self.createOsc(address, [msg]))
        return messages

    def createOsc(self, address, params):
        msg = osc_message_builder(address=address)
        for param in params:
            msg.add_arg(param)
        return msg

    def handleMessage(self):
        ws_msgs = json.loads(self.data)

        bundle = osc_bundle_builder.OscBundleBuilder(osc_bundle_builder.IMMEDIATELY)
        for bone_inf in ws_msgs:
            msg = osc_message_builder.OscMessageBuilder(address="/VMC/Ext/Bone/Rot")
            msg.add_arg(bone_inf[0])
            for bone_i in bone_inf[1:]:
                msg.add_arg(float(bone_i))
            bundle.add_content(msg.build())

        self.oscClient.send(bundle.build())

    def handleConnected(self):
        print(self.address, 'connected')

    def handleClose(self):
        print(self.address, 'closed')


if __name__ == '__main__':
    server = SimpleWebSocketServer('127.0.0.1', 9001, OscBridge)
    server.serveforever()

pythonoscを使って、Unityに送り付けます。

3.3 Unityの設定

[uOSC]を使います。必ず v1.1.0 をDLしてください。

unity の packageをぶち込みます。

V3DReciever.csを作成します。

V3DReciever.cs

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using VRM;

[RequireComponent(typeof(uOSC.uOscServer))]
public class V3DReciever : MonoBehaviour
{
    [SerializeField] GameObject Model;
    private Animator animator;

    [SerializeField] bool isLRBlink = false;
    
    VRMBlendShapeProxy blendShapeProxy = null;
    private static readonly Dictionary<string , BlendShapePreset> BlendPreset = new Dictionary<string, BlendShapePreset>{
        // {"Neutral", BlendShapePreset.Neutral},
        {"A", BlendShapePreset.A},
        {"I", BlendShapePreset.I},
        {"U", BlendShapePreset.U},
        {"E", BlendShapePreset.E},
        {"O", BlendShapePreset.O},
        {"Blink", BlendShapePreset.Blink},
        {"Joy", BlendShapePreset.Joy},
        {"Angry", BlendShapePreset.Angry},
        {"Sorrow", BlendShapePreset.Sorrow},
        {"Fun", BlendShapePreset.Fun},
        {"LookUp", BlendShapePreset.LookUp},
        {"LookDown", BlendShapePreset.LookDown},
        {"Blink_L", BlendShapePreset.Blink_L},
        {"Blink_R", BlendShapePreset.Blink_R},
    };

    private static readonly List<string> BoneNames = new List<string>{
        "J_Bip_C_Hips",
        "J_Bip_C_Spine",
        "J_Bip_C_UpperChest",
        "J_Bip_C_Chest",
        "J_Bip_C_Neck",
        "J_Bip_C_Head",
        "J_Adj_L_FaceEye",
        "J_Adj_R_FaceEye",

        "J_Bip_L_Shoulder",
        "J_Bip_L_LowerArm",
        "J_Bip_L_UpperArm",
        "J_Bip_L_Hand",
        "J_Bip_L_Thumb1",
        "J_Bip_L_Index1",
        "J_Bip_L_Middle1",
        "J_Bip_L_Little1",
        "J_Bip_L_Ring1",
        "J_Bip_L_Thumb2",
        "J_Bip_L_Index2",
        "J_Bip_L_Middle2",
        "J_Bip_L_Little2",
        "J_Bip_L_Ring2",
        "J_Bip_L_Thumb3",
        "J_Bip_L_Index3",
        "J_Bip_L_Middle3",
        "J_Bip_L_Little3",
        "J_Bip_L_Ring3",

        "J_Bip_R_Shoulder",
        "J_Bip_R_LowerArm",
        "J_Bip_R_UpperArm",
        "J_Bip_R_Hand",
        "J_Bip_R_Thumb1",
        "J_Bip_R_Index1",
        "J_Bip_R_Middle1",
        "J_Bip_R_Little1",
        "J_Bip_R_Ring1",
        "J_Bip_R_Thumb2",
        "J_Bip_R_Index2",
        "J_Bip_R_Middle2",
        "J_Bip_R_Little2",
        "J_Bip_R_Ring2",
        "J_Bip_R_Thumb3",
        "J_Bip_R_Index3",
        "J_Bip_R_Middle3",
        "J_Bip_R_Little3",
        "J_Bip_R_Ring3",

        "J_Bip_L_UpperLeg",
        "J_Bip_L_LowerLeg",
        "J_Bip_L_Foot",
        "J_Bip_L_ToeBase",

        "J_Bip_R_UpperLeg",
        "J_Bip_R_LowerLeg",
        "J_Bip_R_Foot",
        "J_Bip_R_ToeBase",
    };

    private static readonly Dictionary<string, string> Uni2VRMBone = new Dictionary<string, string>{
        {"hips", "J_Bip_C_Hips"},
        {"spine", "J_Bip_C_Spine"},
        {"upperChest", "J_Bip_C_UpperChest"},
        {"chest", "J_Bip_C_Chest"},
        {"neck", "J_Bip_C_Neck"},
        {"head", "J_Bip_C_Head"},
        {"leftIris", "J_Adj_L_FaceEye"},
        {"rightIris", "J_Adj_R_FaceEye"},
        
        {"leftShoulder", "J_Bip_L_Shoulder"},
        {"leftLowerArm", "J_Bip_L_LowerArm"},
        {"leftUpperArm", "J_Bip_L_UpperArm"},
        {"leftHand", "J_Bip_L_Hand"},
        {"leftThumbProximal", "J_Bip_L_Thumb1"},
        {"leftThumbIntermediate", "J_Bip_L_Thumb2"},
        {"leftThumbDistal", "J_Bip_L_Thumb3"},
        {"leftIndexProximal", "J_Bip_L_Index1"},
        {"leftIndexIntermediate", "J_Bip_L_Index2"},
        {"leftIndexDistal", "J_Bip_L_Index3"},
        {"leftMiddleProximal", "J_Bip_L_Middle1"},
        {"leftMiddleIntermediate", "J_Bip_L_Middle2"},
        {"leftMiddleDistal", "J_Bip_L_Middle3"},
        {"leftRingProximal", "J_Bip_L_Ring1"},
        {"leftRingIntermediate", "J_Bip_L_Ring2"},
        {"leftRingDistal", "J_Bip_L_Ring3"},
        {"leftLittleProximal", "J_Bip_L_Little1"},
        {"leftLittleIntermediate", "J_Bip_L_Little2"},
        {"leftLittleDistal", "J_Bip_L_Little3"},
        {"leftUpperLeg", "J_Bip_L_UpperLeg"},
        {"leftLowerLeg", "J_Bip_L_LowerLeg"},
        {"leftFoot", "J_Bip_L_Foot"},
        {"leftToes", "J_Bip_L_ToeBase"},

        {"rightShoulder", "J_Bip_R_Shoulder"},
        {"rightLowerArm", "J_Bip_R_LowerArm"},
        {"rightUpperArm", "J_Bip_R_UpperArm"},
        {"rightHand", "J_Bip_R_Hand"},
        {"rightThumbProximal", "J_Bip_R_Thumb1"},
        {"rightThumbIntermediate", "J_Bip_R_Thumb2"},
        {"rightThumbDistal", "J_Bip_R_Thumb3"},
        {"rightIndexProximal", "J_Bip_R_Index1"},
        {"rightIndexIntermediate", "J_Bip_R_Index2"},
        {"rightIndexDistal", "J_Bip_R_Index3"},
        {"rightMiddleProximal", "J_Bip_R_Middle1"},
        {"rightMiddleIntermediate", "J_Bip_R_Middle2"},
        {"rightMiddleDistal", "J_Bip_R_Middle3"},
        {"rightRingProximal", "J_Bip_R_Ring1"},
        {"rightRingIntermediate", "J_Bip_R_Ring2"},
        {"rightRingDistal", "J_Bip_R_Ring3"},
        {"rightLittleProximal", "J_Bip_R_Little1"},
        {"rightLittleIntermediate", "J_Bip_R_Little2"},
        {"rightLittleDistal", "J_Bip_R_Little3"},
        {"rightUpperLeg", "J_Bip_R_UpperLeg"},
        {"rightLowerLeg", "J_Bip_R_LowerLeg"},
        {"rightFoot", "J_Bip_R_Foot"},
        {"rightToes", "J_Bip_R_ToeBase"},
    };
    private Dictionary<string, Transform> ModelBone;
    void Start()
    {
        var server = GetComponent<uOSC.uOscServer>();
        server.onDataReceived.AddListener(OnDataReceived);

        animator = Model.GetComponent<Animator>();
        blendShapeProxy = Model.GetComponent<VRMBlendShapeProxy>();

        ModelBone = new Dictionary<string, Transform>();
        GetAllChildren(Model);
    }

    void Update()
    {
        if (blendShapeProxy == null)
        {
            blendShapeProxy = Model.GetComponent<VRMBlendShapeProxy>();
        }
    }

    void OnDataReceived(uOSC.Message message)
    {
        if (message.address == "/VMC/Ext/Bone/Rot")
        {
            var boneName = (string)message.values[0];
            if (ReferenceEquals(message.values[1], null)){
                return;
            }
            var x = (float)message.values[1];
            var y = (float)message.values[2];
            var z = (float)message.values[3];
            var w = (float)message.values[4];
            if(boneName == "leftEye" || boneName == "rightEye"){
                return;
            }
            if(boneName == "leftIris" || boneName == "rightIris"){
                return;
            }
            if (boneName == "mouth"){
               blendShapeProxy.ImmediatelySetValue(BlendPreset["A"], clipping(x));
               return;
            } 
            if (boneName == "blink"){
                if(isLRBlink){
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink_L"], clipping(x));
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink_R"], clipping(y));
                }else{
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink"], clipping(z));
                }
                return;
            }
            if(boneName == "iris"){
                if((int)w == -1){
                    return;
                }
                ModelBone[Uni2VRMBone["leftIris"]].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
                ModelBone[Uni2VRMBone["rightIris"]].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
                return;
            }
            var VRMBoneName = Uni2VRMBone[boneName];
            ModelBone[VRMBoneName].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
        }
    }

    private void GetAllChildren(GameObject obj){
        Transform children = obj.GetComponentInChildren<Transform> ();
        if (children.childCount == 0){
            return;
        }
        foreach (Transform ob in children){
            if(BoneNames.Contains(ob.name)){
                ModelBone.Add(ob.name, ob);
            }
            GetAllChildren(ob.gameObject);
        }
    }
    public static Quaternion ReverseZ(Quaternion q)
    {
        float angle;
        Vector3 axis;
        q.ToAngleAxis(out angle, out axis);
        return Quaternion.AngleAxis(-angle, ReverseZ(axis));
    }
    public static Vector3 ReverseZ(Vector3 v)
    {
        return new Vector3(v.x, v.y, -v.z);
    }

    private float clipping(float value)
    {
        return Math.Min(Math.Max(value, 0.0f), 1.0f);
    }
}

V3DReciever.cs と UOscServer.csを空のオブジェクトに割り当てます。

これでUnity側の設定は終わりです。

あとは

v3d-web
websocket2udp.py
Unity

の3つを起動すると、モーションキャプチャができます。

お疲れさまでした。