LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@mvm43236(奥田 昌夫)

MindARでThreeJS FaceMeshにcanvasの画像を表示させる

Posted at

FaceMeshでフェイスペイントをしたい

canvasに描いた画像をFaceMeshに表示する「ARフェイスペイント」です。
https://kaihatuiinkai.jp/ardeasobou/facepaint.html
AR_facepaint.png

この基本を解説します。

MindARの「ThreeJS FaceMesh」が基本です

MindAR「Documentation」の「ThreeJS FaceMesh」のページが基本になります。
https://hiukim.github.io/mind-ar-js-doc/more-examples/threejs-face-facemesh

face_mesh.html

<html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script type="importmap">
    {
      "imports": {
        "three": "https://unpkg.com/three@0.160.0/build/three.module.js",
        "three/addons/": "https://unpkg.com/three@0.160.0/examples/jsm/",
        "mindar-face-three":"https://cdn.jsdelivr.net/npm/mind-ar@1.2.5/dist/mindar-face-three.prod.js"
      }
    }
    </script>
    <script type="module">
      import * as THREE from "three";
      import { MindARThree } from "mindar-face-three";
      const mindarThree = new MindARThree({
      	container: document.querySelector("#container"),
      });
      const {renderer, scene, camera} = mindarThree;
      const light = new THREE.HemisphereLight( 0xffffff, 0xbbbbff, 1 );
      scene.add(light);
      const faceMesh = mindarThree.addFaceMesh();
      const texture = new THREE.TextureLoader().load("https://cdn.jsdelivr.net/gh/hiukim/mind-ar-js@1.2.5/examples/face-tracking/assets/canonical_face_model_uv_visualization.png");
      faceMesh.material.map = texture;
      faceMesh.material.transparent = true;
      faceMesh.material.needsUpdate = true;
      scene.add(faceMesh);
      const start = async() => {
        await mindarThree.start();
        renderer.setAnimationLoop(() => {
          renderer.render(scene, camera);
        });
      }
      start();
    </script>
    <style>
      body {
      	margin: 0;
      }
      #container {
        width: 100vw;
        height: 100vh;
        position: relative;
        overflow: hidden;
      }
    </style>
  </head>
  <body>
    <div id="container">
    </div>
  </body>
</html>

作動確認用のサンプルはこちらです。
https://kaihatuiinkai.jp/ardeasobou/face_mesh.html
顔を動かすとメッシュが追随します。目を見開いたり、口を開けたりするとメッシュが変形します。

canvasに描いた画像を表示する

次に、canvasに描いた文字をメッシュの上に表示させてみます。
参考サイト「Three.jsで常に正面を向く長方形の文字ラベルを描画する」
https://astatsuya.medium.com/three-js%E3%81%A7%E5%B8%B8%E3%81%AB%E6%AD%A3%E9%9D%A2%E3%82%92%E5%90%91%E3%81%8F%E9%95%B7%E6%96%B9%E5%BD%A2%E3%81%AE%E6%96%87%E5%AD%97%E3%83%A9%E3%83%99%E3%83%AB%E3%82%92%E6%8F%8F%E7%94%BB%E3%81%99%E3%82%8B-fa606ed6752

scriptの一部を変えるだけです。

face_mesh2.html
    <script type="module">
      import * as THREE from "three";
      import { MindARThree } from "mindar-face-three";
      const mindarThree = new MindARThree({
        container: document.querySelector("#container"),
      });
      const {renderer, scene, camera} = mindarThree;
      const light = new THREE.HemisphereLight( 0xffffff, 0xbbbbff, 1 );
      scene.add(light);
      const faceMesh = mindarThree.addFaceMesh();
      // 貼り付けるcanvasを作成
      const createCanvasForTexture = (canvasWidth, canvasHeight, text, fontSize) => {
        const canvasForText = document.createElement("canvas");
        const ctx = canvasForText.getContext("2d");
        ctx.canvas.width = canvasWidth;
        ctx.canvas.height = canvasHeight; 
        ctx.fillStyle = "rgba(0, 0, 0, 0)";
        ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
        ctx.fillStyle = "black";
        ctx.font = `${fontSize}px serif`;
        ctx.fillText(
          text,
          (canvasWidth - ctx.measureText(text).width) / 2,
          canvasHeight / 2 + ctx.measureText(text).actualBoundingBoxAscent / 2
        );
        return canvasForText;
      };
      // 表示する文字
      var texture = new THREE.CanvasTexture(createCanvasForTexture(512, 512, "Hello World!", 40));
      faceMesh.material.map = texture;
      faceMesh.material.transparent = true;
      faceMesh.material.needsUpdate = true;
      scene.add(faceMesh);
      const start = async() => {
        await mindarThree.start();
        renderer.setAnimationLoop(() => {
          renderer.render(scene, camera);
        });
      }
      start();
    </script>

作動確認用のサンプルはこちらです。
https://kaihatuiinkai.jp/ardeasobou/face_mesh2.html
「Hello World!」文字が顔表面に沿って表示されます。

別canvasに描いた画像を表示する

別canvasが書き換わったとき、FaceMeshに画像を変えるために、renderer.setAnimationLoop内に記載します。

face_mesh3.html
<html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script type="importmap">
    {
      "imports": {
        "three": "https://unpkg.com/three@0.160.0/build/three.module.js",
        "three/addons/": "https://unpkg.com/three@0.160.0/examples/jsm/",
        "mindar-face-three":"https://cdn.jsdelivr.net/npm/mind-ar@1.2.5/dist/mindar-face-three.prod.js"
      }
    }
    </script>
    <script type="module">
      import * as THREE from "three";
      import { MindARThree } from "mindar-face-three";
      const mindarThree = new MindARThree({
        container: document.querySelector("#container"),
      });
      const {renderer, scene, camera} = mindarThree;
      const light = new THREE.HemisphereLight( 0xffffff, 0xbbbbff, 1 );
      scene.add(light);
      const faceMesh = mindarThree.addFaceMesh();
      // 貼り付けるcanvasを作成。
      const canvasForText = document.createElement("canvas");
      const ctx = canvasForText.getContext("2d");
      ctx.canvas.width  = 512;
      ctx.canvas.height = 512;
      var image = document.querySelector("#drawArea").getContext("2d").getImageData(0, 0, 512, 512);
      //  canvasにデータをコピー
      const createCanvasForTexture = () => {
        image = document.querySelector("#drawArea").getContext("2d").getImageData(0, 0, 512, 512);
        ctx.putImageData(image, 0, 0);
        return canvasForText;
      };

      var texture = new THREE.CanvasTexture(createCanvasForTexture());
      texture.colorSpace = THREE.SRGBColorSpace;  // 色の感じを変える
      faceMesh.material.map = texture;
      faceMesh.material.transparent = true;
      faceMesh.material.needsUpdate = true;
      scene.add(faceMesh);
      const start = async() => {
        await mindarThree.start();
        renderer.setAnimationLoop(() => {
          var texture = new THREE.CanvasTexture(createCanvasForTexture());
          faceMesh.material.map = texture;
          renderer.render(scene, camera);
        });
      }
      start();
    </script>
    <style>
      body {
      	margin: 0;
      }
      #container {
        width: 100vw;
        height: 100vh;
        position: relative;
        overflow: hidden;
      }
    </style>
  </head>
  <body>
    <div id="container" style="width: 600px;height: 400px;">
    </div>
    <div style="position:absolute; top: 0px; left:0px">
      <canvas id="drawArea" width= "512px" height= "512px"
           style="z-index: 2;position:absolute; top:0px; left:600px;
          border: 1px solid rgba(0,0,0,1);
          background-color: rgba(255,255,255,0);">
      </canvas>
    </div>
    <script>
      // 別canvasに文字を表示
      const drawArea = document.querySelector("#drawArea");
      const ctxDrawArea = drawArea.getContext("2d");
      ctxDrawArea.fillStyle = "black";
      ctxDrawArea.font = "50px serif";
      ctxDrawArea.fillText("Hello World!", 100, 250);
    </script>
  </body>
</html>

作動確認用のサンプルはこちらです。
https://kaihatuiinkai.jp/ardeasobou/face_mesh3.html
別canvasに描いた「Hello World!」の文字が顔表面に沿って表示されます。
あとは、この「別canvas」に図を描く機能を追加していきます。

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?