How to build MediaPipe hand tracking (iOS) as an Xcode project

Agenda

1. Preparation (MediaPipe)
2. Build from terminal
3. Preparation (Tulsi)
4. Build as an Xcode project

Demo

Hand Tracking on iOS📱#MediaPipe pic.twitter.com/mgHPfUIHwK

— MIWA Tetsushi (@WWWPONTE) August 11, 2020

DEV

• MacBook Pro: Catalina 10.15.4
  • Xcode: 11.6
• iPhone SE (2nd generation): iOS 13.5.1

Preparation (MediaPipe)

1. Install Homebrew

• Copy the command at https://brew.sh
• Paste to your terminal and execute
• The version of Homebrew I used this time was 2.4.7.

$ brew -v
# Homebrew 2.4.7

2. Install Command Line Tools

• Go to https://developer.apple.com/download/more and download Command Line Tools for Xcode XX ※You need to sign in Apple Developer Program.
• Double-click .dmg
• Execute the following command

$ sudo xcodebuild -license

3. Check the Python version

Python is pre-installed on mac by default. Depending on the Python version, the build may not pass, so it is necessary to check the Python version. I confirmed that the build passed with Python 3.7.5, so I recommend building with this version of Python. At that time, there is a version manager called pyenv that can switch the Python version, so I will explain how to use it.

• Clone the pyenv repository

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

• Add to path .zsh_profile

zsh

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zsh_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zsh_profile
$ echo 'eval "$(pyenv init -)"' >> ~/.zsh_profile

• Install Python 3.7.5

$ pyenv install 3.7.5

• Rehash shim

$ pyenv rehash

• Specifying the Python version

The global command sets the global Python version. This can be overridden with other commands, but is useful for ensuring you use a particular Python version by default. If you wanted to use 3.7.5 by default, then you could run this:

$ pyenv global 3.7.5

The local command is often used to set an application-specific Python version. You could use it to set the version to 3.7.5:

$ pyenv local 3.7.5

4. Install six library

Install six library for absorbing the difference between Python 2 and Python 3.

$ pip install –user future six

5. Clone the MediaPipe repository

$ git clone https://github.com/google/mediapipe.git

6. Install Bazel

• Insatll Bazel

$ brew install bazel

• The version of bazel I used this time was 3.3.0.

$ bazel --version
# bazel 3.3.0

7. Install OpenCV and FFmpeg

$ brew install opencv@3

8. Install numpy

$ pip install numpy

9. Check installation with Hello World

• Execute Hello World desktop example

$ export GLOG_logtostderr=1
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
    mediapipe/examples/desktop/hello_world:hello_world

# After building bazel (it takes a few minutes), it is OK if "Hello World!" is displayed 10 times as shown below.
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!

Build from terminal

1. Prepare Provisioning Profile

In order to execute the iOS app on your device, a file for identifying the iOS device or app called the Provisioning Profile is required. Apple Developer Program subscribers can create and download from https://developer.apple.com/jp. Rename the downloaded file to provisioning_profile.mobileprovision and place it in mediapipe/mediapipe/.

2. Change Bundle Identifier

Next, fix BUILD at mediapipe/mediapipe/examples/ios/handtrackinggpu/. Change bundle_id to the same one which you set at Provisioning Profile .

mediapipe/examples/ios/handtrackinggpu/BUILD：36

bundle_id = BUNDLE_ID_PREFIX + ".HandTrackingGpu",
　　　　　　　　　↓
bundle_id = "(Bundle Identifier)",

3. Build

Move mediapipe/ and execute the following command.

$ bazel build -c opt –config=ios_arm64 mediapipe/example/ios/handtrackinggpu:HandTrackingGpuApp

An IPA file is generated at the following directory.

bazel-bin/mediapipe/examples/ios/handtrackinggpu/

Preparation (Tulsi)

1. Clone the Tulsi repository

$ git clone https://github.com/bazelbuild/tulsi.git

2. Apply the patch

$ cd tulsi
$ git fetch origin pull/99/head:xcodefix
$ git checkout xcodefix

3. Execute the build script

sh build_and_run.sh

Occurred an error!!

ERROR: /Users/miwa/tulsi/BUILD:62:18: Linking of rule '//:tulsi.__internal__.apple_binary' failed (Exit 1) wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang -Xlinker -objc_abi_version -Xlinker 2 -fobjc-link-runtime -ObjC -arch x86_64 -filelist ... (remaining 26 argument(s) skipped)

After checking this, I could execute the build script!

tulsi/WORKSPACE：6

tag = "0.17.2",
　　　　　　　　　↓
tag = "0.18.0",

4. Open MediaPipe.tulsiproj and generate the Xcode project

• Launch Tulsi.app and open Mediapipe.tulsiproj at mediapipe/mediapipe/.

• Push Generate button in Configs tab

Build as an Xcode project

Now connect your iPhone to your mac. Then open the generated Xcode project and start the build. After a while, the build will be completed and it will be installed on your iPhone.

Set to use the rear camera.

mediapipe/examples/ios/handtrackinggpu/ViewController.mm：107

_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
　　　　　　　　　↓
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;

Displaying reversed image, so I changed the value of Mirrored.

mediapipe/examples/ios/handtrackinggpu/ViewController.mm：111

_cameraSource.videoMirrored = YES;
　　　　　　　　　↓
_cameraSource.videoMirrored = NO;

Summary

I succeed in implementing high-precision real-time hand tracking on iOS.

In the future, I will develop learning iOS applications for the visually impaired with this technology.