Go to Qiita Advent Calendar 2024 Top
LoginSignup
7
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@wattai

HoloLens を TCP server とした画像の分割受信

Last updated at Posted at 2019-08-22

動機

PC (Python) から HoloLens (C#) へ画像を送信する必要があったため，このコードを書きました．
TCP サーバの受信一回で受信可能な小さな画像でなら記事がいくつか散見されるのですが， HoloLens 環境で複数回に分けて受信する方法が見つからずハマってしまい，試行錯誤したので記録します．

環境

  • Unity 2017.4.24f1
  • HoloLens (C#, server) ※2じゃないよ
  • HoloToolkit-Unity-2017.4.1.0
  • PC (Python3.6, client)

大まかな動作

client 側 (PC, Python)

  1. Python3 で画像を open して byte 列で読み込み．
  2. base64 形式の文字列に encode．
  3. この base64 文字列を送信する byte 数で分割．
  4. 分割した base64 文字列を TCP を使って逐次的に Python から HoloLens(server) に送信．

server 側 (HoloLens, C#)

  1. 分割された base64 文字列を逐次的に処理（受信->decode->ファイルへの書き込みを繰り返す）．
  2. 受信した byte 数が指定した値より小さければ EOF とみなし最後の書き込みを行う．
  3. 受信した画像に応じた処理．

動作確認

  1. ImageReceiver.cs を Unity 内の texture を持つオブジェクトに Add します．（僕は Canvas 内に RawImage を作ってそこへ Add しました．）
  2. HoloLens に書き込み動作させます．
  3. client.py と同じディレクトリに sample.jpg （.png でも大丈夫です）用意します．
  4. client.py を走らせます．
  5. sample.jpg と同じ画像が HoloLens の view に現れたら成功です．

直接 byte 列を送受信するとなぜかファイルが壊れてしまいます．
そこで一旦 base64 という形式の文字列に変換し，これを通して送受信します．

client 側のコード (Python, PC)

host には各自の HoloLens の IP address を入力してください．

client.py
import socket  # Import socket module
import base64

N_byte = 1024*8*8

if __name__ == '__main__':
    host = '192.168.000.000'  # here is your hololens ip address.
    port = 8080  # Reserve a port for your service.
    imgfile = './sample.jpg'

    with socket.socket() as s:
        s.connect((host, port))
        s.settimeout(3)
        with open(imgfile, 'rb') as f:
            imgstring = base64.b64encode(f.read())
            sub_strings = [imgstring[i: i+N_byte] for i in range(0, len(imgstring), N_byte)]
            print('Sending...')
            for sub_string in sub_strings:
                s.send(sub_string)
            print(imgstring[-20:])

        print("Done Sending")
        print(s.recv(N_byte))
        s.shutdown(socket.SHUT_WR)

server 側のコード (C#, HoloLens)

Unity 内で texture を持っているオブジェクトに Add して使います．
僕の環境ではなぜか画像を書き込む直前に少し delay を入れないと受信に失敗したり，受信に成功しても画像の画素値が変わっていたりと，動作が不安定になりました．

ImageReceiver.cs
using UnityEngine;
using System;
using System.IO;
using System.Security.Cryptography;
using UnityEngine.UI;

//<JEM>Ignore unity editor and run this code in the hololens instead</JEM>
#if !UNITY_EDITOR
using Windows.Networking.Sockets;
using Windows.Storage.Streams;
using WinRTLegacy;
using System.Runtime.InteropServices.WindowsRuntime;
using System.Threading.Tasks;
#endif

// Able to act as a reciever 
public class ImageReceiver : MonoBehaviour {
    RawImage rend;
    bool socketClosed = false;
    bool writeStringToFile = false;
    bool loadTexture = false;
    bool logSize = false;
    
    public uint BUFFER_SIZE = 8192;
    public uint PORT = 8080;
    private readonly int DELAYMILLISEC = 10;
    public string textAll = "";
    string error_message;
    string error_source;
    string FILENAME = "received.png"

#if !UNITY_EDITOR
    StreamSocketListener listener;    
#endif
    // Use this for initialization
    void Start() {
#if !UNITY_EDITOR
        rend = this.GetComponent<RawImage>();
        listener = new StreamSocketListener();

        listener.ConnectionReceived += _receiver_socket_ConnectionReceived;
        listener.Control.KeepAlive = true;
        Listener_Start();
#endif
    }

#if !UNITY_EDITOR
    private async void Listener_Start()
    {
        try
        {
            await listener.BindServiceNameAsync(PORT.ToString());
            Debug.Log("Listener started");
            Debug.Log(NetworkUtils.GetMyIPAddress() + " : " + PORT.ToString());

        } catch (Exception e)
        {
            Debug.Log("Error: " + e.Message);
        }
    }

    private async void _receiver_socket_ConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
    {
        try
        {
            if (loadTexture != true) {
                string folderPath = System.IO.Directory.GetCurrentDirectory();

                // Create sample file; replace if exists.
                // Must be set as TemporaryFolder to read files from HoloLens.
                Windows.Storage.StorageFolder storageFolder = Windows.Storage.ApplicationData.Current.TemporaryFolder;

                using (var dr = new DataReader(args.Socket.InputStream)) {
                    using (IInputStream input = args.Socket.InputStream) {
                        using (var imageFile = new FileStream(storageFolder.Path + @"\" + FILENAME, FileMode.Create)) {
                            using (FromBase64Transform myTransform = new FromBase64Transform(
                                FromBase64TransformMode.IgnoreWhiteSpaces)) {
                                
                                byte[] data = new byte[BUFFER_SIZE];
                                IBuffer buffer = data.AsBuffer();
                                uint dataRead = BUFFER_SIZE;
                                byte[] dataTransformed = new byte[BUFFER_SIZE];

                                while (dataRead == BUFFER_SIZE) {
                                    await input.ReadAsync(buffer, BUFFER_SIZE, InputStreamOptions.Partial);
                                    int bytesWritten = myTransform.TransformBlock(data, 0,
                                        (int)BUFFER_SIZE, dataTransformed, 0);
                                    
                                    await Task.Delay(DELAYMILLISEC);
                                    imageFile.Write(dataTransformed, 0, bytesWritten);
                                    dataRead = buffer.Length;
                                }
                                dataTransformed = myTransform.TransformFinalBlock(data, 0, data.Length - (int)dataRead);
                                imageFile.Write(dataTransformed, 0, dataTransformed.Length);
                                myTransform.Clear();
                            }
                            imageFile.Flush();
                        }
                    }
                }
                loadTexture = true;
            }
        }
        catch (Exception e)
        {
            error_source = e.Source;
            error_message = e.Message;
            socketClosed = true;
        }
        finally {
            if (loadTexture == true) {
                using (var dw = new DataWriter(args.Socket.OutputStream)) {
                    dw.WriteString("OK");
                    await dw.StoreAsync();
                    dw.DetachStream();
                }
            } else {
                using (var dw = new DataWriter(args.Socket.OutputStream)) {
                    dw.WriteString("NG");
                    await dw.StoreAsync();
                    dw.DetachStream();
                }
            }
        }
    }

    void Update() {
        if (logSize) {
            Debug.Log("SIZE IS : " + BUFFER_SIZE.ToString());
            logSize = false;
        }
        if (socketClosed) {
            Debug.Log(error_source);
            Debug.Log(error_message);
            Debug.Log("OOPS SOCKET CLOSED ");
            socketClosed = false;
            Debug.Log(textAll);
        }
        if (writeStringToFile) {
            Debug.Log("WRITTEN TO FILE");
            writeStringToFile = false;
        }
        if (loadTexture) {
            Debug.Log("LOADING IMAGE CURRENTLY");

            // Must be set as TemporaryFolder to read files from HoloLens.
            Windows.Storage.StorageFolder storageFolder = Windows.Storage.ApplicationData.Current.TemporaryFolder;

            string imgpath = storageFolder.Path + @"\" + FILENAME;

            Destroy(this.rend.texture);
            this.rend.texture = ReadPngAsTexture(imgpath);
            this.rend.SetNativeSize();

            Debug.Log("LOADED IMAGE");
            Debug.Log(textAll);
            loadTexture = false;
        }
    }
#endif

    private static byte[] ReadPngFile(string path) {
        byte[] values;
        using (FileStream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read)) {
            using (BinaryReader bin = new BinaryReader(fileStream)) {
                values = bin.ReadBytes((int)bin.BaseStream.Length);
            }
        }
        return values;
    }

    private static Texture2D ReadPngAsTexture(string path) {
        byte[] readBinary = ReadPngFile(path);
        Texture2D texture = new Texture2D(1, 1);
        texture.LoadImage(readBinary);
        return texture;
    }
}

既知のバグ

BUFFER_SIZE を下回るサイズの画像を送信すると受信できない不具合を確認しております．

最後に

右も左もわからない状況で HoloLens 用 C# コードを書いております．
問題点等ございましたら，ぜひご指摘お願いします！

参考にした記事

7
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
7
2

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?