LoginSignup
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

@wanic

「Codama」ボードを使った音声認識によるデバイス制御実験（おまけ：認識結果・カメラ映像のモニタリング）

Posted at

ビデオで紹介した今回の実験システムの全体構成です。　Part-1～3で、欠けていた部分を掲載します。(Part-1、Part-2、Part-3）

■　全体構成

codama_21.JPG

■ 実験システムのファイル構成

拡張性と理解しやすさを考えてファイルを分けています。
データの授受（認識結果・ビデオ映像）はRedis D/Bを介して行うことにより分散処理を可能にしています。

・static
　　├ css
　　　　└ style.css
　　├ js
　　　　└ app.js
・template
　　└ index.html
・server.py
・Picam.py
・PiVedeoCamera.py
・define.py (Part-3)
・devcont.py (Part-3)
・DevControl.py (Part-3)

■ 起動方法

exec.sh
python3 Picam.py &
python3 DevControl.py &
python3 server.py

■ Tornade Web Socket Server

認識結果及びビデオ映像をJsonに変換してブラウザに送信しています。

■ Server.py

server.py
# -*- coding: utf-8 -*-
import sys
import os
import json
import time
from struct import *
import random
# Import 3rd-party modules.
from tornado import websocket, web, ioloop
import redis

hostname = '192.168.xxx.xxx'
MAX_FPS = 100

# ============================
# Tornade web socket server
# ============================
class IndexHandler( web.RequestHandler):
    def get( self):
        self.render( 'index.html')

class SocketHandler( websocket.WebSocketHandler):
    def __init__(self, *args, **kwargs):
        super(SocketHandler, self).__init__(*args, **kwargs)
        pool = redis.ConnectionPool( host = hostname, port = 6379, db = 0)
        self._r = redis.StrictRedis( connection_pool = pool)
        self._store = redis.StrictRedis( host='localhost', port='6379')
        self._prec_id = None
        self._prev_image_id = None
        self.result = ""

    def open( self):
        print( "Tornado WebSocket opened")

    def on_message( self, dummy):
        while True:
            time.sleep( 1./MAX_FPS)
            image_id = self._store.get( 'image_id')
            if image_id != self._prev_image_id:
                break
        self._prev_image_id = image_id
        image = self._store.get( 'image').decode( 'utf-8')
        rec_id = self._r.get( "Rec_id")
        if rec_id != self._prec_id:
            # byte型を文字列型に変換
            self.result = self._r.get( "Result").decode( 'utf-8')
            self._prec_id = rec_id

        #　データをDict形式で格納
        tmpDic = { 'Result': self.result, 'Image': image}
        #　jsonデータに変換
        toBrowser = json.dumps( tmpDic)
        #print( "JsonData:", toBrowser, "\n")
        # ブラウザに送信
        self.write_message( toBrowser)

    def on_close(self):
        print( "WebSocket closed")

app = web.Application([
    ( r'/', IndexHandler),
    ( r'/ws', SocketHandler),
    ],
    template_path=os.path.join( os.getcwd(),  "templates"),
    static_path=os.path.join( os.getcwd(),  "static"),
)

if __name__ == '__main__':
    try:
        print( "Tornado Server is running : port 9000")
        app.listen( 9000)
        ioloop.IOLoop.instance().start()

    except KeyboardInterrupt:
        sys.exit(0)

■ index.html

index.html
<!doctype html>
<html lang="ja">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Voice Device Control Test</title>
    <link type="text/css" rel="stylesheet" href="http://code.jquery.com/ui/1.10.3/themes/cupertino/jquery-ui.min.css" />
    <link rel="stylesheet" href="{{ static_url("css/style.css") }}"/>
    <script type="text/javascript" src="http://code.jquery.com/jquery-1.10.2.min.js"></script>
    <script type="text/javascript" src="http://code.jquery.com/ui/1.10.3/jquery-ui.min.js"></script>
    <script type="text/javascript" src="{{ static_url("js/app.js") }}"></script>
</head>
<body>
<h1>Voice Control Device</h1>
<div class="container">
    <dic id="camera_info">
        <img id="cam" src=""/>
    </div>
    <div id="data_info">
        <dl>
            <dt>Result:</dt>
            <dd id="result">@@@</dd>
        </dl>
	</div>
</div>
</body>
</html>

■ app.js

app.js
$(function() {
    var nextRequest = "Send me next";
    var result = "";
    var image = "";
    var ws_path = "ws://192.168.xxx.xxx:9000/ws";
    var ws = new WebSocket( ws_path);

    console.log( ws_path);

    $( "#result").text( result);
        ws.onopen = function() {
        ws.send( nextRequest);
        console.log( "Request send to Tornado Server");
    };

    // ラズパイからのデータ受信
    // e は [object MessageEvent]
    ws.onmessage = function( e) {
        // 受信データをデコード（ラズパイでの定義）
        var decJson = JSON.parse( e.data);
        result = decJson.Result;
        image = decJson.Image;

        $( "#cam").attr( 'src', 'data:image/jpg;base64,' + image);
        $( "#result").text( result);
        ws.send( 1);
    };

    ws.onerror = function( e) {
        console.log( e);
    };
});

■ style.css

style.css
@charset "utf-8";
* {
    margin: 0;
    padding: 0;
}
body {
    padding: 20px;
}
.container {
    position: relative;
    margin-top: 0px;
    margin-left: 10px;
}
#camera_info {
    position: absolute;
    top: 0px;
    left: 15px;
}
#data_info {
    position: absolute;
    top: 560px;
    left: 50px;
}
#data_info dl {
}
#data_info dt {
    padding: 3px;
    font-size: 20px;
    font-weight: bold;
    color: blue;
    float: left;
}
#data_info dd {
    margin-left: 100px;
    padding: 3px;
    font-size: 24px;
    font-weight: bold;
    color: red;
}

■　ラズパイカメラ画像の入力

■ Picam.py

Picam.py
# -*- coding: utf-8 -*-
import sys
import os
import time
import base64
import redis
import numpy
import imutils
import cv2
from PIL import Image
from imutils.video.pivideostream import PiVideoStream

# Create client to the Redis store.
store = redis.StrictRedis( host='localhost', port='6379')

# Create video capture object and socket client.
vs = PiVideoStream().start()

# allow the camera to warmup
time.sleep( 2)

while True:
    frame = vs.read()
    if frame is None:
        time.sleep( 0.5)
        continue

    frame = imutils.resize( frame, width=640)
    # jpegに圧縮
    result, encimg = cv2.imencode( '.jpg', frame)
    # 画像をBase64の文字列に変換
    # prifixは、Javascriptで追加（ 'data:image/jpg;base64,'）
    enc64img = base64.b64encode( encimg)

    store.set( 'image', enc64img)
    image_id = os.urandom( 4)
    store.set( 'image_id', image_id)

■ PiVedeoCamera.py

PiVedeoCamera.py
# -*- coding: utf-8 -*-
# import the necessary packages
from picamera.array import PiRGBArray
from picamera import PiCamera
from threading import Thread

class PiVideoStream:
    def __init__(self, resolution=(320, 240), framerate=32):
        # initialize the camera and stream
        self.camera = PiCamera()
        self.camera.resolution = resolution
        self.camera.framerate = framerate
        self.rawCapture = PiRGBArray( self.camera, size=resolution)
        self.stream = self.camera.capture_continuous( self.rawCapture, format="bgr", use_video_port=True)

        # initialize the frame and the variable used to indicate
        # if the thread should be stopped
        self.frame = None
        self.stopped = False

    def start(self):
        # start the thread to read frames from the video stream
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        # keep looping infinitely until the thread is stopped
        for f in self.stream:
            # grab the frame from the stream and clear the stream in
            # preparation for the next frame
            self.frame = f.array
            self.rawCapture.truncate(0)

            # if the thread indicator variable is set, stop the thread
            # and resource camera resources
            if self.stopped:
                self.stream.close()
                self.rawCapture.close()
                self.camera.close()
                return

    def read(self):
        # return the frame most recently read
        return self.frame

    def stop(self):
        # indicate that the thread should be stopped
        self.stopped = True

参考資料

Websocket／Redis／Picamera　などで大変お世話になったサイトです。

1
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
1
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?