海外お菓子の成分表示を自動翻訳してみた

最近、仕事に忙殺され、趣味コーディングができていなかったので、手前味噌がてら久しぶりに投稿。

画像データに含まれるテキストデータを検出し、好きな言語に自動翻訳できるアプリを実装してみた。
外国人旅行者が増える中、分らない言語の看板や商品説明ページ、食品の成分表...などなど、写真で取れば後は自動で好きな言語に翻訳してくれれば少し幸せなことだと思う。

今回は、AWSの画像認識API(Amazon Rekognition)/翻訳サービス(Amazon Translate)を使用し、以下のような構成で実装してみた。
　　　

1. Lambda側実装

①　Lambdaハンドラー実装

LambdaFunctionHandler.java

package com.amazonaws.lambda.demo;

import java.util.Map;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

public class LambdaFunctionHandler implements RequestHandler<Map<String, String>, String> {

    public String handleRequest(Map<String, String> event, Context context) {

        ImageDetectText imageDetectText = new ImageDetectText();
        TranslateText translateText = new TranslateText(event.get("src"), event.get("target"));

        context.getLogger().log("Received event: " + event);

        String[] imageData = event.get("imageData").split(",");

        try {

            String detectRes = imageDetectText.doDetectText(imageData[1]);
            String translateRes = translateText.doTranslate(detectRes);
            return translateRes;

        } catch (Exception e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.INTERNAL_SERVER_ERROR);
        }
    }

}

②　テキスト抽出処理

ImageDetectText.java

package com.amazonaws.lambda.demo;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.List;

import org.springframework.http.HttpStatus;

import com.amazonaws.services.rekognition.AmazonRekognition;
import com.amazonaws.services.rekognition.AmazonRekognitionClientBuilder;
import com.amazonaws.services.rekognition.model.DetectTextRequest;
import com.amazonaws.services.rekognition.model.DetectTextResult;
import com.amazonaws.services.rekognition.model.Image;
import com.amazonaws.services.rekognition.model.ImageTooLargeException;
import com.amazonaws.services.rekognition.model.InvalidImageFormatException;
import com.amazonaws.services.rekognition.model.TextDetection;
import com.amazonaws.util.IOUtils;

public class ImageDetectText {

    private AmazonRekognition rekognitionClient;

    public ImageDetectText() {
        this.rekognitionClient = AmazonRekognitionClientBuilder.defaultClient();
    }

    public String doDetectText(String encodedImage) {

        StringBuilder sb = new StringBuilder();

        InputStream isImage = new ByteArrayInputStream(
                Base64.getDecoder().decode(encodedImage.getBytes(StandardCharsets.UTF_8)));

        ByteBuffer bbImage = null;
        try {
            bbImage = ByteBuffer.wrap(IOUtils.toByteArray(isImage));
        } catch (IOException e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.INTERNAL_SERVER_ERROR);
        }

        DetectTextRequest request = new DetectTextRequest().withImage(new Image().withBytes(bbImage));

        try {
            DetectTextResult result = rekognitionClient.detectText(request);
            List<TextDetection> textDetections = result.getTextDetections();
            for (TextDetection text : textDetections) {
                sb.append(text.getDetectedText() + " ");
            }
        } catch (InvalidImageFormatException e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.BAD_REQUEST);
        } catch (ImageTooLargeException e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.BAD_REQUEST);
        }

        return new String(sb);
    }

}

③　テキスト翻訳

TranslateText.java

package com.amazonaws.lambda.demo;

import org.springframework.http.HttpStatus;

import com.amazonaws.services.translate.AmazonTranslate;
import com.amazonaws.services.translate.AmazonTranslateClient;
import com.amazonaws.services.translate.model.TextSizeLimitExceededException;
import com.amazonaws.services.translate.model.TranslateTextRequest;
import com.amazonaws.services.translate.model.TranslateTextResult;
import com.amazonaws.services.translate.model.UnsupportedLanguagePairException;

public class TranslateText {

    private AmazonTranslate translate;

    private String srcLanguage;

    private String targetLanguage;

    public TranslateText(String srcLanguage, String targetLanguage) {
        this.translate = AmazonTranslateClient.builder().build();
        this.srcLanguage = srcLanguage;
        this.targetLanguage = targetLanguage;
    }

    public String doTranslate(String inputText) {

        TranslateTextRequest request = new TranslateTextRequest().withText(inputText)
                .withSourceLanguageCode(srcLanguage).withTargetLanguageCode(targetLanguage);

        TranslateTextResult result = null;
        try {
            result = translate.translateText(request);
        } catch (UnsupportedLanguagePairException e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.BAD_REQUEST);
        } catch (TextSizeLimitExceededException e) {
            throw new DemoTranslateException(e.getMessage(), HttpStatus.BAD_REQUEST);
        }
        return result.getTranslatedText();
    }
}

④　リクエスト失敗時例外処理 (※400/500応答のハンドリング)

DemoTranslateException.java

package com.amazonaws.lambda.demo;

import org.springframework.http.HttpStatus;

public class DemoTranslateException extends RuntimeException {

    private static final long serialVersionUID = 1L;

    private static final HttpStatus DEFAULT_STATUS = HttpStatus.INTERNAL_SERVER_ERROR;

    private final HttpStatus status;

    public DemoTranslateException(String msg) {
        super(msg);
        status = DEFAULT_STATUS;
    }

    public DemoTranslateException(Throwable t) {
        super(t);
        status = DEFAULT_STATUS;
    }

    public DemoTranslateException(String msg, HttpStatus status) {
        super(msg);
        this.status = status;
    }

    public HttpStatus getStatus() {
        return status;
    }

}

2. フロント側(Ajax)実装

translate_ajax.js

$(function() {
    var endpoint = "https://{APIGATEWAY_ENDPOINT}/{STAGE}/";

    $('#upfile').on('change', function (e) {
        var reader = new FileReader();
      reader.readAsDataURL(e.target.files[0]);

        reader.onload = function (event) {
           $("#preview").attr('src', event.target.result);
           var data_url = reader.result;
           document.getElementById("image_id").value = data_url;
        }
    });

    $('#btn_id').on('click', function(event) {
        event.preventDefault();
        $.ajax({
            url: endpoint,
            type: 'POST',
            data: JSON.stringify({
                'imageData': $('#image_id').val(),
                'src': $('#lang_from_id').val(),
                'target': $('#lang_to_id').val(),
            }),
            contentType: "application/json; charset=utf-8"
        })
        .done((data) => {
            $("#span1").text(data);
        })
        .fail((data, status, error) => {
            var dataJson = JSON.stringify(data);
            if(data.status == 500){
                $("#span1").text(JSON.parse(dataJson)["errorMessage"]);
            } else if (data.status == 400){
                $("#span1").text(JSON.parse(dataJson)["errorMessage"]);
            }
        })
    })
});

3. 検証

①　画像アップロード

②　翻訳前後の言語を指定
翻訳前後の言語を指定。今回は、英語から日本語を選択。

③　翻訳の実行

ちなみに、フランス語の場合は...(ちゃんと翻訳できてるっぽい？)