LoginSignup
4
0

More than 3 years have passed since last update.

ZOZOテクノロジーズ #3Advent Calendar 2019Day 25zozotech
@BrunoBin株式会社ZOZO

Firebase MLKit

Last updated at Posted at 2019-12-24

Introduction

ML Kit for Firebase is a machine learning toolkit made by Google for Android and iOS. (Well still in Beta version)
With this kit you can use on-device pre-trained APIs :

  • Text recognition
  • Face detection
  • Barcode scanning
  • Image labeling
  • Object detection & tracking
  • Language identification
  • Translation
  • Smart reply generator (only in english)
You can also use cloud APIs for more accurate answer :
  • Text recognition
  • Image labeling
  • Landmark recognition
You can use your custom pre-trained models and you can train your own classification model. (for images labeling only)

Android dependency


In app/build.gradle you should add : 
android {
  //...
  aaptOptions {
      noCompress "tflite"
  }
}  
dependencies {
  //...
  // ml-vision general
  implementation 'com.google.firebase:firebase-ml-vision:24.0.1'
  // Face Detection (contours)
  implementation 'com.google.firebase:firebase-ml-vision-face-model:19.0.0' 
  // Barcode Scanning
  implementation 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.1'
  // Image labeling
  implementation 'com.google.firebase:firebase-ml-vision-image-label-model:19.0.0'
  // Object detection
  implementation 'com.google.firebase:firebase-ml-vision-object-detection-model:19.0.3'
  // ml-natural general
  implementation 'com.google.firebase:firebase-ml-natural-language:22.0.0'
  // Langauge identification
  implementation 'com.google.firebase:firebase-ml-natural-language-language-id-model:20.0.7'
  // Translation
  implementation 'com.google.firebase:firebase-ml-natural-language-translate-model:20.0.7'
  // Smart Replies
  implementation 'com.google.firebase:firebase-ml-natural-language-smart-reply-model:20.0.7'
}
apply plugin: 'com.google.gms.google-services'

If you want to use a custom pre-trained model(AutoML-trined model) to load your own model you will need to add :
implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.3'

Text recognition

// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Create an instance of FirebaseVisionTextRecognizer with on-device model
FirebaseVisionTextRecognizer detector = FirebaseVision.getInstance().getOnDeviceTextRecognizer();
// Or with cloud model
// FirebaseVisionTextRecognizer detector = FirebaseVision.getInstance().getCloudTextRecognizer();

// Process the image
Task<FirebaseVisionText> result =
        detector.processImage(image)
                .addOnSuccessListener(new OnSuccessListener<FirebaseVisionText>() {
                    @Override
                    public void onSuccess(FirebaseVisionText firebaseVisionText) {
                        // Task completed successfully
                        // ...
                    }
                })
                .addOnFailureListener(
                        new OnFailureListener() {
                            @Override
                            public void onFailure(@NonNull Exception e) {
                                // Task failed with an exception
                                // ...
                            }
                        });

FirebaseVisionText will contain bounding box, text, language recognized, paragraph, confidence score.

Face detection


Face detection is done on device only and you can get facial contours too (optional). 
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set Options
FirebaseVisionFaceDetectorOptions options =
          new FirebaseVisionFaceDetectorOptions.Builder()
                .setPerformanceMode(FirebaseVisionFaceDetectorOptions.ACCURATE)
                .setClassificationMode(FirebaseVisionFaceDetectorOptions.ALL_CLASSIFICATIONS)
                .setLandmarkMode(FirebaseVisionFaceDetectorOptions.ALL_LANDMARKS)                                                                                
                .setContourMode(FirebaseVisionFaceDetectorOptions.ALL_CONTOURS)
                .build();

// Create an instance of FirebaseVisionFaceDetector
FirebaseVisionFaceDetector detector = FirebaseVision.getInstance().getVisionFaceDetector(options);

// Process the image
Task<List<FirebaseVisionFace>> result =
        detector.detectInImage(image)
                .addOnSuccessListener(
                        new OnSuccessListener<List<FirebaseVisionFace>>() {
                            @Override
                            public void onSuccess(List<FirebaseVisionFace> faces) {
                                // Task completed successfully
                                // ...
                            }
                        })
                .addOnFailureListener(
                        new OnFailureListener() {
                            @Override
                            public void onFailure(@NonNull Exception e) {
                                // Task failed with an exception
                                // ...
                            }
                        });

As a result FirebaseVisionFace contains : Face bounds, head rotation, eyes ears, mouth nose coordinate, classification probability (smiling, eyes opened, happy ...)
You can also get a tracking Id in case of video streaming.

Barcode scanning


Many different formats are supported :
Code 128, Code 39, Code 93, Codabar, EAN-13, EAN-8, ITF, UPC-A, UPC-E, QR Code, PDF417, Aztec, Data Matrix. 
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set Options
FirebaseVisionBarcodeDetectorOptions options =
        new FirebaseVisionBarcodeDetectorOptions.Builder()
        .setBarcodeFormats(
                FirebaseVisionBarcode.FORMAT_QR_CODE,
                FirebaseVisionBarcode.FORMAT_AZTEC)
        .build();

// Create an instance of FirebaseVisionBarcodeDetector
FirebaseVisionBarcodeDetector detector = FirebaseVision.getInstance().getVisionBarcodeDetector();

// Process the image
Task<List<FirebaseVisionBarcode>> result = detector.detectInImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionBarcode>>() {
            @Override
            public void onSuccess(List<FirebaseVisionBarcode> barcodes) {
                // Task completed successfully
                // ...
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
                // ...
            }
                });

The results will depend on the barcode type

Image labeling


Image labeling can be used on-device with ~400 labels or on-cloud with ~10000 labels 
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Create an instance of FirebaseVisionImageLabeler with on-device model
FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler();
// Or with cloud model
// FirebaseVisionCloudImageLabelerOptions options = new FirebaseVisionCloudImageLabelerOptions.Builder().setConfidenceThreshold(0.7f).build();
// FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance().getOnDeviceImageLabeler(options);

// Process the image
labeler.processImage(image)
    .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
      @Override
      public void onSuccess(List<FirebaseVisionImageLabel> labels) {
        // Task completed successfully
        // ...
      }
    })
    .addOnFailureListener(new OnFailureListener() {
      @Override
      public void onFailure(@NonNull Exception e) {
        // Task failed with an exception
        // ...
      }
    });

Return is just a list of FirebaseVisionImageLabel who contains : label and confidence score.
You can use AutoML Vision Edge to use your own model of classification.

Object detection & tracking


With this you can identify main object and track it (when streaming) 
// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Multiple object detection in static images
FirebaseVisionObjectDetectorOptions options =
        new FirebaseVisionObjectDetectorOptions.Builder()
                .setDetectorMode(FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE)
                .enableMultipleObjects()
                .enableClassification()  // Optional
                .build();

// Create an instance of FirebaseVisionObjectDetector
FirebaseVisionObjectDetector objectDetector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options);

// Process the image
objectDetector.processImage(image)
        .addOnSuccessListener(
                new OnSuccessListener<List<FirebaseVisionObject>>() {
                    @Override
                    public void onSuccess(List<FirebaseVisionObject> detectedObjects) {
                        // Task completed successfully
                        // ...
                    }
                })
        .addOnFailureListener(
                new OnFailureListener() {
                    @Override
                    public void onFailure(@NonNull Exception e) {
                        // Task failed with an exception
                        // ...
                    }
                });

Result is a list of FirebaseVisionObject who contains: tracking Id, bounds, category, confidence score.

Landmark recognition


You can recognize well-known landmarks in an image. This api can only be use on-cloud. 

// Create FirebaseVisionImage Object (here from an url)
FirebaseVisionImage image = FirebaseVisionImage.fromFilePath(context, uri);

// Set the options
FirebaseVisionCloudDetectorOptions options =
        new FirebaseVisionCloudDetectorOptions.Builder()
                .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
                .setMaxResults(15)
                .build();

// Create an instance of FirebaseVisionCloudLandmarkDetector
FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance().getVisionCloudLandmarkDetector(options);

// Process the image
Task<List<FirebaseVisionCloudLandmark>> result = detector.detectInImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionCloudLandmark>>() {
            @Override
            public void onSuccess(List<FirebaseVisionCloudLandmark> firebaseVisionCloudLandmarks) {
                // Task completed successfully
                // ...
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
                // ...
            }
        });

Result is a list of FirebaseVisionCloudLandmark who contains: Name, bounds, latitude, longitude, confidence score.

Language identification


This api doesn't use image but String. 
FirebaseLanguageIdentification languageIdentifier =
        FirebaseNaturalLanguage.getInstance().getLanguageIdentification();
languageIdentifier.identifyAllLanguages(text)
      .addOnSuccessListener(
          new OnSuccessListener<String>() {
            @Override
            public void onSuccess(List<IdentifiedLanguage> identifiedLanguages) {
              for (IdentifiedLanguage identifiedLanguage : identifiedLanguages) {
                String language = identifiedLanguage.getLanguageCode();
                float confidence = identifiedLanguage.getConfidence();
                Log.i(TAG, language + " (" + confidence + ")");
              }
            }
          })
      .addOnFailureListener(
          new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
              // Model couldn’t be loaded or other internal error.
              // ...
            }
          });

Result is a list of IdentifiedLanguage who contains: Language Code and confidence score.

Translation

Translation can be done with a on-device api but it can be use only for casual and simple translation over 59 languages (Japanese is supported).
Model is trained to translate to and from English; so if you choose to translate between non-English languages, English will be used as an intermediate translation, which can affect quality. 
// Create an English-Japanese translator:
FirebaseTranslatorOptions options =
        new FirebaseTranslatorOptions.Builder()
                .setSourceLanguage(FirebaseTranslateLanguage.EN)
                .setTargetLanguage(FirebaseTranslateLanguage.JP)
                .build();
final FirebaseTranslator englishJapaneseTranslator =
        FirebaseNaturalLanguage.getInstance().getTranslator(options);

final String text = "Merry Christmas";

// We need to download the model first
// Each model is around 30MB and are stored locally to be reused
FirebaseModelDownloadConditions conditions = new FirebaseModelDownloadConditions.Builder()
    .requireWifi()
    .build();
englishJapaneseTranslator.downloadModelIfNeeded(conditions)
      .addOnSuccessListener(
          new OnSuccessListener<Void>() {
            @Override
            public void onSuccess(Void v) {
              // Model downloaded successfully. We can start translation
              englishJapaneseTranslator.translate(text)
                  .addOnSuccessListener(
                      new OnSuccessListener<String>() {
                         @Override
                         public void onSuccess(@NonNull String translatedText) {
                            // Translation successful.
                            // translatedText <- "メリークリスマス"
                         }
                  })
                  .addOnFailureListener(
                      new OnFailureListener() {
                          @Override
                          public void onFailure(@NonNull Exception e) {
                             // Error during the translation.
                          }
                      });
            }
          })
      .addOnFailureListener(
          new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
              // Model couldn’t be downloaded or other internal error.
            }
          });

Smart reply generator (only in english)

The model work with the 10 most recent messages and provides a maximum of 3 suggested responses.

// Define a conversation history
// Local User speaks to Remote User
// Smart reply is about what Local User may answer.
List<FirebaseTextMessage> conversation = new ArrayList<>();
conversation.add(FirebaseTextMessage.createForRemoteUser("It's Christmas time", System.currentTimeMillis(),"userId1"));
conversation.add(FirebaseTextMessage.createForLocalUser("Kids are happy", System.currentTimeMillis()));
conversation.add(FirebaseTextMessage.createForRemoteUser("Will Santa Claus come tonight ?", System.currentTimeMillis(),"userId1"));

FirebaseSmartReply smartReply = FirebaseNaturalLanguage.getInstance().getSmartReply();
smartReply.suggestReplies(conversation)
        .addOnSuccessListener(new OnSuccessListener<SmartReplySuggestionResult>() {
            @Override
            public void onSuccess(SmartReplySuggestionResult result) {
                if (result.getStatus() == SmartReplySuggestionResult.STATUS_NOT_SUPPORTED_LANGUAGE) {
                    // The conversation's language isn't supported, so the
                    // the result doesn't contain any suggestions.
                } else if (result.getStatus() == SmartReplySuggestionResult.STATUS_SUCCESS) {
                    // Task completed successfully
                    for (SmartReplySuggestion suggestion : result.getSuggestions()) {
                        String replyText = suggestion.getText();
                    }
                }
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
            }
        });

As result a list of SmartReplySuggestion Object. Each one will contain only a text.
image.png

4
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
4
0