0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@oyahiroki(Hiroki Oya)

NLP4J: Using Different NLP Engines Through One Common Interface

0
Posted at

NLP4J: Using Different NLP Engines Through One Common Interface

Natural language processing libraries often perform similar tasks but expose completely different APIs.

For Japanese, we have NLP libraries and morphological analyzers such as:

  • MeCab
  • Kuromoji
  • Sudachi
  • GiNZA

For English, Apache OpenNLP is another well-known choice.

Each library has its own tokenizer, data structures, POS representation, configuration, and model format.

But from an application developer's point of view, we often want essentially the same information:

Natural language text
        ↓
Tokenization / morphological analysis
        ↓
surface form
lemma / base form
part of speech
character offsets

NLP4J started with a simple idea:

Different NLP engines should be usable through a common programming model.

Recently, I added Apache OpenNLP support to NLP4J.

As a result, Kuromoji and OpenNLP can now be used in almost the same way from an NLP4J application.

The Problem: NLP Libraries Have Different APIs

Suppose we want to analyze Japanese text with Kuromoji.

If we use Kuromoji directly, we need to know Kuromoji-specific classes and APIs such as its tokenizer and token objects.

Likewise, when using Apache OpenNLP directly, we need to work with classes such as:

TokenizerME
POSTaggerME
LemmatizerME

These libraries are good at what they do.

The problem appears in the application layer.

If application code directly depends on a particular NLP library, replacing or combining NLP engines becomes harder.

For example:

Application
    |
    +-- Kuromoji-specific code

Later, if we want to use Sudachi:

Application
    |
    +-- Sudachi-specific code

And if we want to process English with OpenNLP:

Application
    |
    +-- OpenNLP-specific code

The linguistic concepts are similar, but the application code becomes tied to each implementation.

The NLP4J Approach

NLP4J introduces a common layer between NLP engines and applications.

Conceptually:

                Kuromoji
                    |
                 Sudachi
                    |
                  MeCab
                    |
                  GiNZA
                    |
                 OpenNLP
                    |
                    v
          DocumentAnnotator
                    |
                    v
                 Document
                    |
                    v
                 Keyword

An NLP engine is wrapped by an NLP4J DocumentAnnotator.

The result is stored in a common NLP4J Document as Keyword objects.

This means the application mainly needs to understand:

Document
DocumentAnnotator
Keyword

rather than every NLP engine's internal API.

Example 1: Japanese with Kuromoji

Here is the updated Kuromoji example.

import nlp4j.Document;
import nlp4j.impl.DefaultDocument;
import nlp4j.krmj.annotator.KuromojiAnnotator;
import nlp4j.util.DocumentUtil;

public class HelloKuromojiAnnotator {

    public static void main(String[] args) throws Exception {

        Document doc =
                new DefaultDocument("犬が急いで走っている。");

        KuromojiAnnotator ann =
                new KuromojiAnnotator();

        {
            ann.setProperty("target", "text");
        }

        System.out.println(
                DocumentUtil.toJsonPrettyString(doc));

        System.out.println("---");

        ann.annotate(doc);

        System.out.println(
                DocumentUtil.toJsonPrettyString(doc));
    }
}

The important part is very small:

Document doc =
        new DefaultDocument("犬が急いで走っている。");

KuromojiAnnotator ann =
        new KuromojiAnnotator();

ann.annotate(doc);

The sample explicitly configures text as the target field before annotation.

The resulting document contains keywords such as:

{
  "facet": "動詞",
  "upos": "VERB",
  "lex": "走る",
  "str": "走っ",
  "begin": 5,
  "end": 7
}

In this example, Kuromoji's Japanese POS information is preserved in facet, while NLP4J also exposes the normalized Universal POS value VERB through upos.

Example 2: English with OpenNLP

Now compare that with the OpenNLP example.

import nlp4j.Document;
import nlp4j.impl.DefaultDocument;
import nlp4j.opennlp.OpenNLPAnnotator;
import nlp4j.util.DocumentUtil;

public class HelloOpenNLPAnnotator {

    public static void main(String[] args) throws Exception {

        Document doc =
                new DefaultDocument(
                        "Dogs are running quickly.");

        OpenNLPAnnotator ann =
                new OpenNLPAnnotator();

        System.out.println(
                DocumentUtil.toJsonPrettyString(doc));

        System.out.println("---");

        ann.annotate(doc);

        System.out.println(
                DocumentUtil.toJsonPrettyString(doc));
    }
}

The application-side workflow is essentially the same:

Document doc =
        new DefaultDocument(
                "Dogs are running quickly.");

OpenNLPAnnotator ann =
        new OpenNLPAnnotator();

ann.annotate(doc);

The result contains the same kinds of NLP4J properties:

{
  "facet": "VERB",
  "upos": "VERB",
  "lex": "run",
  "str": "running",
  "begin": 9,
  "end": 16
}

Kuromoji and OpenNLP: The Same Programming Model

This is the part I consider important.

Compare the two programs.

Kuromoji

Document doc =
        new DefaultDocument(
                "犬が急いで走っている。");

KuromojiAnnotator ann =
        new KuromojiAnnotator();

ann.annotate(doc);

OpenNLP

Document doc =
        new DefaultDocument(
                "Dogs are running quickly.");

OpenNLPAnnotator ann =
        new OpenNLPAnnotator();

ann.annotate(doc);

The NLP implementation is different.

The language is different.

The models are different.

The internal tokenization algorithms are different.

But from the NLP4J application point of view, the workflow is almost identical:

Create Document
      ↓
Create Annotator
      ↓
annotate(doc)
      ↓
Read Keywords

That is one of the original design goals of NLP4J.

A Common Keyword Representation

NLP4J converts the analysis result into a common keyword representation.

The core properties include:

Property Meaning
str Surface form
lex Lexical / canonical form
upos Universal POS
facet NLP-engine-specific classification
begin Start character offset
end End character offset

For example, Kuromoji produces:

{
  "facet": "動詞",
  "upos": "VERB",
  "lex": "走る",
  "str": "走っ",
  "begin": 5,
  "end": 7
}

while OpenNLP produces:

{
  "facet": "VERB",
  "upos": "VERB",
  "lex": "run",
  "str": "running",
  "begin": 9,
  "end": 16
}

The underlying NLP engines are very different, but both results can be handled as NLP4J keywords.

facet and upos Have Different Roles

One important design decision in NLP4J is that facet and upos are not necessarily the same thing.

facet

facet can preserve an NLP-engine-specific classification.

For example, the Kuromoji sample produces values such as:

名詞
助詞
動詞
記号

The Japanese sample actually records values such as 動詞 in facet.

upos

upos is intended to contain a Universal POS value.

For example:

NOUN
VERB
ADP
AUX
ADV
PUNCT

This gives applications a common POS axis even when the original NLP systems use different POS schemes.

Conceptually:

Kuromoji
    facet = 動詞
    upos  = VERB

OpenNLP
    facet = VERB
    upos  = VERB

In the current OpenNLP model, the engine already produces Universal POS-style values, so facet and upos happen to be identical in many cases. The sample shows NOUN, AUX, VERB, ADV, and PUNCT in both fields.

But they remain conceptually separate fields.

Preserve the Original Information, Normalize What Is Useful

I think this is an important principle for an NLP abstraction layer.

A common representation should not require us to throw away useful information from the original NLP engine.

Different analyzers have genuinely different linguistic models.

For example, a Japanese morphological analyzer may have detailed POS categories that cannot be represented completely by a single Universal POS tag.

Therefore NLP4J can preserve the engine-specific value:

facet

while also exposing a normalized value:

upos

The idea is:

                 NLP Engine
                     |
          +----------+----------+
          |                     |
          v                     v
        facet                  upos
          |                     |
engine-specific          common POS axis
information

Normalization and preservation do not have to be mutually exclusive.

NLP4J Is Not Another Morphological Analyzer

NLP4J is not intended to replace Kuromoji, MeCab, Sudachi, GiNZA, or OpenNLP.

Those projects implement the actual NLP algorithms, dictionaries, tokenizers, statistical models, and linguistic analysis.

NLP4J has a different role.

It provides a layer above these NLP engines.

                    Application
                        |
                        v
                      NLP4J
              Document / Keyword
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
      Kuromoji       OpenNLP        Other
                                     NLP

The objective is to make applications less dependent on one specific NLP implementation.

Switching NLP Engines Becomes Easier

Imagine an application that performs some processing after morphological analysis:

ann.annotate(doc);

for (Keyword kwd : doc.getKeywords()) {
    // text mining
    // statistics
    // indexing
    // aggregation
}

If the downstream application works with NLP4J Keyword objects, it does not need to know whether those keywords were produced by:

Kuromoji
Sudachi
OpenNLP
...

This separation becomes useful when experimenting with different NLP engines.

Instead of:

NLP Engine
    +
Application logic

being tightly coupled, we can have:

NLP Engine
    |
    v
NLP4J common model
    |
    v
Application logic

From Japanese NLP to English NLP

NLP4J originally grew from the need to handle multiple Japanese NLP implementations through a common framework.

The addition of OpenNLP demonstrates that the same abstraction is not inherently limited to Japanese.

Japanese
   |
   +-- Kuromoji ----+
   +-- MeCab -------+
   +-- Sudachi -----+
   +-- GiNZA -------+
                    |
                    v
                  NLP4J
                    ^
                    |
   +-- OpenNLP -----+
   |
English

This changes the way I think about the abstraction.

The fundamental concept is not:

"A common wrapper for Japanese morphological analyzers."

It is more generally:

A common representation for linguistic annotations produced by different NLP engines.

A Small API Is Intentional

One characteristic of NLP4J is that the application-facing API should remain small.

For OpenNLP:

Document doc =
        new DefaultDocument(
                "Dogs are running quickly.");

OpenNLPAnnotator ann =
        new OpenNLPAnnotator();

ann.annotate(doc);

For Kuromoji:

Document doc =
        new DefaultDocument(
                "犬が急いで走っている。");

KuromojiAnnotator ann =
        new KuromojiAnnotator();

ann.annotate(doc);

The application should not need to understand all the internal objects used by each NLP engine unless it explicitly needs engine-specific functionality.

This is intentional.

The abstraction presented to the application is:

Document
    +
DocumentAnnotator
    ↓
Keyword

Why This Matters Beyond Morphological Analysis

A common NLP representation becomes more valuable when the analysis results are used by other systems.

For example:

Natural Language Text
        |
        v
   NLP Annotator
        |
        v
NLP4J Document / Keyword
        |
        +----------------+
        |                |
        v                v
      Search         Aggregation
        |                |
        +--------+-------+
                 |
                 v
            Text Mining

A search or text-mining component does not necessarily need to understand Kuromoji, Sudachi, or OpenNLP individually.

It can consume the common NLP4J representation.

The NLP engine then becomes one interchangeable component in a larger processing pipeline.

Conclusion

There are already many excellent natural language processing libraries.

NLP4J is not trying to create one more tokenizer or morphological analyzer.

Instead, it addresses another problem:

How can applications use different NLP engines through a consistent programming model?

The updated Kuromoji and OpenNLP examples illustrate this idea very directly.

Kuromoji:

Document doc =
        new DefaultDocument(
                "犬が急いで走っている。");

KuromojiAnnotator ann =
        new KuromojiAnnotator();

ann.annotate(doc);

OpenNLP:

Document doc =
        new DefaultDocument(
                "Dogs are running quickly.");

OpenNLPAnnotator ann =
        new OpenNLPAnnotator();

ann.annotate(doc);

Different languages.

Different NLP engines.

Different internal implementations.

But essentially the same NLP4J programming model.

Different NLP Engines
         |
         v
       NLP4J
         |
         v
Common Document / Keyword Model
         |
         v
Reusable Application Logic

That is the idea behind NLP4J:

Use the strengths of different NLP engines while giving applications one common way to work with their results.

With nlp4j-opennlp, this approach now extends naturally from Japanese morphological analysis to English NLP as well.

:eyes:

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?