0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@oyahiroki(Hiroki Oya)

NLP4J Local Search 0.5.1: Exploring Lucene Data with the New `view()` API

0
Posted at

NLP4J Local Search 0.5.1: Exploring Lucene Data with the New view() API

nlp4j-local-search 0.5.1 introduces a new view() API for inspecting and analyzing the contents of a local Lucene index from Python.

Until now, the main interaction with SearchEngine was naturally centered around search:

engine.search(...)

That works well when you already know what you want to search for.

However, text analysis often starts with a different question:

What is actually inside this dataset?

For example:

  • What values appear most frequently?
  • What categories exist?
  • Which parts are especially common among Nissan records?
  • Which keywords are disproportionately frequent in a filtered subset?
  • Can I quickly inspect an index before writing a more specific query?

Version 0.5.1 adds engine.view() for exactly this kind of exploratory analysis.

The interesting part is that view() is not just a convenience wrapper around a count aggregation. When a Lucene query is supplied, it can also calculate relative rates, making it possible to discover values that characterize a particular subset of documents.

What is NLP4J Local Search?

nlp4j-local-search provides a Python interface to a Java-based local search engine implemented directly on top of Apache Lucene.

The idea is simple:

Python
  ↓
nlp4j-local-search
  ↓
Java LocalSearch
  ↓
Apache Lucene

You can therefore use Lucene-based full-text search and text analysis from Python without running a separate Solr or OpenSearch server.

This is particularly useful for:

  • local text analysis
  • notebooks
  • prototyping
  • small and medium-sized document collections
  • offline analysis
  • applications where operating a search server would be unnecessary

The Java implementation has recently gained more text-analysis and analytics functionality, and the Python API is being expanded to expose those capabilities naturally.

view() is one of those additions.

Installation

Install version 0.5.1 with:

pip install nlp4j-local-search==0.5.1

Then import:

from nlp4j_local_search import SearchEngine, ViewResult

Example Dataset

Consider the following very small dataset.

DOCUMENTS = [
    {
        "id": "1",
        "body": "Nissan reported a broken door mirror.",
        "maker": "Nissan",
        "category": "body",
        "part": "door mirror",
    },
    {
        "id": "2",
        "body": "Nissan reported a door mirror failure.",
        "maker": "Nissan",
        "category": "body",
        "part": "door mirror",
    },
    {
        "id": "3",
        "body": "Nissan reported a battery problem.",
        "maker": "Nissan",
        "category": "electrical",
        "part": "battery",
    },
    {
        "id": "4",
        "body": "Toyota reported a brake problem.",
        "maker": "Toyota",
        "category": "brake",
        "part": "brake",
    },
    {
        "id": "5",
        "body": "Toyota reported a battery problem.",
        "maker": "Toyota",
        "category": "electrical",
        "part": "battery",
    },
    {
        "id": "6",
        "body": "Honda reported a brake problem.",
        "maker": "Honda",
        "category": "brake",
        "part": "brake",
    },
]

We can add them to an in-memory search engine:

with SearchEngine("en", auto_analyze=False) as engine:

    for doc in DOCUMENTS:
        engine.add_json(doc)

    engine.commit()

In this example, auto_analyze=False is used because we want to focus on the explicitly supplied structured fields such as maker, category, and part.

1. Inspecting Available Fields

Before analyzing an unfamiliar index, it is useful simply to see what fields exist.

print(engine.fields())

You can also inspect fields that support aggregation:

print(engine.aggregatable_fields())

This is useful for interactive analysis because you do not necessarily need to remember the entire schema before starting exploration.

Conceptually:

engine.fields()

answers:

What fields are stored in this index?

while:

engine.aggregatable_fields()

answers:

Which fields can I summarize by value?

2. Looking at the Entire Index with view()

Calling view() without a field gives a quick overview of all aggregatable fields.

print(engine.view())

Example output:

View: aggregatable fields
Format: field | value (document count)

maker    | Nissan (3), Toyota (2), Honda (1)
category | brake (2), body (2), electrical (2)
part     | brake (2), door mirror (2), battery (2)

This is intended as an inspection API.

Instead of manually executing several aggregations, you can quickly understand the shape of the dataset.

In a Jupyter Notebook, you can also simply write:

engine.view()

as the last expression in a cell.

3. Viewing One Field

If you are interested in only one field:

engine.view("category")

The buckets are shown by document count.

For another field:

engine.view("maker")

we would get values such as:

Nissan  3
Toyota  2
Honda   1

So far, this is similar to a conventional terms aggregation.

But view() becomes more interesting when we add a query.

4. Combining view() with a Lucene Query

Suppose we want to inspect part, but only among Nissan documents.

engine.view("part", "maker:Nissan")

This switches view() into relative-rate mode.

Example output:

View: part
Lucene query: maker:Nissan
Matched documents: 3 / 6
Values are ordered by relative rate.

Rank  Value                Count  All Count  Relative Rate
----  -------------------- -----  ---------  -------------
   1  door mirror              2          2          2.00x
   2  battery                  1          2          1.00x

This gives us substantially more information than a simple filtered count.

What Does Relative Rate Mean?

There are six documents in total.

Three are Nissan documents.

Among the Nissan documents:

door mirror = 2 / 3
battery     = 1 / 3

Across the complete dataset:

door mirror = 2 / 6
battery     = 2 / 6

The relative rate is approximately:

rate in selected documents
--------------------------
rate in all documents

For door mirror:

(2 / 3)
-------
(2 / 6)

= 2.0

So:

door mirror: 2.00x

means that "door mirror" occurs at twice the expected rate in the Nissan subset compared with the complete document collection.

For battery:

(1 / 3)
-------
(2 / 6)

= 1.0

Therefore:

battery: 1.00x

It is not particularly characteristic of Nissan in this dataset.

This is why relative rate can be more useful than raw counts for exploratory text analysis.

Counts Answer “How Many?”

For example:

door mirror = 2
battery     = 1

answers:

How many Nissan documents contain each value?

That is useful, but it does not tell us whether a value is unusual.

Relative Rate Answers “How Characteristic?”

door mirror = 2.00x
battery     = 1.00x

instead answers:

Which values appear disproportionately often in Nissan documents?

This is much closer to the kind of question commonly asked during text mining.

5. Viewing All Fields for a Subset

You do not have to specify a particular field.

engine.view(
    lucene_query="maker:Nissan",
    size=5,
)

This applies the Lucene query and inspects multiple aggregatable fields using relative-rate mode.

That makes the API useful for questions such as:

Show me what is distinctive about the documents matching this query.

For exploratory analysis, this is convenient because you can first define a document subset and then inspect several dimensions of that subset.

6. Sorting Results

ViewResult also supports sort_by().

For example, to sort maker values by ascending document count:

result = (
    engine.view("maker", size=10)
    .sort_by("count", descending=False)
)

print(result)

For relative-rate analysis:

result = (
    engine.view("part", "maker:Nissan")
    .sort_by("relative_rate")
)

print(result)

sort_by() does not mutate the original ViewResult.

It returns a new result object.

This makes operations predictable and easy to chain.

7. Filtering Results

You can also filter buckets.

For example:

result = (
    engine.view("part", "maker:Nissan")
    .filter(min_relative_rate=1.5)
)

print(result)

The result contains only:

door mirror    2.00x

because battery has a relative rate of only 1.00x.

You can also filter by document count:

result = (
    engine.view("maker", size=10)
    .filter(min_count=2)
)

print(result)

In this example, the result contains:

Nissan
Toyota

while Honda is removed because it appears in only one document.

8. Chaining Operations

Because filter() and sort_by() return new ViewResult objects, operations can be chained naturally.

result = (
    engine.view("part", "maker:Nissan")
    .filter(min_count=1)
    .sort_by("relative_rate")
)

print(result)

The style is intentionally similar to exploratory data-processing APIs:

select documents
    ↓
view values
    ↓
filter
    ↓
sort

without requiring users to manually manipulate the underlying Java objects.

9. size vs. candidate_size

Relative-rate calculation introduces an important distinction between two parameters:

engine.view(
    "part",
    "maker:Nissan",
    size=10,
    candidate_size=1000,
)

candidate_size

candidate_size controls how many candidate values are retrieved from the Java analytics layer for relative-rate calculation.

For example:

candidate_size=1000

means that up to 1,000 candidate buckets can participate in the analysis.

A larger candidate set can be useful when working with fields containing many values.

The default is:

1000

size

size controls how many buckets remain in the final ViewResult after processing.

For example:

size=10

means:

Return the top 10 values to Python.

Typical usage is therefore:

candidate_size = 1000
        ↓
analyze a reasonably broad candidate set
        ↓
filter / sort
        ↓
size = 10
        ↓
show the most useful 10 values

This distinction becomes particularly useful with high-cardinality fields.

10. ViewResult Is Also a Data API

view() is designed for humans to read, but its result is not just formatted text.

It returns a ViewResult object.

result = engine.view("part", "maker:Nissan")

assert isinstance(result, ViewResult)

You can access its fields programmatically:

vf = result.fields[0]

print(vf.field)
print(vf.count)
print(vf.total_count)

And individual buckets:

for bucket in vf.buckets:
    print(
        bucket.key,
        bucket.count,
        bucket.all_count,
        bucket.relative_rate,
    )

For the example dataset:

field: part
matched docs: 3 / 6

key='door mirror'
count=2
all_count=2
relative_rate=2.00

key='battery'
count=1
all_count=2
relative_rate=1.00

The important properties are:

bucket.key
bucket.count
bucket.all_count
bucket.relative_rate

This means the same API can be used both interactively and as part of an application.

For example, the result could later be converted into:

  • a Pandas DataFrame
  • a chart
  • a report
  • a dashboard
  • JSON
  • another statistical analysis pipeline

search() and view() Have Different Roles

The distinction I am trying to make in the Python API is:

engine.search(...)

for finding documents, and:

engine.view(...)

for understanding a document collection.

For example:

engine.search(...)

asks:

Which documents match this condition?

while:

engine.view("part", "maker:Nissan")

asks:

Among the documents matching this condition, which values are characteristic?

These two operations complement each other.

A typical workflow might eventually look like:

add documents
     ↓
search
     ↓
inspect a subset with view()
     ↓
discover interesting values
     ↓
refine the query
     ↓
search again

This is particularly useful for text mining, where you often do not know the right query before looking at the data.

Why Put This in the Search Engine?

Of course, similar calculations can be implemented with Pandas.

For a small table, that may be the easiest solution.

However, NLP4J Local Search already maintains:

  • a Lucene index
  • typed fields
  • keyword fields
  • searchable text fields
  • document filters
  • aggregatable values
  • language-analysis results

Performing this analysis directly against the same indexed representation has several advantages.

There is no need to create a separate analytical copy of the data just to answer simple exploratory questions.

For example:

engine.view(
    "part",
    "maker:Nissan AND category:body",
)

can use the same Lucene query syntax as the search layer.

This allows search and analytics to operate over a common document model.

From Search to Text Analysis

One of the broader goals of NLP4J Local Search is to make Lucene useful not only as a search library but also as a lightweight foundation for text analysis.

Lucene itself provides excellent indexing and retrieval primitives.

NLP4J adds another layer around them:

Raw documents
      ↓
Language analysis
      ↓
Lucene index
      ↓
Search
      ↓
Aggregation
      ↓
Relative-rate analysis
      ↓
Inspection with view()

For example, when automatic NLP analysis is enabled, extracted linguistic fields can also become candidates for this kind of exploration.

This starts to make queries such as these possible:

engine.view("word.noun")

or conceptually:

engine.view(
    "word.noun",
    "some_field:some_condition",
)

Instead of only asking which documents contain a word, we can start asking which words are characteristic of a particular group of documents.

That is a small API change, but it changes how the search engine can be used.

Example Use Case: Vehicle Complaint Analysis

The tiny dataset in this article is intentionally simple, but imagine replacing it with thousands or millions of vehicle complaint records.

Fields might include:

maker
model
model_year
component
country
date

while NLP processing could extract fields such as:

word.noun
word.verb
word.adj

You might begin with:

engine.view()

to understand the dataset.

Then:

engine.view("component")

to inspect common components.

And then:

engine.view(
    "word.noun",
    "maker:Nissan",
)

to discover nouns disproportionately associated with Nissan complaints.

Or:

engine.view(
    "component",
    "model_year_i:[2024 TO 2026]",
)

to inspect recent complaints.

This combination of:

Lucene query
+
aggregation
+
relative rate

is where I think view() becomes more interesting than a simple “show top values” function.

Complete Example

Here is a compact version of the full example.

from nlp4j_local_search import SearchEngine

DOCUMENTS = [
    {
        "id": "1",
        "body": "Nissan reported a broken door mirror.",
        "maker": "Nissan",
        "category": "body",
        "part": "door mirror",
    },
    {
        "id": "2",
        "body": "Nissan reported a door mirror failure.",
        "maker": "Nissan",
        "category": "body",
        "part": "door mirror",
    },
    {
        "id": "3",
        "body": "Nissan reported a battery problem.",
        "maker": "Nissan",
        "category": "electrical",
        "part": "battery",
    },
    {
        "id": "4",
        "body": "Toyota reported a brake problem.",
        "maker": "Toyota",
        "category": "brake",
        "part": "brake",
    },
    {
        "id": "5",
        "body": "Toyota reported a battery problem.",
        "maker": "Toyota",
        "category": "electrical",
        "part": "battery",
    },
    {
        "id": "6",
        "body": "Honda reported a brake problem.",
        "maker": "Honda",
        "category": "brake",
        "part": "brake",
    },
]

with SearchEngine("en", auto_analyze=False) as engine:

    for doc in DOCUMENTS:
        engine.add_json(doc)

    engine.commit()

    # Inspect the index
    print(engine.view())

    # Count values
    print(engine.view("maker"))

    # Analyze a subset
    print(engine.view("part", "maker:Nissan"))

    # Keep only characteristic values
    result = (
        engine.view("part", "maker:Nissan")
        .filter(min_relative_rate=1.5)
        .sort_by("relative_rate")
    )

    print(result)

    # Use the result programmatically
    for field in result.fields:
        for bucket in field.buckets:
            print(
                field.field,
                bucket.key,
                bucket.count,
                bucket.relative_rate,
            )

Summary

NLP4J Local Search 0.5.1 adds view() as an exploratory inspection API on top of the existing Lucene-based search engine.

The basic forms are:

# Overview of aggregatable fields
engine.view()

# Values of one field
engine.view("category")

# Analyze one field within a subset
engine.view("part", "maker:Nissan")

# Analyze all fields within a subset
engine.view(lucene_query="maker:Nissan")

Results can then be processed with:

result.filter(...)
result.sort_by(...)

and accessed programmatically through:

result.fields
field.buckets
bucket.key
bucket.count
bucket.all_count
bucket.relative_rate

The most important idea behind the API is the distinction between:

search() → find documents
view()   → understand documents

Search tells us what matches.

view() helps us discover what is characteristic of what matches.

For a Lucene-based library aimed at both search and text mining, I think these two perspectives fit together naturally.

In future versions, I would like to continue strengthening this connection between Lucene search, NLP-generated fields, statistical analysis, and Python-based exploratory workflows.

Links

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?