Try Apache Lucene from Python: Exploring 109,734 NHTSA Complaints with nlp4j-local-search 0.7.2.0

nlp4j-local-search is a lightweight Python library that lets you use Apache Lucene locally without running Elasticsearch, OpenSearch, Solr, or Docker.

In this article, we will install nlp4j-local-search 0.7.2.0 and explore a real-world dataset containing more than 100,000 vehicle complaints from NHTSA (National Highway Traffic Safety Administration).

We will try the following workflow:

1. Install nlp4j-local-search
2. Download a sample NHTSA dataset
3. Start the interactive CLI
4. Load more than 100,000 JSONL documents
5. Inspect available fields
6. Count values with view()
7. Narrow the dataset with Lucene queries
8. Find values that are unusually frequent within a subset
9. Retrieve the actual complaint documents with search()

One of the interesting features is that we can move naturally from:

What models are common?

to:

What components are unusually common for Nissan Sentra?

and finally to:

Show me the actual complaints about those components.

without setting up a search server.

Environment

This article uses:

nlp4j-local-search 0.7.2.0

Project page:

The package requires Python 3.9 or later and a Java runtime environment.

Internally, nlp4j-local-search uses Java and Apache Lucene, but normal users can operate it from Python or from its interactive CLI without writing Java code.

1. Install nlp4j-local-search

Install version 0.7.2.0 from PyPI:

pip install nlp4j-local-search==0.7.2.0

For example:

Collecting nlp4j-local-search==0.7.2.0
  Downloading nlp4j_local_search-0.7.2.0-py3-none-any.whl.metadata (23 kB)
Collecting jpype1>=1.4.0 (from nlp4j-local-search==0.7.2.0)
  Downloading jpype1-1.7.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (4.3 kB)
Collecting packaging (from jpype1>=1.4.0->nlp4j-local-search==0.7.2.0)
  Downloading packaging-26.3-py3-none-any.whl.metadata (3.5 kB)
Downloading nlp4j_local_search-0.7.2.0-py3-none-any.whl (38.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.3/38.3 MB 10.4 MB/s eta 0:00:00
Downloading jpype1-1.7.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (438 kB)
Downloading packaging-26.3-py3-none-any.whl (129 kB)
Installing collected packages: packaging, jpype1, nlp4j-local-search
Successfully installed jpype1-1.7.1 nlp4j-local-search-0.7.2.0 packaging-26.3

The wheel is relatively large because the package includes the components required to provide the Lucene-based local search functionality.

2. Download the sample NHTSA complaint data

For this example, I prepared a compressed JSONL dataset based on NHTSA complaint data.

Download it with curl:

curl https://nlp4j-2.sakura.ne.jp/data/nhtsa/nhtsa_complaints_2025_2.jsonl.gz \
  --output data.gz

The compressed file is about 20 MB.

100 20.4M  100 20.4M

Because the file is already compressed with gzip, we do not need to extract it before loading it.

nlp4j-local-search can load the .jsonl.gz file directly.

3. Start the interactive CLI

Because the complaint text is English, start the CLI with:

nlp4j-local-search --lang en

You should see something similar to:

nlp4j-local-search
Language: en
Auto analyze: False
Type 'help' or '?' for help.

>>

We are now inside the interactive shell.

4. Load the NHTSA data

Load the downloaded file:

>> load("data.gz")

Example result:

Loading documents...
Loading: 109,734 documents | 5,794 docs/sec | 18.9s
Loaded 109,734 documents in 18.94 seconds (5,794 docs/sec).

In this example, 109,734 documents were loaded in about 19 seconds.

The exact speed will naturally depend on the machine.

This gives us a reasonably large dataset for experimenting with local full-text search and analytics without running an external search server.

5. Inspect the available fields

Before searching, it is useful to see what fields exist.

Run:

>> fields

Example:

id
text
data
odi_no_s
manufacturer_s
make_s
model_s
model_year_i
crash_s
fire_s
injured_i
deaths_i
component_s
state_s
incident_state_s
date_added
date
complaint_type_s
product_type_s
police_report_s
medical_attention_s
vehicles_towed_s
fuel_type_s
drive_train_s
transmission_s
vehicle_speed_i
incident_date_dt
miles_i

The dataset contains not only the complaint text but also structured information such as:

• manufacturer
• vehicle make
• model
• model year
• component
• state
• injuries
• deaths
• fuel type
• transmission
• mileage
• incident date

This means that we can combine full-text search with structured field queries.

6. Find the most common vehicle models

Let's first inspect the model_s field.

>> view("model_s")

Example output:

View: model_s
Values are ordered by document count.

Rank  Value                   Count  Full Value
----  -------------------- --------  ----------------------------------------
   1  F-150                    4005
   2  ESCAPE                   3225
   3  EXPLORER                 2940
   4  1500                     2875
   5  SILVERADO 1500           2564
   6  WRANGLER                 2475
   7  SIERRA 1500              2212
   8  ACCORD                   1909
   9  CR-V                     1763
  10  GRAND CHEROKEE           1731

With a single command, we can see the most frequent values in a field.

For example, F-150 appears in 4,005 complaint documents.

This is the simplest use of view():

view(field)

When no query is specified, values are ranked by document count.

7. Narrow the analysis to Nissan Sentra

Counting values globally is useful, but a more interesting question is:

What kinds of components are disproportionately represented in complaints about the Nissan Sentra?

We can pass a Lucene query as the second argument of view():

>> view("component_s","model_s:SENTRA")

The result is:

View: component_s
Lucene query: model_s:SENTRA
Matched documents: 391 / 109,734
Values are ordered by relative rate.

Rank  Value                   Count  All Count  Relative Rate  Full Value
----  -------------------- -------- ---------- --------------  ----------------------------------------
   1  STEERING:COLUMN             1          5         56.13x
   2  SERVICE BRAKES, H...        1         12         23.39x  SERVICE BRAKES, HYDRAULIC:PEDALS AND LINKAGES
   3  EXTERIOR LIGHTING...        1         13         21.59x  EXTERIOR LIGHTING:HEADLIGHTS:BULBS
   4  TRACTION CONTROL ...        1         47          5.97x  TRACTION CONTROL SYSTEM
   5  VEHICLE SPEED CON...        1         54          5.20x  VEHICLE SPEED CONTROL:ACCELERATOR PEDAL
   6  POWER TRAIN:AUTOM...        2        119          4.72x  POWER TRAIN:AUTOMATIC TRANSMISSION
   7  WHEELS                      9        748          3.38x
   8  VISIBILITY                  2        169          3.32x
   9  VEHICLE SPEED CON...       19       2469          2.16x  VEHICLE SPEED CONTROL
  10  FORWARD COLLISION...       16       2334          1.92x  FORWARD COLLISION AVOIDANCE: AUTOMATIC EMERGENCY BRAKING

This is where view() becomes more than a simple aggregation command.

The query:

model_s:SENTRA

matches 391 documents out of 109,734.

Instead of simply sorting component_s values by their count among those 391 documents, view() sorts them by relative rate.

8. What does "Relative Rate" mean?

Consider this row:

POWER TRAIN:AUTOMATIC TRANSMISSION

Count:          2
All Count:    119
Relative Rate: 4.72x

There are:

391

Sentra complaint documents, and:

109,734

documents in the complete dataset.

The component appears in:

2 / 391

of Sentra complaints.

Across the complete dataset, it appears in:

119 / 109734

of complaints.

The relative rate can therefore be interpreted approximately as:

(2 / 391) / (119 / 109734)

which is:

4.72

So:

Relative Rate = 4.72x

means that this component appears about 4.7 times as frequently among Sentra complaints as we would expect from its overall frequency in the dataset.

This type of analysis is often more informative than simply looking at the largest counts.

For example, a very common component may have many complaints for every model. Its raw count may therefore be high without being particularly distinctive for the Sentra.

Relative rate helps us discover values that are disproportionately associated with the subset we are investigating.

9. Why relative rate is useful

Suppose we only sorted Sentra components by complaint count.

We might mostly find generally common categories.

With relative rate, the question changes from:

Which components have many complaints?

to:

Which components occur unusually often in this particular group compared with the dataset as a whole?

This is useful for exploratory analysis of:

• product complaints
• customer feedback
• support tickets
• defect reports
• incident reports
• survey responses
• log data

and many other datasets containing both text and structured fields.

It is similar to asking which terms or categories are characteristic of a selected population.

10. Narrow the query further

From the previous result, we found:

POWER TRAIN:AUTOMATIC TRANSMISSION

We can now narrow the Lucene query to that exact component.

Because the value contains spaces and a colon, putting the value inside quotes is convenient:

>> view("component_s",'model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"')

Result:

View: component_s
Lucene query: model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"
Matched documents: 2 / 109,734
Values are ordered by relative rate.

Rank  Value                   Count  All Count  Relative Rate  Full Value
----  -------------------- -------- ---------- --------------  ----------------------------------------
   1  POWER TRAIN:AUTOM...        2        119        922.13x  POWER TRAIN:AUTOMATIC TRANSMISSION

Now only two documents match.

Notice the Lucene query:

model_s:SENTRA
AND
component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"

This demonstrates how structured conditions can be combined using normal Lucene query syntax.

The very large 922.13x value in this particular query is expected because we have explicitly restricted the result set to documents containing that component.

For exploratory analysis, the previous query:

view("component_s","model_s:SENTRA")

is generally the more interesting one because it allows the component values themselves to be discovered.

The second query is useful as the next step: verifying and narrowing down what we found.

11. Retrieve the actual complaints

Aggregation tells us that something may be interesting.

The next question is:

What did people actually report?

Use the same Lucene query with search():

>> search('model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"')

Two documents are returned.

For example:

[2089598] score=5.6629
See attached document.

Another complaint describes repeated CVT transmission failures and reports that the vehicle lost acceleration while driving:

[2091762] score=5.6629
My 2016 Nissan Sentra SV has had two complete CVT transmission failures.
...
The car lost acceleration and came to a complete halt while driving...

This illustrates an important workflow:

view()
   ↓
discover an interesting category
   ↓
add it to the Lucene query
   ↓
search()
   ↓
read the actual documents

We are not limited to aggregated statistics.

We can move immediately from the statistical signal back to the original text.

12. Try another component

The relative-rate analysis also showed:

WHEELS

with:

Count          9
All Count    748
Relative Rate 3.38x

Let's inspect those documents:

>> search('model_s:SENTRA AND component_s:"WHEELS"')

The results contain various reports involving tires, wheels, drive shafts, traction, and related vehicle behavior.

This demonstrates how view() can act as an exploratory interface before full-text document inspection.

Instead of guessing every search condition in advance, we can let the data suggest what to investigate next.

13. The full workflow

The complete session can be summarized as follows.

First, install the package:

pip install nlp4j-local-search==0.7.2.0

Download the sample data:

curl https://nlp4j-2.sakura.ne.jp/data/nhtsa/nhtsa_complaints_2025_2.jsonl.gz \
  --output data.gz

Start the CLI:

nlp4j-local-search --lang en

Load the data:

load("data.gz")

Inspect fields:

fields

Check the most common models:

view("model_s")

Analyze components for Sentra:

view("component_s","model_s:SENTRA")

Narrow the condition:

view("component_s",'model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"')

Finally, retrieve the original complaint text:

search('model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"')

When finished:

exit

14. Why I find this workflow useful

Apache Lucene is an extremely powerful search library, but developers often encounter it indirectly through systems such as:

• Elasticsearch
• OpenSearch
• Apache Solr

Those systems are excellent when building a search service, but there are situations where starting and maintaining an external server is unnecessary.

For example, I may simply want to:

JSONL
  ↓
Lucene index
  ↓
inspect fields
  ↓
aggregate
  ↓
filter
  ↓
search

on my local machine.

That is the use case I am targeting with nlp4j-local-search.

The index runs locally, so it is convenient for experimentation, data exploration, NLP prototyping, search experiments, and RAG-related work.

15. Search and analytics in the same workflow

One aspect I particularly want to emphasize is the combination of search and inspection/analytics.

Traditional search usually starts from a query:

query
  ↓
documents

But when investigating an unfamiliar dataset, we may not yet know what query we should write.

In that situation, the workflow can instead be:

dataset
  ↓
view(field)
  ↓
discover important values
  ↓
view(field, lucene_query)
  ↓
discover characteristic values
  ↓
search(lucene_query)
  ↓
inspect original documents

The NHTSA example makes this easy to see.

We started without knowing which Sentra components might be interesting.

view("component_s","model_s:SENTRA")

showed us values that were relatively frequent.

We then converted one of those observations directly into a search query:

search('model_s:SENTRA AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"')

This tight connection between exploration, aggregation, and document retrieval is one of the ideas behind nlp4j-local-search.

Conclusion

With nlp4j-local-search, we loaded:

109,734 documents

and explored them locally using Apache Lucene.

No Elasticsearch server was required.

No OpenSearch server was required.

No Solr server was required.

No Docker container was required.

The basic workflow was only:

pip install
    ↓
download JSONL.gz
    ↓
load()
    ↓
fields
    ↓
view()
    ↓
Lucene query
    ↓
search()

For small and medium-sized datasets, NLP experiments, search prototyping, and exploratory text analysis, being able to perform this workflow directly from Python or a local CLI can be very convenient.

In particular, the relative-rate mode of view() provides a simple way to move beyond raw counts and ask:

What is unusually frequent in this subset of documents?

And once we find something interesting, we can immediately use the same Lucene query to inspect the original documents.

That makes nlp4j-local-search useful not only as a local search engine, but also as a lightweight tool for interactive data exploration.

Links

PyPI:

Sample NHTSA complaint dataset used in this article:

Suggested Qiita tags

Python Lucene NLP Search NHTSA