Exploring NHTSA Complaint Data with nlp4j-local-search 0.7.3.0: fields → view → search
nlp4j-local-search is a lightweight local search and analysis tool built on Apache Lucene.
One workflow I wanted to make easy is:
Understand the dataset
↓
Find interesting values
↓
Narrow the data step by step
↓
Inspect characteristic values
↓
Search the actual documents
In version 0.7.3.0, the CLI makes this workflow much more practical.
In this article, I will use a dataset containing 109,734 NHTSA vehicle complaints and explore it interactively.
The interesting part is that we do not need to decide the final search query at the beginning.
Instead, we can discover the query while looking at the data.
1. Start the CLI
For development from a local checkout:
python3 -m pip install -e .
Then start the CLI:
nlp4j-local-search --lang en
Example:
nlp4j-local-search
Language: en
Auto analyze: False
Type 'help' or '?' for help.
>>
In this example, automatic linguistic analysis is disabled.
The dataset already contains structured fields such as manufacturer, make, model, component, date, mileage, and vehicle speed.
2. Load the NHTSA complaint dataset
Load a JSONL or compressed JSONL file with load():
>> load("examples_demo/nhtsa_complaints_2025_2.jsonl.gz")
Example output from this run:
Loading documents...
Loading: 109,734 documents | 6,038 docs/sec | 18.2s
Loaded 109,734 documents in 18.17 seconds (6,038 docs/sec).
The exact speed depends on the environment, but the important point is that after loading, the data is immediately available for local Lucene search and analysis.
3. Start with fields
Before writing queries, I first want to understand the dataset.
Version 0.7.3.0 extends the fields command so that it shows more than just field names.
>> fields
Example:
Documents: 109,734
Field Type Aggregatable Coverage Unique Diversity Example
------------------- ----------- ------------ -------- ------ --------- ------------------------------
id KEYWORD false - - - 2052305
text TEXT false - - - A-pillar trim flew off whil...
data STORED_ONLY false - - - {"id":"2052305","odi_no_s":...
odi_no_s KEYWORD true 100.00% 74,354 67.76% 11633472
manufacturer_s KEYWORD true 100.00% 272 0.25% Ford Motor Company
make_s KEYWORD true 100.00% 324 0.30% FORD
model_s KEYWORD true 100.00% 1,762 1.61% EXPLORER
model_year_i INTEGER true 100.00% - - 2016
component_s KEYWORD true 100.00% 436 0.40% STRUCTURE:BODY
state_s KEYWORD true 100.00% 61 0.06% NV
date DATE true 100.00% - - 2025-01-01T00:00:00+09:00
vehicle_speed_i INTEGER true 46.94% - - 75
incident_date_dt DATE true 100.00% - - 2024-07-01T00:00:00+09:00
miles_i INTEGER true 16.45% - - 200000
This gives us a quick profile of the dataset.
What the columns mean
Type
The field type used by the local Lucene index.
Examples include:
KEYWORD
TEXT
INTEGER
DATE
STORED_ONLY
Aggregatable
Whether the field can be used for aggregation and view().
For example:
make_s KEYWORD true
model_s KEYWORD true
component_s KEYWORD true
These are good candidates for interactive exploration.
Coverage
The percentage of documents containing a value for the field.
For example:
vehicle_speed_i 46.94%
miles_i 16.45%
This immediately tells us that mileage is available for only part of the dataset.
Unique
For aggregatable KEYWORD fields, this is the number of distinct values.
For example:
make_s 324
model_s 1,762
component_s 436
Diversity
Diversity is:
Unique / DocumentsWithValue
A low diversity value often indicates a useful categorical field.
A high value may indicate something closer to an identifier.
For example:
odi_no_s 67.76%
manufacturer_s 0.25%
make_s 0.30%
component_s 0.40%
odi_no_s has very high cardinality, while manufacturer, make, and component are much more suitable for grouping and exploration.
For numeric and date fields, version 0.7.3.0 intentionally does not calculate Unique or Diversity.
Numeric statistics such as minimum, maximum, average, and distribution can be treated separately in the future.
4. Explore the most common vehicle makes
Now that we know make_s is an aggregatable KEYWORD field, we can inspect it.
>> view("make_s")
Result:
View: make_s
Values are ordered by document count.
Rank Value Count
---- -------------------- --------
1 FORD 19991
2 HONDA 10631
3 CHEVROLET 9652
4 JEEP 7845
5 HYUNDAI 6545
6 TOYOTA 5985
7 KIA 5629
8 GMC 5378
9 RAM 4204
10 NISSAN 4033
Ford is the largest group in this dataset:
FORD: 19,991 complaints
Let's explore Ford further.
5. Find Ford models
The second argument of view() can be a Lucene query.
>> view("model_s", "make_s:FORD")
Result:
View: model_s
Lucene query: make_s:FORD
Matched documents: 19,991 / 109,734
Values are ordered by relative rate.
Rank Value Count All Count Relative Rate
---- -------------------- -------- ---------- -------------
1 F-150 4005 4005 5.49x
2 EXPLORER 2940 2940 5.49x
3 FUSION 1214 1214 5.49x
4 BRONCO SPORT 592 592 5.49x
5 F-250 SD 524 524 5.49x
6 F-350 SD 355 355 5.49x
7 MAVERICK HYBRID 319 319 5.49x
8 BRONCO 314 314 5.49x
9 FLEX 301 301 5.49x
10 MUSTANG MACH E 287 287 5.49x
The F-150 has:
4,005 complaints
So the next question becomes:
What complaint components are especially characteristic of the F-150?
6. Inspect components associated with the F-150
We can use the result from the previous step as the next filter:
>> view("component_s", "model_s:F-150")
Now view() calculates the relative rate of each component among F-150 complaints compared with the entire dataset.
For example:
Rank Value Count All Count Relative Rate
---- -------------------- -------- ---------- -------------
1 ELECTRICAL SYSTEM... 1 1 27.40x
2 ENGINE AND ENGINE... 1 1 27.40x
3 ENGINE AND ENGINE... 1 1 27.40x
4 SERVICE BRAKES, A... 1 1 27.40x
5 SERVICE BRAKES, H... 1 1 27.40x
6 POWER TRAIN:AUTOM... 2 3 18.27x
...
This is a useful distinction from simply sorting by count.
Count tells us how many matching documents contain the value.
All Count tells us how many documents in the entire dataset contain it.
Relative Rate tells us how disproportionately common the value is in the filtered subset.
7. Look at more candidates
Very rare values can naturally have a high relative rate.
For example, a component that appears only once in the entire dataset and happens to belong to an F-150 will have a very high relative rate.
So instead of automatically selecting rank 1, it is often useful to look further down the list.
We can request more rows:
>> view("component_s", "model_s:F-150", 30)
Among the results we find:
Rank Value Count All Count Relative Rate
---- ----------------------------------------- ------ ---------- -------------
15 POWER TRAIN:AUTOMATIC TRANSMISSION 24 119 5.53x
18 ENGINE ... CATALYTIC CONVERTOR 12 67 4.91x
23 PARKING BRAKE 12 78 4.22x
25 POWER TRAIN 1520 12118 3.44x
POWER TRAIN:AUTOMATIC TRANSMISSION is interesting because it is not just a one-document anomaly.
There are:
24 F-150 complaints
119 complaints overall
Relative Rate: 5.53x
This gives us a reasonable candidate for document-level investigation.
8. Turn the exploration into a search query
We have now discovered the conditions interactively:
make_s = FORD
model_s = F-150
component_s = POWER TRAIN:AUTOMATIC TRANSMISSION
We can turn them into a normal Lucene query.
>> search(
'make_s:FORD AND model_s:F-150 AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"',
30
)
Or, because the model condition already identifies the target in this exploration:
>> search(
'model_s:F-150 AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"',
30
)
The search results contain the actual complaint text.
Examples include reports describing:
- transmissions skipping between gears
- hard shifts and jerking
- unexpected downshifting
- transmission replacement
- CDF drum failures
- vehicles failing to accelerate
- vehicles entering limp mode
- transmission-related crashes
At this point, we are no longer looking only at statistics.
We can read the original complaint descriptions and understand what the underlying records actually say.
Important: use uppercase AND
Lucene Query Parser boolean operators should be written as:
AND
OR
NOT
For example:
model_s:F-150 AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"
Do not rely on lowercase:
model_s:F-150 and component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"
Using uppercase boolean operators makes the intended query structure explicit.
This is especially important when moving from exploratory view() commands to a final search() query.
9. The workflow: discover the query instead of writing it first
The important point of this example is not the specific Ford F-150 result.
It is the workflow.
We started without a complex query.
First:
fields
told us what data was available.
Then:
view("make_s")
showed us the major groups.
Then:
view("model_s", "make_s:FORD")
narrowed the analysis to Ford models.
Then:
view("component_s", "model_s:F-150", 30)
showed components disproportionately associated with the F-150.
Finally:
search(
'model_s:F-150 AND component_s:"POWER TRAIN:AUTOMATIC TRANSMISSION"',
30
)
let us inspect the actual records.
In other words:
fields
↓
view("make_s")
↓
view("model_s", "make_s:FORD")
↓
view("component_s", "model_s:F-150", 30)
↓
search(...)
This is a form of interactive query construction.
Instead of starting with:
What exact search query should I write?
we can start with:
What is in this dataset?
and progressively discover useful conditions.
10. view() and search() have different roles
I think this distinction is important.
view()
Use view() to understand a population.
Typical questions are:
What values are common?
What values are characteristic?
What should I filter on next?
Examples:
view("make_s")
view("model_s", "make_s:FORD")
view("component_s", "model_s:F-150")
search()
Use search() when you already have conditions and want to inspect actual documents.
Typical questions are:
What do the individual records say?
What happened in the reported cases?
Does the statistical pattern make sense when I read the text?
This gives a natural loop:
aggregate
→ discover
→ filter
→ aggregate again
→ search
→ read
11. Why fields matters in this workflow
The enhanced fields command in 0.7.3.0 makes the beginning of this workflow much easier.
Previously, seeing only field names required some prior knowledge of the dataset.
Now we can immediately see information such as:
model_s KEYWORD aggregatable=true
component_s KEYWORD aggregatable=true
vehicle_speed_i INTEGER coverage=46.94%
miles_i INTEGER coverage=16.45%
date DATE
That helps answer several questions before running any analysis:
Which fields are categorical?
Which fields can be aggregated?
How complete is each field?
Is a field low-cardinality or almost an identifier?
What kind of value does the field contain?
For unfamiliar JSONL datasets, this is especially useful.
12. A local Lucene workflow for both humans and LLMs
This kind of interface is also useful beyond interactive human use.
The commands are simple and structured:
fields
view(...)
search(...)
An LLM or agent could follow essentially the same process:
1. Inspect available fields
2. Choose a useful aggregation field
3. Inspect values
4. Construct the next filter
5. Repeat
6. Retrieve actual documents
The search engine therefore does not have to be only an interface for a human who already knows the schema.
It can also become a tool for discovering the schema and constructing searches dynamically.
That is one of the directions I am exploring with nlp4j-local-search.
Conclusion
With nlp4j-local-search 0.7.3.0, the CLI can now support a practical exploratory workflow:
fields
→ view
→ filtered view
→ more focused view
→ search
→ inspect original documents
Using the NHTSA complaint dataset, we were able to move from more than 100,000 documents to a focused set of Ford F-150 automatic-transmission complaints without designing the complete query in advance.
The key idea is simple:
Use
view()to discover the query, then usesearch()to inspect the evidence.
For exploratory local search and text analysis, I think this is a useful pattern—especially when working with a dataset whose schema and value distributions are not yet familiar.