From JSONL to Search and Analysis with nlp4j-local-search 0.6.0
nlp4j-local-search provides a simple Python API for loading documents into a local Lucene index, searching them with Lucene Query Syntax, and inspecting the data with aggregation-based analysis.
Version 0.6.0 makes it possible to write a compact end-to-end workflow that covers:
- importing JSONL data
- renaming fields before indexing
- loading documents into the search engine
- adding additional documents from Python
- searching with Lucene Query Syntax
- inspecting field distributions with
view()
This article demonstrates the complete flow in one small Python program.
Installation
In Google Colab or Jupyter:
!pip install -q nlp4j-local-search==0.6.0
Or from a terminal:
pip install nlp4j-local-search==0.6.0
Sample data
Suppose we have JSONL data like this:
{"id":"1","text":"Kyoto is a historic city in Japan.","category":"city"}
{"id":"2","text":"Nintendo is headquartered in Kyoto.","category":"company"}
{"id":"3","text":"Tokyo is the capital city of Japan.","category":"city"}
For this example, however, the JSONL file is generated directly from Python so that the entire example can run as a single script.
Complete example
import json
import tempfile
from pathlib import Path
from nlp4j_local_search import SearchEngine
# ------------------------------------------------------------
# 1. Create a sample JSONL file
# ------------------------------------------------------------
documents = [
{
"id": "1",
"text": "Kyoto is a historic city in Japan.",
"category": "city",
},
{
"id": "2",
"text": "Nintendo is headquartered in Kyoto.",
"category": "company",
},
{
"id": "3",
"text": "Tokyo is the capital city of Japan.",
"category": "city",
},
]
with tempfile.TemporaryDirectory() as tmpdir:
jsonl_path = Path(tmpdir) / "sample.jsonl"
with jsonl_path.open("w", encoding="utf-8") as f:
for doc in documents:
f.write(json.dumps(doc, ensure_ascii=False) + "\n")
# --------------------------------------------------------
# 2. Import JSONL and rename fields before indexing
#
# text -> body
# category -> category_s
# --------------------------------------------------------
with SearchEngine("en", auto_analyze=False) as engine:
result = (
engine.data(jsonl_path)
.rename("text", "body")
.rename("category", "category_s")
.load()
)
print("=== Load ===")
print(result)
# ----------------------------------------------------
# 3. Add another document from Python
# ----------------------------------------------------
engine.add(
"4",
"Osaka is a major city in Japan.",
fields={"category_s": "city"},
)
engine.commit()
# ----------------------------------------------------
# 4. Full-text search
# ----------------------------------------------------
print("\n=== Search: Kyoto ===")
for r in engine.search("Kyoto", 10):
print(f"[{r.id}] {r.body}")
# ----------------------------------------------------
# 5. Field search with Lucene Query Syntax
# ----------------------------------------------------
print("\n=== Search: category_s:city ===")
for r in engine.search("category_s:city", 10):
print(f"[{r.id}] {r.body}")
# ----------------------------------------------------
# 6. Inspect the field distribution
# ----------------------------------------------------
print("\n=== View: category_s ===")
print(engine.view("category_s"))
Output
The program produces output similar to the following:
=== Load ===
Loaded 3 documents in 0.16 seconds.
=== Search: Kyoto ===
[2] Nintendo is headquartered in Kyoto.
[1] Kyoto is a historic city in Japan.
=== Search: category_s:city ===
[1] Kyoto is a historic city in Japan.
[3] Tokyo is the capital city of Japan.
[4] Osaka is a major city in Japan.
=== View: category_s ===
View: category_s
Values are ordered by document count.
Rank Value Count
---- -------------------- --------
1 city 3
2 company 1
1. Importing and transforming JSONL
The first important part is the DataPipeline.
result = (
engine.data(jsonl_path)
.rename("text", "body")
.rename("category", "category_s")
.load()
)
Instead of preprocessing the JSONL file separately, transformations can be defined before loading the documents into the search engine.
In this example:
text
↓
body
and:
category
↓
category_s
The transformed documents are then loaded into the index with:
.load()
Conceptually, the flow is:
JSONL
↓
DataPipeline
↓
rename()
↓
rename()
↓
load()
↓
SearchEngine
This is useful when the field names in an external dataset do not exactly match the field names you want to use in your search application.
2. Adding documents programmatically
Documents do not have to come only from JSONL files.
After loading the initial dataset, another document can be added directly:
engine.add(
"4",
"Osaka is a major city in Japan.",
fields={"category_s": "city"},
)
After adding documents, call:
engine.commit()
to make them searchable.
The indexed dataset now contains four documents.
3. Searching with Lucene Query Syntax
In version 0.6.0, SearchEngine.search() accepts Lucene Query Syntax directly.
A simple query:
engine.search("Kyoto", 10)
searches the default text field.
The result is:
[2] Nintendo is headquartered in Kyoto.
[1] Kyoto is a historic city in Japan.
Field-specific queries can also be written directly:
engine.search("category_s:city", 10)
This returns the three documents whose category_s value is city.
[1] Kyoto is a historic city in Japan.
[3] Tokyo is the capital city of Japan.
[4] Osaka is a major city in Japan.
Because search() uses Lucene Query Syntax, more expressive queries can also be used.
For example:
engine.search("Kyoto AND historic")
engine.search("category_s:city AND Kyoto")
engine.search("category_s:city OR category_s:company")
engine.search('body:"historic city"')
The same query language can therefore be used for both full-text and structured field searches.
4. Search and analysis are different operations
One design idea in nlp4j-local-search is to distinguish between searching and viewing.
A search starts with something you want to find:
engine.search("Kyoto")
You already have a query or search intention.
Analysis is different.
Sometimes you simply want to look at the data and discover what is common or unusual.
For example:
engine.view("category_s")
produces:
View: category_s
Values are ordered by document count.
Rank Value Count
---- -------------------- --------
1 city 3
2 company 1
No search keyword was required.
Simply looking at the distribution tells us that the current dataset contains three city documents and one company document.
This leads to a useful distinction:
search()
"I want to find something."
view()
"I want to look at the data and discover its characteristics."
5. Looking at the entire dataset
view() can also be called without a field name.
print(engine.view())
This gives an overview of aggregatable fields in the index.
This can be useful when exploring an unfamiliar dataset.
Instead of starting with a query, you can first inspect the data, identify interesting fields or values, and then decide what to search for.
A typical exploratory workflow can therefore be:
load data
↓
view()
↓
view("some_field")
↓
discover an interesting pattern
↓
search(...)
This is slightly different from a traditional search-first workflow.
6. Data preparation, search, and analysis in one API
The example combines several operations:
JSONL import
↓
field transformation
↓
indexing
↓
additional document insertion
↓
Lucene query search
↓
field analysis
In Python, the core workflow is compact:
with SearchEngine("en", auto_analyze=False) as engine:
(
engine.data("sample.jsonl")
.rename("text", "body")
.rename("category", "category_s")
.load()
)
engine.add(
"4",
"Osaka is a major city in Japan.",
fields={"category_s": "city"},
)
engine.commit()
results = engine.search("category_s:city")
print(engine.view("category_s"))
The same SearchEngine instance is used for both document retrieval and exploratory analysis.
7. A useful starting point for larger datasets
The sample dataset contains only four documents, but the same structure can be used with normal JSONL datasets.
For example, a dataset might contain fields such as:
{
"id": "10001",
"text": "The customer reported a battery problem.",
"maker": "Nissan",
"year": "2026",
"category": "electrical"
}
The fields can be normalized before indexing:
(
engine.data("documents.jsonl")
.rename("text", "body")
.rename("category", "category_s")
.load()
)
Then search can be performed with Lucene Query Syntax:
engine.search(
"maker:Nissan AND category_s:electrical"
)
while distributions can be inspected independently:
engine.view("maker")
engine.view("category_s")
This makes it possible to move naturally between document search and exploratory analysis.
Summary
With nlp4j-local-search 0.6.0, a small amount of Python code can cover the complete flow from raw JSONL data to search and analysis.
engine.data(...)
.rename(...)
.load()
engine.add(...)
engine.search(...)
engine.view(...)
The main concepts are intentionally separate:
DataPipeline
prepares data
search()
retrieves documents using Lucene Query Syntax
view()
helps inspect distributions and discover patterns
For users working with JSONL datasets, this provides a simple way to prepare the data, search it, and inspect its characteristics from the same Python workflow.