From JSONL to Lucene Search and Data Exploration in Google Colab with nlp4j-local-search 0.6.0
nlp4j-local-search is a lightweight Python library that lets you use Lucene-based local search without running Elasticsearch, OpenSearch, Solr, or a separate search server.
With version 0.6.0, we can build a simple workflow that combines:
- JSONL import
- field transformation
- local Lucene indexing
- document insertion
- Lucene Query Syntax search
- aggregation and exploratory analysis
In this article, we will try each step interactively in Google Colab.
The goal is not to hide everything inside one large program.
Instead, we will inspect the data and experiment with the search engine step by step.
1. Install nlp4j-local-search
Run the following cell in Google Colab:
!pip install -q nlp4j-local-search==0.6.0
Then import the library:
from nlp4j_local_search import SearchEngine
No Elasticsearch, OpenSearch, Solr, or Docker setup is required.
The Lucene-based search engine runs locally.
2. Create a small JSONL dataset
Let's create a simple dataset containing cities and companies.
import json
documents = [
{
"id": "1",
"text": "Kyoto is a historic city in Japan.",
"category": "city",
},
{
"id": "2",
"text": "Nintendo is headquartered in Kyoto.",
"category": "company",
},
{
"id": "3",
"text": "Tokyo is the capital city of Japan.",
"category": "city",
},
]
with open("sample.jsonl", "w", encoding="utf-8") as f:
for doc in documents:
f.write(json.dumps(doc, ensure_ascii=False) + "\n")
Because this is Colab, we can immediately inspect the file:
!cat sample.jsonl
Output:
{"id": "1", "text": "Kyoto is a historic city in Japan.", "category": "city"}
{"id": "2", "text": "Nintendo is headquartered in Kyoto.", "category": "company"}
{"id": "3", "text": "Tokyo is the capital city of Japan.", "category": "city"}
This is ordinary JSONL: one JSON object per line.
3. Start the local search engine
Create an English search engine:
engine = SearchEngine("en", auto_analyze=False)
We intentionally do not use:
with SearchEngine(...) as engine:
in this notebook.
Keeping engine alive makes it possible to use the same search index from multiple Colab cells.
At the end of the notebook, we will explicitly call:
engine.close()
4. Import JSONL and rename fields
The source dataset contains:
text
category
Suppose we want to use:
body
category_s
inside our search application.
We can transform the fields while loading the JSONL data:
result = (
engine.data("sample.jsonl")
.rename("text", "body")
.rename("category", "category_s")
.load()
)
print(result)
Example output:
Loaded 3 documents in 0.16 seconds.
The pipeline can be understood as:
sample.jsonl
text -----------------> body
category -------------> category_s
|
v
Lucene index
This is useful when an external dataset uses different field names from the schema you want to use in your application.
5. Inspect the available fields
Before searching, let's look at the fields registered in the index:
engine.fields()
You can also inspect fields that can be aggregated:
engine.aggregatable_fields()
This is useful when exploring an unfamiliar JSONL dataset interactively.
Instead of immediately writing queries, we can first ask:
What fields are available?
6. Add another document from Python
JSONL is not the only way to add data.
We can add another document directly:
engine.add(
"4",
"Osaka is a major city in Japan.",
fields={
"category_s": "city",
},
)
engine.commit()
We now have four documents in the index.
Let's check:
engine.count()
Expected result:
4
7. Try a simple full-text search
Now let's search for Kyoto.
results = engine.search("Kyoto", 10)
for r in results:
print(f"[{r.id}] {r.body}")
Example output:
[2] Nintendo is headquartered in Kyoto.
[1] Kyoto is a historic city in Japan.
The results are ordered according to Lucene's search score.
A bare query such as:
engine.search("Kyoto")
searches the default text field.
For an English SearchEngine, the internal text field is text_en.
8. Search a specific field
search() accepts Lucene Query Syntax.
For example, let's retrieve only documents whose category is city:
results = engine.search(
"category_s:city",
10,
)
for r in results:
print(f"[{r.id}] {r.body}")
Output:
[1] Kyoto is a historic city in Japan.
[3] Tokyo is the capital city of Japan.
[4] Osaka is a major city in Japan.
Structured fields and full-text conditions can be combined in the same query.
For example:
engine.search(
"category_s:city AND Kyoto",
10,
)
Try displaying the result:
for r in engine.search("category_s:city AND Kyoto", 10):
print(f"[{r.id}] {r.body}")
Only the Kyoto city document should match.
9. Experiment with Lucene Query Syntax
Because search() directly accepts Lucene Query Syntax, we can try different expressions in separate Colab cells.
AND
engine.search(
"Kyoto AND historic",
10,
)
OR
engine.search(
"Kyoto OR Tokyo",
10,
)
Field query
engine.search(
"category_s:company",
10,
)
Combination of full-text and field conditions
engine.search(
"Kyoto AND category_s:company",
10,
)
Phrase query
engine.search(
'"historic city"',
10,
)
This notebook-style workflow is useful because we can modify only the query cell and immediately see how Lucene interprets different expressions.
10. Validate a query before running it
Version 0.6.0 also provides a query validation API.
For example:
validation = engine.validate_query(
"category_s:city AND Kyoto"
)
print(validation)
We can inspect:
validation.valid
For an invalid query:
validation = engine.validate_query(
"category_s:city AND ("
)
print("valid:", validation.valid)
print("message:", validation.message)
This can be useful when building interactive search applications or command-line tools where users enter Lucene queries themselves.
11. Search and view() have different purposes
So far, we have been searching for documents.
For example:
engine.search("Kyoto")
means:
I want to find documents related to Kyoto.
But sometimes we do not yet know what to search for.
We may simply want to look at the dataset.
That is the role of view().
Let's inspect the category distribution:
print(
engine.view("category_s")
)
Output:
View: category_s
Values are ordered by document count.
Rank Value Count
---- -------------------- --------
1 city 3
2 company 1
No search keyword was required.
We simply looked at the dataset and discovered that:
city = 3 documents
company = 1 document
This distinction is intentional.
search()
"I know what I want to find."
view()
"I want to look at the data and discover something."
12. Start with view() before writing a query
We can also call:
print(engine.view())
This gives an overview of aggregatable fields.
The idea is similar to exploratory data analysis.
Instead of starting with a search query, we can follow this workflow:
Load data
↓
View the dataset
↓
Notice an interesting field/value
↓
Search or analyze it further
For example:
print(engine.view("category_s"))
shows that city is common.
We can then retrieve those documents:
results = engine.search("category_s:city")
for r in results:
print(r.body)
This creates a natural connection between exploration and retrieval.
13. View a subset of the data
view() can also use a Lucene query to define the subset being analyzed.
For example:
print(
engine.view(
"category_s",
"Kyoto",
)
)
Now we are not simply counting categories in the entire dataset.
We are looking at the category distribution of documents matching Kyoto and comparing it with the complete dataset.
This mode uses relative-rate analysis.
That makes view() useful for questions such as:
Which values are unusually common in documents related to this topic?
This is different from simply returning matching documents.
14. Programmatic aggregation is also available
view() is designed for inspection.
If we need aggregation results as structured Python data, we can use aggregate().
result = engine.aggregate(
"category_s",
size=10,
)
result
The result uses an OpenSearch-like aggregation structure.
For example:
for bucket in result["aggregations"]["category_s"]["buckets"]:
print(
bucket["key"],
bucket["doc_count"],
)
Output:
city 3
company 1
This gives us two related APIs:
view()
human-friendly exploration
aggregate()
programmatic aggregation
15. Count documents with a Lucene query
We can also count documents without retrieving them.
All documents:
engine.count()
Result:
4
Only city documents:
engine.count(
"category_s:city"
)
Result:
3
Only documents mentioning Kyoto:
engine.count("Kyoto")
Result:
2
This is useful when only the size of a result set is needed.
16. A useful Colab exploration pattern
At this point, we can interactively move between different operations.
For example:
print(engine.view("category_s"))
Then:
engine.count("category_s:city")
Then:
results = engine.search(
"category_s:city"
)
for r in results:
print(r.body)
Then try a narrower query:
results = engine.search(
"category_s:city AND Kyoto"
)
for r in results:
print(r.body)
This is one reason a notebook can be a convenient environment for search and text analytics.
We do not need to decide every query beforehand.
We can inspect the data, formulate a query, look at the result, and continue exploring.
17. Add more data and immediately search again
Because the same SearchEngine stays alive across notebook cells, we can continue adding documents.
engine.add(
"5",
"Apple is a technology company based in Cupertino.",
fields={
"category_s": "company",
},
)
engine.commit()
Check the new total:
engine.count()
Now:
5
And inspect the category distribution again:
print(
engine.view("category_s")
)
We should now see:
city 3
company 2
The notebook becomes an interactive local search environment.
18. Close the search engine
When finished, close the engine:
engine.close()
This releases the Lucene resources used by the current SearchEngine.
If you run the notebook again from the beginning, simply create a new engine:
engine = SearchEngine(
"en",
auto_analyze=False,
)
The complete workflow
The notebook demonstrated this flow:
JSONL
|
v
DataPipeline
|
+-- rename fields
|
v
Lucene index
|
+-- add()
|
+-- count()
|
+-- search()
|
+-- aggregate()
|
+-- view()
|
+-- validate_query()
The corresponding Python API remains compact:
engine = SearchEngine(
"en",
auto_analyze=False,
)
(
engine.data("sample.jsonl")
.rename("text", "body")
.rename("category", "category_s")
.load()
)
engine.add(
"4",
"Osaka is a major city in Japan.",
fields={
"category_s": "city",
},
)
engine.commit()
results = engine.search(
"category_s:city"
)
print(
engine.view("category_s")
)
engine.close()
Search vs. exploration
One of the useful ideas behind this API is that document retrieval and exploratory analysis do not have to be the same operation.
A search begins with an intention:
engine.search(
"Kyoto AND category_s:city"
)
We already know something about what we want.
Exploration can begin without that intention:
engine.view()
or:
engine.view(
"category_s"
)
We can simply look at the data and notice a pattern.
That may then lead to the next query.
view
↓
discover
↓
search
↓
view again
↓
refine
For interactive environments such as Google Colab, this combination works particularly well.
Conclusion
With nlp4j-local-search 0.6.0, we can go from raw JSONL to Lucene-based search and exploratory analysis using a small Python API:
engine.data(...)
engine.add(...)
engine.search(...)
engine.count(...)
engine.aggregate(...)
engine.view(...)
There is no separate search server to configure.
More importantly, the workflow does not have to start with a search query.
We can load the data, look at its distributions, discover an interesting pattern, and then decide what to search for.
For experimentation in Google Colab, that makes nlp4j-local-search useful not only as a search library, but also as a lightweight environment for exploring text-oriented datasets.