Introduction
I published a Python package called nlp4j-local-search-embedding on PyPI.
This package provides a simple way to perform embedding-based semantic search locally from Python.
pip install nlp4j-local-search-embedding==0.1.0
GitHub:
https://github.com/oyahiroki/nlp4j-local-search-embedding
PyPI:
https://pypi.org/project/nlp4j-local-search-embedding/0.1.0/
The important point is that this package is not just a small Python wrapper around an embedding model.
It is built on top of nlp4j-local-search, which uses Apache Lucene as the local search engine layer.
Why Apache Lucene matters
Apache Lucene is one of the most important open-source search libraries in the world.
Many developers may know Elasticsearch, OpenSearch, or Apache Solr as search platforms, but under the hood, these systems are built on top of Lucene.
Lucene provides the core search engine technology, including:
- inverted indexes
- full-text search
- scoring
- analyzers
- indexing
- vector search support in recent versions
In other words, Lucene is not just another search library.
It is one of the foundational technologies behind many famous search systems.
However, using Lucene directly usually means writing Java code.
Using Elasticsearch or OpenSearch usually means running a server process, preparing an index, managing configuration, and communicating through HTTP APIs.
nlp4j-local-search-embedding aims to provide a much simpler experience:
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
With just a few lines of Python, you can try local semantic search backed by Lucene-based vector indexing.
What this package does
Traditional keyword search mainly depends on whether query terms appear in document text.
Embedding-based search converts both documents and queries into vectors and searches by semantic similarity.
For example, suppose we have the following documents:
Query: bicycle
Documents:
- A car is running on the road.
- I want to buy a bicycle.
- Cycling is my favorite hobby.
- I went shopping by bike.
A keyword search can easily find documents containing bicycle, but it may not find documents containing related words such as cycling or bike.
Embedding-based search can retrieve these documents because they are semantically related.
bicycle
cycling
bike
This is useful when you want to search not only by exact word matching but also by meaning.
Installation
You can install the package from PyPI.
pip install nlp4j-local-search-embedding==0.1.0
In Google Colab, run:
!pip install -q nlp4j-local-search-embedding==0.1.0
The package internally uses Java/Lucene through nlp4j-local-search, so you can also check whether Java is available:
!java -version
I confirmed that it works on Google Colab with OpenJDK 17.
openjdk version "17.0.19"
OpenJDK Runtime Environment
OpenJDK 64-Bit Server VM
Minimal example
Here is a minimal example of semantic search.
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "A car is running on the road.",
"doc2": "I want to buy a bicycle.",
"doc3": "Cycling is my favorite hobby.",
"doc4": "I went shopping by bike.",
})
app.commit()
results = app.search("bicycle", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
Example output
The output will look like this:
=== Search results ===
number of results: 4
result[0].id: doc2
result[0].text: I want to buy a bicycle.
result[0].score: 0.93
result[0].metadata: {}
---
result[1].id: doc3
result[1].text: Cycling is my favorite hobby.
result[1].score: 0.91
result[1].metadata: {}
---
result[2].id: doc4
result[2].text: I went shopping by bike.
result[2].score: 0.90
result[2].metadata: {}
---
result[3].id: doc1
result[3].text: A car is running on the road.
result[3].score: 0.86
result[3].metadata: {}
---
Scores may vary depending on the model version and runtime environment.
The important point is that documents containing related words such as cycling and bike can be retrieved even when the query is bicycle.
Another example
Here is another example using city and technology-related documents.
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "Kyoto is a historic city in Japan.",
"doc2": "Tokyo is the capital city of Japan.",
"doc3": "Python is a popular programming language.",
"doc4": "Nintendo is a video game company headquartered in Kyoto.",
})
app.commit()
results = app.search("an old Japanese capital", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
For the query an old Japanese capital, the document Kyoto is a historic city in Japan. is expected to appear near the top.
Adding documents with metadata
Documents can also have metadata.
from nlp4j_local_search_embedding import SemanticSearch
documents = [
{
"id": "doc1",
"text": "Kyoto is a historic city in Japan with many temples and shrines.",
"metadata": {
"category": "city",
"country": "Japan",
},
},
{
"id": "doc2",
"text": "Nintendo is a video game company headquartered in Kyoto.",
"metadata": {
"category": "company",
"country": "Japan",
},
},
{
"id": "doc3",
"text": "Python is widely used for data science and machine learning.",
"metadata": {
"category": "technology",
},
},
]
app = SemanticSearch("en")
app.add(documents)
app.commit()
results = app.search("a Japanese game company", limit=3)
for result in results:
print(result.id)
print(result.text)
print(result.score)
print(result.metadata)
print("---")
Each result contains:
id
text
score
metadata
Architecture
nlp4j-local-search-embedding is designed as a thin embedding search layer on top of nlp4j-local-search.
The architecture is roughly as follows:
text documents
|
v
embedding model
|
v
vectors
|
v
nlp4j-local-search
|
v
Apache Lucene vector index
|
v
semantic search results
The base package, nlp4j-local-search, provides local search and vector indexing using Lucene.
This package adds text embedding support and provides a convenient SemanticSearch API.
Why not just use Elasticsearch or OpenSearch?
Elasticsearch and OpenSearch are powerful and widely used search platforms.
However, for small experiments, local tools, notebooks, prototypes, or embedded applications, running a separate search server can feel too heavy.
You may need to think about:
- starting a server
- managing ports
- preparing an index
- sending HTTP requests
- maintaining server configuration
- cleaning up test data
For many Python experiments, I wanted something simpler.
app = SemanticSearch("en")
app.add(documents)
app.commit()
results = app.search("query")
That is the main motivation behind this package.
It gives Python users an easy way to try Lucene-based local semantic search without setting up a full search server.
Difference from keyword search
In keyword search, exact words in the query are very important.
For example, when searching for bicycle, keyword search can easily find:
I want to buy a bicycle.
However, it may not find documents such as:
Cycling is my favorite hobby.
I went shopping by bike.
Embedding search can retrieve these documents because they are semantically related.
bicycle
cycling
bike
This is useful for synonym search, semantic retrieval, and lightweight local search applications.
Default embedding model
The default model is:
intfloat/multilingual-e5-large
E5 models commonly use different prefixes for documents and queries:
passage: <document text>
query: <search query>
nlp4j-local-search-embedding handles these prefixes internally.
Therefore, users can simply add plain text and search with plain text.
app.add({
"doc1": "Kyoto is a historic city in Japan.",
})
results = app.search("old Japanese capital")
Running on Google Colab
You can try it on Google Colab with the following cells.
Install
!pip install -q nlp4j-local-search-embedding==0.1.0
Run semantic search
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "A car is running on the road.",
"doc2": "I want to buy a bicycle.",
"doc3": "Cycling is my favorite hobby.",
"doc4": "I went shopping by bike.",
})
app.commit()
results = app.search("bicycle", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
The first run may take some time because the embedding model needs to be downloaded and loaded.
Relationship with nlp4j-local-search
nlp4j-local-search is the lower-level local search package.
It can perform keyword search and vector search locally.
nlp4j-local-search-embedding adds an embedding layer on top of it.
nlp4j-local-search
local keyword search
local vector search
Lucene-based indexing
nlp4j-local-search-embedding
text-to-embedding
semantic search
simple Python API
This separation is intentional.
The base local search package can stay relatively lightweight, while the embedding package can depend on heavier machine learning libraries such as sentence-transformers.
Future plans
I plan to improve the package with features such as:
- index save/load support
- larger document collection examples
- integration with RAG packages
- integration with
nlp4j-local-search-rag - better result formatting
- Google Colab example notebooks
Summary
nlp4j-local-search-embedding makes it easy to try local embedding-based semantic search from Python.
pip install nlp4j-local-search-embedding==0.1.0
A minimal example looks like this:
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "Kyoto is a historic city in Japan.",
"doc2": "Tokyo is the capital city of Japan.",
})
app.commit()
results = app.search("old Japanese capital", limit=10)
for result in results:
print(result.id, result.score, result.text)
This package is useful when you want to search local documents by meaning rather than exact keywords.
It is also a convenient way to use the power of Apache Lucene from Python without setting up Elasticsearch, OpenSearch, or Solr.
nlp4j-local-search-embedding is designed for developers who want to try local semantic search from Python without running a full vector database server.
It uses Apache Lucene underneath, so it is closer to a local search engine library than a heavyweight vector database platform.
Not only vector search: keyword search also matters
Many vector databases focus primarily on vector search.
However, real-world search applications often need both keyword search and semantic search.
For example:
- exact keyword matching
- phrase matching
- language-specific text analysis
- BM25-style scoring
- metadata filtering
- semantic search with embeddings
- hybrid search combining keywords and vectors
This is where Apache Lucene is very powerful.
Lucene has been one of the most important search engine libraries for many years.
It is widely known as the core search technology behind systems such as Elasticsearch, OpenSearch, and Apache Solr.
nlp4j-local-search-embedding is built on top of nlp4j-local-search, which uses Lucene as the underlying search engine layer.
This means that the package is not just a small vector search wrapper.
It is designed as a path toward local search where keyword search and embedding-based search can coexist naturally.
In small experiments, notebooks, and local tools, you may not want to run a full search server or a production-grade vector database.
With this package, you can start with a simple Python API:
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add(documents)
app.commit()
results = app.search("old Japanese capital", limit=10)
At the same time, the underlying direction is Lucene-based search, not just array-based vector comparison.
This is important because many search applications are not purely semantic.
They often need both exact keyword matching and semantic similarity.
- Local Semantic Search in Python Powered by Apache Lucene
- Easy Lucene-Based Semantic Search in Python
Not a heavyweight vector database.
Not just an embedding demo.
A lightweight Lucene-based local search package for Python.
Vector search should also be usable like a desktop tool
Large-scale vector databases are very useful.
If you need to handle massive datasets, high availability, distributed indexing, and production workloads, systems such as vector databases are the right choice.
However, not every use case starts at that scale.
Sometimes, you simply want to search a small collection of local documents:
- notes
- Markdown files
- CSV records
- exported documents
- technical articles
- small knowledge bases
- experimental datasets
For these use cases, setting up a full vector database server can feel too heavy.
I believe vector search should also be available in a more casual way — almost like using Excel or Word.
You open your data, add documents, and search by meaning.
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "A car is running on the road.",
"doc2": "I want to buy a bicycle.",
"doc3": "Cycling is my favorite hobby.",
"doc4": "I went shopping by bike.",
})
app.commit()
results = app.search("bicycle", limit=10)
This is the direction of nlp4j-local-search-embedding.
It is not trying to replace large-scale vector databases.
Instead, it aims to make local semantic search easy enough to use in notebooks, scripts, prototypes, and personal tools.
Why Lucene is important here
Another important point is that this package is built on top of Apache Lucene through nlp4j-local-search.
Lucene is not only about vector search.
It is a mature search engine library with a long history of keyword search, analyzers, indexing, scoring, and text search.
This matters because real-world search is often not purely vector-based.
In many cases, we need both:
- keyword search
- semantic search
- metadata filtering
- hybrid search
Vector search is powerful, but exact words still matter.
For example, product names, error codes, class names, function names, IDs, and technical terms often need keyword-based search.
This is where a Lucene-based approach becomes attractive.
nlp4j-local-search-embedding provides a small Python API for semantic search, while keeping a path toward Lucene-based keyword search and hybrid search.
Large vector databases are important for large-scale production systems.
But vector search should not be limited to large infrastructure.
There is also value in making semantic search available as a lightweight local tool, just like opening a spreadsheet or a document.
nlp4j-local-search-embedding is designed for that use case:
easy local semantic search from Python, powered by Apache Lucene.
A lightweight Lucene-based semantic search tool for Python scripts, notebooks, and local productivity use cases.