Introduction
I published a Python package called nlp4j-local-search-embedding on PyPI.
This package provides a simple way to perform embedding-based semantic search locally from Python.
pip install nlp4j-local-search-embedding==0.1.0
GitHub:
https://github.com/oyahiroki/nlp4j-local-search-embedding
PyPI:
https://pypi.org/project/nlp4j-local-search-embedding/0.1.0/
What this package does
Traditional keyword search mainly depends on whether query terms appear in document text.
Embedding-based search, on the other hand, converts both documents and queries into vectors and searches by semantic similarity.
For example, suppose we have the following documents:
Query: bicycle
Documents:
- A car is running on the road.
- I want to buy a bicycle.
- Cycling is my favorite hobby.
- I went shopping by bike.
A keyword search can easily find documents containing bicycle, but it may not find documents containing related words such as cycling or bike.
Embedding-based search can retrieve these documents because they are semantically related.
bicycle
cycling
bike
This is useful when you want to search not only by exact word matching but also by meaning.
Installation
You can install the package from PyPI.
pip install nlp4j-local-search-embedding==0.1.0
In Google Colab, run:
!pip install -q nlp4j-local-search-embedding==0.1.0
The package internally uses Java/Lucene through nlp4j-local-search, so you can also check whether Java is available:
!java -version
I confirmed that it works on Google Colab with OpenJDK 17.
openjdk version "17.0.19"
OpenJDK Runtime Environment
OpenJDK 64-Bit Server VM
Minimal example
Here is a minimal example of semantic search.
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "A car is running on the road.",
"doc2": "I want to buy a bicycle.",
"doc3": "Cycling is my favorite hobby.",
"doc4": "I went shopping by bike.",
})
app.commit()
results = app.search("bicycle", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
Example output
The output will look like this:
=== Search results ===
number of results: 4
result[0].id: doc2
result[0].text: I want to buy a bicycle.
result[0].score: 0.93
result[0].metadata: {}
---
result[1].id: doc3
result[1].text: Cycling is my favorite hobby.
result[1].score: 0.91
result[1].metadata: {}
---
result[2].id: doc4
result[2].text: I went shopping by bike.
result[2].score: 0.90
result[2].metadata: {}
---
result[3].id: doc1
result[3].text: A car is running on the road.
result[3].score: 0.86
result[3].metadata: {}
---
Scores may vary depending on the model version and runtime environment.
The important point is that documents containing related words such as cycling and bike can be retrieved even when the query is bicycle.
Another example
Here is another example using city and technology-related documents.
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "Kyoto is a historic city in Japan.",
"doc2": "Tokyo is the capital city of Japan.",
"doc3": "Python is a popular programming language.",
"doc4": "Nintendo is a video game company headquartered in Kyoto.",
})
app.commit()
results = app.search("an old Japanese capital", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
For the query an old Japanese capital, the document Kyoto is a historic city in Japan. is expected to appear near the top.
Adding documents with metadata
Documents can also have metadata.
from nlp4j_local_search_embedding import SemanticSearch
documents = [
{
"id": "doc1",
"text": "Kyoto is a historic city in Japan with many temples and shrines.",
"metadata": {
"category": "city",
"country": "Japan",
},
},
{
"id": "doc2",
"text": "Nintendo is a video game company headquartered in Kyoto.",
"metadata": {
"category": "company",
"country": "Japan",
},
},
{
"id": "doc3",
"text": "Python is widely used for data science and machine learning.",
"metadata": {
"category": "technology",
},
},
]
app = SemanticSearch("en")
app.add(documents)
app.commit()
results = app.search("a Japanese game company", limit=3)
for result in results:
print(result.id)
print(result.text)
print(result.score)
print(result.metadata)
print("---")
Each result contains:
id
text
score
metadata
Design concept
nlp4j-local-search-embedding is not intended to be a large all-in-one framework.
It is designed as a thin embedding search layer on top of nlp4j-local-search.
The architecture is roughly as follows:
text documents
|
v
embedding model
|
v
vectors
|
v
nlp4j-local-search vector index
|
v
semantic search results
The base package, nlp4j-local-search, provides local search and vector indexing using Lucene.
This package adds text embedding support and provides a convenient SemanticSearch API.
Difference from keyword search
In keyword search, exact words in the query are very important.
For example, when searching for bicycle, keyword search can easily find:
I want to buy a bicycle.
However, it may not find documents such as:
Cycling is my favorite hobby.
I went shopping by bike.
Embedding search can retrieve these documents because they are semantically related.
bicycle
cycling
bike
This is useful for synonym search, semantic retrieval, and lightweight local search applications.
Default embedding model
The default model is:
intfloat/multilingual-e5-large
E5 models commonly use different prefixes for documents and queries:
passage: <document text>
query: <search query>
nlp4j-local-search-embedding handles these prefixes internally.
Therefore, users can simply add plain text and search with plain text.
app.add({
"doc1": "Kyoto is a historic city in Japan.",
})
results = app.search("old Japanese capital")
Running on Google Colab
You can try it on Google Colab with the following cells.
Install
!pip install -q nlp4j-local-search-embedding==0.1.0
Run semantic search
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "A car is running on the road.",
"doc2": "I want to buy a bicycle.",
"doc3": "Cycling is my favorite hobby.",
"doc4": "I went shopping by bike.",
})
app.commit()
results = app.search("bicycle", limit=10)
print("=== Search results ===")
print(f"number of results: {len(results)}")
for i, result in enumerate(results):
print(f"result[{i}].id: {result.id}")
print(f"result[{i}].text: {result.text}")
print(f"result[{i}].score: {result.score}")
print(f"result[{i}].metadata: {result.metadata}")
print("---")
The first run may take some time because the embedding model needs to be downloaded and loaded.
Future plans
I plan to improve the package with features such as:
- index save/load support
- larger document collection examples
- integration with RAG packages
- integration with
nlp4j-local-search-rag - better result formatting
- Google Colab example notebooks
Summary
nlp4j-local-search-embedding makes it easy to try local embedding-based semantic search from Python.
pip install nlp4j-local-search-embedding==0.1.0
A minimal example looks like this:
from nlp4j_local_search_embedding import SemanticSearch
app = SemanticSearch("en")
app.add({
"doc1": "Kyoto is a historic city in Japan.",
"doc2": "Tokyo is the capital city of Japan.",
})
app.commit()
results = app.search("old Japanese capital", limit=10)
for result in results:
print(result.id, result.score, result.text)
This package is useful when you want to search local documents by meaning rather than exact keywords.