Local Japanese Keyword Search and Vector Search from Python without Elasticsearch: nlp4j-local-search

When working on NLP or RAG experiments in Python, you may want to quickly try search functionality such as:

• Japanese full-text keyword search
• Vector search
• Comparing keyword search and embedding search
• Building a small local search index in Jupyter Notebook or Google Colab
• Testing search behavior without installing Elasticsearch, OpenSearch, Solr, or Docker

For this purpose, I added vector search support to nlp4j-local-search.

GitHub:

https://github.com/oyahiroki/nlp4j-local-search

What is nlp4j-local-search?

nlp4j-local-search is a lightweight local search library that can be used from Python.

It allows you to create a local search index, add documents, and search them directly from Python code.

Main features:

• Usable from Python
• Japanese keyword search
• English keyword search
• Vector search
• JSON document support
• No Elasticsearch server required
• No OpenSearch server required
• No Solr server required
• No Docker required
• Easy to use in Jupyter Notebook and Google Colab
• Internally based on Apache Lucene search functionality

This library is not intended to replace a large-scale production search system.

Instead, it is designed for cases where you want to quickly try search functionality for NLP, RAG, embedding evaluation, or small experiments.

Installation

You can install it directly from GitHub.

!pip install git+https://github.com/oyahiroki/nlp4j-local-search.git

After installation, you can check the installed package.

!pip list | grep nlp4j

Example:

nlp4j-local-search    0.2.0

Japanese keyword search example

First, let’s try Japanese keyword search.

from nlp4j_local_search import SearchEngine

# Initialize the search engine in Japanese mode
engine = SearchEngine("ja")

Add documents.

engine.add("1", "東京都は日本の都道府県のひとつです")
engine.add("2", "京都は日本の都市です")
engine.add("3", "京都市には任天堂の本社があります")
engine.add_json({"id": "4", "body": "京都府は広いです"})  # JSON format is also supported
engine.add_json({"id": "5", "body": "大阪は関西の大都市です"})

Commit the documents to the index.

engine.commit()
print("Index commit completed")

Search with the keyword 京都.

results = engine.search("京都", limit=10)

for r in results:
    print(f"ID: {r.id}, Score: {r.score:.4f}")
    print(f"Body: {r.body}")
    print("-" * 50)

Example output:

ID: 2, Score: 0.2729
Body: 京都は日本の都市です
--------------------------------------------------
ID: 4, Score: 0.2729
Body: 京都府は広いです
--------------------------------------------------
ID: 3, Score: 0.2450
Body: 京都市には任天堂の本社があります
--------------------------------------------------

This is not just a simple substring search.

For example, if you use simple substring matching, 東京都 may also match the query 京都, because the character sequence 京都 appears inside 東京都.

However, Japanese full-text search analyzes Japanese text as search terms instead of treating everything as a raw character sequence.

This makes it useful for Japanese search experiments.

Vector search example

Next, let’s try vector search.

For simplicity, this example uses 2-dimensional vectors.

from nlp4j_local_search import SearchEngine

# Japanese mode + 2-dimensional vector search
engine = SearchEngine(lang="ja", vector_dimension=2)

Add simple vector data.

engine.add("1 East",  [1.0,  0.0])   # East
engine.add("2 North", [1.0,  1.0])   # North-East
engine.add("3 West",  [-1.0, 0.0])   # West
engine.add("4 South", [-1.0, -1.0])  # South-West

engine.commit()

Search by query vector.

query_vector = [0.9, 0.1]

print("Query vector:", query_vector)

for r in engine.search(query_vector, limit=10):
    print(f"{r.id}: body={r.body} score={r.score}")

Example output:

Query vector: [0.9, 0.1]
1 East: body=None score=0.9969
2 North: body=None score=0.8904
4 South: body=None score=0.1095
3 West: body=None score=0.0036

The query vector [0.9, 0.1] is close to [1.0, 0.0], so East is returned with the highest score.

In this way, once vectors are added to the index, you can search for items that are close to a query vector.

Keyword search and vector search with a similar interface

With nlp4j-local-search, both keyword search and vector search can be used from the same SearchEngine interface.

Keyword search:

results = engine.search("京都", limit=10)

Vector search:

results = engine.search([0.9, 0.1], limit=10)

This is useful when you want to compare different search methods in NLP experiments.

For example:

• Compare keyword search results and embedding search results
• Prototype the retrieval part of a RAG application
• Validate search logic with a small dataset
• Create a temporary search index for unit tests
• Check search behavior in Google Colab or Jupyter Notebook

Before setting up Elasticsearch or OpenSearch, you can quickly test the search behavior locally.

No Elasticsearch server required

Normally, if you want to use a search engine, you may need to prepare things such as:

• Installing Elasticsearch
• Installing OpenSearch
• Installing Solr
• Preparing Docker Compose
• Configuring ports and memory settings
• Defining index mappings
• Creating index settings

These are important for production systems.

However, for small NLP experiments or notebook-based validation, this setup can feel too heavy.

With nlp4j-local-search, you can initialize the search engine directly in Python, add documents, and search them immediately.

from nlp4j_local_search import SearchEngine

with SearchEngine("ja") as engine:
    engine.add("1", "京都は日本の都市です")
    engine.add("2", "大阪は関西の大都市です")
    engine.commit()

    for r in engine.search("京都", limit=10):
        print(r.id, r.body, r.score)

You do not need to start a search server.

You can begin search experiments with only Python code.

Use cases

nlp4j-local-search is especially useful for the following use cases.

1. NLP experiments

You can use it to search small text datasets, dictionary data, Wikipedia-derived text, JSONL data, and other local datasets.

2. RAG prototyping

Before preparing a full vector database or search server, you can test the retrieval part locally.

3. Comparing keyword search and vector search

When evaluating embedding models, it is often useful to compare keyword search results with vector search results.

For small evaluation datasets, this can be done locally and quickly.

4. Google Colab and Jupyter Notebook experiments

You can create a temporary search index inside a notebook, run searches, and discard the index after the experiment.

5. Unit testing

You can create a temporary local search index for tests that involve search behavior.

Notes

At this stage, nlp4j-local-search is mainly intended for experimentation and validation.

For large-scale production search systems, Elasticsearch, OpenSearch, Solr, Milvus, or other dedicated systems may be more appropriate.

However, nlp4j-local-search is useful when you want to:

• Quickly check search behavior
• Build a small proof of concept
• Run search experiments in a notebook
• Compare keyword search and vector search
• Validate search logic before introducing a search server

Summary

Vector search support has been added to nlp4j-local-search.

Now you can easily use the following features from Python:

• Japanese keyword search
• English keyword search
• JSON document indexing
• Vector search
• Local in-memory search

You do not need to install Elasticsearch, OpenSearch, Solr, or Docker.

This makes nlp4j-local-search useful as a lightweight local search engine for NLP experiments, RAG prototyping, embedding evaluation, and search logic validation.