0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@oyahiroki(Hiroki Oya)

What If You Need Lucene, but Not a Search Server? Building Local Search Directly in Java

0
Posted at

What If You Need Lucene, but Not a Search Server? Building Local Search Directly in Java

When Java developers think about full-text search, the names that often come to mind are:

  • Apache Solr
  • OpenSearch
  • Elasticsearch
  • and, more recently, vector databases such as Milvus

I like Solr and OpenSearch myself. They are powerful platforms and are the right choice for many systems.

But there is another option worth remembering:

Apache Lucene itself is a Java library.

That means a search engine can live directly inside your Java application.

In this article, I would like to introduce LocalSearch, a small application-oriented API in NLP4J built on Apache Lucene.

The basic idea is:

Java Application
      |
      v
   LocalSearch
      |
      v
 Apache Lucene
      |
      v
 Local Index

No HTTP request is required between your application and the search engine.

No separate search process is required.

For some applications, that simplicity can be very useful.

Lucene Is Already a Search Engine Library

Apache Lucene is a high-performance search engine library written in Java.

It provides the foundation for several well-known search technologies.

So Java developers already have a very capable search engine available as a library.

The interesting architectural question is therefore:

Do I need a search server, or do I only need search functionality inside my application?

These are different requirements.

Search Server vs Search Library

Solr and OpenSearch provide much more than just indexing and querying.

They are designed for requirements such as:

  • distributed search
  • replication
  • clustering
  • REST APIs
  • remote access
  • operational management
  • horizontal scaling

Those features are extremely valuable.

But consider applications such as:

Desktop application
Batch application
NLP pipeline
Command-line tool
Local analytics application
Integration test
Proof of concept
Small internal application

For these applications, it may be reasonable to keep the search engine inside the Java process.

Instead of:

Java Application
      |
      | HTTP / REST
      v
Solr / OpenSearch
      |
      v
    Lucene

we can also have:

Java Application
      |
      v
 LocalSearch
      |
      v
    Lucene

This is not a replacement for Solr or OpenSearch.

It is simply another deployment model.

Add It to Your Maven Project

nlp4j-lucene can be added as a normal Maven dependency.

For version 1.5.0.0, add the following to your pom.xml:

<dependency>
    <groupId>org.nlp4j</groupId>
    <artifactId>nlp4j-lucene</artifactId>
    <version>1.5.0.0</version>
</dependency>

Then the search engine can be used directly from Java:

import nlp4j.lucene.LocalSearch;
import nlp4j.lucene.SearchResult;

There is no separate search server installation step.

The current nlp4j-lucene 1.5.0.0 build targets Java 17 and uses Apache Lucene 9.12.2 internally.

The application architecture can therefore remain quite simple:

pom.xml
   |
   | Maven dependency
   v
nlp4j-lucene
   |
   v
Apache Lucene

This is one of the main ideas behind the project:

Add search functionality as a Java dependency, rather than always introducing search infrastructure.

A Minimal Keyword Search Example

Here is a complete example:

import nlp4j.lucene.LocalSearch;
import nlp4j.lucene.SearchResult;

public class ExampleKeywordSearch {

    public static void main(String[] args) throws Exception {

        try (LocalSearch search = new LocalSearch("en")) {

            search.addJson("""
                {"id":"1","body":"Kyoto is a historic city in Japan."}
                """);

            search.addJson("""
                {"id":"2","body":"Nintendo is headquartered in Kyoto, Japan."}
                """);

            search.addJson("""
                {"id":"3","body":"Tokyo is the capital city of Japan."}
                """);

            search.addJson("""
                {"id":"4","body":"Paris is the capital city of France."}
                """);

            search.addJson("""
                {"id":"5","body":"Sony is a Japanese multinational company based in Tokyo."}
                """);

            search.commit();

            SearchResult[] results =
                search.search("Kyoto", 10);

            for (SearchResult r : results) {
                System.out.println(
                    r.id + " : "
                    + r.body + " : "
                    + r.score
                );
            }
        }
    }
}

The essential part is very small:

try (LocalSearch search = new LocalSearch("en")) {

    search.add(
        "1",
        "Kyoto is a historic city."
    );

    search.add(
        "2",
        "Tokyo is the capital of Japan."
    );

    search.commit();

    SearchResult[] results =
        search.search("Kyoto", 10);
}

There is no search server to start.

The search engine exists inside the Java application.

Lucene Query Syntax

For developers already familiar with Lucene or Solr, Lucene-style query expressions can also be used.

Suppose we have these documents:

try (LocalSearch search = new LocalSearch("en")) {

    search.add(
        "1",
        "Nintendo is a company based in Kyoto."
    );

    search.add(
        "2",
        "Sony is a company based in Tokyo."
    );

    search.add(
        "3",
        "Microsoft is a company based in the United States."
    );

    search.commit();

    // ...
}

We can use a Boolean query such as:

String q = "company OR Kyoto";

SearchResult[] results =
    search.searchLucene(q, 10);

Or:

String q = "company AND Kyoto";

SearchResult[] results =
    search.searchLucene(q, 10);

For developers who have used Lucene or Solr query syntax, this should look familiar.

Structured Fields

Practical search applications usually need more than text.

Documents often contain fields such as:

year
price
timestamp
category
status

LocalSearch therefore supports schema-aware fields.

For example:

try (LocalSearch search =
        LocalSearch.builder("en")
            .autoAnalyze(false)
            .field(
                "year_i",
                FieldTypeDef.integer()
                    .stored(true)
                    .aggregatable(true)
            )
            .field(
                "price_d",
                FieldTypeDef.doubleNumber()
                    .stored(true)
                    .aggregatable(true)
            )
            .build()) {

    // ...
}

Documents can then be added as JSON:

search.addJson("""
    {
      "id":"1",
      "text":"Product A",
      "year_i":2024,
      "price_d":80.0
    }
    """);

search.addJson("""
    {
      "id":"2",
      "text":"Product B",
      "year_i":2025,
      "price_d":150.0
    }
    """);

search.addJson("""
    {
      "id":"3",
      "text":"Product C",
      "year_i":2026,
      "price_d":250.0
    }
    """);

search.commit();

We can then query an exact numeric value:

SearchResult[] results =
    search.searchLucene(
        "year_i:2025",
        10
    );

Or a numeric range:

SearchResult[] results =
    search.searchLucene(
        "year_i:[2025 TO 2026]",
        10
    );

Exclusive bounds can also be expressed:

year_i:{2025 TO 2026]
year_i:[2025 TO 2026}

And the same idea works for double fields:

SearchResult[] results =
    search.searchLucene(
        "price_d:[100 TO 200]",
        10
    );

This starts to look less like a simple text-search wrapper and more like an application-level search API.

Date Range Queries

Date fields can be defined in the schema as well.

For example:

try (LocalSearch search =
        LocalSearch.builder("en")
            .autoAnalyze(false)
            .field(
                "created_dt",
                FieldTypeDef.date()
                    .stored(true)
                    .aggregatable(true)
            )
            .build()) {

    search.addJson("""
        {
          "id":"1",
          "text":"Event A",
          "created_dt":"2026-07-01"
        }
        """);

    search.addJson("""
        {
          "id":"2",
          "text":"Event B",
          "created_dt":"2026-08-19"
        }
        """);

    search.addJson("""
        {
          "id":"3",
          "text":"Event C",
          "created_dt":"2026-09-30"
        }
        """);

    search.commit();

    // ...
}

We can search only August 2026:

SearchResult[] results =
    search.searchLucene(
        "created_dt:[2026-08-01 TO 2026-09-01}",
        10
    );

Or search everything on or after July 1:

created_dt:[2026-07-01 TO *]

Open lower bounds are also possible:

created_dt:[* TO 2026-07-31T23:59:59Z]

This is useful when a search index contains application data rather than text alone.

The Index Can Be Persisted

An in-memory or temporary index is convenient for experiments.

But an application may also want to build an index once and reopen it later.

For example:

Path indexDir =
    Files.createTempDirectory("local-search-index");

try (LocalSearch search =
        LocalSearch.builder("en").build()) {

    search.add(
        "1",
        "Kyoto is a historic city."
    );

    search.add(
        "2",
        "Tokyo is the capital of Japan."
    );

    search.add(
        "3",
        "Paris is the capital of France."
    );

    search.commit();

    search.saveIndexTo(indexDir);
}

The saved index can later be reopened:

try (LocalSearch search =
        LocalSearch.builder("en")
            .loadIndexFrom(indexDir)
            .build()) {

    SearchResult[] results =
        search.search("Kyoto", 10);

    for (SearchResult r : results) {
        System.out.println(r.id + " : " + r.body);
    }
}

This can be useful when a Lucene index naturally belongs to the application itself.

Where Does This Fit Compared with Solr and OpenSearch?

I do not see this approach as competing with Solr or OpenSearch.

I use and like those technologies too.

They simply operate at a different architectural level.

A simplified view is:

Local Lucene Solr OpenSearch
Runs inside Java application Yes Usually no No
Separate server No Yes Yes
REST API required No Usually yes Yes
Distributed cluster No Yes Yes
Full-text search Yes Yes Yes
Structured search Yes Yes Yes
Lucene-based Yes Yes Yes
Good for embedded/local use Yes Different model Different model
Good for distributed search Not the goal Yes Yes

The important question is not:

Which search engine is better?

A more useful question is:

Where should the search engine live?

For a distributed web service:

Applications
      |
      v
Solr / OpenSearch Cluster

is often exactly the right architecture.

For a standalone Java application:

Application
   |
   +-- Business Logic
   |
   +-- NLP
   |
   +-- Lucene Search

can also make sense.

Both are valid designs.

What About Milvus?

Milvus addresses another important use case.

It is primarily associated with vector search and applications such as:

  • embedding search
  • semantic search
  • RAG
  • large vector collections

Lucene, however, also supports vector search in addition to its traditional strengths in:

Full-text search
Structured fields
Boolean queries
Range queries
Local indexing

So for a Java application that already needs conventional search as well as vector functionality, Lucene can be an interesting foundation.

Again, this does not make one system universally better than another.

They start from different architectural requirements.

An Interesting Middle Ground

There is an interesting space between:

grep / SQL LIKE

and:

Distributed Search Cluster

That space is:

Embedded Search Engine

For Java, Lucene is particularly interesting here because it already is a library.

For example:

+----------------------------------+
| Java Application                 |
|                                  |
| Business Logic                   |
|                                  |
| NLP / Text Analysis              |
|                                  |
| LocalSearch                      |
|    |                             |
|    +-- Keyword Search            |
|    +-- Numeric Search            |
|    +-- Date Search               |
|    +-- Aggregation               |
|    +-- Vector Search             |
|                                  |
| Apache Lucene                    |
+----------------------------------+

No network boundary is required between the application and the search engine.

Sometimes that is exactly what we want.

English Text Analysis Is Built In

Another aspect of LocalSearch is that text analysis can be integrated into the indexing process.

For example:

try (LocalSearch search = new LocalSearch("en")) {

    search.add(
        "1",
        "The developer is searching documents."
    );

    search.add(
        "2",
        "The application searched the local index."
    );

    search.add(
        "3",
        "Users can search documents without a server."
    );

    search.commit();

    SearchResult[] results =
        search.search("search", 10);

    for (SearchResult r : results) {
        System.out.println(
            r.id + " : " + r.body
        );
    }
}

The application can work with ordinary English text while the search layer handles text analysis internally.

This is particularly useful for NLP-oriented applications where:

Raw Text
   |
   v
Language Analysis
   |
   v
Lucene Index
   |
   +--> Search
   |
   +--> Aggregation
   |
   +--> Text Analytics

need to live close together.

Why Not Use Lucene Directly?

Of course, you can.

And for many Lucene developers, that is absolutely the correct choice.

If your application requires detailed control over:

  • IndexWriter
  • Directory
  • Analyzer
  • Query
  • IndexSearcher
  • collectors
  • codecs
  • scoring
  • custom query implementations

then using Lucene directly makes sense.

LocalSearch is not intended to replace the Lucene API.

Instead, it targets another situation:

I want Lucene inside my application, but I do not want every application to implement the same indexing, schema, JSON handling, query parsing, persistence, and NLP integration code.

In other words:

Apache Lucene

is the search engine library.

LocalSearch

is an application-oriented layer built on top of it.

There Is a Python API Too

The same project also exposes Lucene-based search functionality to Python.

The Python package is:

nlp4j-local-search

It can be installed with:

pip install nlp4j-local-search==0.5.0

A minimal example is:

from nlp4j_local_search import SearchEngine

with SearchEngine("en") as engine:

    engine.add(
        "1",
        "Kyoto is a historic city."
    )

    engine.add(
        "2",
        "Tokyo is the capital of Japan."
    )

    engine.commit()

    for result in engine.search(
        "Kyoto",
        limit=10
    ):
        print(
            result.id,
            result.body,
            result.score
        )

Internally, this uses the Java implementation and Apache Lucene.

So the project can be viewed roughly as:

              +----------------+
              |    Java API    |
              +-------+--------+
                      |
              +-------v--------+
              |  LocalSearch   |
              +-------+--------+
                      |
              +-------v--------+
              | Apache Lucene  |
              +----------------+

                       ^
                       |
              +--------+-------+
              |   Python API   |
              |  SearchEngine  |
              +----------------+

For Java developers, Lucene remains an embedded Java library.

For Python users, the same idea becomes available through a Python-friendly interface.

This also makes environments such as Jupyter Notebook and Google Colab interesting places for experimenting with Lucene-backed search.

When Would I Use This?

I think an embedded Lucene approach is particularly interesting for several kinds of applications.

Desktop applications

A desktop Java application can keep its own search index without asking the user to install a separate search server.

NLP applications

Analyzed text can immediately become searchable inside the same process.

Batch processing

A batch job can create a Lucene index, analyze it, and discard it afterward.

CLI tools

A command-line application can include full-text search without external infrastructure.

Automated tests

Tests can create real Lucene indexes without requiring Docker or an external service.

Proofs of concept

Search behavior can be evaluated before deciding whether distributed infrastructure is necessary.

Small internal applications

Sometimes a search cluster would simply be more infrastructure than the application requires.

Start Small, Scale When Necessary

Using an embedded Lucene index does not mean an application must always remain embedded.

A project may start as:

Java Application
      |
      v
Local Lucene Index

and later evolve into:

Applications
      |
      v
Solr / OpenSearch Cluster

when requirements change.

Those requirements may include:

More users
More data
Replication
High availability
Remote clients
Distributed indexing
Operational monitoring

At that point, a dedicated search platform becomes very attractive.

Starting with a library is therefore not necessarily a statement about the final architecture.

Sometimes it simply means:

For this application, today, a library is enough.

Conclusion

Many Java developers encounter Lucene indirectly through larger search platforms.

But Lucene itself is already a powerful Java search library.

That gives us another architectural option:

No separate search server.
No HTTP boundary.
No additional process.

Java Application
       +
   LocalSearch
       +
 Apache Lucene

nlp4j-lucene is my attempt to make this model easier to use for application development, especially where search and NLP need to live close together.

The Maven dependency is simply:

<dependency>
    <groupId>org.nlp4j</groupId>
    <artifactId>nlp4j-lucene</artifactId>
    <version>1.5.0.0</version>
</dependency>

The current implementation includes examples for:

  • keyword search
  • Lucene query syntax
  • numeric fields
  • numeric range queries
  • date fields
  • date range queries
  • index persistence
  • English text analysis
  • Japanese text analysis
  • aggregation
  • vector search

For large distributed search systems, Solr and OpenSearch are excellent choices.

For vector-database-centered systems, technologies such as Milvus are worth considering.

But when writing a Java application whose requirement is simply:

"I need search inside this application."

there is another option worth remembering:

Lucene can simply be a library dependency.

Sometimes, that is all you need.

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?