0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@oyahiroki(Hiroki Oya)

Try Japanese Full-Text Search with Apache Lucene in Google Colab — nlp4j-local-search 0.6.1

0
Posted at

Try Japanese Full-Text Search with Apache Lucene in Google Colab — nlp4j-local-search 0.6.1

Introduction

If you want to experiment with full-text search, you may think of systems such as:

  • Elasticsearch
  • OpenSearch
  • Apache Solr

These are powerful search platforms, but sometimes you just want to try searching a local dataset without setting up a search server or Docker environment.

nlp4j-local-search is a Python package that provides a lightweight way to use Apache Lucene locally.

It can also be used through an interactive command-line interface:

nlp4j-local-search --lang ja

In this article, we will use Google Colab to try the CLI step by step.

We will:

  1. Install nlp4j-local-search
  2. Download a Japanese Wikipedia sample dataset
  3. Start the interactive CLI
  4. Load approximately 30,000 documents
  5. Run Japanese full-text searches
  6. Try phrase searches
  7. Search structured fields
  8. Run date-range queries
  9. Explore categories with view()

No Elasticsearch, OpenSearch, Solr, or Docker server is required.

About nlp4j-local-search

nlp4j-local-search is a Python interface for running local search with Apache Lucene.

The basic architecture looks like this:

Python / CLI
     |
     v
nlp4j-local-search
     |
     v
Java
     |
     v
Apache Lucene

The user works mainly with Python or the CLI, while Apache Lucene is used internally as the search engine.

PyPI:

https://pypi.org/project/nlp4j-local-search/0.6.1/

GitHub:

https://github.com/oyahiroki/nlp4j-local-search

In this tutorial, we will use version 0.6.1.

Step 1 — Open Google Colab

Create a new notebook in Google Colab.

A GPU is not required for this tutorial.

The standard CPU runtime is sufficient for trying the Lucene-based text search features shown here.

Step 2 — Install nlp4j-local-search

Run the following cell:

!pip install -q nlp4j-local-search==0.6.1 pexpect

Then confirm that the CLI command is available:

!nlp4j-local-search --help

You should see the command-line help.

Step 3 — Check Java

nlp4j-local-search uses Java and Apache Lucene internally.

You can check the Java runtime available in Colab:

!java -version

You should see output similar to:

openjdk version ...

You do not need to write Java code yourself.

The Java layer is used internally by nlp4j-local-search.

Step 4 — Download the Japanese sample dataset

For this tutorial, we will use a sample dataset generated from the Japanese Wikipedia dump.

The dataset contains manga-related Wikipedia articles.

Download it with wget:

!wget -q \
  https://nlp4j-2.sakura.ne.jp/data/wiki/jawiki-20260801-pages-articles_compact_manga.jsonl.gz \
  -O /content/jawiki_manga.jsonl.gz

Check the downloaded file:

!ls -lh /content/jawiki_manga.jsonl.gz

The file is gzip-compressed JSON Lines.

nlp4j-local-search can read .jsonl.gz directly, so you do not need to decompress it first.

Step 5 — Inspect the data

Let's look at the first few records.

import gzip

with gzip.open(
    "/content/jawiki_manga.jsonl.gz",
    "rt",
    encoding="utf-8",
) as f:
    for _ in range(3):
        print(f.readline())

The dataset contains fields such as:

id
title_s
text_ja
timestamp_dt
category_s

For example:

  • text_ja — Japanese full-text field
  • category_s — keyword/string field
  • timestamp_dt — date field

These field names can later be used directly in Lucene queries.

Step 6 — Start the nlp4j-local-search CLI

In a normal terminal, you would start the CLI like this:

nlp4j-local-search --lang ja

The --lang ja option tells nlp4j-local-search to use Japanese text processing.

In Google Colab, we want the CLI process to stay alive across multiple cells.

For that, we can use pexpect.

Run:

import pexpect

cli = pexpect.spawn(
    "nlp4j-local-search --lang ja",
    encoding="utf-8",
    timeout=120,
)

cli.expect(">> ")
print(cli.before)

You should see something similar to:

nlp4j-local-search
Language: ja
Auto analyze: False
Type 'help' or '?' for help.

The CLI process is now running in the background.

Step 7 — Create a helper function

Let's create a small helper so that we can send commands to the CLI from individual Colab cells.

def run_cli(command, timeout=120):
    print(f">> {command}")
    cli.sendline(command)
    cli.expect(">> ", timeout=timeout)
    print(cli.before)

Now we can execute CLI commands like this:

run_cli('search("京都")')

This makes the notebook easy to experiment with step by step.

Step 8 — Show the CLI help

Run:

run_cli("help")

The CLI provides commands such as:

load(...)
fields
aggregatable_fields
count
search(...)
view(...)
exit

The syntax intentionally looks similar to Python function calls.

Step 9 — Load the Japanese Wikipedia dataset

Now load the sample file into Lucene:

run_cli(
    'load("/content/jawiki_manga.jsonl.gz")',
    timeout=300,
)

You should see output similar to:

Loaded 30,956 documents in ... seconds (... docs/sec).

The exact time depends on the Colab environment.

At this point, the documents have been indexed by Apache Lucene and are ready to search.

Step 10 — Check the document count

Run:

run_cli("count")

For this sample dataset, the result should be around:

30,956

Now we can start searching.

Step 11 — Try a Japanese full-text search

Let's search for the manga artist Rumiko Takahashi:

run_cli(
    'search("高橋留美子", 5)'
)

The result contains Lucene scores and document text.

Example:

[... ] score=...
『...』は、高橋留美子による日本の漫画。...

[... ] score=...
...

The result IDs and scores depend on the indexed data.

This is not simply equivalent to:

"高橋留美子" in text

The search is performed through Apache Lucene using the Japanese text field configuration.

Step 12 — Change the search term

One advantage of using Colab is that you can copy a cell and change only the query.

For example:

run_cli(
    'search("漫画家", 5)'
)

Try another one:

run_cli(
    'search("少年漫画", 5)'
)

You can immediately compare the results.

This interactive style works well for learning search behavior.

Step 13 — Inspect the indexed fields

Run:

run_cli("fields")

You may see fields such as:

id
body
text
text_en
text_ja
category_s
title_s
timestamp_dt
timestamp_year_i
timestamp_month_i
timestamp_day_i
...

These fields can be referenced directly in Lucene queries.

Step 14 — Try a Japanese phrase search

Apache Lucene supports phrase queries.

For example, search for the phrase:

週刊少年サンデー

Run:

run_cli(
    """search('text_ja:"週刊少年サンデー"', 5)"""
)

The query:

text_ja:"週刊少年サンデー"

means:

Search the Japanese text field text_ja for the phrase "週刊少年サンデー".

This is one example of using Lucene Query Parser syntax directly from the CLI.

Step 15 — Search a structured field

The sample dataset also contains Wikipedia categories.

Let's search for documents in the category:

恋愛漫画

Run:

run_cli(
    'search("category_s:恋愛漫画", 10)'
)

Here:

category_s:

specifies the field to search.

This is different from normal full-text search.

category_s is treated as a structured keyword field.

Step 16 — Combine full-text search and structured fields

Lucene queries can combine multiple conditions.

For example:

run_cli(
    """search('text_ja:"高橋留美子" AND category_s:恋愛漫画', 10)"""
)

This means:

Japanese text contains "高橋留美子"
AND
category is 恋愛漫画

This is one of the useful properties of Lucene:

full-text search
+
structured field search

can be expressed in the same query.

Step 17 — Try OR queries

You can also use OR.

For example:

run_cli(
    """search('category_s:恋愛漫画 OR category_s:ギャグ漫画', 10)"""
)

Lucene Query Parser syntax supports operators such as:

AND
OR
NOT

Colab makes it easy to duplicate the cell and experiment with different combinations.

Step 18 — Try a date-range query

The sample data contains a date field:

timestamp_dt

Let's search for documents dated from January 1, 2026 onward:

run_cli(
    'search("timestamp_dt:[2026-01-01 TO *]", 10)'
)

The Lucene range:

[2026-01-01 TO *]

means:

2026-01-01 or later

So the same search interface can handle both text and date conditions.

Step 19 — Search a derived year field

The dataset also exposes derived integer fields such as:

timestamp_year_i

For example:

run_cli(
    'search("timestamp_year_i:[2020 TO 2026]", 10)'
)

This searches for documents whose year is between 2020 and 2026.

With nlp4j-local-search, one Lucene index can contain fields such as:

Text
Keyword
Integer
Date

and they can be queried together.

Step 20 — Show aggregatable fields

Now let's look at another feature.

Run:

run_cli(
    "aggregatable_fields"
)

This shows fields that can be explored using view().

For this dataset, one particularly useful field is:

category_s

Step 21 — Explore popular Wikipedia categories

Run:

run_cli(
    'view("category_s", 20)'
)

You may see results similar to:

Rank  Value                  Count
----  ------------------- --------
   1  日本の漫画家               7007
   2  存命人物                 7005
   3  生年未記載                2644
   4  継続中の作品               1857
   5  恋愛漫画                 1639
...

This is not a search result list.

Instead, it answers a different question:

What kinds of values are common in this field?

So the same Lucene index can also be used for simple data exploration.

Step 22 — Filter rare values

The third argument of view() can be used as a minimum count.

For example:

run_cli(
    'view("category_s", 20, 3)'
)

This removes values that occur only a very small number of times.

It can be useful when a field contains many categories.

Step 23 — Explore categories related to a search query

view() can also be combined with a Lucene query.

For example:

run_cli(
    """view("category_s", 'text_ja:"高橋留美子"', 20, 3)"""
)

Conceptually, this performs:

1. Search for documents containing "高橋留美子"
2. Collect their category_s values
3. Compare their occurrence with the full dataset

Normal search tells us:

Which documents matched?

view() can help answer:

What characteristics are common among the matched documents?

This makes it possible to use the Lucene index not only for retrieval but also for lightweight text exploration.

Step 24 — Try your own Japanese queries

Once everything is working, try modifying the queries.

For example:

run_cli(
    'search("category_s:SF漫画", 10)'
)

or:

run_cli(
    'search("category_s:少女漫画", 10)'
)

or:

run_cli(
    """search('text_ja:"タイムスリップ"', 10)"""
)

You can also combine conditions:

run_cli(
    """search('text_ja:"タイムスリップ" AND category_s:恋愛漫画', 10)"""
)

This is where the notebook format becomes especially useful.

You can change one part of the query, rerun the cell, and immediately inspect the result.

Step 25 — Exit the CLI

Because exit terminates the CLI process instead of returning another >> prompt, send it directly:

cli.sendline("exit")
cli.expect(pexpect.EOF)
print(cli.before)

You should see:

bye

The interactive search session is now closed.

The complete Colab workflow

The overall workflow is very small.

1. Install

!pip install -q nlp4j-local-search==0.6.1 pexpect

2. Download sample data

!wget -q \
  https://nlp4j-2.sakura.ne.jp/data/wiki/jawiki-20260801-pages-articles_compact_manga.jsonl.gz \
  -O /content/jawiki_manga.jsonl.gz

3. Start the CLI

import pexpect

cli = pexpect.spawn(
    "nlp4j-local-search --lang ja",
    encoding="utf-8",
    timeout=120,
)

cli.expect(">> ")
print(cli.before)

4. Define the helper

def run_cli(command, timeout=120):
    print(f">> {command}")
    cli.sendline(command)
    cli.expect(">> ", timeout=timeout)
    print(cli.before)

5. Load the data

run_cli(
    'load("/content/jawiki_manga.jsonl.gz")',
    timeout=300,
)

6. Search Japanese text

run_cli(
    'search("高橋留美子", 5)'
)

7. Search a structured field

run_cli(
    'search("category_s:恋愛漫画", 10)'
)

8. Search by date

run_cli(
    'search("timestamp_dt:[2026-01-01 TO *]", 10)'
)

9. Explore categories

run_cli(
    'view("category_s", 20)'
)

That is enough to start experimenting with Apache Lucene search in Google Colab.

What about English documents?

The sample in this article uses Japanese Wikipedia data, so we started the CLI with:

nlp4j-local-search --lang ja

nlp4j-local-search can also be used with English data.

For an English dataset, start the CLI with:

nlp4j-local-search --lang en

In Colab, that would be:

cli = pexpect.spawn(
    "nlp4j-local-search --lang en",
    encoding="utf-8",
    timeout=120,
)

Then an English text field such as:

text_en

can be queried with Lucene syntax.

For example:

search("artificial intelligence")

or:

search('text_en:"natural language processing"')

So the language option determines the language mode used by the search engine:

--lang ja    Japanese
--lang en    English

The tutorial uses Japanese simply because the provided sample dataset is Japanese.

Why use Apache Lucene directly?

Apache Lucene is the underlying search library used by many well-known search platforms.

However, if your goal is something like:

Load a JSONL file
↓
Create a local index
↓
Try some queries
↓
Inspect the results

running a full search server may be more infrastructure than you need.

nlp4j-local-search aims to make this smaller workflow easy:

JSONL
  ↓
Python
  ↓
Apache Lucene

In this tutorial, we only needed:

pip install
↓
wget
↓
load()
↓
search()

to search tens of thousands of Japanese Wikipedia documents with Lucene.

Why Google Colab works well for this

Search experiments are naturally interactive.

You might first try:

search("高橋留美子")

Then wonder:

What happens if I restrict the category?

So you try:

search('text_ja:"高橋留美子" AND category_s:恋愛漫画')

Then you may want to inspect the dataset itself:

view("category_s", 20)

A notebook is convenient because every step stays visible.

You can:

  • change one query
  • rerun one cell
  • compare outputs
  • keep notes next to the experiments

This makes Google Colab a convenient environment for learning Lucene query behavior and exploring text datasets.

Summary

In this article, we used nlp4j-local-search 0.6.1 from Google Colab to experiment with Japanese full-text search using Apache Lucene.

We tried:

  • Apache Lucene-based local search
  • Japanese full-text search
  • Google Colab
  • interactive CLI usage
  • .jsonl.gz loading
  • Lucene Query Parser syntax
  • phrase search
  • field-specific search
  • AND and OR
  • date-range queries
  • numeric-range queries
  • category exploration with view()

No Elasticsearch, OpenSearch, Solr, or Docker server was required.

For Japanese data:

nlp4j-local-search --lang ja

For English data:

nlp4j-local-search --lang en

If you want to experiment with Lucene from Python or simply search a local JSONL dataset without building a search server first, nlp4j-local-search may be useful as a small search environment.

Links

PyPI

https://pypi.org/project/nlp4j-local-search/0.6.1/

GitHub

https://github.com/oyahiroki/nlp4j-local-search

Japanese Wikipedia sample dataset

https://nlp4j-2.sakura.ne.jp/data/wiki/jawiki-20260801-pages-articles_compact_manga.jsonl.gz

Qiita tags

Python
Lucene
NLP
FullTextSearch
GoogleColab
0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?