0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

@oyahiroki(Hiroki Oya)

I Built `nlp4j-jsonl`, a Comfortable CLI Tool for Editing JSONL Files

0
Posted at

Note: The following article was created using a personal IBM Bob account. The content itself is original.

Introduction

In data science and NLP (Natural Language Processing) projects, we often work with data in JSONL (JSON Lines) format. Especially when preparing training datasets for large language models or building RAG (Retrieval-Augmented Generation) pipelines, editing JSONL files becomes a routine task.

However, existing tools often come with several problems:

  • 🔧 Complex dependencies — Installation can be painful
  • 💻 Programming required — Even small edits need custom scripts
  • 🐌 High memory usage — Large files are loaded entirely into memory
  • 🔒 Security concerns — Some implementations rely on dangerous eval() calls

To solve these issues, I developed a zero-dependency CLI tool called nlp4j-jsonl.

What is nlp4j-jsonl?

nlp4j-jsonl is a Python command-line tool designed to make editing JSONL files simple and safe.

Main Features

  • 🎮 Three execution modes

    • Interactive mode (REPL)
    • Batch command mode
    • Traditional CLI mode

  • 📦 Zero dependencies — Uses only Python standard libraries

  • 🔒 Safe expression evaluation — No eval(), uses AST parsing

  • 💾 Streaming processing — Efficient even for large files

  • 🛡️ Safe in-place editing — Uses temporary files for reliability

Installation

Option 1: Use Without Installation

# Clone the repository
git clone https://github.com/nlp4j/nlp4j-python.git
cd nlp4j-python/tools/jsonl-editor

# Run directly
python nlp4j-jsonl.py

Option 2: Install as a Package

cd nlp4j-python/tools/jsonl-editor
pip install .

After installation, the nlp4j-jsonl command becomes available.

Usage

1. Interactive Mode

Run the tool without arguments to start interactive editing.

$ nlp4j-jsonl
nlp4j-jsonl interactive mode
Type "help" for available commands, "exit" to quit

>> load test.jsonl
loaded successfully
current attributes: ["key1", "key2", "key3"]

>> head 2
{"key1":"value1","key2":"value2","key3":"value3"}
{"key1":"x1","key2":"x2","key3":"x3"}

>> remove key1
removed key1
current attributes: ["key2", "key3"]

>> rename key2 name
renamed key2 name
current attributes: ["name", "key3"]

>> add email 'concat(name, "@example.com")'
added email
current attributes: ["name", "key3", "email"]

>> head 1
{"name":"value2","key3":"value3","email":"value2@example.com"}

>> write output.jsonl
wrote successfully

>> exit

Available Commands

Command Description
load <file> Load a JSONL file
head [n] Show first n records (default: 5)
tail [n] Show last n records (default: 5)
count Show record count
remove <key> Remove a key
rename <old> <new> Rename a key
select <key1> [key2...] Keep only specified keys
add <key> <expr> Add a key using an expression
write <file> Save to file
show Show current attributes
help Display help
exit Exit

2. Batch Command Mode

You can execute multiple commands at once by separating them with semicolons. Great for scripting!

# Basic usage
nlp4j-jsonl -c "load test.jsonl; remove key1; write output.jsonl"

# Complex transformation pipeline
nlp4j-jsonl -c "load users.jsonl; remove id; rename first firstname; rename last lastname; add fullname 'concat(firstname, \" \", lastname)'; write users_transformed.jsonl"

# Data exploration
nlp4j-jsonl -c "load data.jsonl; count; show; head 3"

3. Traditional CLI Mode

Traditional subcommand-based CLI usage is also supported.

# Remove a key
nlp4j-jsonl remove --input test.jsonl key1 --output out.jsonl

# Rename a key (in-place with backup)
nlp4j-jsonl rename --input test.jsonl key2 key2_new --inplace --backup

# Select specific keys
nlp4j-jsonl select --input test.jsonl name email --output filtered.jsonl

# Add a key using an expression
nlp4j-jsonl add --input test.jsonl full_text "concat(title, ' ', body)" --inplace

Expression DSL (Domain Specific Language)

The add command supports a safe expression language for creating new fields.

Supported Elements

Element Example Description
Field reference name Reference a field
String literal 'text' String constant
Numeric literal 123, 3.14 Numeric constant
Boolean literal True, False Boolean value
Null literal None Null value
concat() concat(a, ' ', b) Concatenate strings
coalesce() coalesce(a, b, 'default') Return first non-null value

Expression Examples

# Concatenate fields
nlp4j-jsonl add --input data.jsonl full_text "concat(title, ' ', body)" --inplace

# Set default values
nlp4j-jsonl add --input data.jsonl display_name "coalesce(nickname, username, 'Anonymous')" --inplace

# Mix literals and fields
nlp4j-jsonl add --input data.jsonl email "concat(username, '@example.com')" --inplace

# Complex expression
nlp4j-jsonl add --input data.jsonl summary "concat('[', category, '] ', title, ' - ', coalesce(description, 'No description'))" --inplace

Practical Examples

Example 1: Transforming User Data

Input File (users.jsonl)

{"id":1,"first":"Taro","last":"Yamada","dept":"Development"}
{"id":2,"first":"Hanako","last":"Sato","dept":"Sales"}

Command

nlp4j-jsonl -c "load users.jsonl; remove id; rename first first_name; rename last last_name; add fullname 'concat(last_name, \" \", first_name)'; write users_transformed.jsonl"

Output File (users_transformed.jsonl)

{"first_name":"Taro","last_name":"Yamada","dept":"Development","fullname":"Yamada Taro"}
{"first_name":"Hanako","last_name":"Sato","dept":"Sales","fullname":"Sato Hanako"}

Example 2: Filtering Product Data

Input File (products.jsonl)

{"id":1,"name":"Widget","price":999,"stock":100,"internal_code":"W001"}
{"id":2,"name":"Gadget","price":1999,"stock":50,"internal_code":"G001"}

Command

nlp4j-jsonl select --input products.jsonl name price --output products_public.jsonl

Output File (products_public.jsonl)

{"name":"Widget","price":999}
{"name":"Gadget","price":1999}

Example 3: Preparing NLP Datasets

# Create a text field by combining title and body
nlp4j-jsonl -c "load articles.jsonl; add text 'concat(title, \". \", body)'; select text label; write training_data.jsonl"

# Handle missing values
nlp4j-jsonl -c "load data.jsonl; add description 'coalesce(description, summary, \"No description\")'; write data_cleaned.jsonl"

Technical Highlights

1. Zero Dependencies

The tool uses only Python standard libraries.

import json
import ast
import argparse
import sys
from pathlib import Path
from typing import Any, Dict, List

2. Safe Expression Evaluation

Instead of using eval(), the tool safely evaluates expressions using AST parsing.

def eval_expr(expr: str, record: Dict[str, Any]) -> Any:
    """Safe expression evaluation"""
    tree = ast.parse(expr, mode="eval")
    return _eval_node(tree.body, record)

This prevents arbitrary code execution and improves security.

3. Streaming Processing

Files are processed line by line, making the tool memory efficient.

def read_jsonl(path: Path) -> Iterator[Tuple[int, Dict[str, Any]]]:
    """Streaming reader"""
    with path.open("r", encoding="utf-8") as fp:
        for line_no, line in enumerate(fp, start=1):
            yield line_no, json.loads(line.strip())

4. Safe In-Place Editing

The --inplace option uses temporary files to safely overwrite originals.

# Write to temporary file
with tempfile.NamedTemporaryFile(...) as tmp:
    # processing
    pass

# Replace original file after success
shutil.move(tmp_path, input_path)

Architecture

The project is organized into the following modules:

src/nlp4j_jsonl/
├── __init__.py      # Version info
├── cli.py           # CLI entry point
├── commands.py      # Transformations and file processing
├── interactive.py   # REPL session
├── expr.py          # Expression engine
└── jsonl_io.py      # JSONL I/O

Each module follows the single responsibility principle for maintainability and testability.

Use Cases

1. NLP Dataset Preprocessing

nlp4j-jsonl -c "load dataset.jsonl; add text 'coalesce(content, body, description)'; select text label; write preprocessed.jsonl"

2. Preparing Data for RAG Pipelines

nlp4j-jsonl -c "load documents.jsonl; add metadata 'concat(\"source:\", source, \",date:\", date)'; write documents_with_metadata.jsonl"

3. Building Search Indexes

nlp4j-jsonl select --input full_data.jsonl id title content embedding --output index_data.jsonl

4. Data Cleansing

nlp4j-jsonl -c "load raw_data.jsonl; remove _id; remove internal_notes; remove debug_info; write clean_data.jsonl"

Performance

Thanks to streaming processing:

  • ✅ Process 1 million JSONL records (~500MB) in seconds
  • ✅ Constant memory usage regardless of file size
  • ✅ Interactive mode keeps data in memory for easy exploration

Troubleshooting

Common Issues

FileNotFoundError: Input file not found

Check that the file path is correct.

Invalid JSON at line X

Make sure each line is a valid JSON object.

ExpressionError: Unsupported function

Only concat() and coalesce() are currently supported.

Permission denied when using --inplace

Check file and directory write permissions.

Conclusion

nlp4j-jsonl is a JSONL editing tool with the following advantages:

  • 📦 Zero dependencies — Easy installation
  • 🎮 Three execution modes — Flexible workflows
  • 🔒 Safe — No dangerous eval() usage
  • 💾 Efficient — Streaming support for large files
  • 🛡️ Reliable — Safe in-place editing

If you frequently work with JSONL files in NLP or data science projects, I hope this tool helps simplify your workflow.

Give it a try!

Links

License

Apache License 2.0

If you found this article useful, I’d appreciate a ⭐ on GitHub!

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?