#166 Building a Multi-Crate IDL Generator for Solana Programs

Introduction

When developing complex Solana programs that span multiple crates, generating Interface Definition Language (IDL) files becomes a crucial step in your build process.
IDL files serve as the bridge between your on-chain programs and client applications, providing essential metadata about instructions, accounts, types, and events.

In this post, we'll explore shank-idl-generator, a new crate available on crates.io that automatically generates consolidated IDL files from multi-crate Solana programs.
We'll examine how it works, why this approach is valuable for modern Solana development, and how you can start using it today.

What Problem Does This Solve?

My current program often follow a modular architecture, splitting functionality across multiple crates:

• Core crate: Contains shared types and business logic
• SDK crate: Provides client-side utilities and helpers
• Program crate: Houses the actual on-chain instruction handlers

The Solution: A Multi-Crate IDL Aggregator

Our IDL generator solves this by:

1. Scanning multiple crates for IDL-relevant code
2. Extracting IDL components from each crate
3. Merging everything into a single consolidated IDL file
4. Writing the result as a JSON file ready for client consumption

Code Walkthrough

Configuration Structure

struct IdlConfiguration {
    program_id: String,
    name: &'static str,
    paths: Vec<&'static str>,
}

The IdlConfiguration struct defines how to process a multi-crate program:

• program_id: The on-chain program address
• name: The output IDL filename
• paths: List of crate directories to scan

Environment Setup

let envs = envfile::EnvFile::new(crate_root.join("config").join("program.env"))?;
let ncn_portal_program_id = envs
    .get("NCN_PORTAL_PROGRAM_ID")
    .ok_or_else(|| anyhow!("NCN_PORTAL_PROGRAM_ID not found"))?
    .to_string();

The program reads configuration from an environment file, keeping sensitive data like program IDs separate from the codebase. This is a best practice for managing different deployment environments (devnet, testnet, mainnet).

Multi-Crate Processing

The heart of the utility lies in its ability to process multiple crates:

for path in idl.paths {
    let cargo_toml = crate_root.join(path).join("Cargo.toml");
    let manifest = Manifest::from_path(&cargo_toml)?;
    let lib_rel_path = manifest.lib_rel_path()?;
    
    let opts = ParseIdlOpts {
        program_address_override: Some(idl.program_id.to_string()),
        ..ParseIdlOpts::default()
    };
    let idl = extract_idl(lib_full_path, opts)?;
    idls.push(idl);
}

For each crate path:

1. Locates the Cargo.toml to understand the crate structure
2. Finds the library entry point using the manifest
3. Extracts IDL components using the shank_idl crate
4. Collects results for later merging

IDL Consolidation

The most sophisticated part is merging multiple IDL files:

let mut accumulator = idls.pop().unwrap();
for other_idls in idls {
    accumulator.constants.extend(other_idls.constants);
    accumulator.instructions.extend(other_idls.instructions);
    accumulator.accounts.extend(other_idls.accounts);
    accumulator.types.extend(other_idls.types);
    
    // Handle optional fields
    if let Some(events) = other_idls.events {
        if let Some(accumulator_events) = &mut accumulator.events {
            accumulator_events.extend(events);
        } else {
            accumulator.events = Some(events);
        }
    }
}

This merging strategy:

• Combines all constants, instructions, accounts, and types from each crate
• Carefully handles optional fields like events and errors
• Preserves the structure expected by IDL consumers

Key Benefits

1. Modular Development

Teams can maintain clean separation of concerns across crates while still generating unified IDL files.

2. Automated Build Process

No manual IDL maintenance - the generator runs as part of your build pipeline.

3. Environment Management

Program IDs are externalized to environment files, supporting multiple deployment targets.

4. Type Safety

Leverages Rust's type system and the shank framework for reliable IDL extraction.

Usage in Practice

This utility would typically run as part of your build process:

# Generate IDL files
shank-idl-generator

# IDL files are written to ./idl/ directory
ls idl/
# ncn_portal.json

The generated JSON files can then be consumed by:

• TypeScript clients using @coral-xyz/anchor

Best Practices

When implementing similar IDL generation:

1. Use environment files for configuration that varies between deployments
2. Implement comprehensive error handling for file operations and IDL extraction
3. Add logging to help debug issues during generation
4. Validate the output to ensure all expected components are present
5. Version your IDL files alongside your program deployments

Conclusion

This IDL generator demonstrates how thoughtful tooling can solve the complexity that emerges from modular Solana program architecture. By automating the consolidation of IDL components across multiple crates, it enables teams to maintain clean code organization without sacrificing the developer experience for client applications.