Building an Agentic RAG System for Technical Troubleshooting in the Automotive Industry
Introduction
Enterprise IT support in the automotive industry can be surprisingly complex.
Modern automotive companies depend on many interconnected systems, including business applications, databases, workload schedulers, infrastructure platforms, and operational tools.
When a serious incident occurs, support engineers must identify which component is responsible, locate the relevant technical information, and determine the appropriate response.
Traditional document search and FAQ systems can help with known issues. However, they are often less effective when a question spans multiple technical domains or when the exact problem has not been documented as a single known case.
In this article, I introduce an anonymized proof of concept in which we developed an agentic retrieval-augmented generation system, or agentic RAG, for an automotive-related enterprise.
The objective was not simply to build another chatbot. It was to explore whether multiple specialized AI agents could collaborate to support complex technical investigations.
This article intentionally omits the customer name, confidential data, internal system names, commercial information, and project-specific implementation details.
1. The Business Challenge
The customer operated a large and complex enterprise IT environment supporting business-critical automotive operations.
When an incident occurred, the investigation could involve multiple technical areas, such as:
- Business applications
- Database systems
- Workload scheduling
- Infrastructure components
- Product manuals and technical specifications
- Historical support records
- Troubleshooting knowledge held by experienced engineers
The main challenge was that the required knowledge was distributed across different documents and technical domains.
A question that initially appeared to be a database problem could actually be related to workload scheduling, application behavior, infrastructure configuration, or the interaction between several components.
This created several practical difficulties.
1.1 Knowledge was fragmented
Technical information existed in manuals, specifications, support documents, and other knowledge sources.
Engineers had to search several repositories and then interpret the relationships between the results.
1.2 Troubleshooting depended on experienced specialists
Experienced engineers could often identify useful search terms, recognize technical relationships, and decide which component should be investigated next.
Less experienced engineers did not always know:
- Which repository to search
- Which technical terms to use
- Which product area was responsible
- Whether additional investigation was required
- When to escalate the issue to another specialist
1.3 Conventional RAG was not always sufficient
A conventional RAG application typically performs the following sequence:
- Receive a question.
- Search a predefined knowledge base.
- Retrieve relevant passages.
- Send the passages to a large language model.
- Generate an answer.
This works well when the question and the knowledge source are relatively clear.
However, technical troubleshooting often requires multiple steps. The system may first need to determine the relevant domain, generate several search queries, inspect results from different sources, and revise the investigation plan.
That requirement led us to an agent-based architecture.
2. Why Agentic RAG?
We defined agentic RAG as a system in which AI agents do more than retrieve a fixed set of documents.
An agent can:
- Interpret the user's objective
- Decide which technical area should be investigated
- Select an appropriate search tool
- Generate or reformulate search queries
- Review the retrieved information
- Ask another specialized agent for assistance
- Combine evidence from multiple sources
- Produce a grounded response
The important difference is that the investigation path is not completely fixed in advance.
For example, consider the following question:
A batch process stopped after a database-related warning. What should we investigate?
A basic RAG system might search all documents using the original question.
An agentic RAG system can take a more structured approach:
- A routing agent analyzes the question.
- It identifies database and workload scheduling as possible domains.
- A database specialist agent searches database manuals and support information.
- A workload scheduler specialist agent searches operational documents.
- Each agent returns relevant evidence.
- The routing agent compares the findings.
- The system prepares a consolidated answer with supporting sources.
This approach resembles how a human technical support team collaborates.
3. Conceptual Architecture
The proof of concept used a multi-agent architecture consisting of a routing agent and multiple domain-specific agents.
User
|
v
User Interface
|
v
Routing Agent
|
+------------------------+
| |
v v
Database Specialist Workload Scheduler Specialist
Agent Agent
| |
v v
Database Knowledge Scheduler Knowledge
| |
+-----------+------------+
|
v
Grounded Answer
3.1 Routing agent
The routing agent acts as the first point of contact.
Its responsibilities include:
- Understanding the user's question
- Identifying the relevant technical domains
- Selecting one or more specialist agents
- Coordinating the investigation
- Combining the returned evidence
- Presenting the final answer
The routing agent does not need to contain all domain knowledge itself. Its main responsibility is to determine how the investigation should proceed.
3.2 Domain-specific agents
Each specialist agent behaves like an AI expert for a particular technical domain.
A specialist agent has access to the documents and search tools relevant to its area. For example:
- A database agent searches database manuals and support information.
- A workload scheduler agent searches scheduling manuals and operational documents.
- An application agent searches application specifications and troubleshooting guides.
- A source-code agent analyzes source code and related design documents.
This separation improves maintainability because knowledge sources, prompts, search strategies, and evaluation criteria can be managed independently for each domain.
3.3 Knowledge sources
The agents can retrieve information from sources such as:
- Product manuals
- Technical specifications
- Support documents
- Operational procedures
- Historical inquiry records
- Application design documents
- Source code
- Public technical information
The knowledge sources available to each agent should be selected according to its responsibility.
4. The Role of Hybrid Search
Technical documents contain product names, commands, error codes, abbreviations, version numbers, and configuration parameters.
Vector search is useful for finding semantically similar passages, but exact keyword matching remains important in technical support.
For example, the following elements may require lexical matching:
- Error codes
- Command names
- Parameter names
- Product identifiers
- Log messages
- Version-specific expressions
For this reason, we evaluated a hybrid search approach that combines semantic and lexical retrieval.
User Question
|
+--> Query analysis and reformulation
|
+--> Vector search
|
+--> Keyword or field-based search
|
+--> Result filtering and ranking
|
+--> Context passed to the LLM
The purpose of hybrid search was not simply to retrieve more documents.
It was to retrieve a smaller and more useful set of passages that the language model could use as evidence.
This is particularly important in enterprise RAG systems. A powerful language model cannot produce a reliable, organization-specific answer if the retrieval layer does not find the correct information.
5. Technology Considerations
The architecture was designed to combine agent orchestration, application logic, enterprise search, vector retrieval, and foundation models.
Depending on the deployment environment and customer requirements, relevant components can include:
- IBM watsonx Orchestrate for agent orchestration
- IBM watsonx.ai for foundation-model inference
- IBM Watson Discovery for document search and text analysis
- IBM watsonx.data and Milvus for vector retrieval
- Python or Node.js for application logic and tool implementation
- IBM Cloud Object Storage for storing source data and processed artifacts
- React or another web framework for the user interface
The proof of concept was not designed around the assumption that every IBM product had to be used.
Instead, each component needed a clear responsibility:
- The LLM generates and synthesizes language.
- The retrieval components locate relevant evidence.
- The agents plan and coordinate tasks.
- The application layer controls data processing and integration.
- The user interface presents the answer and its supporting information.
This separation of responsibilities also makes it easier to replace or extend individual components.
6. From Documents to an AI-Accessible Knowledge Base
Building the knowledge base was as important as building the agents.
The source documents included technical manuals, specifications, and support information. These documents could not simply be uploaded without considering their structure and intended use.
The data preparation process included the following activities.
6.1 Understanding the documents
Before designing the index, we examined:
- Document types
- Document structures
- Headings and sections
- Product and version metadata
- Relationships between documents
- Technical terms and abbreviations
- Expected user questions
This activity helped us determine how the documents should be divided, enriched with metadata, and assigned to specialist agents.
6.2 Dividing documents into searchable units
Chunking must preserve enough context for a retrieved passage to be useful.
If chunks are too small, important technical relationships may be lost. If they are too large, a search result may contain too much unrelated information.
For technical documents, headings, product names, versions, and surrounding sections can be important metadata.
A chunk should ideally provide enough information for the system to understand:
- What product or component the passage describes
- Which version it applies to
- Which operation or error condition is involved
- Where the passage appears in the original document
6.3 Creating embeddings and indexes
The processed content was prepared for semantic retrieval using an embedding model and a vector store.
At the same time, searchable fields were maintained for exact terms such as commands, error messages, product identifiers, and configuration parameters.
This made it possible to combine semantic retrieval with keyword-based retrieval.
6.4 Connecting knowledge to specialist agents
Each agent was connected to the knowledge relevant to its technical responsibility.
This reduced unnecessary searching and allowed us to configure different retrieval strategies for different domains.
For example, an agent responsible for database investigation could use database-specific metadata and terminology, while a workload scheduling agent could use a different index and search strategy.
7. Handling Unknown or Multi-Domain Questions
One of the project's key themes was how to investigate situations that were not already represented as a known question-and-answer pair.
We treated these as questions that could not be answered by simply looking up an existing support record.
An unknown issue does not necessarily mean that no relevant knowledge exists. The necessary information may be distributed across several sources.
For example:
- One manual explains the error condition.
- Another document explains a configuration dependency.
- A support document describes a similar symptom.
- A product specification explains the relationship between two components.
The agentic approach allows the system to search these sources step by step and construct a grounded response.
A simplified investigation flow may look like this:
1. Interpret the user's question
2. Identify possible technical domains
3. Select specialist agents
4. Generate domain-specific search queries
5. Retrieve relevant passages
6. Compare evidence from multiple sources
7. Identify confirmed facts and possible hypotheses
8. Generate a grounded response
9. Escalate to a human specialist when necessary
This does not mean that the AI should make unrestricted assumptions.
The agents must distinguish between:
- Information explicitly supported by retrieved documents
- A technical hypothesis derived from multiple pieces of evidence
- Information that could not be confirmed
That distinction is essential for enterprise use.
8. Example Investigation Flow
Suppose a user submits the following question:
A scheduled process failed after a database connection error. What should I check?
The routing agent may identify two relevant domains:
- Database
- Workload scheduling
The routing agent then sends focused questions to the specialist agents.
Database specialist agent
The database agent may investigate:
- The meaning of the database error
- Connection-related settings
- Authentication conditions
- Resource or capacity conditions
- Product-version-specific restrictions
Workload scheduler specialist agent
The scheduling agent may investigate:
- Job dependencies
- Execution timing
- Retry behavior
- Credential configuration
- The propagation of return codes
- Relevant operational procedures
Consolidation by the routing agent
The routing agent compares the results and generates an answer such as:
- Confirm the exact database error and timestamp.
- Check whether the scheduler used the expected credentials.
- Compare the job execution time with the database availability period.
- Review the relevant database and scheduler documentation.
- Escalate the case if the available evidence is insufficient.
The actual response should include citations or links to the retrieved source passages.
The routing agent should also clearly identify which parts of the response are confirmed and which parts remain hypotheses.
9. Human Evaluation Was Part of the Design
The proof of concept included evaluation by customer-side technical specialists.
We did not assume that a fluent answer was automatically a correct or useful answer.
The evaluation focused on questions such as:
- Did the system select the appropriate technical domain?
- Did it retrieve relevant documents?
- Did the answer reflect the retrieved evidence?
- Did the system identify multiple technical possibilities when necessary?
- Did it avoid presenting unsupported assumptions as facts?
- Was the response helpful for determining the next investigation step?
For high-impact technical support, a useful system should also be able to say:
The available information is not sufficient to determine the cause.
This is preferable to producing an unsupported but confident answer.
Human evaluation was therefore not only a final acceptance activity. It was also part of the iterative development process.
Feedback from domain experts helped improve:
- Agent instructions
- Document selection
- Search queries
- Chunking strategies
- Metadata design
- Answer structure
- Escalation conditions
10. What We Learned
Lesson 1: Retrieval quality is more important than answer fluency
Users often focus first on the language model.
In practice, enterprise RAG quality depends heavily on whether the system retrieves the correct information.
A fluent answer based on irrelevant passages is still an incorrect answer.
Lesson 2: Domain decomposition improves extensibility
Separating agents by technical domain made the architecture easier to expand.
A new technical area could be added as another specialist agent without redesigning the entire system.
This also allowed each domain to have its own:
- Knowledge sources
- Search configuration
- Agent instructions
- Tools
- Evaluation criteria
Lesson 3: Keyword search still matters
Technical support contains many strings for which semantic similarity alone is insufficient.
Hybrid retrieval is particularly useful when documents contain:
- Exact error codes
- Commands
- Product names
- Configuration values
- Log messages
- Version identifiers
Lesson 4: Agent autonomy should be controlled
More autonomous behavior is not automatically better.
The available tools, permitted data sources, number of retries, and possible actions should be explicitly defined.
Investigation logs should also be retained so that the system's behavior can be reviewed.
Lesson 5: Evaluation data must reflect real work
Generic questions are not enough to evaluate an enterprise system.
The most useful test cases are based on actual investigation patterns, including:
- Ambiguous questions
- Incomplete information
- Cross-domain issues
- Questions with several possible causes
- Questions that cannot be answered from the available sources
- Cases that should be escalated to a human expert
Lesson 6: A PoC should validate an architecture, not just a demonstration
A visually impressive chatbot does not prove that the architecture can support real operations.
A useful proof of concept should evaluate:
- Data preparation
- Retrieval quality
- Agent routing
- Answer grounding
- Failure behavior
- Maintainability
- Extensibility
- Security requirements
- Operational requirements
11. Considerations for Production Use
A successful proof of concept is only the beginning.
Before introducing an agentic RAG system into production, the following areas should be considered.
11.1 Access control
The system must ensure that users and agents can retrieve only the information they are authorized to access.
11.2 Source traceability
Users should be able to identify which documents or passages support an answer.
11.3 Logging and monitoring
The system should retain sufficient information to review:
- User questions
- Selected agents
- Search queries
- Retrieved passages
- Generated answers
- Errors and failed tool calls
11.4 Knowledge maintenance
Technical documents change over time.
The production process must define how documents are added, updated, reprocessed, versioned, and removed.
11.5 Human escalation
The system should define conditions under which an issue must be transferred to a human specialist.
Examples include:
- Insufficient evidence
- Conflicting documents
- High-impact operational decisions
- Security-related questions
- Actions that could affect production systems
11.6 Continuous evaluation
Retrieval quality and answer quality should be evaluated continuously as documents, products, and user behavior change.
12. Outcome
The proof of concept demonstrated that a multi-agent RAG architecture could provide useful responses for complex, multi-step technical questions.
It also showed that a routing agent and domain-specific agents could provide a flexible foundation for future expansion.
The architecture can potentially be extended by adding agents for:
- Application investigation
- Source-code analysis
- Incident classification
- Technical support history
- Infrastructure diagnostics
- Operational procedures
- Test-case generation
- Voice-based support interfaces
The project also reinforced an important principle:
Enterprise AI is not only about selecting a powerful language model. It is about connecting models to the right knowledge, tools, workflows, controls, and human experts.
Conclusion
In complex automotive IT environments, technical investigation rarely belongs to a single knowledge domain.
Agentic RAG offers a practical way to coordinate multiple specialized knowledge sources while preserving the transparency and grounding required for enterprise use.
The most important design principles from this project were:
- Start with the real investigation workflow.
- Separate responsibilities among specialized agents.
- Combine semantic and lexical retrieval.
- Treat document preparation as a major engineering activity.
- Evaluate answers with domain experts.
- Design explicitly for uncertainty and escalation.
- Keep the architecture modular so that new agents and technologies can be added later.
Agentic RAG should not be viewed simply as an advanced chatbot.
It is better understood as a framework for representing how specialists investigate problems, access knowledge, collaborate, and make evidence-based decisions.
References
- RAG development with IBM watsonx.ai
- IBM watsonx RAG pattern documentation
- Building context-aware AI agents with watsonx Orchestrate
Disclaimer
The views and opinions expressed in this article are those of the author and do not necessarily reflect the official policy or position of IBM.
The architecture described in this article is a generalized and anonymized example based on project experience. It does not disclose any customer's confidential information, internal system configuration, or contractual details.
Product features and service availability may change over time. Please refer to the official product documentation for the latest information.