How to Build an Agentic RAG Chatbot That Avoids Hallucinations
Most AI chatbots suffer from two critical flaws: they make up answers when they don't know (hallucinations) and falsely attribute information to documents. This CDF Hackathon solution shows how to build a retrieval-augmented chatbot with persistent memory that solves both problems while guarding against prompt injection attacks.
The Hallucination Problem in AI Chatbots
Every business using AI chatbots faces the same frustrating reality - when the system doesn't know an answer, it often makes one up anyway. At 3:45 in the video demonstration, we see this exact scenario play out when asking about a project manager's name that wasn't in the documents. Traditional chatbots might invent a name, but this solution correctly responds "This document does not include information about project manager."
The second major issue is false citations - where chatbots attribute information to documents that don't actually contain it. The demonstration shows how this system handles general knowledge questions (like capitals of countries) without falsely citing uploaded documents when the information comes from its base knowledge instead.
Key insight: Hallucinations occur in 15-20% of chatbot responses according to Stanford research. This solution reduces that to near-zero through strict retrieval gating.
Solution Architecture Overview
The agentic RAG chatbot combines three key components: retrieval-augmented generation for document-based answers, persistent memory for maintaining context, and strict security measures against prompt injection. Built with Streamlit for the frontend (chosen for its rapid prototyping capabilities), the system demonstrates several innovative approaches:
- File-based memory storage for transparency and simplicity
- Separate memory streams for user context and organizational knowledge
- Visual memory display in the sidebar showing exactly what the system remembers
- Clear source citations showing which document provided each piece of information
At 7:20 in the video, we see how uploading a project info file allows the chatbot to accurately summarize project goals while properly citing the source document - a capability most business chatbots lack.
Implementing Persistent Memory
Unlike standard chatbots that forget everything when the session ends, this system maintains two types of persistent memory:
- User Memory: Stores personal context like "User is a data scientist" (shown at 5:30 in the demo)
- Company Memory: Retains organizational knowledge that could help colleagues while avoiding sensitive information
The memory implementation uses simple text files rather than a database - a deliberate tradeoff that allows direct inspection and editing of the agent's "brain." While this limits scalability, it provides crucial transparency for internal business use cases where understanding the system's knowledge is as important as the answers it provides.
Implementation tip: The memory system follows strict rules about what gets stored - personal facts go in user memory, while reusable organizational knowledge goes in company memory after vetting.
Strict Retrieval Gating Against Hallucinations
The system's most valuable feature is its strict retrieval gating mechanism that prevents hallucinations. When asked a question:
- It first attempts to retrieve relevant information from uploaded documents
- If found, it responds with the information and cites the exact source
- If not found, it explicitly states the information isn't available rather than guessing
This approach solves both hallucination and false citation problems simultaneously. At 4:15 in the video, we see this in action when asking about Microsoft's location - the system correctly stores this in company memory without falsely attributing it to any uploaded document.
Security Against Prompt Injection
Prompt injection attacks represent a growing threat to AI systems, where malicious inputs attempt to override the system's instructions. This solution includes specific defenses:
- Detection and rejection of documents containing phrases like "ignore prior instructions"
- Automatic filtering of requests that attempt to reveal secrets
- Clear "I cannot assist with that request" responses to suspicious queries
At 9:40 in the demonstration, we see the system successfully defend against a malicious.txt file containing prompt injection attempts - a critical capability for business environments where document uploads come from multiple sources.
Key Design Tradeoffs
Every architecture decision involves tradeoffs. This solution prioritized:
- Transparency over scalability: File-based memory allows direct inspection but limits multi-user support
- Accuracy over creativity: Strict retrieval gating prevents interesting but potentially false responses
- Security over convenience: Some legitimate requests might get blocked to prevent prompt injection
The Streamlit frontend choice (over more complex frameworks like React) reflects another tradeoff - faster development with fewer lines of code at the cost of some customization flexibility. As shown at 11:20, this allowed rapid implementation of the memory visualization sidebar that helps users understand the system's knowledge state.
Future Improvements
While the current implementation solves core business needs, several enhancements could make it even more powerful:
- Semantic chunking: Replace fixed-size text chunks with meaning-based segmentation for better retrieval
- Multi-user support: Migrate from file storage to a SQL database to handle multiple users simultaneously
- Fine-grained permissions: Add role-based access control for sensitive organizational knowledge
- Conversation threading: Allow following up on specific points from earlier in long conversations
These improvements would build on the already solid foundation demonstrated in the hackathon solution while maintaining its core advantages of transparency and accuracy.
Watch the Full Tutorial
See the complete implementation walkthrough including memory system demonstrations, security tests against prompt injection, and the sanity check validation system that generates JSON reports of the chatbot's capabilities.
Frequently Asked Questions
Common questions about agentic RAG chatbots
Agentic RAG chatbots combine retrieval-augmented generation with persistent memory, allowing them to maintain context across conversations while avoiding hallucinations by strictly grounding responses in retrieved documents.
This creates a more reliable and trustworthy experience for business users who need accurate information rather than creative but potentially false responses.
- Maintains context across multiple conversations
- Only provides answers when information is available
- Clearly cites sources for all document-based responses
The system implements strict retrieval gating - if information isn't found in the uploaded documents, it explicitly states that rather than making up answers. This prevents false information generation.
Unlike standard LLMs that might guess at answers, this system has been programmed to acknowledge its limitations when information isn't available in its knowledge base.
- Explicit "I don't know" responses when information is unavailable
- No false attribution to documents
- Clear distinction between general knowledge and document-based answers
The chatbot detects and rejects documents containing phrases like "ignore prior instructions" that attempt prompt injection, responding with "I cannot assist with that request" to maintain security.
This creates an important safeguard for business environments where documents might come from untrusted sources or contain malicious content attempting to override system instructions.
- Pattern matching against known injection phrases
- Conservative response policy to suspicious inputs
- Clear rejection of requests attempting to bypass safeguards
The system maintains separate memory streams - user memory for personal context and company memory for organizational knowledge. This persists across sessions via file storage.
This dual-memory approach allows the system to remember important details about individual users while also building up institutional knowledge that can benefit the entire organization.
- User memory stores personal context (e.g., "User is a data scientist")
- Company memory retains reusable organizational knowledge
- Memory persists between sessions via simple text files
File storage was chosen for transparency and simplicity, allowing direct inspection/editing of memory files. This trades scalability for lower complexity in single-user scenarios.
The decision reflects the system's initial focus as an internal tool where understanding and modifying the agent's knowledge is more important than supporting thousands of concurrent users.
- Easier debugging and manual adjustments
- No database setup or maintenance required
- Limited to single-user operation in current implementation
The chatbot only cites documents when information is actually retrieved from them. For general knowledge questions, it provides answers without false citations.
This creates a clear distinction between information pulled from uploaded documents versus the system's base knowledge, preventing misleading source attribution.
- Explicit source citations for document-based answers
- No citations provided for general knowledge responses
- Clear visual distinction between different information types
Future versions could implement semantic chunking (breaking text by meaning rather than character count) and multi-user support through database migration.
These enhancements would build on the current foundation while addressing some of the limitations inherent in the initial prototype's design choices.
- Semantic chunking for more accurate document retrieval
- Database backend for multi-user operation
- Role-based access controls for sensitive information
GrowwStacks specializes in building custom AI agent solutions with RAG architectures tailored to your specific business needs. Our team can design, implement, and deploy secure chatbot solutions with proper hallucination guards and memory systems.
We offer end-to-end implementation of agentic RAG systems including integration with your existing knowledge bases, security requirements, and user interface preferences.
- Custom RAG architecture design
- Hallucination prevention implementation
- Persistent memory system configuration
- Security against prompt injection attacks
Ready to Deploy a Hallucination-Free Chatbot for Your Business?
Every day without a reliable AI assistant costs your team productivity and risks spreading misinformation. GrowwStacks can implement a custom agentic RAG solution tailored to your specific needs in as little as 2 weeks.