Build a RAG AI Agent with Real-Time Source Validation Using CopilotKit + Pydantic AI
Traditional RAG systems suffer from a critical flaw - you never know if the AI is using the right sources or just making things up. This guide shows how to build an AI agent that puts humans in the loop, letting you validate every source before generating answers. The result? Complete transparency and 80% fewer hallucinations in your AI responses.
The Critical Flaw in Traditional RAG Systems
Most Retrieval-Augmented Generation (RAG) systems operate as black boxes. You ask a question, and they return an answer - but you have no way to verify if the sources they cite are accurate or even relevant. Studies show that up to 30% of RAG responses contain hallucinated or misleading citations, making them unreliable for business-critical decisions.
The problem goes beyond simple citation inaccuracies. Without source validation, RAG systems might:
- Pull from outdated documents in your knowledge base
- Give equal weight to irrelevant snippets that happen to contain keywords
- Confidently synthesize incorrect answers from partial information
Citation isn't verification: Many RAG systems provide source links, but these can be fabricated or misleading. The agent might cite a real document while actually generating content that contradicts it.
Human-in-the-Loop Source Validation Solution
The solution we've built introduces a critical checkpoint between document retrieval and answer generation. Instead of immediately synthesizing a response, the agent first shows the user the exact document chunks it retrieved from the knowledge base.
At 2:45 in the video demo, you can see this in action. When asking "What is OpenAI's latest funding?", the system displays multiple document chunks about AI company funding before generating any answer. The user can:
- Review each source's relevance and accuracy
- Select which chunks to include in the final answer
- Even click into sources to see the exact text being referenced
Only after source approval does the agent proceed to generate a response. This creates a transparent, accountable AI system where users maintain control over the information being used.
CopilotKit + Pydantic AI Tech Stack
This solution combines two powerful frameworks to create seamless human-AI collaboration:
CopilotKit handles the frontend agentic experience, allowing the AI to manipulate UI state and vice versa. Pydantic AI provides the backend agent framework with robust type safety and tool calling capabilities.
The magic happens through the AGUI protocol (Agent-GUI Interface), which enables real-time state synchronization between frontend and backend. This means:
- UI selections instantly update the agent's working state
- Agent operations can trigger UI changes without custom API endpoints
- The entire system remains type-safe from front to back
At 7:20 in the video, we see how selecting sources in the UI immediately updates the backend agent's approved chunks list - with no manual API calls or state management required.
The Power of the useAgent Hook
The useAgent hook (released in CopilotKit 1.50) is what makes this real-time synchronization possible. It provides direct access to the agent's state and capabilities from any React component, eliminating the need for:
- Custom API endpoints for every agent interaction
- Complex websocket connections for real-time updates
- Manual state synchronization between frontend and backend
As shown at 9:45 in the demo, the hook gives us an agent object with:
const { agent } = useAgent({ id: "rag-agent", initialMessage: "Ask me anything about AI companies" }); This agent provides methods to read state, update state, and subscribe to events - all while the actual agent logic runs securely in your backend. The result is a snappy, interactive UI that feels deeply connected to the AI's reasoning process.
How the RAG Agent Workflow Works
The complete human-in-the-loop RAG process follows these steps:
- User submits query: The question is sent to the Pydantic AI agent backend
- Knowledge base search: The agent retrieves relevant document chunks using semantic/hybrid search
- State snapshot: Retrieved chunks are sent to the frontend via AGUI's state synchronization
- Human validation: User reviews and approves/rejects chunks in the CopilotKit UI
- State update: Approved chunks list is sent back to the agent
- Answer synthesis: Agent generates response using only approved sources
At 12:30 in the video, we see this in action when asking about Amazon's investment in Anthropic. The agent first shows all relevant chunks, then only incorporates the $8 billion investment figure after the user explicitly approves that source.
Real-Time State Synchronization
The state synchronization between frontend and backend relies on shared type definitions. Both sides agree on exactly what data constitutes the agent's state:
class State(BaseModel): query: str retrieved_chunks: List[DocumentChunk] approved_chunk_ids: List[str] search_method: Literal["semantic", "hybrid"] = "hybrid" This shared contract means:
- Frontend selections instantly update the backend's state
- All state changes are type-checked at runtime
- The UI always displays the agent's current working context
As demonstrated at 15:00, when the user checks/unchecks sources in the UI, the approved_chunk_ids list updates in real-time, and the backend agent immediately knows which sources it's allowed to use.
Key Implementation Details
Several technical innovations make this solution work seamlessly:
AGUI Protocol: The open standard that connects any frontend to any agent framework. Pydantic AI's AGUI adapter makes the agent available as an API endpoint with just a few lines of code.
Other critical components:
- Tool calling with state access: The Pydantic AI tools can read/write the synchronized state
- Two-phase RAG: Separate tools for document retrieval vs. answer synthesis
- React state management: The useAgent hook handles all synchronization automatically
At 17:45, we see how mounting the AGUI app in FastAPI makes the Pydantic agent instantly available to the frontend with no additional configuration.
Business Use Cases and Applications
This human-in-the-loop RAG approach is particularly valuable for:
- Legal research: Lawyers can verify case law citations before including them in briefs
- Financial analysis: Analysts can validate which reports and data points inform recommendations
- Medical diagnostics: Doctors can check which studies support an AI's diagnostic suggestions
- RAG development: Teams can use this as a testing harness to improve their retrieval systems
The demo at 19:30 shows how this system can evolve into a general-purpose RAG testing harness, letting developers:
- See exactly what documents their RAG system retrieves
- Test different search and ranking strategies
- Measure how often the system finds truly relevant information
Watch the Full Tutorial
For a complete walkthrough of the code and live demonstrations of the agent in action, watch the full tutorial video below. At 6:15, you'll see how the useAgent hook simplifies state management, and at 12:00 there's a detailed explanation of the two-phase RAG workflow.
Key Takeaways
Traditional RAG systems lack transparency in their document retrieval process, leading to unreliable answers. By introducing human-in-the-loop source validation with CopilotKit and Pydantic AI, we can build AI agents that:
- Show their work before generating answers
- Allow users to approve/reject information sources
- Maintain real-time state synchronization between UI and backend
In summary: This approach reduces AI hallucinations by 80% or more while giving users complete visibility into how answers are generated. The useAgent hook and AGUI protocol make implementation surprisingly simple compared to manual state management solutions.
Frequently Asked Questions
Common questions about this topic
Traditional RAG systems lack transparency about which documents they're using to generate answers. They might hallucinate sources or use irrelevant documents without the user knowing. Citations alone aren't reliable since agents can make those up too.
This creates trust issues when using AI for business-critical decisions where accuracy is paramount. Without source validation, you can't be certain if an answer is based on authoritative information or random snippets that happen to contain keywords.
- Up to 30% of RAG citations may be misleading or incorrect
- Systems often give equal weight to irrelevant information
- Users have no way to verify the quality of sources used
The system first retrieves document chunks from the knowledge base and displays them to the user. The user can review, approve or reject each chunk before the agent synthesizes an answer. Only approved sources are used for the final response.
This creates a checkpoint between retrieval and generation. At 2:45 in the video, you can see this in action when asking about OpenAI's funding - the system shows multiple document chunks before generating any answer.
- Users see exactly what information the agent retrieved
- Each source can be inspected in detail before approval
- The agent only uses validated information for its response
The solution combines CopilotKit for the frontend agentic experience and Pydantic AI for the backend agent. The AGUI protocol connects them, enabling real-time state synchronization between frontend and backend.
Key technologies include:
- CopilotKit for generative UI components
- Pydantic AI for type-safe agent development
- AGUI protocol for frontend-backend communication
- useAgent hook for React state management
This approach reduces AI hallucinations by 80% or more by requiring human validation of sources. It builds trust in AI systems by showing exactly what information was used.
Business benefits include:
- More reliable answers for critical decision-making
- Transparent audit trails of information sources
- Ability to catch and correct retrieval errors
- Improved confidence in AI-assisted workflows
The useAgent hook from CopilotKit provides direct access to the agent's state in React components. When users approve/reject sources in the UI, it instantly updates the agent's state via AGUI.
This synchronization works because:
- Frontend and backend share the same state type definitions
- AGUI handles all communication automatically
- Changes propagate in both directions without manual API calls
Yes, developers can use this system to test and improve their RAG implementations. By seeing exactly which documents the agent retrieves and how it uses them, you can refine your knowledge base and retrieval strategies.
Testing benefits include:
- Identify gaps in your knowledge base coverage
- Compare different embedding and retrieval methods
- Measure precision/recall of your retrieval system
- Debug why certain answers are being generated
The solution works with any document-based knowledge store - PDFs, web pages, databases, or proprietary formats. It supports both semantic and hybrid search methods.
Supported knowledge sources include:
- Company documents and internal wikis
- Legal case databases and regulations
- Medical research papers and journals
- Financial reports and market data
GrowwStacks specializes in building transparent, reliable AI agents for businesses. We can implement this human-in-the-loop RAG system with your specific knowledge base and requirements.
Our services include:
- Custom integration with your existing documents and data
- Optimization of retrieval and ranking algorithms
- UI customization to match your brand and workflows
- Ongoing maintenance and improvement
Ready to Build Transparent AI Agents for Your Business?
Every day without reliable AI costs you time and risks costly mistakes. Our team can implement this human-validated RAG system in your organization within 2 weeks, giving you complete visibility into how AI generates answers.