How Navan Built a Voice AI That Calls Hotels Like a Human — Real-World Evals
Most travel companies struggle with the "last mile" of hotel payments — manual calls to confirm virtual cards, late check-ins, and multilingual support. Navan's AI agent "Miles" handles thousands of these calls daily with 89% accuracy, while their evaluation system prevents regression and continuously improves performance.
The $47B Hotel Payment Problem AI Solves
Every day, thousands of business travelers arrive at hotels only to discover their virtual payment cards aren't accepted. Front desk agents—overwhelmed and untrained on digital payment systems—often default to requesting physical cards, creating reconciliation nightmares for finance teams.
Navan (formerly TripActions) identified this "last mile" problem while processing $4.7B in annual travel spend. Their solution? An AI voice agent named Miles that calls hotels pre-arrival to:
Key stat: 73% of hotels still require manual payment verification calls despite having API connections available. Navan's voice AI handles these at 1/10th the cost of human agents while maintaining 89% first-call resolution.
- Confirm virtual card acceptance
- Verify booking details
- Handle late check-in requests
- Escalate payment link requirements
How Miles the AI Agent Works (With Real Call Example)
At 2:15 in the video, you'll hear an actual call between Miles and a hotel front desk agent. Notice three critical behaviors most voice AIs fail to replicate:
- Intentional hesitation: The AI pauses slightly before answering, mimicking human cognitive processing
- Contextual humor handling: When the agent jokes about April Fool's Day, Miles acknowledges without derailing the transaction
- Precise financial verification: Confirms last four card digits while complying with PCI security standards
This level of nuance comes from Navan's evaluation system analyzing thousands of call recordings to identify successful interaction patterns.
Building the Evaluation System That Achieved 89% Accuracy
Traditional software testing fails for voice AI because:
- Human conversations have infinite variability
- Audio contains tone/subtext text transcripts miss
- Success requires both task completion AND relationship building
Navan partnered with BrainTrust to create an evaluation framework that:
Critical insight: Treat your evaluation prompt as a machine learning classifier. Navan improved their F1 score from 0.56 to 0.89 by optimizing prompts like training a model, using production data splits for validation.
- Processes raw audio files directly through GPT models (no error-prone transcription step)
- Classifies calls into 5 buckets: Success, IVR Failure, Manual Needed, Disconnect, Unknown
- Logs all intermediate reasoning steps for debugging (visible at 34:50 in the video)
Chain-of-Thought Reasoning: Why It Beat Simple Classifiers
Simple "pass/fail" evaluation fails for voice AI because:
- Front desk agents provide partial information
- Cultural norms affect communication styles
- Payment protocols vary by hotel brand
Navan's solution uses chain-of-thought prompting that mimics human evaluation:
1. Sanity Check: Is audio quality sufficient? (Yes/No) 2. Task Verification: Were all required data points confirmed? - Booking ID - Card authorization - Incidental hold amount 3. Relationship Quality: Did agent seem satisfied with interaction? 4. Final Classification: Success (all criteria met) or specify failure reason This multi-step approach reduced false positives by 43% compared to single-step classifiers.
How Production Data Feeds Continuous Improvement
Navan's system evolves through a tight feedback loop:
- Weekly: Sample 200 calls for manual review, identifying new edge cases
- Bi-weekly: Update evaluation criteria based on observed patterns
- Monthly: Retest entire evaluation system against fresh production data
Key metric: 92% of hotel calls now complete without human intervention, up from 68% at initial launch. The remaining 8% primarily involve payment link requests or VIP exceptions requiring ops team follow-up.
This process ensures the evaluation system stays aligned with real-world hotel behaviors as they evolve.
When Humans Still Need to Step In (And How AI Knows)
Despite 89% automation, certain scenarios still require human follow-up:
- Payment Links (42% of escalations): Hotels requesting digital payment forms
- VIP Exceptions (23%): Properties requiring physical cards for high-status guests
- Multilingual Needs (19%): Front desk agents preferring local language
The evaluation system flags these through specific audio cues:
- Phrases like "we need a physical card" or "can you email the payment link"
- Extended pauses indicating confusion
- Background noise suggesting front desk distraction
Flagged calls route to Navan's payment ops team within 30 minutes for resolution.
Implementation Timeline: What to Expect
Navan's deployment took 6 months, but most businesses can implement core functionality in 8-12 weeks:
| Phase | Duration | Key Activities |
|---|---|---|
| Call Analysis | 2-3 weeks | Listen to 100+ real calls, identify patterns |
| Flow Design | 3-4 weeks | Map conversation paths, exception handling |
| Eval Setup | 4-6 weeks | Build classification rubric, integrate BrainTrust |
| Pilot Testing | 2-3 weeks | Run 500+ test calls, refine prompts |
The heaviest lift? Developing the evaluation system—accounting for 60% of implementation time but driving 90% of long-term success.
Key Takeaways
Voice AI transforms industries relying on phone-based verification—from travel to healthcare to financial services. Navan's implementation proves three critical lessons:
In summary: 1) Treat evaluations as ML classifiers, 2) Process raw audio to capture tone/subtext, 3) Maintain human review for edge cases. This approach reduced Navan's payment reconciliation costs by 60% while improving hotel satisfaction scores.
For teams considering similar implementations, start small: identify your highest-volume, most repetitive calls, and build evaluation criteria around those before expanding.
Frequently Asked Questions
Common questions about voice AI implementations
Navan's AI agent Miles confirms payment details by verifying booking IDs, card authorization forms, and incidental hold amounts. It achieves 89% accuracy in these financial verifications by processing raw audio files directly through GPT models, avoiding transcription errors.
The system flags any discrepancies for human review by Navan's payment ops team, typically within 30 minutes of call completion. This hybrid approach maintains compliance while automating the majority of routine verifications.
- Validates last four card digits without violating PCI standards
- Confirms authorization forms were received by hotel
- Verifies incidental hold amounts match policy
Navan tracks F1 scores (improved from 0.56 to 0.89) across multiple call categories: successful task completion, IVR navigation failures, manual intervention needs, and disconnections.
They evaluate calls end-to-end using chain-of-thought reasoning prompts that mimic human evaluation patterns, logging all intermediate steps for debugging. This provides visibility into where classification decisions break down.
- Daily monitoring of 5 key call categories
- Weekly manual review of 200+ calls for edge cases
- Monthly retesting against fresh production data
The system categorizes 11% of calls as requiring human follow-up, primarily for payment link requests or VIP exceptions. It maintains conversational flow by injecting appropriate hesitation (like nervous human agents) and clearly stating when it's an AI.
All unusual responses are logged in BrainTrust for ops team review within 30 minutes. The evaluation system tracks these exceptions separately to identify emerging patterns needing workflow adjustments.
- Flags 7 specific exception triggers in real-time
- Maintains call context during handoffs
- Prioritizes cases by traveler status/urgency
Navan processes raw audio files directly through GPT models without intermediate transcription. They use a multi-step reasoning classifier with chain-of-thought prompts that evaluate: call success (70% of calls), failure reasons (20%), and follow-up needs (10%).
The system integrates with Navan's travel payment APIs to validate card details in real-time. All components log to BrainTrust for centralized monitoring and continuous improvement.
- GPT-4 Turbo for audio processing
- Custom classification layers
- Real-time payment system checks
Navan maintains a continuous feedback loop: 1) They sample 200 calls weekly for manual review, 2) Update evaluation criteria based on new edge cases, 3) Test updates against a validation dataset before deployment.
This process reduced false positives by 43% in their production environment. The system also tracks performance across hotel brands to identify chain-specific communication patterns needing customization.
- Version-controlled evaluation prompts
- Separate test/validation datasets
- Automated alerting on metric drifts
Currently optimized for English with 89% accuracy, but designed for multilingual expansion. The evaluation system tracks language-specific success rates separately, with plans to add Spanish and French support in by training on localized hotel call patterns.
Navan's architecture supports adding languages without reworking core evaluation criteria. Each new language undergoes 4 weeks of localized call analysis before deployment.
- English-only currently
- Spanish/French roadmap
- Culture-specific tone adjustments
Navan's initial deployment took 6 months including 3 months for evaluation system development. Key time investments were: 1) 200+ hours listening to real calls, 2) Building the classification rubric, 3) Implementing the BrainTrust integration.
Most businesses can deploy a basic version in 8-12 weeks with proper automation support. The evaluation system typically accounts for 60% of implementation time but drives 90% of long-term success.
- 2-3 weeks call analysis
- 4-6 weeks eval setup
- 2-3 weeks pilot testing
GrowwStacks specializes in building customized voice AI solutions for hospitality, healthcare, and customer service. Our 30-day implementation package includes: 1) Call flow design based on your use cases, 2) Evaluation system setup with BrainTrust integration, 3) Continuous improvement framework.
We've helped clients reduce call center costs by 60% while maintaining 85%+ task completion rates. Our team handles everything from initial call analysis to ongoing performance optimization.
- Free workflow assessment
- Custom evaluation criteria
- Ongoing performance tuning
Ready to Automate Your Phone-Based Verifications?
Manual verification calls cost your team hours daily and create reconciliation headaches. GrowwStacks can implement a Navan-style voice AI solution in 30 days that handles 85%+ of calls automatically while maintaining human oversight for exceptions.