How to Build Low-Latency Voice AI Agents with Go in
Most voice AI solutions feel robotic with noticeable delays between turns. Discover how Sinflow achieves natural 2.8-second response times using Go's concurrency patterns, LLM fan-out techniques, and hybrid finite state machines - all while processing 100,000+ daily calls.
Why Go for Voice AI Infrastructure
When Sinflow first prototyped their voice AI system, they used a different programming language that struggled under production loads. The team switched to Go when they realized they needed to handle over 100,000 daily calls with minimal overhead.
Go's lightweight concurrency model proved ideal for their streaming architecture. Channels became the perfect abstraction for passing audio chunks between transcription, LLM processing, and text-to-speech components. Contexts provided built-in cancellation when users interrupted responses mid-flow.
Key advantage: Go's goroutines allow Sinflow to process thousands of simultaneous calls with predictable memory usage. Their average deployment handles 150 concurrent conversations per server with 2.8-second latency targets.
The 2.8-Second Latency Target
Research shows conversations feel unnatural when response delays exceed 2.8 seconds. Users perceive longer pauses as the bot "stopping" rather than thinking, breaking the illusion of a fluid dialogue.
Sinflow's pipeline must complete multiple steps within this window: speech-to-text transcription, LLM processing, response generation, and text-to-speech conversion. Their critical metric is Time To First Token (TTFT) - how quickly they can generate the initial audio chunk that keeps users engaged.
Pro tip: The first audible segment matters more than complete response generation. Once playback begins, users tolerate longer processing times for remaining content.
LLM Fan-Out Pattern
LLMs introduce unpredictable latency - sometimes responding in 500ms, other times taking 5 seconds for identical prompts. Sinflow's solution leverages Go's concurrency to fire multiple identical requests across different endpoints simultaneously.
The first response that provides sufficient content (with punctuation for natural TTS breaks) "wins," while contexts cancel the remaining requests. This pattern improved their P90 latency by 37% compared to single-endpoint calls.
Implementation note: They fan out to 3-5 endpoints per call, balancing redundancy against compute costs. The system monitors each endpoint's historical performance to optimize distribution.
Adaptive Latency Router
While fan-out helped, Sinflow needed smarter routing to handle regional traffic spikes and random LLM slowdowns. Their latency router tracks response times across 20-25 Azure deployments, using weighted averages that prioritize recent performance.
The system follows a 90/10 rule: 90% of traffic routes to historically fast endpoints while 10% explores alternatives. This maintains optimization while discovering new fast paths as conditions change. Endpoints showing errors get temporarily blacklisted for graceful degradation.
Result: The router reduced 99th percentile latency from 4.2s to 3.1s while improving reliability through automatic failover during regional outages.
Hybrid FSM Architecture
Pure LLM conversations often go off-script - skipping required steps or making illogical jumps. Sinflow combines finite state machines (FSMs) with LLMs to enforce business process integrity while maintaining natural language flexibility.
The FSM defines required data collection (slots) before allowing state transitions. For example, a healthcare bot must confirm symptoms before booking appointments. Meanwhile, the LLM handles the actual conversation flow within each state's constraints.
Key innovation: Customers can choose between deterministic rules ("if income > X") and LLM-evaluated conditions ("does user sound upset?") for state transitions, blending structure with emotional intelligence.
Two-Phase Command Processing
Sinflow's command protocol lets the LLM set slots, utter responses, and request state transitions in a single output. Their two-phase processing validates these commands against projected future state before execution.
When the LLM correctly predicts valid transitions (90% of cases), the system executes everything in one round-trip. For incorrect predictions (10%), it makes corrective LLM calls while still executing valid portions. This balances latency with correctness.
Example: At 12:30 in the video, Omar explains how this prevents saying "transferring you now" before verifying transfer eligibility - avoiding confusing user experiences.
Real-Time Voice Challenges
Voice interfaces introduce unique timing complexities. Sinflow implemented several heuristics to make conversations feel natural:
- Interrupt handling: Configurable word-count thresholds prevent "uh-huh" from cutting off important responses
- Debouncing: Filters overlapping speech during bot/user cross-talk
- Variable endpointing: Longer silence thresholds for phone numbers/emails
- Non-cancelable contexts: Critical actions (like bookings) continue even if interrupted
These rules require constant tuning since what feels right varies by use case and customer. The team maintains different configurations for healthcare, finance, and other verticals.
Key Takeaways
Building natural voice AI requires optimizing multiple latency-sensitive components while maintaining conversation integrity. Sinflow's architecture demonstrates how to balance these competing demands at scale.
In summary: Go's concurrency patterns + LLM fan-out + hybrid FSM + adaptive routing creates voice AI that feels human. The system handles 100K+ daily calls with 2.8s latency while preventing nonsensical conversation flows.
Watch the Full Tutorial
See Omar and his colleague demonstrate these techniques live, including a deep dive into the FSM implementation at 18:45 and latency router metrics at 24:30.
Frequently Asked Questions
Common questions about this topic
Sinflow chose Go because its lightweight concurrency patterns allow handling thousands of simultaneous calls efficiently. They process about 100,000 calls daily with predictable resource usage.
Go's channels proved ideal for streaming audio data between components, and contexts helped manage cancellation of in-flight operations when users interrupt conversations. The language's native support for these patterns reduced infrastructure overhead compared to their original prototype.
Sinflow aims for 2.8 seconds maximum latency between user speech and bot response. Research shows responses longer than this feel laggy, causing users to think the bot stopped working.
They measure Time To First Token (TTFT) as the critical metric since users remain engaged once audio generation begins. The first audible segment (with punctuation for natural breaks) matters more than complete response generation time.
The fan-out pattern fires multiple identical LLM requests across different endpoints simultaneously. The first response that provides audible content (with punctuation for TTS) wins, while contexts cancel the remaining requests.
This approach improved their P90 latency significantly by leveraging Go's native concurrency features. They typically fan out to 3-5 endpoints per call, balancing redundancy against compute costs.
The latency router tracks response times across 20-25 Azure deployments, using weighted averages that prioritize recent performance. It routes 90% of traffic to historically fast endpoints while exploring alternatives with 10% of calls.
This adaptive system handles regional traffic spikes and random LLM slowdowns automatically. Endpoints showing errors get temporarily blacklisted, improving overall reliability while maintaining low latency.
Finite State Machines provide deterministic business logic guarantees that pure LLMs cannot. While LLMs handle natural language understanding, the FSM enforces required data collection before state transitions (like verifying income before transfers).
This hybrid approach prevents nonsensical conversation flows that could occur with LLMs alone. Customers can configure when to use strict rules vs. LLM-evaluated conditions for state transitions.
Two-phase processing lets the LLM speculatively generate responses for predicted next states during assessment. If validated (90% of cases), these execute immediately without a second LLM call.
When wrong (10%), the system makes corrective calls while still executing valid command portions. This pattern minimizes round trips while maintaining conversation naturalness, as demonstrated in the 12:30 video segment.
Sinflow implements configurable word-count thresholds before registering interrupts, debouncing overlapping speech, and variable silence detection (longer for phone numbers/emails).
They maintain message buffers for non-interruptible responses while capturing subsequent user input. These heuristics balance responsiveness against premature cutoffs that would frustrate users.
GrowwStacks helps businesses implement automation workflows, AI integrations, and scalable systems tailored to their operations.
Whether you need a custom workflow, AI automation, or a full multi-platform automation system, the GrowwStacks team can design, build, and deploy a solution that fits your exact requirements.
- Custom automation workflows built for your business
- Integration with your existing tools and platforms
- Free consultation to discuss your automation goals
Need Natural-Sounding Voice AI for Your Business?
Every second of latency costs you customer trust and engagement. GrowwStacks builds custom voice AI solutions that feel human - with response times under 3 seconds and business logic guarantees.