Minimizing Latency in LiveKit Voice AI Agents
From Cloud Mesh to Streaming STT/TTS: Techniques for Sub-1.5s End-to-End Latency
1. LiveKit Cloud vs Self-Hosted for Lowest Latency
How deployment choice affects latency
LiveKit Cloud uses a global mesh of SFUs, connecting each client to the nearest server. This keeps regional round-trip latency under ~100ms.
Use region pinning to lock sessions to a region (e.g., India) to avoid media hop to another geography.
Self-hosting is optimal only when you can co-locate the LiveKit server and AI stack (STT/TTS/LLM) in one low-latency region like Mumbai.
✅ Recommendation: Use Cloud if serving multiple regions. Use self-host only for ultra-local deployments.
2. Region Selection (AWS & Others)
Choose the best region for your voice pipeline
📍 Choose regions like ap-south-1 (Mumbai), ap-south-2 (Hyderabad), or Singapore for Indian users.
🚫 Avoid inter-region latency by keeping STT, TTS, and LLM in the same region.
🧭 For APIs with no region control (e.g., OpenAI), benchmark from your LiveKit server's location to select best route.
3. STT (Speech-to-Text) Optimization
Fast and intelligent speech recognition strategies
- ✅ Use streaming STT (AWS, Google, Azure) instead of batch mode.
- 🧠 Enable VAD (Voice Activity Detection) to reduce silence.
- ⚡ Use preemptive processing to feed transcripts to the LLM as they arrive.
- 📦 Consider local STT inference (e.g., Whisper on GPU) if cloud STT adds too much latency.
4. LLM Optimization
Reducing generation lag with the right models and prompts
- 💡 Prefer streaming LLM APIs (e.g., GPT-4o Realtime).
- 🪶 Use compact models like GPT-3.5 or Llama 3 for faster responses.
- ✂️ Optimize prompts: fewer tokens = faster generation.
- 📤 Stream early tokens to TTS before full LLM output is ready.
5. TTS (Text-to-Speech) Optimization
Fast audio response strategies
- 🔊 Use streaming TTS (e.g., AWS Polly, Azure TTS).
- 🎙️ Choose fast voices — neural voices may sound better, but compare latency.
- 🎧 Pre-synthesize common responses to avoid real-time generation.
- ✂️ Chunk long responses with
<break/>using SSML for faster synthesis start.
6. LiveKit Audio Settings for Low Latency
Fine-tuning audio encoding and SFU routing
- 🎚️ Use
OPUS@64kbpsmono for high-quality voice. - 🎼 Consider
16kHzsample rate for reduced processing. - ⏱️ Use
20msaudio frames (or 10ms for ultra-low latency). - 🌀 Disable echo cancellation or AGC if not needed — each adds ms of delay.
- 🚫 Avoid transcoding — use passthrough modes (WHIP, SIP, etc.) in LiveKit.
7. Designing the Real-Time Pipeline
Architecting the fastest possible voice AI loop
🎯 End-to-End Streaming: user → LiveKit → STT → LLM → TTS → LiveKit → user.
📦 Avoid file I/O or HTTP uploads. Use in-memory buffers for audio streaming.
🧩 Use on_user_speaking, interrupt(), and VAD to control flow adaptively.
⏩ Enable preemptive generation to start LLM reply before input ends.
⛓️ Pipeline each stage to work concurrently, not sequentially.
Summary: Bringing Latency Below 1.5 Seconds
Best Practices Recap
- 🌍 Deploy in the nearest region (LiveKit, STT, LLM, TTS).
- 🔁 Use streaming APIs and overlap each stage (STT, LLM, TTS).
- 🎛️ Tune LiveKit and WebRTC: OPUS mono, 20ms frames, minimal buffering.
- ⏱️ Avoid cold starts, file saves, or long prompts.
- 📶 Leverage LiveKit Cloud’s regional mesh with region-pinning if needed.
✅ With these steps, sub-1.5s latency is practical even across cloud components.