Latency Is the New Uptime: Why 200ms Is the Voice AI Threshold

The web trained us to think about uptime in nines. 99.9%, 99.99%—the more nines, the more reliable. But in voice AI, uptime is table stakes. The metric that determines success or failure is latency, measured in milliseconds.

Human conversational dynamics are unforgiving. Research from the Max Planck Institute shows that the average gap between conversational turns is 200 milliseconds. When an AI system exceeds this threshold, users subconsciously interpret the pause as hesitation—a signal of uncertainty or incompetence.

At 500ms, the effect is catastrophic. Call abandonment rates spike. User trust collapses. The AI, regardless of how accurate its response, is perceived as broken. This is not a UX preference—it is a neurological response hardwired by millions of years of social evolution.

At 500ms, the AI—regardless of accuracy—is perceived as broken.

Achieving sub-200ms round-trip latency in a voice AI pipeline requires optimization at every layer. Transcription must be streaming, not batch. Intent resolution must use pre-computed embeddings, not real-time inference. Synthesis must begin before the full response is generated—a technique we call "progressive utterance."

The infrastructure implications are significant. Edge deployment is not optional—it is mandatory. Model inference must happen within 50ms of the user, which means regional GPU clusters, not centralized cloud. Orbit Shift operates inference points in 12 regions, with automatic failover and latency-based routing.

Edge deployment is not optional—it is mandatory.

We measure and publish our P50, P95, and P99 latency metrics in real-time. Current P95 sits at 187ms for the complete transcription-to-synthesis pipeline. This is not a benchmark—it is a production metric under load.