Scope note. A capacity result applies only to the tested workload, model, voice, hardware, software and serving configuration. This guide provides a method for producing a result, not a universal calls-per-GPU number.
Key takeaways
- Measure end-of-turn to first playable audio for the user experience, and retain component timestamps for diagnosis.
- Drive tests with production-shaped arrivals, session behavior, text and audio duration. A fixed concurrency loop is not a substitute for an arrival model.
- Define usable capacity as the highest tested load that still meets every latency, failure, queue and recovery objective.
- Treat p99, real-time factor and GPU utilization as evidence with a stated scope, not as standalone capacity claims.
- Price the tested usable capacity, including failed and cancelled work, instead of dividing cost by theoretical peak throughput.
Last updated August 2026. No KugelAudio hardware capacity or latency result is claimed here.
Which latency does a voice AI user actually experience?
The user hears the interval from the end of their turn to the start of useful playback. Depending on the application, that includes turn detection, transcription, application logic, language-model output, text normalization, TTS, network transfer, decoding and a playback buffer. Model inference time alone does not describe that experience.
Instrument one trace with monotonic timestamps at these boundaries:
user turn end
-> transcript ready
-> first stable text sent to TTS
-> TTS request admitted
-> synthesis starts
-> first audio byte received
-> enough valid audio is decoded and buffered
-> playback starts
-> synthesis completes or is cancelled
Name the metric after its endpoints. For example, TTS time to first audio (TTFA) might run from request admission to the first audio chunk, while user-visible first playable audio runs from turn end to playback start. The first network byte can be a header or partial frame, so it is not automatically playable audio. Record client region, connection reuse, codec, sample rate and playback-buffer rule with the result.
KugelAudio's streaming and chunking guide documents how its server-side chunking and client flush behavior affect TTFA. It does not remove the need to measure the complete application path.
Best for: execute these as named scenarios. Combining cold starts, bursts and steady traffic into one aggregate conceals the cause of a slow percentile.
How should a capacity test model production load?
Start with a versioned workload manifest, not a generic request count. Include:
- session-arrival rate and burst distribution;
- calls in progress, turns per session and think time between turns;
- text-length, language, voice and normalization mix;
- generated-audio duration, format and sample rate;
- streaming chunk and flush behavior;
- connection reuse, reconnects, deadlines and retries;
- interruptions, cancellation timing and abandoned sessions;
- noisy-neighbor or batch traffic sharing the same dependencies.
Use sanitized production distributions when available. Otherwise publish the text corpus or deterministic generator, random seed and distribution parameters. Hash the corpus and record the load-generator revision so a later run uses the same work.
Choose the load-generator model deliberately. A closed-loop generator waits for one response before issuing the next request; when the service slows, it sends less work and can hide overload. It is still useful when fixed active concurrency is itself the production condition. An open-loop generator schedules arrivals independently of response completion. Google Cloud's load-testing guidance explains this distinction. For a call product, model session starts as arrivals and preserve the within-session turn timing. Report both scheduled and actually sent work, because a saturated load generator invalidates the run.
Run cold and warm experiments separately. Define cold precisely, such as a new replica with the model not loaded, and record every startup phase. For a warm run, complete the declared warm-up, then begin measurement only after throughput, queue and resource signals meet a predeclared stability rule. Do not mix the first request into a warm percentile or discard it without reporting the cold result.
How do concurrency, Little's Law and real-time factor fit together?
"Concurrent calls" can mean open sockets, live sessions, turns waiting in a queue or streams actively generating audio. Publish each separately. Active work consumes inference resources, queued work adds wait and occupies service resources, and idle callers still matter for connections and memory.
For a stable system, Little's Law relates long-run averages:
L = lambda * W
L average work in the measured system
lambda effective throughput per second
W average time each item spends in that system, in seconds
Use it as a consistency check and an initial workload estimate. It does not predict tail latency, maximum throughput or required headroom. Define the same boundary for all three terms, count completed work rather than silently using offered work, and do not apply the steady-state relationship to a window in which the queue is still growing.
Real-time factor (RTF) answers a different question:
RTF = synthesis elapsed time / generated audio duration
RTFx = generated audio duration / synthesis elapsed time = 1 / RTF
Lower RTF is faster; higher RTFx is faster. NVIDIA's TTS performance documentation uses RTFx and publishes its test inputs, parallel streams and iterations. State which convention and timing boundary you use. Compute-only RTF excludes queue and network time; end-to-end RTF does not. Neither captures first playable audio, so report TTFA and chunk gaps separately. A single-stream RTF below one also does not prove that an interactive service can accept unlimited parallel work.
How much evidence supports a latency percentile?
Report the observation count, test duration, repeat count, quantile method and histogram buckets beside every percentile. With only 100 successful observations, p99 is determined by roughly one tail observation, depending on the estimator. That is a description of a small sample, not a stable commitment. Google Cloud also warns in its latency-metric guidance that low-volume p99 can simply be the slowest request.
There is no useful universal sample count. Predefine the acceptable uncertainty, then choose duration and independent repeats for the traffic's variability. If the run cannot support a tail estimate, report a lower percentile or the fraction of requests meeting a declared latency threshold. Define the percentile's population explicitly. Report timeouts, rejects and cancellations as outcomes, and include them in the denominator of any all-attempts success rate.
Keep raw events when practical and retain mergeable histograms for operations. The OpenTelemetry metrics data model describes histogram aggregation. Put bucket boundaries around the application's decision thresholds. Do not average precomputed per-instance percentiles; Prometheus explains why that is invalid in its histogram guidance.
How should a voice service behave under overload?
Overload must be bounded and visible. Set admission limits from the interaction deadline and tested service capacity. When work cannot begin in time, reject or route it explicitly instead of accepting an unbounded queue. A short spike should not become minutes of stale speech or consume memory until the service fails.
Test the complete cycle:
- establish a stable baseline at a known offered rate;
- apply a versioned burst or remove a replica;
- observe admissions, rejects, queue age, latency and active work;
- cancel selected streams and verify their compute work stops;
- remove the fault or burst;
- verify the queue drains and all signals return to the baseline range;
- check that accepted streams were not mixed, duplicated or truncated.
Retry only when the remaining user deadline makes another attempt useful. Bound attempts and use backoff with jitter; synchronized retries can become new load. Google's guidance on cascading failures connects realistic capacity testing, small queues, early rejection and retry control. The fallback can be a pre-approved message, a separately provisioned service or human handover, but it must be exercised under load.
How should autoscaling and GPU telemetry be tested?
Autoscaling begins after demand has already changed. Measure the full scale-up delay: metric collection, decision, scheduling, image availability, model load, health checks and warm-up. Admission control and reserved capacity must protect requests during that delay. Test scale-down, connection draining, rolling updates and oscillation as well as scale-up.
Kubernetes documents custom and external metrics plus stabilization behavior in its horizontal pod autoscaling guide. A generic CPU target is not automatically appropriate for GPU TTS. Queue age, admitted active work and a saturation signal may react more directly, but use only metrics that are timely, bounded and validated against user-facing latency.
GPU utilization, memory, temperature, power and hardware-error counters explain why a result changed; they do not define usable capacity. Correlate them with queue wait, synthesis time, first playable audio, completion and failures. NVIDIA's Data Center GPU Manager documentation describes available health and telemetry capabilities. Record the metric name, sampling interval, hardware and exporter version rather than reporting a lone "GPU percent" chart.
How do you turn results into replicas and cost?
For each model and workload mix, increase offered load in steps and repeat the boundary runs. Usable per-replica capacity is the highest tested rate at which all predeclared latency, failure, queue stability and recovery gates pass. Then validate multiple replicas: routing, connection affinity, shared normalization, storage and control-plane dependencies can prevent linear scaling.
A first planning estimate is:
replicas = ceiling(forecast peak offered rate / validated per-replica rate)
This is not the final deployment count. Add capacity for the chosen failure scenario, maintenance, forecast error and startup delay, and rerun that topology. The headroom is a business and reliability decision, not a universal percentage. Google SRE recommends demand forecasting and regular load tests to relate raw resources to service capacity in its capacity-planning guidance.
Compare economics only at the same passing objectives:
cost per delivered audio hour =
total deployment cost during the window / successful playable audio hours
cost per 1,000 completed interactions =
total deployment cost during the window / completed interactions * 1,000
Include compute, idle replicas, orchestration, licenses and attributable network costs. Failed, rejected and cancelled work remains in the numerator, so waste is not hidden. Publish utilization and the pricing date. A cheaper saturation result that misses the latency objective is not a cheaper usable system.
What must a reproducible capacity report contain?
A report should provide the workload manifest and corpus hash; load-generator revision and topology; client and server regions; clock source; model, voice, normalization and serving versions; hardware, driver and power mode; replica, batching and queue configuration; warm-up and measurement windows; raw or mergeable latency distributions; offered, admitted, rejected and completed work; output-audio duration; every failure and cancellation class; resource traces; and the overload and recovery timeline.
End with decisions, not adjectives: the passing load, failed gate, selected replica count, failure reserve, autoscaling bounds and rerun trigger. Google's SRE guidance on service-level objectives is a useful framework for deriving those gates from user expectations.
For customer-operated KugelAudio, the self-hosted deployment guide documents delivery to Kubernetes with a Helm chart. It does not publish a hardware-to-concurrency guarantee. Size the selected model, hardware and workload with the method above.
What are the limitations of this guide?
This article does not benchmark a named GPU, model, voice or deployment. It also does not prescribe a conversational-latency target, queue size or safety margin. Those decisions depend on the call flow, user expectation, traffic variation, failure policy and evidence produced by the actual system.
FAQ
Is time to first byte the same as time to first playable audio?
No. The first byte may be a header or partial frame. First playable audio is the point at which the client has valid decoded audio and enough buffer to begin playback under the declared player policy.
Why is average latency insufficient?
An average hides the distribution and says nothing about failures. Report the count, selected percentiles, threshold pass rate, errors, timeouts, rejects and cancellations for the same measurement window.
How many calls can one GPU handle?
Only a matching benchmark can answer that. Model, hardware, serving build, text and audio duration, streaming behavior, workload mix and latency objective all change usable concurrency.
Is RTF below one enough for a real-time voice agent?
No. It shows that the measured synthesis interval was shorter than the generated audio duration. It does not establish first playable audio, queue delay, tail latency, cancellation behavior or multi-stream capacity.
Should GPU utilization trigger autoscaling?
Only if tests show that it predicts user pressure early enough. Compare it with queue age, active generation, admission rate, user-facing latency and the actual time required for a new replica to become ready.
What should happen when capacity is exhausted?
Stop admitting work that cannot meet its deadline, emit an explicit overload outcome and invoke the tested fallback. Do not hide excess demand in an unbounded queue or immediate retry loop.
How often should capacity tests be repeated?
Repeat them after material changes to the model, runtime, driver, hardware, normalization, routing, autoscaling policy or workload. Also rerun before making or renewing a production capacity commitment.
Build with KugelAudio
Put European voice infrastructure into production.
Use the EU endpoint or discuss a customer-operated Kubernetes deployment.