How to Build an Open German and Dialect TTS Benchmark

Methodology, not results. This guide specifies a German TTS benchmark. It does not report scores or name a winner.

Key takeaways

• Measure intelligibility, entity correctness, perceived quality, dialect authenticity, latency, and reliability separately.
• Treat ASR-based character error rate as a pipeline metric, not a direct measurement of what a person understands.
• Define the listening question, sample allocation, exclusions, and statistical analysis before collecting ratings.
• Identify dialect samples by named variety and locality. “German dialect” is not a useful evaluation slice.
• Archive exact inputs, requests, outputs, versions, and analysis code. Label an API whose backend revision cannot be pinned as a dated observation.
• Keep deployment and data-governance facts outside the audio-quality score.

Last updated August 2026.

How should an open German TTS benchmark work?

Freeze the protocol before generating audio. Each eligible system receives the same text within a declared track. Store the unmodified response, make playback copies with one published conversion pipeline, run a fixed ASR configuration, conduct a blinded listening study, and measure latency from one controlled client. Do not tune prompts after seeing test-set results.

Each result row needs the case ID, system and voice identifiers, full request, API region or local hardware, software version, timestamp, retry outcome, output checksum, and protocol version. A practical release layout is:

benchmark/
  PROTOCOL.md
  CHANGELOG.md
  LICENSES.md
  dataset/cases.jsonl
  dataset/provenance.csv
  configs/systems/*.yaml
  environment/container-digest.txt
  outputs/raw/<system>/<case-id>.*
  outputs/playback/<system>/<case-id>.wav
  asr/raw-transcripts.jsonl
  listening-study/design.json
  listening-study/anonymized-ratings.csv
  results/metrics.csv
  results/latency-events.jsonl
  scripts/

Publish an immutable release with checksums, not just a changing branch. The ACM artifact criteria are a useful standard of evidence: an artifact should be documented, complete, exercisable, and consistent with its claims. Availability alone does not prove that another evaluator can reproduce a result.

Which metrics should a TTS benchmark report?

Do not collapse these rows into a weighted “quality score.” The weights would be a product decision disguised as measurement. For deployment sizing, use a separate TTS capacity-planning protocol.

What belongs in the German test set?

Build the set from the intended workload, then stratify it. A customer-service benchmark might allocate cases across plain dialogue, long compounds, names, dates and times, currency, phone numbers, alphanumeric identifiers, abbreviations, and German-English code-switching. Publish the count and source for every category. A narration benchmark needs different material and should not inherit this mix without justification.

Every dataset row should include at least:

{
  "case_id": "de_money_001",
  "input_text": "Der Betrag ist 3.847,26 Euro.",
  "spoken_reference": "Der Betrag ist dreitausendachthundertsiebenundvierzig Euro und sechsundzwanzig Cent.",
  "category": "currency",
  "language_tag": "de-DE",
  "variety_label": "Standard German",
  "locality": null,
  "source": "benchmark-authored",
  "license": "CC-BY-4.0"
}

The spoken reference must be written before synthesis. It prevents an ASR that returns digits from deciding what counts as a correct reading after the fact. For fields with legitimate alternatives, list those alternatives in advance. Keep entity accuracy separate from CER because a single wrong digit can be operationally serious while changing little in a sentence-level edit rate. The text-normalization guide explains why dates, currency, and identifiers need explicit expected readings.

Use held-out cases for the published evaluation. The development subset can be public from the start; publish each held-out version with its results and rotate future test cases. This does not eliminate training-data contamination, so the release must state when each model and test set became public.

How should dialect varieties be sampled and labelled?

Do not use filenames such as de-BY to mean Bavarian. In BCP 47, a two-letter region subtag denotes an ISO country code, not a German federal state. Use a valid tag from RFC 5646, then record the subnational variety and locality in separate fields. For example, language_tag: de-DE can accompany variety_label: Munich Bavarian and a documented locality. Use gsw-CH or nds-DE only when those registered language subtags accurately describe the sample.

“Bavarian,” “Alemannic,” and “Swiss German” each cover internal variation. Existing corpora make this visible: STT4SG-350 balances Swiss German material across dialect regions, while the Betthupferl corpus publishes separate Franconian, Bavarian, Alemannic, and Standard German subsets. A benchmark should therefore report each recruited locality separately before showing any broader group summary.

Have speakers of the named variety author or review the text, spelling, intended pronunciation, and variety label. Record the reviewer criteria, such as where they grew up, where they currently live, and how often they use the variety. Those are sample descriptors, not tests of whether someone is “authentic.”

Run two different tracks if needed:

1. Common-input track: every system receives identical dialect text and no undisclosed repair.
2. Documented-capability track: a system may use its documented accent control, prompt, or reference voice, but the exact configuration is public.

Never mix these tracks in one rank. Dialect listeners should answer intelligibility and regional fit as separate questions; a clear Standard German reading of dialect text can score well on one and poorly on the other.

How is ASR-based character error rate calculated?

Synthesize the input, transcribe the audio with one frozen ASR system, normalize the hypothesis and spoken reference with the same published function, then compute insertions, deletions, and substitutions divided by reference length. The NIST SCTK documentation documents this edit-distance family of speech-recognition scores.

Publish the raw ASR output and every normalized string. Prefer conservative NFC Unicode normalization, case handling, and explicit punctuation rules. The Unicode normalization specification warns that compatibility forms can erase distinctions, so “apply NFKC” is not a neutral default. Do not map an incorrectly spoken number to the expected number.

The metric should be named after the ASR configuration, for example CER_ASR-X_v1, because it measures TTS, ASR, and normalization together. If resources allow, rerun with a second frozen ASR as a sensitivity check and show both results rather than averaging them. Do not compare a dialect slice with Standard German as though ASR error were equally distributed between them.

How should the blinded listening study be run?

Start from ITU-T P.85, which addresses subjective assessment of speech output devices. Use ITU-T P.808 when the test is conducted by crowdsourcing. These recommendations are starting points, not proof that any five-point web form is a valid MOS study.

Pre-register the listening question and analysis. “How natural does this speech sound?” is not interchangeable with overall quality, speaker similarity, listening effort, or dialect authenticity. Ask one construct at a time or report each answer separately.

The released design should state:

• the number of voices per system, samples, listeners, and ratings per sample;
• listener language, dialect familiarity, and recruitment geography;
• randomized order, system blinding, and how samples were allocated;
• playback format, listening instructions, and device or environment checks;
• reference and control samples, attention checks, and predeclared exclusions;
• the exact question, response labels, raw rating distribution, and analysis code.

Use a pilot to choose a sample size that meets a stated precision or power goal; there is no universal listener count for every design. Ratings repeat across listeners and utterances, so the analysis unit matters. Declare whether the confidence interval is per file or per condition and calculate it accordingly; ITU-T P.1401 discusses both forms. If a winner is claimed, publish the predeclared comparison model and treatment of multiple comparisons. A mean plus an unexplained “95% CI” is not enough.

Employees, sample producers, and anyone who can recognize system outputs should be reported as a separate expert panel or excluded from the blind panel. If only one voice represents a system, the conclusion applies to that voice and setup, not to the provider's entire catalogue.

How should time to first audio and reliability be measured?

Define time to first audio as elapsed monotonic-clock time from immediately before the client sends the synthesis request until the first audio sample that the application can decode and play. A response header or container header is not playable audio. Store the event timestamps so others can recompute the metric.

Run and report separate scenarios:

• cold connection, including DNS, TCP, and TLS where applicable;
• warm reused connection at concurrency one;
• a declared arrival rate or concurrency representative of production;
• native lossless output and, if relevant, one common 8 kHz telephony conversion applied after archiving the original.

Interleave system requests to reduce time-of-day bias. Fix the client location, provider region, text set, format, sample rate, timeout, retry policy, and connection reuse. Disable automatic retries or log every attempt. Timeouts, invalid audio, rate limits, and server errors belong in the denominator and in named failure categories; silently dropping them improves both latency and reliability on paper.

Publish the count, empirical distribution, p50, and p90. Publish p99 only with its uncertainty and enough observations for the intended precision. The MLPerf Inference rules show why tail-latency confidence requires far more observations than a median; a p99 calculated from a few dozen calls should not drive a procurement claim. Provider-reported processing latency may be included in a separate column, but it must not be mixed with end-to-end client measurements.

Which systems can be included fairly?

Define the roster in the versioned benchmark release, not in a marketing article. Apply the same admission rule to KugelAudio and every competitor.

A closed API can be useful in an open benchmark even when its implementation is not open source. The benchmark must state the resulting limitation. For local alternatives, consult the self-hosted TTS model guide, then verify the exact checkpoint's model card and licence at release time.

What must be published for licensing and reproduction?

Track rights per artifact. Benchmark code, test text, reference recordings, voice prompts, generated audio, model weights, and provider terms can all have different licences or redistribution conditions. A top-level LICENSE for the harness does not grant rights to the dataset or audio.

Record source, author or rightsholder, licence identifier or terms URL, retrieval date, required attribution, and whether redistribution is allowed. SPDX provides standardized metadata for software, AI models, and datasets, including provenance, licensing, and checksums. Obtain documented consent for any reference voice and state the allowed benchmark uses.

If provider terms prevent publishing generated audio, disclose that before the run and mark the system as only partially reproducible. Do not replace the files with cherry-picked examples. Archive each release, its terms snapshot where permitted, and a correction log. Another evaluator should be able to rebuild all reported tables from the released artifacts with one documented command.

How should readers interpret the results?

Every conclusion is bounded by the sampled voices, text, dialect localities, listeners, ASR, region, load, codec, and date. Report per-case data and slice distributions before summaries. Do not turn several locality results into a universal “German dialect score.”

KugelAudio sells a system that may appear in the evaluation. Any KugelAudio-run release must state that conflict beside every result table, apply the published protocol without post-hoc exceptions, and invite independent reruns. Use the benchmark to narrow a shortlist, then rerun it with your own traffic mix, critical entities, locations, concurrency, and voices. Quality evidence also does not establish privacy or compliance; assess those separately using the on-premise voice AI infrastructure guide.

FAQ

How is German TTS quality measured?

Use separate evidence for ordinary-text intelligibility, critical entities, human-perceived quality, dialect fit, latency, and failures. No single metric supports all of those conclusions.

What is round-trip CER?

It is the character error rate between a declared spoken reference and the output of a frozen ASR system applied to synthesized audio. It is repeatable when the ASR and normalization code are pinned, but it is not a pure TTS or human-intelligibility score.

How many listeners does a MOS study need?

There is no universal number. Choose it from a pilot and a stated precision or power target, then publish listeners, ratings per sample, exclusions, and the analysis unit. More ratings do not repair a biased listener panel or test set.

Can a benchmark report p99 from 100 requests?

It can calculate a sample percentile, but that estimate contains very little tail information. Report the raw distribution, sample count, and uncertainty; do not present a small-run p99 as a stable production guarantee.

Which German TTS system is most accurate?

This methodology does not name one. A defensible answer must specify the text slice, voices, model snapshots, ASR configuration, listening panel, and date, and it must make the underlying evidence available.

Must every included model be open source?

No. A versioned commercial API can be evaluated transparently. If its backend cannot be pinned or its outputs cannot be redistributed, label that limitation and keep it out of claims about exact reproducibility.

Does better audio quality imply better privacy or compliance?

No. Hosting region, retention, subprocessors, contracts, and on-premise controls are separate procurement evidence. Evaluate them independently from audio scores.