AI addendum

The five AI-specific things a new engineer must understand in one read: what each translated minute costs us, how evals gate every deploy, why a provider can't lock us in, what telemetry compounds into a moat, and how prompts are versioned. Numbers inherited from tech.md §8 as the single source of truth — not re-derived.

1 · AI cost-of-goods model

Purpose: every engineer can name what a translated minute costs and which leg dominates. Bar: you can recompute the per-active-hour number from the cells below without asking the founder. Refresh: quarterly + on any provider price change

The make-or-break insight: STT is ~60% of dub COGS, not TTS. Anyone who tells you "TTS is 80% of the bill" is reading a stale doc. Per-leg unit prices (May 2026): Deepgram Nova-3 multilingual $0.0092/min, Cartesia Sonic $0.006/min, Gemini 2.5 Flash-Lite $0.10/$0.40 per M tok.

Sensitivity — what breaks the margin

2 · Eval harness spec

Purpose: no deploy ships without passing babelio-evals. Bar: a new engineer can read the gate, run it locally, and know exactly which metric blocked a PR. Refresh: continuous (runs every push)

For a real-time translator the eval that matters is not BLEU — it's end-to-end p95 latency on real clips. A perfect translation that lands 1.2s late is a failed dub. So we gate on two harnesses at once: Promptfoo for MT quality + glossary, and a custom latency-budget harness that replays labeled audio clips through the full STT→MT→TTS path. Either regresses → CI goes red → deploy blocked. Online Braintrust/Langfuse traces catch prod-vs-eval drift.

# babelio-evals — CI gate (pseudo)
npx promptfoo eval -c evals/mt.yaml      # WER, hallucination, chrF, glossary
python -m evals.latency --clips data/   # replay STT→MT→TTS, assert p95

assert p95_ms      < 700
assert wer         < 0.12
assert halluc      < 0.02
assert chrf_drop   < 2.0
assert glossary    > 0.95   # any failure → exit 1 → merge blocked

3 · Model lock-in mitigation

Purpose: no single provider can hold our margin or uptime hostage. Bar: any engineer can shadow-test an alternate provider on real traffic within one day, no code rewrite. Refresh: quarterly provider-price review

Every provider call goes through a self-hosted LiteLLM proxy (on Fly). The client never holds provider keys — it authenticates to the proxy with a device-license JWT, and the proxy owns key custody, fallback, and spend caps. Swapping a provider is a config change, not a deploy. Pre-wired fallbacks per leg:

Provider-swap procedure — shadow-test in one day

1. Add the candidate model to litellm config.yaml under the leg's model_list with a shadow: tag. ~15 min
2. Mirror a sampled % of live traffic to the shadow model — it runs but its output is not served; results land in Braintrust. ~30 min
3. Run babelio-evals against the shadow model on the labeled clip set (latency + WER + hallucination). ~1 hr
4. Compare shadow vs primary on the eval bar and live cost. If it passes the gate and is cheaper/faster, flip shadow: → primary in config. ~5 min
5. No client release needed — the proxy reloads config. Roll back instantly by reverting one line. instant

4 · Data flywheel

Purpose: the one defensible moat — instrument it day-1 or every frontier release erodes us to parity with free. Bar: an engineer knows exactly which signals to log on the first shipped build and why. Refresh: continuous (logged every session)

This is a workflow-graph + eval-correction flywheel (two of the four "real" types). The schema already exists in the architecture — usage_ledger (per-leg cost, p95 latency, prompt version per billable second) and evals (clip, src/ref/hyp, quality, latency). Logging it from the first shipped build is non-negotiable, not a Growth-stage add. When models commoditize, this capture+eval layer is where value accrues — it's the asset the B2B/SDK platform thesis sells.

When it compounds (12-month estimate)

Labeled per-app sessions accumulate super-linearly with installs. The asset isn't volume — it's coverage of the (app × codec × network) matrix. Around M6–M9 we should hold enough per-app profiles that a tuned capture+VAD config measurably beats a cold competitor on the same app. That delta is the moat candidate — and the proof point for the SDK pitch.

5 · Prompt management

Purpose: prompts are code — versioned, reviewed, regression-gated. Bar: an engineer can change the MT prompt safely, because a PR can't merge if it regresses the eval set. Refresh: per prompt change (PR-gated)

No prompt CMS day one. The MT system prompt + per-user glossary live as versioned modules in git (prompts/mt-v*.ts), and every deployed version is stamped into mt_prompt_versions (version, system_prompt, glossary_hash, deployed_at). Each billable second in usage_ledger records which prompt version produced it — so we can attribute a quality or cost shift to an exact prompt change. The glossary term-lock injector (names, jargon as a prompt prefix) is the cheap quality lever raw-NMT competitors skip.

1. Edit a prompts/mt-v*.ts module in a branch — never hot-patch a prompt in prod.
2. Open a PR. CI runs babelio-evals against the new prompt on the full clip set — same gate as code.
3. Regression blocks merge. Any drop in chrF/COMET, glossary-hit, or hallucination → red → PR can't merge. A second engineer reviews the diff + the eval delta.
4. Deploy stamps the version. New row in mt_prompt_versions; usage_ledger starts attributing sessions to it. Roll back = redeploy the prior version row.

6 · AI Act / GDPR / safety

Purpose: the compliance gates that bind a synthetic-audio product reaching EU users — what applies, the response, and who owns it. Bar: an engineer knows which gates block a release vs which can wait. Refresh: quarterly + on any regime change