Voice-agent bugs don’t reproduce. The call that failed existed once, in real time, shaped by a caller who will never say those words that way again, a speech model that will never phrase its reply the same way twice, and a transcription layer that misheard one syllable at exactly the wrong moment. You cannot attach a debugger to a conversation that no longer exists.
So the discipline that matters isn’t reproduction — it’s reconstruction: when a call goes wrong, the logs alone must let an engineer rebuild it turn by turn and name the root cause down to the file and line. This post covers the observability practices that made that routine for a production voice agent, and the coding patterns that follow once you accept the second premise: increasingly, the engineer doing that reconstruction is an LLM. Your code now has two audiences, and the second one will be asked to fix a bug at 2 a.m. with nothing but the repository and a log stream.
Log decisions, not events
Standard logging records what happened: request in, response out. Voice agents fail in the gaps between events — in the decisions: a recovery timer that chose to fire while the model was mid-answer, a gate that approved a hangup, an extraction that silently declined to write a field. The shift that changed everything for me: one structured, greppable line per consequential decision, carrying the verdict and the inputs it was based on.
Not “call ended.” Instead: which path requested the end, what the gate saw (fields collected, what was missing, who was speaking), and what it ruled. When a required answer went missing from a record, the decision trail showed me the exact mechanism in minutes: the model acknowledged the answer without recording it, the recovery deferred four times while audio drained, and a completion path ended the call on an unverified claim. Three subsystems, one greppable timeline, no guesswork — and the same trail later proved a different bug was not in the dialog logic at all but in transcript bookkeeping.
Two constraints make this sustainable. Privacy: log field names, lengths, booleans, verdicts — never values; “recorded field X on turn 3” debugs as well as the value itself and can live in production logs safely. Naming: stable machine-parseable keys (source=, reason=, turn=, verdict=) so the log stream is queryable by a human with grep or by an LLM with a prompt.
The trust boundary
The most expensive bug class in LLM-integrated systems is believed claims. My canonical example: an internal result object carried the reason string “caller farewell after required fields collected,” and a code path ended calls on the strength of that sentence — while a sibling code path could plainly see a required field missing. The reason string was a claim. Nothing verified it. The call hung up with the data unrecorded.
The rules that fixed it generalize to any system where model outputs touch state:
- Verify, never trust. A model’s (or upstream component’s) claim about state is a hypothesis. Check it against the actual data at the moment of action — at one shared gate, because five copies of a check are five chances for the copies to disagree, and they will.
- Evidence grounding for LLM-written data. When an LLM extracts values from a transcript, every value must come with a verbatim quote, and the code verifies the quote actually appears in the source — speaker-attributed, so the agent’s own words can’t masquerade as the caller’s. No quote, no write. This single rule is the difference between recovery and hallucination.
- Backfill-only writes. LLM-derived data fills empty slots; it never overwrites what was captured live. Worst case is an unfilled field — never a corrupted one.
- Format validation stays in code. Dates parse, digits count. The model formats; the code enforces. And keep validation language-agnostic — the moment you write a keyword regex for how people phrase something, you are re-implementing the model badly, one language at a time.
Prompts and tool schemas are code
In a tool-driven agent, the model’s behavior is programmed by three text surfaces: the system instructions, the tool descriptions, and — most underrated — the tool result messages, which steer the model’s next move (“recorded; ask for the next missing field: X”). These strings are load-bearing. A one-sentence instruction that told the model to use one language “for the remainder of this call” caused it to answer a later switch-back request in the wrong language; one sentence, reviewed like prose instead of like code, shipped a bug. Treat steering text as an API surface: versioned, reviewed for contradictions across surfaces (two instructions disagreeing about whether to confirm a number is a race condition written in English), and covered by tests that assert the contracts — “no surface tells the model to ask for confirmation” — rather than exact wording.
An anti-pattern bestiary
Recurring bugs from the intersection of async code and LLM calls — all of which I shipped, none of which I’ll ship again:
- The warm-then-read guard. Fire an async classifier, then synchronously read its cache in the same tick, treating “no answer yet” as “no.” The guard literally cannot work on first evaluation — it silently returns the default every time it matters. Await with a bounded timeout, or register a callback for when the verdict lands; never pretend an async answer is sync.
- Errors cached as verdicts. A classifier timeout cached as
falsepoisons that input’s verdict for every future call on the process. Cache answers, never failures. - Counting the wrong thing. A “we’ve asked three times, give up” counter that incremented per internal tool dispatch instead of per actual question to the caller — batched tool calls burned the budget without a single question being asked. Count what the user experienced, not what the code did.
- Artifacts that diverge from reality. Stored transcripts that drop or mislabel audio the caller actually heard corrupt everything downstream — debugging, evaluation, and the record itself. The artifact of record must match the experienced reality, always.
The forensic loop
Put together, debugging becomes a loop that improves the system every pass: a failure is reported; the decision logs reconstruct the call turn by turn; the root cause gets named at file-and-line precision; the fix lands with a deterministic regression test; and — the step most teams skip — wherever the reconstruction required guesswork, a new decision log is added, so the next failure of that kind is self-explanatory. Observability isn’t a dashboard you build once; it’s a debt you pay down one missing log line at a time.
This loop is also precisely what makes the codebase maintainable by an AI. An LLM engineer can’t set breakpoints in a phone call any more than you can — but given decision-level logs with stable keys, constraint-stating comments (why this check must stay, not what the next line does), single-owner state machines instead of flag webs, and deterministic tests for every past failure, an LLM can reconstruct, diagnose, and fix voice-agent bugs from artifacts alone. I know, because that’s how most of the bugs in this series were found and fixed. Write code for the maintainer who can only see what you logged — whichever species they are.
This closes the series: part one covered the architecture — who owns the next turn — and part two covered testing with simulated callers and LLM-as-a-judge.