Inside a Production AI Voice Architecture

An AI voice agent is easy to demo and hard to deploy. In a demo you dial a number, a pleasant voice answers, it books a fake appointment, everyone nods. None of the things that actually break in production are in the room: the number the business already owns and cannot afford to lose, the carrier that decides your text messages look like spam, the caller who switches to Spanish mid-sentence, the customer who hangs up halfway through giving you their name.

This is a teardown of the system behind Daniel Monzon Automation (DMA) — a bilingual (English / Caribbean-Spanish) AI receptionist that answers real small-business phone lines and turns each call into a structured lead. It comes in two parts, and the order matters: first what actually shipped for the first client, assembled by hand, and then what productizing it required once there was a second one. I'll walk each architecture quickly and spend most of the post on the five problems that took real work to solve, because those are the parts you can't learn from a demo.

Part one: the first deployment

The first system was built and configured by hand for one client. Nothing about it was generic, and that was the right call — you can't factor out the shared parts of a thing you've only built once. The runtime is smaller than people expect; the hard part isn't the number of boxes, it's what happens between them.

Conversation layer — Retell. Retell orchestrates the live call: answering, turn-taking, routing, and firing the webhooks that drive everything downstream. It's the brain of the call while the caller is on the line.

Voice — ElevenLabs. Text-to-speech, including a dedicated Caribbean-Spanish voice so Spanish calls sound native rather than translated. More on why that distinction mattered below.

Telephony. Carrier numbers, call-forwarding bridges, a Google Voice port, and Twilio for SMS. This layer is where "it's just a phone number" stops being true.

Post-call pipeline. When a call ends, a webhook fires into a Zapier extraction step that pulls eight structured fields (intent, contact, appointment details, and so on). Those fields write to Postgres as the source of truth, then fan out to a HubSpot CRM record and an email/SMS lead alert to the owner.

Web concierge — Chatbase. A site assistant ("Mona") with its own lead-alert path, so the website captures leads on the same footing as the phone.

Infra. A Hetzner VPS with Postgres.

One decision is worth stating before the failures, because it shaped how I recovered from all of them: Postgres is the source of truth, not the CRM. A CRM and a Zapier chain are integration surfaces, and integration surfaces hiccup — a rate limit, a dropped webhook, a schema change on someone else's side. I didn't want any of those to mean a lost lead. So every call lands in Postgres first, and everything downstream is a projection of that record. If HubSpot is having a bad day, the lead still exists and still reaches the owner.

How a call moves

A caller dials in. Retell answers in the selected language and runs the conversation. The call ends. A webhook fires. Zapier extracts the eight fields. The record is written to Postgres, pushed to HubSpot, and turned into an email/SMS alert. Within seconds of hangup, the business owner has a structured, actionable lead — not a voicemail they'll listen to later, if ever.

That's the happy path. Here's where production disagreed with it.

1. Porting a number without dropping a call

A client's existing business number had to move onto the new system. The problem: porting a number (in this case out of Google Voice) takes days, and a service business cannot miss a single call while it happens. Every missed call is a missed customer.

The reframe that solved it was to stop treating the cutover as a configuration change and start treating it as a live-traffic migration. I stood up call forwarding as a bridge: the AI agent began answering immediately on a forwarded path while the port completed in the background. Callers reached the agent the whole time; the plumbing moved underneath them without anyone noticing.

2. When the bug isn't in your code

SMS lead notifications stopped delivering. The instinct is to go read your own code, and I did — and there was nothing wrong with it. The messages were being gated behind toll-free verification with the carrier. It wasn't an application bug; it was a compliance state living several layers below anything I'd written.

The fix wasn't more code, it was sequencing. I'd already separated the delivery channels, so voice and email lead delivery could ship immediately while SMS waited on verification. The moment the carrier cleared the number, SMS layered in cleanly with no rework. The system went live without being held hostage by a process I didn't control.

3. "Speaks Spanish" is not the bar

Generic Spanish TTS sounded flat and non-native to a Caribbean-Spanish-speaking customer base. On paper the requirement was "handle Spanish calls," and technically it did. But a customer doesn't hear a checkbox; they hear an accent that isn't theirs, and they clock it instantly.

So I built and tuned a dedicated Caribbean-Spanish voice, so Spanish calls sound like they're coming from someone local rather than from a translation engine. The work wasn't adding a capability — it was raising the quality of one that already technically existed to the point where it holds up with real people.

4. Designing for the caller who goes off-script

Real callers do not behave like test inputs. They give partial information, pick the wrong language, talk over the agent, and hang up early. Any one of those threatened the structured output the whole system exists to produce — an extraction step that assumes a clean, complete call will fail the moment a call isn't one.

I built the post-call extraction to degrade gracefully instead of failing: whatever the caller did or didn't provide, the pipeline still produces the best structured lead it can across all eight fields, and the owner always receives something usable. A half-complete lead that arrives beats a perfect lead that never does.

Part two: productizing it

Then there was a second client, and the work started repeating: prompt wiring, voice selection, call routing, post-call follow-up. Doing that by hand a third and fourth time was obviously not the plan, but the interesting part was why it could be avoided. The differences between clients turned out to be narrow and describable — a greeting, business hours, how calls route, which voice, what happens after the call — while the hard parts were shared.

So I factored the shared parts into a kit and pushed the differences into config. That's dma-deploy-kit, which is public and MIT-licensed. Worth being precise about what it is: a parallel, productized system built after the fact, not a rewrite of the hand-configured deployment above. That one still answers its phone line the way it was originally built.

In the kit, a client is one YAML file, validated by a strict schema, compiled into a fixed-section prompt by a pure function — no templating engine, no network call. A plan/apply engine diffs that against what's actually deployed and writes a per-client lockfile, so re-running converges to a no-op instead of quietly mutating a live agent. When a call ends, a FastAPI endpoint verifies the webhook's HMAC-SHA256 signature, parses the call into a lead, resolves which client owns that agent from the lockfiles, and dispatches the alert. Booking SMS goes out only when consent, a booking URL, a captured consent flag, and a normalizable US phone number all hold — and it's written to a send-once ledger so a webhook retry can never double-text a real customer.

One rule shaped all of it: everything outside the live conversation is deterministic Python. The model is confined to the call itself. Deploy, diffing, lead routing, SMS gating, and the evals are all plain code — so the parts I have to audit and reproduce are ones I can actually read.

Onboarding a new receptionist became editing a file rather than rebuilding an agent. Which bought a new problem.

5. Proving a change didn't break a deployment

Once one kit ships the same change to every deployment at once, editing a prompt stops being free. "It still sounds fine" is not evidence, and a regression doesn't announce itself — it shows up on someone's actual call, days later, as a booking that didn't happen.

So the behavior got a test suite: four layers, each answering a different question. Static prompt-policy checks gate CI. Deterministic transcript assertions and latency budgets run over real calls. Then an LLM judge scores each transcript against a fixed rubric — and this is the part I'd defend hardest: every "fail" it returns must quote a verbatim span from a turn it cites, or the verdict is downgraded. The judge is structurally barred from making a claim I can't check, rather than merely instructed to be honest.

The subtler problem is that a test suite rots quietly. A check that can no longer fail still passes, and a green CI run stops meaning anything. So a golden suite of synthetic fixtures — calls engineered to trip specific checks — gates CI alongside the real ones. Every push has to prove the checks still discriminate, not just that the code imports. Each run also pins the sha256 of every compiled prompt, so comparing two runs flags a regression only when a check newly fires. Prompts stay free to change until one of them actually breaks something.

What production actually taught me

The demo is the easy twenty percent. The other eighty is telephony that fights you, carriers with their own compliance calendars, a dialect that has to be right, and human beings who will use your system in every way you didn't plan for. The interesting engineering isn't getting the agent to talk — it's everything that has to be true for it to keep working on someone else's live phone line, and to keep working the day after you hand it over.

That's the work I like: owning a system past the demo, into production, and through the failures that only show up once real customers are on the other end of the line.