Inside a Production AI Voice Architecture

An AI voice agent is easy to demo and hard to deploy. In a demo you dial a number, a pleasant voice answers, it books a fake appointment, everyone nods. None of the things that actually break in production are in the room: the number the business already owns and cannot afford to lose, the carrier that decides your text messages look like spam, the caller who switches to Spanish mid-sentence, the customer who hangs up halfway through giving you their name.

This is a teardown of the system behind Daniel Monzon Automation (DMA) — a bilingual (English / Caribbean-Spanish) AI receptionist that answers real small-business phone lines and turns each call into a structured lead. I'll walk the architecture quickly, then spend most of the post on the four problems that took real work to solve, because those are the parts you can't learn from a demo.

The system at a glance

The runtime is smaller than people expect; the hard part isn't the number of boxes, it's what happens between them.

Conversation layer — Retell. Retell orchestrates the live call: answering, turn-taking, routing, and firing the webhooks that drive everything downstream. It's the brain of the call while the caller is on the line.

Voice — ElevenLabs. Text-to-speech, including a dedicated Caribbean-Spanish voice so Spanish calls sound native rather than translated. More on why that distinction mattered below.

Telephony. Carrier numbers, call-forwarding bridges, a Google Voice port, and Twilio for SMS. This layer is where "it's just a phone number" stops being true.

Post-call pipeline. When a call ends, a webhook fires into a Zapier extraction step that pulls eight structured fields (intent, contact, appointment details, and so on). Those fields write to Postgres as the source of truth, then fan out to a HubSpot CRM record and an email/SMS lead alert to the owner.

Web concierge — Chatbase. A site assistant ("Mona") with its own lead-alert path, so the website captures leads on the same footing as the phone.

Infra. A Hetzner VPS with Postgres.

One decision is worth stating before the failures, because it shaped how I recovered from all of them: Postgres is the source of truth, not the CRM. A CRM and a Zapier chain are integration surfaces, and integration surfaces hiccup — a rate limit, a dropped webhook, a schema change on someone else's side. I didn't want any of those to mean a lost lead. So every call lands in Postgres first, and everything downstream is a projection of that record. If HubSpot is having a bad day, the lead still exists and still reaches the owner.

How a call moves

A caller dials in. Retell answers in the selected language and runs the conversation. The call ends. A webhook fires. Zapier extracts the eight fields. The record is written to Postgres, pushed to HubSpot, and turned into an email/SMS alert. Within seconds of hangup, the business owner has a structured, actionable lead — not a voicemail they'll listen to later, if ever.

That's the happy path. Here's where production disagreed with it.

1. Porting a number without dropping a call

A client's existing business number had to move onto the new system. The problem: porting a number (in this case out of Google Voice) takes days, and a service business cannot miss a single call while it happens. Every missed call is a missed customer.

The reframe that solved it was to stop treating the cutover as a configuration change and start treating it as a live-traffic migration. I stood up call forwarding as a bridge: the AI agent began answering immediately on a forwarded path while the port completed in the background. Callers reached the agent the whole time; the plumbing moved underneath them without anyone noticing.

2. When the bug isn't in your code

SMS lead notifications stopped delivering. The instinct is to go read your own code, and I did — and there was nothing wrong with it. The messages were being gated behind toll-free verification with the carrier. It wasn't an application bug; it was a compliance state living several layers below anything I'd written.

The fix wasn't more code, it was sequencing. I'd already separated the delivery channels, so voice and email lead delivery could ship immediately while SMS waited on verification. The moment the carrier cleared the number, SMS layered in cleanly with no rework. The system went live without being held hostage by a process I didn't control.

3. "Speaks Spanish" is not the bar

Generic Spanish TTS sounded flat and non-native to a Caribbean-Spanish-speaking customer base. On paper the requirement was "handle Spanish calls," and technically it did. But a customer doesn't hear a checkbox; they hear an accent that isn't theirs, and they clock it instantly.

So I built and tuned a dedicated Caribbean-Spanish voice, so Spanish calls sound like they're coming from someone local rather than from a translation engine. The work wasn't adding a capability — it was raising the quality of one that already technically existed to the point where it holds up with real people.

4. Designing for the caller who goes off-script

Real callers do not behave like test inputs. They give partial information, pick the wrong language, talk over the agent, and hang up early. Any one of those threatened the structured output the whole system exists to produce — an extraction step that assumes a clean, complete call will fail the moment a call isn't one.

I built the post-call extraction to degrade gracefully instead of failing: whatever the caller did or didn't provide, the pipeline still produces the best structured lead it can across all eight fields, and the owner always receives something usable. A half-complete lead that arrives beats a perfect lead that never does.

What production actually taught me

The demo is the easy twenty percent. The other eighty is telephony that fights you, carriers with their own compliance calendars, a dialect that has to be right, and human beings who will use your system in every way you didn't plan for. The interesting engineering isn't getting the agent to talk — it's everything that has to be true for it to keep working on someone else's live phone line, and to keep working the day after you hand it over.

That's the work I like: owning a system past the demo, into production, and through the failures that only show up once real customers are on the other end of the line.