← Back to work

Sofia — Real-time voice AI agent

Real-time voice AI agent

Sofia

2026Private repo
  • Python 3.12
  • LiveKit Agents
  • Deepgram Nova-3
  • Claude Haiku 4.5
  • Gemini 2.5 Flash
  • ElevenLabs Flash
  • Silero VAD
  • PostgreSQL
  • Docker

Overview

Sofia is a 24/7 AI voice receptionist for clinics. A caller dials a normal phone number and talks to a real-time agent that books, captures details, and hands off to a human when unsure — in natural speech, under a second and a half per turn.

Architecture

caller
STTDeepgram Nova-3
LLMClaude Haiku 4.5 · Gemini 2.5 Flash
TTSElevenLabs Flash
caller

A real-time, EU-resident cascade. Every stage is independently observable and measurable.

LiveKit transport · Silero VAD + LM end-of-turn detector · Postgres tool-calls

~416 ms
STT
~882 ms
LLM
~163 ms
TTS
~1.46 s
median e2e

Latency (per turn)

0500100015001.5 s · hang-up threshold1461turn 11394turn 21286turn 31467turn 41530turn 5median ~1461 msSTTLLMTTS
Median ~1.46 s end-to-end. All five turns sit under the line where callers give up.

Highlights

  • Real-time cascade: Deepgram Nova-3 (STT) → an LLM (Claude Haiku 4.5, with Gemini 2.5 Flash as a runtime-swappable failover) → ElevenLabs Flash (TTS), orchestrated over LiveKit.
  • Median ~1.46s end-to-end latency per turn — measured and logged per turn (STT / LLM / TTS breakdown), because past ~1.5s callers assume voicemail and hang up.
  • Idempotent tool-calling: the model calls save_caller_name once; a Postgres unique index enforces exactly one row per call, so retries can’t duplicate.
  • Silero VAD + an LM end-of-turn detector decide when the caller has actually finished — no mid-sentence cut-offs.
  • Typed (mypy --strict), ≥90% test coverage on business logic, Dockerized, CI-gated. EU-resident stack for GDPR.