Pipecat Voice AI with Twilio and Web Chat

A FastAPI + Pipecat Voice AI system that supports inbound, outbound Twilio calls, and browser-based text/voice chat from a shared assistant prompt. It is designed primarily for local deployment, with the client and server running in the same environment, but it can also be deployed on a public server by adjusting the deployment strategy and using a public server URL instead of ngrok.

• Email: rishikksh20@gmail.com

Important Note :

This repository is a basic showcase of a Voice AI agent built with Pipecat, supporting text chat, live browser voice chat, and inbound/outbound telephony calls. The same foundation can be extended into production Voice AI platforms with custom tools, long-term memory, RAG, workflow automation, analytics, and full control over the voice pipeline.

Production use cases can include sales agents, support agents, telemarketing automation, personal assistants, enterprise workflows, research assistants, and domain-specific voice agents with CRM/database integrations, human handoff, call summaries, lead qualification etc.

Although this demo uses Pipecat, I can also build similar real-time Voice AI systems with LiveKit, depending on product requirements. I can support integrations with Twilio, Telnyx, custom WebSocket serializers, OpenAI Realtime, Gemini Live, and other live voice/TTS/STT pipelines.

I am available for paid, freelance, long-term, or research-based collaborations related to advanced Voice AI products, telephony agents, real-time AI agents, TTS systems, RAG-based assistants, long-term memory, and screen-share-enabled AI agents.

If you are building or planning something in this space, or have ideas you want to explore, feel free to connect with me at rishikksh20@gmail.com.

Architecture :

┌────────────────────┐       ┌────────────────────────┐
│                    │       │                        │
│ Static Web Client  │──────▶│ FastAPI routes         │
│ Calls + Chat + RTC │       │ server/routes.py       │
│                    │       │                        │
└─────────┬──────────┘       └───────────┬────────────┘
          │                              │
          │                              │
          │                     ┌────────▼─────────┐
          │                     │ Shared settings  │
          │                     │ prompts + LLM    │
          │                     └────────┬─────────┘
          │                              │
┌─────────▼──────────┐       ┌───────────▼────────────┐       ┌─────────────────┐
│ Web text chat      │       │ Browser voice chat     │       │ Twilio calls    │
│ /web/chat/stream   │       │ /web/voice/* WebRTC    │       │ /twilio/*       │
│ WebConversationSvc │       │ WebVoiceService        │       │ TwilioService   │
└─────────┬──────────┘       └───────────┬────────────┘       └────────┬────────┘
          │                              │                             │
          │                    ┌─────────▼─────────────────────────────▼─────────┐
          │                    │ Pipecat realtime voice pipeline                 │
          │                    │ transport -> AssemblyAI STT -> OpenAI ->        │
          │                    │ Cartesia TTS -> transport                       │
          │                    └─────────────────────────────────────────────────┘
          │
          └──────────────────▶ OpenAI LLM Pipecat text pipeline

The static client uses one page with separate sections for inbound calls, outbound calls, web text chat, and browser WebRTC voice chat. Telephony and browser voice share the Pipecat realtime voice pipeline in server/bot.py. Web text uses server/conversation_service.py, which runs a lightweight Pipecat LLM pipeline and streams NDJSON events to the browser. Shared prompts are built in server/prompts/ through server/llm_service.py and inject the current date/day using the UTC/GMT offset from APP_TIMEZONE.

Interface :

Tech Stack :

• Transport: Twilio Media Streams
• LLM: OpenAI GPT-4o-mini
• STT: AssemblyAI realtime streaming
• TTS: Cartesia
• Server: FastAPI + Python
• Voice pipeline: Pipecat 1.5.0
• Web chat: Pipecat OpenAILLMService streaming
• Browser voice chat: WebRTC through Pipecat SmallWebRTC
• Browser audio transcription: OpenAI transcription API
• Client: Static HTML + JavaScript
• Python tooling: uv
• Container runtime: Docker / Docker Compose

Project Structure :

.
├── client/
│   └── index.html
├── server/
│   ├── __init__.py
│   ├── bot.py
│   ├── call_store.py
│   ├── config.py
│   ├── conversation_service.py
│   ├── llm_service.py
│   ├── main.py
│   ├── prompts/
│   ├── routes.py
│   └── twilio_service.py
├── scripts/
│   └── check_env.py
├── tests/
├── .dockerignore
├── .env.example
├── .python-version
├── Dockerfile
├── AGENTS.md
├── docker-compose.yml
├── pyproject.toml
└── README.md

Environment :

Copy the example file and fill in your real values:

cp .env.example .env

The real .env file is intentionally ignored by Git and by Docker builds.

Variable	Required	Used for	Notes
OPENAI_API_KEY	Yes	LLM responses	OpenAI API key for GPT-4o-mini.
OPENAI_MODEL	No	LLM responses	Defaults to gpt-4o-mini.
OPENAI_TRANSCRIPTION_MODEL	No	Browser audio transcription	Defaults to whisper-1.
ASSEMBLYAI_API_KEY	Yes	Speech-to-text	AssemblyAI key for realtime streaming STT.
CARTESIA_API_KEY	Yes	Text-to-speech	Cartesia key for generated voice audio.
CARTESIA_VOICE_ID	No	Text-to-speech voice	Defaults to a0e99841-438c-4a64-b679-ae501e7d6091.
TWILIO_ACCOUNT_SID	Yes	Twilio REST API	Required for outbound calls and ending calls.
TWILIO_AUTH_TOKEN	Yes	Twilio REST API	Required with TWILIO_ACCOUNT_SID.
TWILIO_PHONE_NUMBER	Yes	Inbound/outbound calls	Must be an active Twilio number, for example +1234567890.
SERVER_URL	Yes for Twilio calls	Webhooks and outbound calls	Public HTTPS URL from ngrok or your deployed server, without trailing slash. You can set or update it from the Env Variables tab after deployment.
PORT	No	Local server port	Defaults to 8000.
APP_TIMEZONE	No	Prompt date/day context	UTC/GMT offset for prompt injection, for example UTC+05:30; defaults to UTC+05:30.
PROMPT_INTRO	No	Prompt customization	Optional assistant intro override.
PROMPT_BODY	No	Prompt customization	Optional assistant behavior/body override.
PROMPT_OUTRO	No	Prompt customization	Optional default conversation-ending message.
HOST_PORT	No	Docker Compose host port	Defaults to 8000; only used by docker-compose.yml.

Local Deployment with ngrok :

For local development on the same PC, the web UI and peer-to-peer WebRTC voice chat can use http://localhost:8000. A public HTTPS URL is required when Twilio must reach the local server, or when the web client is opened from another device.

1. In terminal, expose the same port through ngrok:

   ngrok http 8000

2. Copy the HTTPS forwarding URL shown by ngrok and set it in .env without a trailing slash, or open the app and save it from the Env Variables tab:

   SERVER_URL=https://abc123.ngrok-free.app

3. Start/restart the FastAPI server after changing .env manually. If you update SERVER_URL from the Env Variables tab, the running process uses the new value immediately.

   • http://localhost:8000 when using the same PC.
   • The ngrok HTTPS URL when testing from another device or through the public internet.

SERVER_URL is important because Twilio cannot call localhost. The server uses this public base URL to build outbound-call TwiML and status callback URLs. Configure the inbound Twilio webhook with the same public URL:

https://abc123.ngrok-free.app/twilio/incoming

Use POST as the webhook method.

Browser voice chat uses the origin from which the web page was loaded for /web/voice/* WebRTC signaling requests. It does not read SERVER_URL directly. Localhost is accepted as a secure browser context, but remote microphone access requires HTTPS, which the ngrok URL provides.

ngrok exposes the web application and WebRTC signaling endpoints; it does not relay peer-to-peer WebRTC audio. Connections across different networks or strict NAT/firewall environments may additionally require STUN/TURN configuration. Also remember that a temporary ngrok URL can change when ngrok restarts. Update SERVER_URL from the Env Variables tab or .env, and update the Twilio webhook whenever it changes.

Deployment URL Handling

Some hosts, including Vercel-style platforms, only provide the final public URL after deployment. In that case:

1. Deploy with the provider credentials configured, leaving SERVER_URL empty if the URL is not known yet.
2. Open the deployed app over HTTPS.
3. Go to Env Variables, set SERVER_URL to the deployed HTTPS URL, then save.

The save reloads env-backed settings in the live FastAPI process immediately. If the deployment filesystem is read-only, the UI will show that the value is active only for the current process. For durable values across cold starts, redeploys, or multiple instances, also set SERVER_URL in the hosting provider's environment variable settings.

Run Without Docker

Use this for local development. The commands below run from the repo root.

Install dependencies with uv:

uv sync --frozen

Start the server:

uv run python -m server.main

Open the app:

http://localhost:8000

Run With Docker

Build and run with Docker Compose:

docker compose up --build

Compose reads values from your shell or local .env file. The container listens on port 8000; set HOST_PORT if you want a different host port:

HOST_PORT=8080 docker compose up --build

You can also run plain Docker:

docker build -t pipecat-voice-ai-agent .
docker run --rm -p 8000:8000 --env-file .env pipecat-voice-ai-agent

Open the app:

http://localhost:8000

Configure Twilio :

For inbound calls, configure your Twilio phone number webhook:

• URL: ${SERVER_URL}/twilio/incoming
• Method: POST

See Local Deployment with ngrok when running the server on your PC.

Usage

Inbound calls:

1. Open http://localhost:8000.
2. Select the Receive Call tab.
3. Dial your Twilio phone number.
4. Talk to the AI assistant.

Outbound calls:

1. Open http://localhost:8000.
2. Select the Make Call tab.
3. Enter a phone number with country code.
4. Click Call.

Web chat and browser voice:

1. Open http://localhost:8000.
2. Select the Web Chat tab.
3. Type a message for streamed text chat.
4. Click Start Voice Chat in a browser that supports WebRTC to use live microphone input and spoken replies.
5. While a voice session is connected, typed messages can also be sent into the same voice session.

Call ending:

• The assistant prompt advertises an end_conversation tool.
• When a caller clearly says they are done, says goodbye, or the conversation naturally finishes, the model calls that tool.
• The bot queues a short thank-you/outro message first, then ends the Twilio media stream so inbound and outbound calls close naturally.

API Endpoints :

Endpoint	Method	Description
/	GET	Serve the web UI
/api/info	GET	Server info and configured Twilio number
/health	GET	Health check
/twilio/incoming	POST	Twilio webhook for incoming calls
/twilio/ws/{call_id}	WS	Twilio Media Streams WebSocket
/twilio/outbound-handler/{call_id}	GET/POST	TwiML handler for outbound calls
/twilio/status/{call_id}	POST	Twilio call status callback
/call/start	POST	Prepare the UI for an inbound call
/call/outbound	POST	Start an outbound call
/call/end/{call_id}	POST	End an active call
/call/status/{call_id}	GET	Get call status
/web/chat	POST	Send a text message to the shared assistant
/web/chat/stream	POST	Stream a text-chat assistant response as NDJSON events
/web/transcribe	POST	Transcribe uploaded browser audio
/web/voice/offer	POST	Start a browser voice WebRTC session from an SDP offer
/web/voice/ice	POST	Add ICE candidates for a browser voice session
/web/voice/message	POST	Send a text message into a browser voice session
/web/voice/end/{session_id}	POST	End a browser voice session
/web/session/{session_id}/transcript	GET	Get a web chat or browser voice session transcript
/env	GET/POST	Read and update supported runtime env vars
/prompt	GET/POST	Read and update prompt overrides
/active-calls	GET	List active calls

Validation and Tests :

Run credential format checks without calling provider APIs:

uv run python scripts/check_env.py

Run automated tests:

uv run pytest

Use scripts/check_env.py --live only when you intentionally want to call OpenAI, AssemblyAI, Cartesia, and Twilio APIs.

Pipecat Version Notes :

The project is migrated to pipecat-ai[assemblyai,cartesia,openai,silero,webrtc,websocket]==1.5.0.

• The bot uses LLMContext and LLMContextAggregatorPair.
• Prompt text is centralized in server/prompts/ and injects the current date, ISO date, and day on each run.
• The Pipecat context exposes the shared end_conversation tool so the model can end telephony and browser voice conversations after speaking an outro.
• OpenAI, AssemblyAI, and Cartesia configuration now uses Pipecat 1.5 service Settings objects where applicable.
• VAD is configured on LLMUserAggregatorParams, not the FastAPI WebSocket transport.
• The AssemblyAI wrapper only handles extra stream events that are not part of the normal 1.5 transcription path.
• Re-test one inbound and one outbound Twilio call after deployment because live Twilio media streams are not covered by unit tests.
• Re-test browser WebRTC voice after deployment because NAT, HTTPS, and TURN/STUN behavior depends on the hosting environment.

Notes :

• .env is intentionally ignored by Git and Docker build context.
• pyproject.toml and uv.lock are the dependency sources.
• Keep PORT=8000 inside the container unless you also change the Docker port mapping.

License

MIT