voice-ai-engine-development
Skill35 starsChanged 6 months agoMITView on GitHub
Build production-ready real-time conversational voice AI engines using async worker pipelines, Gemini Live API for streaming STT, Gemini TTS for synthesis, LangGraph or Google ADK agents, interrupt handling, and FastAPI WebSocket integration. Use when building voice assistants, real-time voice bots, or conversational AI pipelines with full interrupt support.
- Reads credentials
3 other projects use a version of this.
Adopted by
Most recently taken up by
- ahmedibm9-cyber/VibeGuardtheir own version, first seen Oct 20, 2026
- xfstudio/skillstheir own version, first seen Oct 13, 2026
- ai4brands-design/claude-skillstheir own version, first seen Mar 22, 2026
Other versions3 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 3
- xfstudio/skills★ 9same name
- ai4brands-design/claude-skills★ 2same name
- ahmedibm9-cyber/VibeGuard★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository157 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- a2ui-angularSkill
- accessibility-auditSkill
- adk-deploy-guideSkill
- adk-dev-guideSkill
- adk-eval-guideSkill
- adk-observability-guideSkill
- agentic-ai-coding-standardSkill
- agentic-ai-devSkill
- ai-auditSkill
- ai-chatSkill
- ai-decision-recordSkill
- ai-incident-responseSkill
- ai-launch-checkSkill
- ai-playbookSkill
- angular-best-practicesSkill
- angularSkill
- angular-spaSkill
- angular-ui-patternsSkill
- api-design-principlesSkill
- app-store-optimizationSkill
- architect-reviewSkill
- architecture-decision-recordsSkill
- architecture-designSkill
- asc-cli-usageSkill
- asc-crash-triageSkill
- asc-id-resolverSkill
- asc-release-flowSkill
- asc-signing-setupSkill
- asc-submission-healthSkill
- asc-testflight-orchestrationSkill
- browser-testingSkill
- changelog-generatorSkill
- claude-actions-auditorSkill
- clean-codeSkill
- codebase-onboardingSkill
- code-explainerSkill
- code-reviewerSkill
- code-simplifierSkill
- comment-analyzerSkill
- database-schema-designerSkill
- ddd-architectSkill
- decision-frameworksSkill
- dedup-code-agentSkill
- deployment-ci-cdSkill
- design-systemSkill
- dockerSkill
- documentation-generationSkill
- domain-finderSkill
- error-detectiveSkill
- eval-guideSkill
- feature-forgeSkill
- firebase-basicsSkill
- firebase-hosting-basicsSkill
- fixing-accessibilitySkill
- fixing-motion-performanceSkill
- flutter-animationsSkill
- flutter-genuiSkill
- flutter-mobileSkill
- flutter-security-expertSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
4 rules. 0 are common, 2 are distinctive, and 2 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
2
Distinctive
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 62
Before generating code, consult these sources:
- Very specific: line 96
Iron rule: Every except block in a worker loop MUST:
- Very specific: line 117
No
print()anywhere: Use structlog.get_logger(). - Distinctive: line 121
After writing voice engine code, dispatch:
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Iron Law
NO VOICE ENGINE WITHOUT INTERRUPT HANDLING AND ASYNC WORKER ISOLATION — EVERY COMPONENT RUNS IN ITS OWN QUEUE
A voice engine without interrupts is a demo, not a product. Every worker (transcriber, agent, synthesizer) MUST run in an isolated asyncio.Queue-based loop. Interrupt handling MUST be wired before any other feature is added. No exceptions.
Voice AI Engine Development — Gemini Live API + Python 3.14 + FastAPI
Quick Scaffold
uv init voice-engine && cd voice-engine
uv add "google-genai>=1.0.0" "fastapi>=0.135.2" "uvicorn[standard]" \
"websockets>=13.0" pydantic pydantic-settings structlog \
"langgraph>=1.0.7" "langchain-core>=1.2.8" \
"google-adk>=1.28.0" pydub numpy aiolimiter
uv add --dev pytest pytest-asyncio httpx ruff mypyProcess
- Scaffold —
uv init+ installgoogle-genai(NOT google-generativeai) and dependencies - Configure —
core/config.pywith pydantic-settings,.env, structured logging (noprint()) - Implement BaseWorker — asyncio.Queue input/output,
start(),_run_loop(),terminate() - Implement GeminiTranscriberWorker — WebSocket to Gemini Live API, mute/unmute for echo prevention
- Implement Agent Worker — either
LangGraphAgentWorker(StateGraph) orADKAgentWorker(SequentialAgent) - Implement GeminiSynthesizerWorker —
gemini-2.5-flash-tts-previeworgemini-2.5-pro-tts-preview - Wire Pipeline — transcriber → agent → synthesizer via asyncio.Queue;
StreamingConversationorchestrator - Add Interrupt System —
InterruptibleEvent,broadcast_interrupt(), rate-limited chunk playback, mute logic - Expose WebSocket — FastAPI
/conversationendpoint,asynccontextmanagerlifespan (NEVER@app.on_event) - Test — unit test each worker in isolation, integration test full pipeline, test interrupt path
- Error Handling — every except block MUST log + rethrow; no silent returns; no mock data
Key Patterns
| Pattern | Implementation | Reference |
| Worker Pipeline | BaseWorker → asyncio.Queue → isolated _run_loop | reference/worker-pipeline.md |
| Gemini Live STT | GeminiTranscriberWorker — WebSocket to gemini-live-2.5-flash-native-audio | reference/worker-pipeline.md |
| Gemini TTS | GeminiSynthesizerWorker — gemini-2.5-flash-tts-preview / gemini-2.5-pro-tts-preview | reference/worker-pipeline.md |
| Interrupt System | InterruptibleEvent + broadcast_interrupt() + rate-limited chunks + transcriber mute | reference/interrupt-handling.md |
| Provider Factory | VoiceComponentFactory — config-driven worker creation | reference/worker-pipeline.md |
| WebSocket Integration | FastAPI /conversation + asynccontextmanager lifespan + WebsocketOutputDevice | reference/worker-pipeline.md |
| Error Recovery | Worker loop catches + logs + reraises; session reconnect for Gemini Live timeouts | reference/interrupt-handling.md |
Documentation Sources
Before generating code, consult these sources:
| Source | URL / Tool | Purpose |
| Gemini Live API | https://ai.google.dev/gemini-api/docs/llms.txt | Live API WebSocket protocol, audio format, barge-in |
| google-genai SDK | Context7 MCP → resolve google-genai | Current SDK API signatures |
| LangGraph | https://langchain-ai.github.io/langgraph/llms-full.txt | StateGraph, nodes, edges |
| Google ADK | Context7 MCP → resolve google-adk | SequentialAgent, session patterns |
| FastAPI | Context7 MCP → resolve fastapi | WebSocket, lifespan patterns |
Reference Files
| File | Content | When to Use |
reference/gemini-provider-setup.md | google-genai install, Live API WebSocket setup, model names, audio format, auth, rate limits | First — before writing any Gemini code |
reference/worker-pipeline.md | BaseWorker/BaseTranscriber/BaseAgent/BaseSynthesizer + GeminiTranscriberWorker + GeminiSynthesizerWorker + LangGraphAgentWorker + ADKAgentWorker + FastAPI wiring | Core pipeline implementation |
reference/interrupt-handling.md | InterruptibleEvent, broadcast_interrupt, rate-limited playback, transcriber mute, state machine, graceful shutdown | Interrupt system implementation |
reference/provider-comparison.md | Gemini vs Deepgram vs ElevenLabs vs Azure — latency, cost, features, when to use alternatives | Provider selection |
reference/common-pitfalls.md | Audio jumping, echo feedback, interrupts not working, Gemini-specific traps, session timeouts | Debugging and prevention |
Common Commands
uvicorn src.main:app --reload # Run dev server with hot reload
pytest -q # Run all tests
pytest -q tests/test_workers.py # Test worker isolation
pytest -q tests/test_interrupt.py # Test interrupt path
ruff check --fix . # Lint and auto-fix
ruff format . # Format code
mypy src/ # Type checkError Handling
Iron rule: Every except block in a worker loop MUST:
- Log with
structlogaterrorlevel (include full exception context) - Either rethrow OR return a proper error state (never return empty/None/mock)
# Required pattern in every worker loop
async def _run_loop(self) -> None:
while self.active:
try:
item = await self.input_queue.get()
await self.process(item)
except asyncio.CancelledError:
raise # Never swallow CancelledError
except Exception as exc:
self._log.error("worker_error", worker=self.__class__.__name__, error=str(exc), exc_info=True)
# Continue loop — worker recovers from transient errors
# For fatal errors: self.terminate(); raiseGemini Live session timeouts: Sessions expire after ~10 minutes. Implement reconnect with exponential backoff. See reference/interrupt-handling.md § Error Recovery.
No print() anywhere. Use structlog.get_logger().
Post-Code Review
After writing voice engine code, dispatch:
security-reviewer— WebSocket input validation, API key handlingagentic-ai-reviewer(if LangGraph agent) — graph correctness, iteration limitscode-reviewer— async patterns, resource cleanup, error handling
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

