agentleFS
kumaran-iskumaran-is/claude-code-onboarding/.claude/skills/voice-ai-engine-development/SKILL.md

voice-ai-engine-development

Skill35 starsChanged 6 months agoMITView on GitHub

Build production-ready real-time conversational voice AI engines using async worker pipelines, Gemini Live API for streaming STT, Gemini TTS for synthesis, LangGraph or Google ADK agents, interrupt handling, and FastAPI WebSocket integration. Use when building voice assistants, real-time voice bots, or conversational AI pipelines with full interrupt support.

  • Reads credentials

3 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • 0

    Common

    Most of the 3 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Before generating code, consult these sources:

    line 62
  2. Very specific:

    Iron rule: Every except block in a worker loop MUST:

    line 96
  3. Very specific:

    No print() anywhere: Use structlog.get_logger().

    line 117
  4. Distinctive:

    After writing voice engine code, dispatch:

    line 121

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Iron Law

NO VOICE ENGINE WITHOUT INTERRUPT HANDLING AND ASYNC WORKER ISOLATION — EVERY COMPONENT RUNS IN ITS OWN QUEUE

A voice engine without interrupts is a demo, not a product. Every worker (transcriber, agent, synthesizer) MUST run in an isolated asyncio.Queue-based loop. Interrupt handling MUST be wired before any other feature is added. No exceptions.

Voice AI Engine Development — Gemini Live API + Python 3.14 + FastAPI

Quick Scaffold

uv init voice-engine && cd voice-engine
uv add "google-genai>=1.0.0" "fastapi>=0.135.2" "uvicorn[standard]" \
  "websockets>=13.0" pydantic pydantic-settings structlog \
  "langgraph>=1.0.7" "langchain-core>=1.2.8" \
  "google-adk>=1.28.0" pydub numpy aiolimiter
uv add --dev pytest pytest-asyncio httpx ruff mypy

Process

  1. Scaffold — uv init + install google-genai (NOT google-generativeai) and dependencies
  2. Configure — core/config.py with pydantic-settings, .env, structured logging (no print())
  3. Implement BaseWorker — asyncio.Queue input/output, start(), _run_loop(), terminate()
  4. Implement GeminiTranscriberWorker — WebSocket to Gemini Live API, mute/unmute for echo prevention
  5. Implement Agent Worker — either LangGraphAgentWorker (StateGraph) or ADKAgentWorker (SequentialAgent)
  6. Implement GeminiSynthesizerWorker — gemini-2.5-flash-tts-preview or gemini-2.5-pro-tts-preview
  7. Wire Pipeline — transcriber → agent → synthesizer via asyncio.Queue; StreamingConversation orchestrator
  8. Add Interrupt System — InterruptibleEvent, broadcast_interrupt(), rate-limited chunk playback, mute logic
  9. Expose WebSocket — FastAPI /conversation endpoint, asynccontextmanager lifespan (NEVER @app.on_event)
  10. Test — unit test each worker in isolation, integration test full pipeline, test interrupt path
  11. Error Handling — every except block MUST log + rethrow; no silent returns; no mock data

Key Patterns

PatternImplementationReference
Worker PipelineBaseWorker → asyncio.Queue → isolated _run_loopreference/worker-pipeline.md
Gemini Live STTGeminiTranscriberWorker — WebSocket to gemini-live-2.5-flash-native-audioreference/worker-pipeline.md
Gemini TTSGeminiSynthesizerWorker — gemini-2.5-flash-tts-preview / gemini-2.5-pro-tts-previewreference/worker-pipeline.md
Interrupt SystemInterruptibleEvent + broadcast_interrupt() + rate-limited chunks + transcriber mutereference/interrupt-handling.md
Provider FactoryVoiceComponentFactory — config-driven worker creationreference/worker-pipeline.md
WebSocket IntegrationFastAPI /conversation + asynccontextmanager lifespan + WebsocketOutputDevicereference/worker-pipeline.md
Error RecoveryWorker loop catches + logs + reraises; session reconnect for Gemini Live timeoutsreference/interrupt-handling.md

Documentation Sources

Before generating code, consult these sources:

SourceURL / ToolPurpose
Gemini Live APIhttps://ai.google.dev/gemini-api/docs/llms.txtLive API WebSocket protocol, audio format, barge-in
google-genai SDKContext7 MCP → resolve google-genaiCurrent SDK API signatures
LangGraphhttps://langchain-ai.github.io/langgraph/llms-full.txtStateGraph, nodes, edges
Google ADKContext7 MCP → resolve google-adkSequentialAgent, session patterns
FastAPIContext7 MCP → resolve fastapiWebSocket, lifespan patterns

Reference Files

FileContentWhen to Use
reference/gemini-provider-setup.mdgoogle-genai install, Live API WebSocket setup, model names, audio format, auth, rate limitsFirst — before writing any Gemini code
reference/worker-pipeline.mdBaseWorker/BaseTranscriber/BaseAgent/BaseSynthesizer + GeminiTranscriberWorker + GeminiSynthesizerWorker + LangGraphAgentWorker + ADKAgentWorker + FastAPI wiringCore pipeline implementation
reference/interrupt-handling.mdInterruptibleEvent, broadcast_interrupt, rate-limited playback, transcriber mute, state machine, graceful shutdownInterrupt system implementation
reference/provider-comparison.mdGemini vs Deepgram vs ElevenLabs vs Azure — latency, cost, features, when to use alternativesProvider selection
reference/common-pitfalls.mdAudio jumping, echo feedback, interrupts not working, Gemini-specific traps, session timeoutsDebugging and prevention

Common Commands

uvicorn src.main:app --reload                          # Run dev server with hot reload
pytest -q                                              # Run all tests
pytest -q tests/test_workers.py                        # Test worker isolation
pytest -q tests/test_interrupt.py                      # Test interrupt path
ruff check --fix .                                     # Lint and auto-fix
ruff format .                                          # Format code
mypy src/                                              # Type check

Error Handling

Iron rule: Every except block in a worker loop MUST:

  1. Log with structlog at error level (include full exception context)
  2. Either rethrow OR return a proper error state (never return empty/None/mock)
# Required pattern in every worker loop
async def _run_loop(self) -> None:
    while self.active:
        try:
            item = await self.input_queue.get()
            await self.process(item)
        except asyncio.CancelledError:
            raise  # Never swallow CancelledError
        except Exception as exc:
            self._log.error("worker_error", worker=self.__class__.__name__, error=str(exc), exc_info=True)
            # Continue loop — worker recovers from transient errors
            # For fatal errors: self.terminate(); raise

Gemini Live session timeouts: Sessions expire after ~10 minutes. Implement reconnect with exponential backoff. See reference/interrupt-handling.md § Error Recovery.

No print() anywhere. Use structlog.get_logger().

Post-Code Review

After writing voice engine code, dispatch:

  • security-reviewer — WebSocket input validation, API key handling
  • agentic-ai-reviewer (if LangGraph agent) — graph correctness, iteration limits
  • code-reviewer — async patterns, resource cleanup, error handling

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.