Speech to Speech

The speech-to-speech library from Hugging Face assembles a four-stage voice-agent pipeline — voice activity detection, speech-to-text, a tool-calling LLM, and text-to-speech — into a single pip-installable package. The pipeline is exposed through an OpenAI Realtime-compatible WebSocket API, so any client already speaking that protocol connects without modification. CLI flags let you redirect the LLM slot to a local server, making fully air-gapped operation a configuration choice rather than a fork.

The differentiating design decision is that every component is independently swappable. You are not buying a bundled stack — you are wiring together open-source models where the contract between stages is the protocol, not the vendor. That means swapping the STT model for a domain-specific alternative, or routing LLM calls to llama.cpp on embedded hardware, is a flag change rather than a system rewrite. The Apache-2.0 license means you can modify and ship it without royalty conversations.

This fits developers building local voice agents, robotics teams running inference on-device, and engineers who need an OpenAI Realtime-compatible backend without the API bill. It breaks down when you need guaranteed low-latency behavior under load: the modular architecture means latency compounds across stages, and there is no managed infrastructure absorbing the spike. Teams shipping to consumers who expect sub-second response at scale typically instrument each stage separately and often replace one or more components with optimized alternatives as usage grows.

Installation is via pip, with Docker and Docker Compose files in the repo for containerized deployment, including an ARM64 Dockerfile for embedded targets. The repo documents integration with HF Inference Providers and local OpenAI-compatible servers such as vLLM and llama.cpp, covering the range from cloud-backed to fully offline deployments.