Voice AI · Pipeline

Voice Dubbing

Self-hosted, open-source AI video dubbing: translate a video's speech and re-synthesize it, timed to match the original, for a fraction of commercial tool pricing.

$0.08–0.13 per 5-min video vs $0.5–2.5 / min commercial Live demo
Try the live demo View on GitHub

Problem

AI dubbing services such as ElevenLabs, HeyGen, and Rask AI charge roughly $0.5–$2.5 per minute. For indie creators, educators, and small teams who dub regularly, that puts wider reach out of budget. Using your own OpenAI API key avoids the markup, so cost stays close to the raw API price.

Approach

The input is a video upload or a YouTube URL plus a target language. Seven modules, each with a defined input/output contract:

Download upload or YouTube via yt-dlp (+ existing captions) Extract audio ffmpeg Transcribe captions if present, else Whisper → timed segments Translate GPT per segment, timestamps untouched Synthesize TTS per segment Align & stitch ffmpeg atempo fits each clip to its window Remux dubbed track into the original video → .mp4
  1. Download: save the upload, or fetch with yt-dlp (also grabbing existing .srt captions).
  2. Extract audio with ffmpeg.
  3. Transcribe: parse the captions if present, otherwise use Whisper. Output is a list of timed segments.
  4. Translate each segment with GPT, leaving timestamps untouched.
  5. Synthesize speech per segment with TTS.
  6. Align and stitch: stretch or compress each clip to its original timing window (ffmpeg atempo), then join the clips into one track.
  7. Remux the dubbed track into the original video and output an .mp4.

Results

  • Estimated $0.08–$0.13 per 5-minute video in API cost, against $2.50–$12.50 for the same five minutes on commercial tools.
  • MVP live on Streamlit (bring your own key), plus local runs with uv or Docker Compose.
  • Written PRD, architecture, engineering, and resource-estimate docs ship with the code.

Scope decisions

v1 deliberately limits scope: videos under 5 minutes, a single speaker, a generic TTS voice with no cloning, no lip-sync, and the original background audio replaced. Diarization, voice cloning, and background preservation are planned for later phases.

Stack

PythonWhisperGPTOpenAI TTSffmpegyt-dlpStreamlitDocker