# WordBeat Captioner

**Version 0.1.2**

Local, open-source, multilingual, frame-accurate phrase-building captions.

WordBeat analyzes the **actual audio** in a video, obtains word-level acoustic timing through WhisperX forced alignment, groups the spoken words into complete conceptual phrases, and generates a compositor timeline where each phrase grows by one word at a time.

## Core timing rule

For each word `N`:

1. Keep the existing phrase visible.
2. Reveal word `N` only after word `N-1` has finished.
3. Reveal word `N` no later than the first video frame at its measured acoustic onset.
4. Keep word `N` visible while it is spoken.
5. Do not reveal word `N+1` until its own transition frame.
6. Do not begin a new phrase until the current phrase is complete.

When a real silent gap exists, WordBeat places the reveal inside that gap. When speech is coarticulated and no gap exists, the reveal occurs on the first frame whose timestamp is at or after the measured onset of the new word.

## Outputs

- `words.json`: measured word onset/offset/confidence data
- `timeline.json`: frame-accurate phrase-building events for an avatar/video compositor
- `captions.srt`: conventional accessibility captions
- `wordbeat.ass`: phrase-building captions suitable for FFmpeg rendering
- `preview.mp4`: optional burned-in preview

## Requirements

- Windows 11, Linux, or macOS
- Windows installer automatically installs Python 3.11 and FFmpeg when missing
- Python 3.10 or 3.11 on Linux/macOS
- FFmpeg on Linux/macOS
- NVIDIA GPU recommended; CPU mode is supported but slower
- Internet access on first setup to download open models

## Windows quick start

Open PowerShell in this folder and run. The installer checks disk space, automatically installs Python 3.11 and FFmpeg with WinGet when needed, installs WordBeat/WhisperX, then installs and verifies the correct official CUDA 12.8 PyTorch build last so WhisperX cannot replace it with CPU-only Torch:

```powershell
Set-ExecutionPolicy -Scope Process Bypass
.\install-windows.ps1
.\.venv\Scripts\Activate.ps1
python -m wordbeat.cli process "C:\path\video.mp4" --output "C:\path\wordbeat-output" --device cuda --fps 30
```

For CPU:

```powershell
python -m wordbeat.cli process "C:\path\video.mp4" --output "C:\path\wordbeat-output" --device cpu --compute-type int8 --fps 30
```

## Optional transcript correction

WhisperX first transcribes the actual audio. To correct spelling without replacing the spoken content, edit `transcript.txt`, then run:

```powershell
python -m wordbeat.cli realign "C:\path\video.mp4" "C:\path\transcript.txt" --output "C:\path\wordbeat-output" --device cuda --fps 30
```

The correction transcript must contain only words actually spoken in the recording.

## Render a preview

```powershell
python -m wordbeat.cli render "C:\path\video.mp4" "C:\path\wordbeat-output\wordbeat.ass" --output "C:\path\wordbeat-output\preview.mp4"
```

## Design goals

- Local and private
- No mandatory cloud service
- Multilingual
- Measured alignment, never evenly interpolated timing
- Explicit confidence and review flags
- Deterministic frame quantization
- Open timeline format for transparent avatars, IP-video backgrounds, QR layers, and other compositor elements

## Accuracy

WhisperX combines speech recognition, VAD, and phoneme-based forced alignment to provide word-level timestamps. For critical material, WordBeat marks low-confidence or unaligned words for review instead of silently inventing timing. WhisperX documents forced alignment and word-level timestamps; Montreal Forced Aligner can later be added as a second refinement backend.

## License

AGPL-3.0-or-later. See `LICENSE`.