49 lines
2.2 KiB
Markdown
49 lines
2.2 KiB
Markdown
# Speech-to-Text worker — Whisper (RunPod Serverless)
|
|
|
|
Multilingual voice-to-text via **faster-whisper** — lets a worker **speak**
|
|
instead of typing annotation text, in both the image and video editors. 99
|
|
languages incl. Hindi/regional + English. Chosen over Parakeet/NeMo because it
|
|
has no heavy dependencies (no NeMo, no torch), so it builds cleanly.
|
|
|
|
## Contract
|
|
|
|
Request → `POST /runsync`:
|
|
```json
|
|
{ "input": { "audio": "<base64 audio>", "language": "en", "timestamps": false } }
|
|
```
|
|
- `audio`: base64 wav/mp3/m4a (any sample rate — resampled internally)
|
|
- `language`: optional (e.g. `"hi"`, `"en"`); omit to auto-detect
|
|
- `timestamps`: optional — per-segment start/end
|
|
|
|
Response → `{ "output": { "transcript": "...", "language": "en" } }`
|
|
(with `segments: [{text, start_sec, end_sec}]` if `timestamps: true`), or `{ "output": { "error": "..." } }`.
|
|
|
|
## Deploy (new endpoint — one per model)
|
|
|
|
1. **RunPod → Serverless → New Endpoint → Deploy from a GitHub repository** → `Sumit-Pluto/photo_gallery`.
|
|
2. **Dockerfile path:** `/workers/voice-to-text/Dockerfile`
|
|
3. Config:
|
|
- **GPU:** light — **16 GB (T4 / A4000)** is plenty (`large-v3` ≈ 5 GB VRAM).
|
|
- **Network volume:** attach your weights volume (caches the model).
|
|
- **Workers:** Max 1, Active 0, FlashBoot on. **Execution timeout** ~300s (first-run download).
|
|
4. **Warm up once** (downloads the model, ~1.5 GB):
|
|
```bash
|
|
curl -X POST https://api.runpod.ai/v2/<NEW_ID>/runsync \
|
|
-H "Content-Type: application/json" -H "Authorization: Bearer <API_KEY>" \
|
|
-d '{"input":{"audio":"<base64-wav>","language":"en"}}'
|
|
```
|
|
5. In **Vercel**, set `RUNPOD_STT_URL` = `https://api.runpod.ai/v2/<NEW_ID>/runsync` (server-only).
|
|
|
|
## Tuning (env vars)
|
|
|
|
| Var | Default | Purpose |
|
|
|---|---|---|
|
|
| `WHISPER_MODEL` | `large-v3` | accuracy vs speed — `medium` / `small` are faster/lighter |
|
|
| `WHISPER_COMPUTE` | `float16` | `int8_float16` uses less VRAM |
|
|
|
|
## App integration (next step, not yet built)
|
|
|
|
Mic-record button in the editor's text/annotation tool → `/api/ai/transcribe`
|
|
(proxies here) → inserts the transcript. Client `rpTranscribe()` already exists in
|
|
`apps/web/src/lib/runpod/endpoints.ts`.
|