diff --git a/.claude/memory/status.md b/.claude/memory/status.md index ba5afbf..f4a4ba1 100644 --- a/.claude/memory/status.md +++ b/.claude/memory/status.md @@ -298,7 +298,54 @@ Phases 1–3 complete & verified (see [requirements.md](requirements.md)). `pnpm DEPLOY: `docs/DEPLOY-MANUAL.md` — 3 no-Git paths: (A) Vercel CLI `vercel --prod` from repo root, Root Dir=apps/web (recommended; keeps SSR+nonce CSP+AI route); (B) self-host Node (`pnpm build` → `pnpm --filter web start` on Render/Railway/VPS); (C) static export to any host — requires `output:'export'` + DISABLING the nonce middleware/layout headers (security trade-off, no server AI route). SDK ships as source via transpilePackages so "whole project" = build the demo (no npm publish). AI: `apps/web/src/app/api/ai/edit/route.ts` REFACTORED to a pluggable backend via `AI_EDIT_PROVIDER` (auto|local|huggingface|gemini|none). auto = local→huggingface→gemini. `local` = own Stable Diffusion (AUTOMATIC1111/Forge/SD.Next `/sdapi/v1/img2img`, env LOCAL_SD_URL+denoise/steps/sampler; free+private). `huggingface` = free Inference API instruct-pix2pix (HF_API_TOKEN, HF_IMAGE_MODEL; 503 cold-start handled). `gemini` kept (needs BILLED key for image). MAX_BASE64 lowered to 4M (under Vercel 4.5MB). Server-side fetches → no CSP change. KEY POINT documented in `docs/AI-SETUP.md`: analysis (objects/faces/OCR/embeddings via TF.js/face-api/tesseract/transformers) + Remove Background (@imgly) already run FREE in-browser, no key — only generative pixel edits needed a backend. `.env.example` rewritten with Supabase + all AI options. typecheck + build pass. Did NOT git commit. +- S-runpod (2026-07-18): **RunPod endpoint integration (model-api-spec.docx) — image edits + detection, all server-proxied.** + Planned via an `ultracode` Workflow (5 parallel readers → Opus blueprint). GOAL: make the app talk to the 15 RunPod + endpoints (server still down; wired + typecheck/build-verified, not yet live-tested against real endpoints). + NEW server-only client module `apps/web/src/lib/runpod/{types,base64,client,endpoints}.ts`: `runpodCall` transport + (bearer `RUNPOD_API_KEY`, `{input}` envelope, `/runsync` inline + `/run`→`/status/{id}` poll under a 55s budget), + `pickOutputImage` (handles image|image_png|images[0]|nested), and one typed `rp*` fn per endpoint. `RunpodError.status` + surfaces as the editor's red error text. + EDIT ROUTE `apps/web/src/app/api/ai/edit/route.ts`: added `'runpod'` provider (auto-preferred when RUNPOD_API_KEY + + an SD URL set). `editRunPod` op→endpoint map: restore→#7 ESRGAN(+face), upscale→#7, colorize→#8 DDColor, prompt→#10 + SD img2img, replace-sky→#9 inpaint IF mask else #10 low-strength, magic-eraser/generative-fill→#9 masked inpaint + (400 if no mask). Body now accepts `maskBase64` + `params` (negativePrompt/strength/steps/seed/guidanceScale, all + clamped); image+mask share the ONE ~4MB body cap. RUNPOD_ONLY_OPS (upscale/magic-eraser/generative-fill) 400 on + non-runpod backends. + MASK FLOW (the SD 3.5 ask): client `createDemoAIProvider.generativeEdit` rasterizes `op.mask` (ImageData) → PNG + matched to the downscaled image dims via new `lib/ai/imageEncode.ts maskToBase64`, strips the non-serializable mask + from the wire op, sends `maskBase64`. Output interpreted unchanged (base64→Blob→aiResultUrl→Save), so no UI change + needed to SHOW results. STILL MISSING: a brush-mask UI in PhotoEditor to CREATE masks (magic-eraser/generative-fill + have no button yet + return the 400 until then); replace-sky degrades to img2img until a mask/sky-seg exists. + DETECTION #1: new `apps/web/src/app/api/ai/classify/route.ts` + `lib/ai/runpodYoloProvider.ts` (POSTs image+dims, + normalizes YOLO boxes→DetectedObject fractions 0..1 — `normalizeBox` assumes ultralytics xyxy pixels, VERIFY vs real + endpoint). Gated by `NEXT_PUBLIC_APG_RUNPOD_DETECT=true`, else in-browser COCO-SSD (also the automatic fallback). + ALREADY LOCAL (no RunPod): screenshot #4 / geo #15 (exifr+classify.ts), tag-rename #5 (state), bg-remove #6 (@imgly). + SCAFFOLD ONLY (typed rp* + env, NO route/UI): tilt #2, voice #3, audio-denoise #12; DOC-ONLY: video #13/#14 (need + out-of-band upload+queue, incompatible with Vercel 4.5MB/60s + in-browser bake). All flagged in docs/AI-SETUP.md. + CSP UNCHANGED (server-side fetch bypasses browser CSP; middleware excludes /api/*). ENV: `.env.example` RunPod block; + docs/AI-SETUP.md "Option 0 — RunPod" (op→endpoint table + mask/detection notes) + docs/DEPLOY.md server-env table + (RUNPOD_* never NEXT_PUBLIC). VERIFIED: `pnpm -r typecheck` clean (both pkgs); `pnpm build` passes, both new routes + compiled (/api/ai/edit, /api/ai/classify dynamic), /gallery 199KB, only the pre-existing benign face-api warning. + Did NOT git commit. NEXT: brush-mask UI (unlocks true #9 for eraser/fill + real sky mask); outpaint #11 (client pads + canvas+mask→generative-fill); #5 alias rename map; then Vercel deploy once endpoints are up. VERIFY normalizeBox + + the `{input}`/`output` field names against the live endpoints before trusting detection/edit results. + ADVERSARIAL REVIEW (ultracode Workflow, 3 dims find→verify, 20 agents; 12 CONFIRMED, 5 refuted) — ALL 12 FIXED & + re-verified (typecheck + build pass, both routes compiled, /gallery 199KB): (1) normalizeBox rewritten to handle + object-form xyxy `{x1,y1,x2,y2}`/`{left,top,right,bottom}` (ultralytics tojson shape — was dropped!) + array boxes + keyed correctly (xyxy vs xywh/COCO bbox=[x,y,w,h] vs generic box=xyxy); (2) label priority fixed — string `name` + before numeric `class` index (was returning "0" for ultralytics), empty-string label falls through to class_; + (3) envNum treats blank env ('' → 0) as unset; (4) rpUpscale no longer lets RUNPOD_UPSCALE_SCALE override the + per-request factor (removed the env knob); (5) client envelope: a raw /runsync body with a `status` field but no + `id` is now correctly treated as inline output (was misrouted to poll→throw); (6) poll loop clamps sleep+abort to + the remaining budget so it can't overshoot Vercel 60s; (7) pickOutputImage validates strings under generic + output/result/data keys (looksLikeImageData) so a status/id string isn't returned as an image; (8) auto provider + detects runpod on ANY RUNPOD_*_URL (not just SD) so an upscale/colorize-only deploy works; (9) maskToBase64 sets + imageSmoothingEnabled=false so binary masks stay crisp (no grey halo). Dropped RUNPOD_UPSCALE_SCALE from .env.example. + REFUTED (correctly, no change): status-URL query-string, combined-body-cap rejecting masks, replace-sky client + strength, 'mask' in op undefined-crash, mask-op has no UI button (all non-bugs / already-safe). + ## Key files +- RunPod integration (S-runpod): `apps/web/src/lib/runpod/*` (server) + `apps/web/src/app/api/ai/{edit,classify}/route.ts` + `apps/web/src/lib/ai/{runpodYoloProvider,imageEncode}.ts` (client). Env: `RUNPOD_API_KEY` + `RUNPOD_*_URL`. - Advanced video editor (S-advanced): `packages/photo-sdk/src/lib/videoBake.ts` + `lib/videoTimeline.ts` + `components/editor/VideoEditor.tsx`. - AI edit backend (S-deploy-ai): `apps/web/src/app/api/ai/edit/route.ts` (env AI_EDIT_PROVIDER). - SDK: `packages/photo-sdk/src/{components,store,adapters,ai,lib,icons,styles}`. diff --git a/.env.example b/.env.example index 7a540a8..ae4906a 100644 --- a/.env.example +++ b/.env.example @@ -13,7 +13,23 @@ # NOT required to run the app: object detection, faces, OCR, and Remove Background # all run FREE in-browser with no key. Full guide: docs/AI-SETUP.md # --------------------------------------------------------------------------- -# AI_EDIT_PROVIDER=auto # auto | local | huggingface | gemini | none +# AI_EDIT_PROVIDER=auto # auto | runpod | local | huggingface | gemini | none +# +# Option 0 — RunPod serverless GPU endpoints (one per model; key stays server-side). +# `auto` picks this first when RUNPOD_API_KEY + an SD URL are set. Deploy endpoints so +# their URLs end in /runsync. Full map (op → endpoint) in docs/AI-SETUP.md. +# AI_EDIT_PROVIDER=runpod +# RUNPOD_API_KEY=rpa_xxxxxxxx +# RUNPOD_SD_IMG2IMG_URL=https://api.runpod.ai/v2//runsync # #10 prompt (img2img) +# RUNPOD_SD_INPAINT_URL=https://api.runpod.ai/v2//runsync # #9 replace-sky / magic-eraser / generative-fill (masked) +# RUNPOD_UPSCALE_URL=https://api.runpod.ai/v2//runsync # #7 restore / upscale (Real-ESRGAN) +# RUNPOD_COLORIZE_URL=https://api.runpod.ai/v2//runsync # #8 colorize (DDColor) +# RUNPOD_BG_REMOVE_URL=https://api.runpod.ai/v2//runsync # #6 (optional; default is in-browser @imgly) +# RUNPOD_YOLO_URL=https://api.runpod.ai/v2//runsync # #1 detection (needs the flag below) +# NEXT_PUBLIC_APG_RUNPOD_DETECT=false # true = use #1 YOLO server detection (else free in-browser COCO-SSD) +# Optional SD tuning: RUNPOD_SD_STEPS RUNPOD_SD_STRENGTH RUNPOD_SD_GUIDANCE RUNPOD_SD_NEGATIVE_PROMPT +# Scaffold-only (typed client, no route/UI yet): RUNPOD_TILT_URL (#2) RUNPOD_STT_URL (#3) +# RUNPOD_AUDIO_DENOISE_URL (#12) RUNPOD_VIDEO_FPS_URL (#13) RUNPOD_VIDEO_UPSCALE_URL (#14) # # Option 1 — your own local Stable Diffusion (AUTOMATIC1111/Forge/SD.Next; free + private): # LOCAL_SD_URL=http://127.0.0.1:7860 diff --git a/apps/web/src/app/api/ai/classify/route.ts b/apps/web/src/app/api/ai/classify/route.ts new file mode 100644 index 0000000..26ba892 --- /dev/null +++ b/apps/web/src/app/api/ai/classify/route.ts @@ -0,0 +1,57 @@ +import { type NextRequest, NextResponse } from 'next/server'; + +import { RunpodError } from '../../../../lib/runpod/client'; +import { rpDetect } from '../../../../lib/runpod/endpoints'; + +export const runtime = 'nodejs'; +export const maxDuration = 60; + +const MAX_BASE64 = 4_000_000; // ~3 MB decoded — under serverless body limits + +/** + * Object-detection proxy for the RunPod YOLO construction-material classifier (#1). + * The key + endpoint URL stay server-side. The client (runpodYoloProvider) calls + * this only when NEXT_PUBLIC_APG_RUNPOD_DETECT is on; otherwise detection runs + * fully in-browser (COCO-SSD) with no server round-trip. Returns the SDK's + * DetectedObject[] shape (box as 0..1 fractions) so it drops straight into the + * existing Objects browser / smart albums / search. + */ +export async function POST(req: NextRequest) { + if (!process.env.RUNPOD_API_KEY || !process.env.RUNPOD_YOLO_URL) { + return NextResponse.json( + { error: 'RunPod detection is not configured (set RUNPOD_API_KEY + RUNPOD_YOLO_URL).' }, + { status: 503 }, + ); + } + + let body: unknown; + try { + body = await req.json(); + } catch { + return NextResponse.json({ error: 'Invalid request body.' }, { status: 400 }); + } + + const { imageBase64, width, height } = (body ?? {}) as { + imageBase64?: unknown; + width?: unknown; + height?: unknown; + }; + if (typeof imageBase64 !== 'string' || imageBase64.length === 0 || imageBase64.length > MAX_BASE64) { + return NextResponse.json({ error: 'Invalid or oversized image.' }, { status: 400 }); + } + const w = Number(width); + const h = Number(height); + + try { + const objects = await rpDetect( + imageBase64, + Number.isFinite(w) && w > 0 ? w : 1, + Number.isFinite(h) && h > 0 ? h : 1, + ); + return NextResponse.json({ objects }); + } catch (err) { + const message = err instanceof Error ? err.message : 'Detection failed.'; + const status = err instanceof RunpodError ? err.status : 502; + return NextResponse.json({ error: message }, { status }); + } +} diff --git a/apps/web/src/app/api/ai/denoise/route.ts b/apps/web/src/app/api/ai/denoise/route.ts new file mode 100644 index 0000000..33c3099 --- /dev/null +++ b/apps/web/src/app/api/ai/denoise/route.ts @@ -0,0 +1,40 @@ +import { type NextRequest, NextResponse } from 'next/server'; + +import { RunpodError } from '../../../../lib/runpod/client'; +import { rpDenoiseAudio } from '../../../../lib/runpod/endpoints'; + +export const runtime = 'nodejs'; +export const maxDuration = 60; // cold-start denoise worker can take a while + +/** + * Audio noise-removal proxy. Accepts base64 WAV (48 kHz mono PCM16, produced + * in-browser by lib/audioCapture) and returns a cleaned base64 WAV. Calls the + * RunPod audio-denoise endpoint (RUNPOD_AUDIO_DENOISE_URL) — key stays + * server-side. Used before transcription on noisy sites. See docs/AI-SETUP.md. + */ + +const MAX_BASE64 = 12_000_000; // ~9 MB decoded WAV + +export async function POST(req: NextRequest) { + let body: { audio?: unknown }; + try { + body = await req.json(); + } catch { + return NextResponse.json({ error: 'Invalid JSON body.' }, { status: 400 }); + } + + const audio = typeof body.audio === 'string' ? body.audio : ''; + if (!audio) return NextResponse.json({ error: 'Missing audio.' }, { status: 400 }); + if (audio.length > MAX_BASE64) { + return NextResponse.json({ error: 'Audio too long — keep it under ~30s.' }, { status: 413 }); + } + + try { + const { audioB64 } = await rpDenoiseAudio(audio); + return NextResponse.json({ audio: audioB64 }); + } catch (e) { + const msg = + e instanceof RunpodError ? e.message : e instanceof Error ? e.message : 'Denoise failed.'; + return NextResponse.json({ error: msg }, { status: 502 }); + } +} diff --git a/apps/web/src/app/api/ai/edit/route.ts b/apps/web/src/app/api/ai/edit/route.ts index 61dda6f..8a3c2b3 100644 --- a/apps/web/src/app/api/ai/edit/route.ts +++ b/apps/web/src/app/api/ai/edit/route.ts @@ -1,5 +1,8 @@ import { type NextRequest, NextResponse } from 'next/server'; +import { RunpodError } from '../../../../lib/runpod/client'; +import { rpImg2Img, rpInpaint, rpRemoveBackground, rpUpscale } from '../../../../lib/runpod/endpoints'; + export const runtime = 'nodejs'; export const maxDuration = 60; // SD / cold-start models can take a while @@ -7,6 +10,12 @@ export const maxDuration = 60; // SD / cold-start models can take a while * Generative image-edit proxy. The BACKEND is pluggable so you are not tied to a * paid model — pick one with env `AI_EDIT_PROVIDER` (default `auto`): * + * - `runpod` → your RunPod serverless GPU endpoints (one per model, per + * docs/model-api-spec). Maps each op → endpoint: restore/upscale + * → Real-ESRGAN (#7), colorize → DDColor (#8), replace-sky/ + * magic-eraser/generative-fill → SD 3.5 masked inpaint (#9), + * prompt → SD 3.5 img2img (#10). Env: RUNPOD_API_KEY + per-model + * RUNPOD_*_URL. Key stays server-side. See docs/AI-SETUP.md. * - `local` → your own Stable Diffusion server (Automatic1111 / Forge / * SD.Next img2img API). Env: LOCAL_SD_URL (e.g. http://127.0.0.1:7860). * 100% free + private, needs a GPU box you run. Most reliable. @@ -14,10 +23,10 @@ export const maxDuration = 60; // SD / cold-start models can take a while * Env: HF_API_TOKEN (free), HF_IMAGE_MODEL (default instruct-pix2pix). * - `gemini` → Google Gemini image model (needs a billed key for image output). * Env: GEMINI_API_KEY, GEMINI_IMAGE_MODEL. - * - `auto` → first configured of: local → huggingface → gemini. + * - `auto` → first configured of: runpod → local → huggingface → gemini. * - * NOTE: `remove-background` never reaches here — it runs fully in-browser (@imgly, - * free, no key). Object detection / faces / OCR / embeddings are also 100% in-browser. + * NOTE: `remove-background` runs in-browser by default (@imgly, free, no key), so + * it usually never reaches here. Object detection uses its own route (/api/ai/classify). * See docs/AI-SETUP.md. */ @@ -31,19 +40,39 @@ const OP_PROMPTS: Record = { const MAX_BASE64 = 4_000_000; // ~3 MB decoded — stays under serverless body limits (e.g. Vercel ~4.5MB) -type Provider = 'local' | 'huggingface' | 'gemini' | 'none'; +type Provider = 'runpod' | 'local' | 'huggingface' | 'gemini' | 'none'; function resolveProvider(): Provider { const explicit = (process.env.AI_EDIT_PROVIDER || 'auto').toLowerCase(); - if (explicit === 'local' || explicit === 'huggingface' || explicit === 'gemini') return explicit; + if ( + explicit === 'runpod' || + explicit === 'local' || + explicit === 'huggingface' || + explicit === 'gemini' + ) + return explicit; if (explicit === 'none') return 'none'; - // auto: prefer a private local server, then free HF, then Gemini. + // auto: prefer RunPod GPU endpoints, then a private local server, then free HF, then Gemini. + // Detect RunPod when the key + ANY image endpoint URL is set (an upscale/colorize-only + // deployment is valid — not just the SD ones). + if ( + process.env.RUNPOD_API_KEY && + (process.env.RUNPOD_SD_IMG2IMG_URL || + process.env.RUNPOD_SD_INPAINT_URL || + process.env.RUNPOD_UPSCALE_URL || + process.env.RUNPOD_COLORIZE_URL || + process.env.RUNPOD_BG_REMOVE_URL) + ) + return 'runpod'; if (process.env.LOCAL_SD_URL) return 'local'; if (process.env.HF_API_TOKEN) return 'huggingface'; if (process.env.GEMINI_API_KEY) return 'gemini'; return 'none'; } +/** Ops that only the RunPod (mask/fixed-function) backend can serve. */ +const RUNPOD_ONLY_OPS = new Set(['upscale', 'magic-eraser', 'generative-fill']); + interface EditResult { imageBase64: string; mimeType: string; @@ -55,7 +84,7 @@ export async function POST(req: NextRequest) { return NextResponse.json( { error: - 'AI image editing is not configured. Set LOCAL_SD_URL (own Stable Diffusion), HF_API_TOKEN (free Hugging Face), or GEMINI_API_KEY. Background removal and all analysis still work with no key. See docs/AI-SETUP.md.', + 'AI image editing is not configured. Set AI_EDIT_PROVIDER=runpod + RUNPOD_API_KEY + the per-model RUNPOD_*_URL vars (RunPod GPU), or LOCAL_SD_URL (own Stable Diffusion), HF_API_TOKEN (free Hugging Face), or GEMINI_API_KEY. Background removal and all analysis still work with no key. See docs/AI-SETUP.md.', }, { status: 503 }, ); @@ -68,45 +97,136 @@ export async function POST(req: NextRequest) { return NextResponse.json({ error: 'Invalid request body.' }, { status: 400 }); } - const { imageBase64, mimeType, op } = (body ?? {}) as { + const { imageBase64, mimeType, op, maskBase64, params } = (body ?? {}) as { imageBase64?: unknown; mimeType?: unknown; - op?: { type?: string; prompt?: string }; + op?: { type?: string; prompt?: string; factor?: number }; + maskBase64?: unknown; + params?: unknown; }; - if (typeof imageBase64 !== 'string' || imageBase64.length === 0 || imageBase64.length > MAX_BASE64) { - return NextResponse.json({ error: 'Invalid or oversized image (try a smaller image).' }, { status: 400 }); + if (typeof imageBase64 !== 'string' || imageBase64.length === 0) { + return NextResponse.json({ error: 'Invalid image.' }, { status: 400 }); + } + const hasMask = typeof maskBase64 === 'string' && maskBase64.length > 0; + // Image + mask share one request body — budget them together against the cap. + if (imageBase64.length + (hasMask ? (maskBase64 as string).length : 0) > MAX_BASE64) { + return NextResponse.json({ error: 'Image (plus mask) is too large — try a smaller image.' }, { status: 400 }); } const safeMime = typeof mimeType === 'string' && /^image\/(jpeg|png|webp)$/.test(mimeType) ? mimeType : 'image/jpeg'; + const opType = op?.type ?? ''; + if (provider !== 'runpod' && RUNPOD_ONLY_OPS.has(opType)) { + return NextResponse.json( + { error: 'This edit needs the RunPod backend (set AI_EDIT_PROVIDER=runpod).' }, + { status: 400 }, + ); + } + // Build the instruction from an allow-listed op (never trust arbitrary server prompts). - let instruction: string; - if (op?.type === 'prompt') { - const p = typeof op.prompt === 'string' ? op.prompt.trim() : ''; + let instruction = ''; + if (opType === 'prompt' || opType === 'generative-fill') { + const p = typeof op?.prompt === 'string' ? op.prompt.trim() : ''; if (!p) return NextResponse.json({ error: 'Empty prompt.' }, { status: 400 }); instruction = p.slice(0, 500); - } else if (op?.type === 'replace-sky' && typeof op.prompt === 'string' && op.prompt.trim()) { - instruction = `Replace the sky with: ${op.prompt.trim().slice(0, 300)}. Keep the foreground unchanged and photorealistic.`; - } else if (op?.type && OP_PROMPTS[op.type]) { - instruction = OP_PROMPTS[op.type]!; - } else { + } else if (opType === 'replace-sky') { + instruction = + typeof op?.prompt === 'string' && op.prompt.trim() + ? `Replace the sky with: ${op.prompt.trim().slice(0, 300)}. Keep the foreground unchanged and photorealistic.` + : OP_PROMPTS['replace-sky']!; + } else if (opType === 'magic-eraser') { + instruction = 'Fill the selected region with a clean, seamless, plausible background. Photorealistic.'; + } else if (OP_PROMPTS[opType]) { + instruction = OP_PROMPTS[opType]!; + } else if (opType !== 'upscale') { return NextResponse.json({ error: 'Unsupported operation.' }, { status: 400 }); } try { let result: EditResult; - if (provider === 'local') result = await editLocal(instruction, imageBase64); + if (provider === 'runpod') + result = await editRunPod(op ?? {}, imageBase64, instruction, hasMask ? (maskBase64 as string) : undefined, params); + else if (provider === 'local') result = await editLocal(instruction, imageBase64); else if (provider === 'huggingface') result = await editHuggingFace(instruction, imageBase64); else result = await editGemini(instruction, imageBase64, safeMime); return NextResponse.json(result); } catch (err) { const message = err instanceof Error ? err.message : 'AI request failed.'; - const status = err instanceof AiError ? err.status : 502; + const status = err instanceof AiError || err instanceof RunpodError ? err.status : 502; return NextResponse.json({ error: message }, { status }); } } +// --------------------------------------------------------------------------- +// Backend: RunPod serverless GPU endpoints (one model per endpoint). +// Each op maps to the endpoint from the spec; the API key + URLs stay server-side. +// --------------------------------------------------------------------------- +interface SdParams { + negativePrompt?: string; + strength?: number; + steps?: number; + seed?: number; + guidanceScale?: number; +} + +function sanitizeParams(raw: unknown): SdParams { + const p = (raw ?? {}) as Record; + const out: SdParams = {}; + if (typeof p.negativePrompt === 'string' && p.negativePrompt.trim()) + out.negativePrompt = p.negativePrompt.trim().slice(0, 300); + const strength = Number(p.strength); + if (Number.isFinite(strength)) out.strength = Math.max(0, Math.min(1, strength)); + const steps = Number(p.steps); + if (Number.isFinite(steps)) out.steps = Math.max(1, Math.min(60, Math.round(steps))); + const guidance = Number(p.guidanceScale); + if (Number.isFinite(guidance)) out.guidanceScale = Math.max(1, Math.min(20, guidance)); + const seed = Number(p.seed); + if (Number.isFinite(seed)) out.seed = Math.max(0, Math.min(2_147_483_647, Math.round(seed))); + return out; +} + +async function editRunPod( + op: { type?: string; prompt?: string; factor?: number }, + imageBase64: string, + instruction: string, + maskBase64: string | undefined, + rawParams: unknown, +): Promise { + const params = sanitizeParams(rawParams); + switch (op.type) { + case 'remove-background': + // U²-Net via rembg (#6) — a real endpoint replacing the flaky in-browser remover. + return rpRemoveBackground(imageBase64); + case 'restore': + // Real-ESRGAN (#7) with the GFPGAN face pass = "Restore & Enhance". + return rpUpscale(imageBase64, 4, true); + case 'upscale': + return rpUpscale(imageBase64, op.factor === 4 ? 4 : 2, false); + case 'colorize': + // The dedicated DDColor endpoint kept hard-crashing (modelscope). Route + // colorize through the img2img model as an instruction instead — reliable, + // and high quality once img2img is FLUX. (RUNPOD_COLORIZE_URL now unused.) + return rpImg2Img({ imageB64: imageBase64, prompt: instruction, ...params }); + case 'prompt': + return rpImg2Img({ imageB64: imageBase64, prompt: instruction, ...params }); // SD 3.5 img2img (#10) + case 'replace-sky': + // True sky replacement is masked inpaint (#9). Without a mask (no in-app sky + // segmentation yet) degrade to a low-strength img2img (#10) so the foreground + // is mostly preserved. Add a mask (brush UI / sky-seg) to get real #9. + if (maskBase64) + return rpInpaint({ imageB64: imageBase64, maskB64: maskBase64, prompt: instruction, ...params }); + return rpImg2Img({ imageB64: imageBase64, prompt: instruction, ...params, strength: params.strength ?? 0.4 }); + case 'magic-eraser': + case 'generative-fill': + // SD 3.5 masked inpaint (#9) — white in the mask = the region to regenerate. + if (!maskBase64) throw new AiError('This edit needs a mask/selection.', 400); + return rpInpaint({ imageB64: imageBase64, maskB64: maskBase64, prompt: instruction, ...params }); + default: + throw new AiError('Unsupported operation.', 400); + } +} + class AiError extends Error { status: number; constructor(message: string, status = 502) { diff --git a/apps/web/src/app/api/ai/tilt/route.ts b/apps/web/src/app/api/ai/tilt/route.ts new file mode 100644 index 0000000..7f63083 --- /dev/null +++ b/apps/web/src/app/api/ai/tilt/route.ts @@ -0,0 +1,44 @@ +import { type NextRequest, NextResponse } from 'next/server'; + +import { RunpodError } from '../../../../lib/runpod/client'; +import { rpTilt } from '../../../../lib/runpod/endpoints'; + +export const runtime = 'nodejs'; +export const maxDuration = 60; // cold-start tilt worker can take a while + +/** + * Camera-tilt estimation proxy. Accepts a base64 image and returns + * {rollDegrees, pitchDegrees, fovDegrees} from the RunPod tilt endpoint + * (RUNPOD_TILT_URL) so the editor can auto-straighten. Requires the tilt + * endpoint to be deployed; errors clearly if RUNPOD_TILT_URL is unset. + */ + +const MAX_BASE64 = 4_000_000; // ~3 MB decoded + +export async function POST(req: NextRequest) { + let body: { image?: unknown }; + try { + body = await req.json(); + } catch { + return NextResponse.json({ error: 'Invalid JSON body.' }, { status: 400 }); + } + + const image = typeof body.image === 'string' ? body.image : ''; + if (!image) return NextResponse.json({ error: 'Missing image.' }, { status: 400 }); + if (image.length > MAX_BASE64) { + return NextResponse.json({ error: 'Image too large.' }, { status: 413 }); + } + + try { + const tilt = await rpTilt(image); + return NextResponse.json(tilt); + } catch (e) { + const msg = + e instanceof RunpodError + ? e.message + : e instanceof Error + ? e.message + : 'Tilt estimate failed.'; + return NextResponse.json({ error: msg }, { status: 502 }); + } +} diff --git a/apps/web/src/app/api/ai/transcribe/route.ts b/apps/web/src/app/api/ai/transcribe/route.ts new file mode 100644 index 0000000..4c809d4 --- /dev/null +++ b/apps/web/src/app/api/ai/transcribe/route.ts @@ -0,0 +1,45 @@ +import { type NextRequest, NextResponse } from 'next/server'; + +import { RunpodError } from '../../../../lib/runpod/client'; +import { rpTranscribe } from '../../../../lib/runpod/endpoints'; + +export const runtime = 'nodejs'; +export const maxDuration = 60; // cold-start STT worker can take a while + +/** + * Speech-to-text proxy for voice annotations. Accepts base64 WAV (16 kHz mono + * PCM16, produced in-browser by lib/audioCapture) and returns the transcript. + * Calls the RunPod voice-to-text endpoint (RUNPOD_STT_URL) — the key stays + * server-side. See docs/AI-SETUP.md. + */ + +const MAX_BASE64 = 8_000_000; // ~6 MB decoded WAV — stays under serverless body limits + +export async function POST(req: NextRequest) { + let body: { audio?: unknown; language?: unknown }; + try { + body = await req.json(); + } catch { + return NextResponse.json({ error: 'Invalid JSON body.' }, { status: 400 }); + } + + const audio = typeof body.audio === 'string' ? body.audio : ''; + const language = typeof body.language === 'string' ? body.language : undefined; + if (!audio) return NextResponse.json({ error: 'Missing audio.' }, { status: 400 }); + if (audio.length > MAX_BASE64) { + return NextResponse.json({ error: 'Audio too long — keep it under ~30s.' }, { status: 413 }); + } + + try { + const { transcript, segments } = await rpTranscribe(audio, { language, punctuation: true }); + return NextResponse.json({ transcript, segments }); + } catch (e) { + const msg = + e instanceof RunpodError + ? e.message + : e instanceof Error + ? e.message + : 'Transcription failed.'; + return NextResponse.json({ error: msg }, { status: 502 }); + } +} diff --git a/apps/web/src/lib/ai/createDemoAIProvider.ts b/apps/web/src/lib/ai/createDemoAIProvider.ts index 66f6592..d583681 100644 --- a/apps/web/src/lib/ai/createDemoAIProvider.ts +++ b/apps/web/src/lib/ai/createDemoAIProvider.ts @@ -2,17 +2,24 @@ import type { AIProvider } from '@photo-gallery/sdk'; import { createClipProvider } from './clipProvider'; import { createFaceProvider } from './faceProvider'; +import { imageToBase64, maskToBase64 } from './imageEncode'; import { createOCRProvider } from './ocrProvider'; +import { createRunpodYoloProvider } from './runpodYoloProvider'; import { createTensorflowProvider } from './tensorflowProvider'; /** - * The demo's AI provider, combining free in-browser capabilities + Gemini: + * The demo's AI provider, combining free in-browser capabilities + a + * server-proxied generative-edit backend (RunPod / local SD / Hugging Face / Gemini): * - object detection: TensorFlow.js COCO-SSD, fully in-browser (no key) * - face detection + recognition: face-api.js, in-browser → clustered into People * - OCR: tesseract.js, in-browser → searchable text + the Documents album (no key) * - background removal: @imgly, in-browser WASM (no key) — see generativeEdit - * - other generative edits: Gemini, proxied through /api/ai/edit so the API key - * stays server-side and is never exposed to the browser. + * - other generative edits: proxied through /api/ai/edit (RunPod / local SD / + * Hugging Face / Gemini, chosen by env) so the API key stays server-side. + * + * Object detection uses the in-browser COCO-SSD by default, or the RunPod YOLO + * construction-material classifier (#1) when NEXT_PUBLIC_APG_RUNPOD_DETECT=true + * (proxied through /api/ai/classify, with COCO-SSD as the automatic fallback). */ export function createDemoAIProvider(): AIProvider { const tf = createTensorflowProvider(); @@ -20,9 +27,15 @@ export function createDemoAIProvider(): AIProvider { const ocr = createOCRProvider(); const clip = createClipProvider(); + // Opt-in server-side detection; defaults to free in-browser COCO-SSD. + const detectObjects = + process.env.NEXT_PUBLIC_APG_RUNPOD_DETECT === 'true' + ? createRunpodYoloProvider(tf).detectObjects + : tf.detectObjects; + return { - name: 'demo-ai (coco-ssd + face-api + tesseract + clip + gemini)', - detectObjects: tf.detectObjects, + name: 'demo-ai (coco-ssd/yolo + face-api + tesseract + clip + runpod-edit)', + detectObjects, detectFaces: face.detectFaces, ocr: ocr.ocr, embedImage: clip.embedImage, @@ -30,18 +43,91 @@ export function createDemoAIProvider(): AIProvider { async generativeEdit(item, image, op) { // Remove Background runs fully in-browser (free, no key) via @imgly — works - // even when Gemini is unavailable. Other ops go through the Gemini route. + // even with no backend. Other ops go through the /api/ai/edit route. if (op.type === 'remove-background') { + // Prefer the RunPod U²-Net endpoint when enabled; fall back to in-browser + // @imgly if it's off or the request fails, so this always produces a result. + if (process.env.NEXT_PUBLIC_APG_RUNPOD_BG === 'true') { + try { + const { data, mimeType } = imageToBase64(image, 1600); + const res = await fetch('/api/ai/edit', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + imageBase64: data, + mimeType, + op: { type: 'remove-background' }, + params: {}, + }), + }); + if (res.ok) { + const { imageBase64: out, mimeType: outMime } = (await res.json()) as { + imageBase64: string; + mimeType?: string; + }; + return base64ToBlob(out, outMime || 'image/png'); + } + } catch { + /* fall through to the in-browser remover */ + } + } const inputBlob = await canvasBlob(image, 1600); const { removeBackground } = await import('@imgly/background-removal'); return removeBackground(inputBlob, { output: { format: 'image/png' } }); } - const { data, mimeType } = imageToBase64(image, 1280); + // Outpaint / expand-canvas: pad the image with a neutral border, mark that + // border WHITE in the mask, and run it through the same inpaint path as + // generative-fill — no extra backend route needed. + if (op.type === 'outpaint') { + const { imageBase64: padded, maskBase64: border } = padForOutpaint( + image, + typeof op.factor === 'number' ? op.factor : 1.5, + ); + const outParams: Record = { + strength: typeof op.strength === 'number' ? op.strength : 0.85, + }; + const outRes = await fetch('/api/ai/edit', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + imageBase64: padded, + mimeType: 'image/png', + op: { + type: 'generative-fill', + prompt: + op.prompt || + 'Extend and continue the scene naturally, matching lighting, colors and perspective.', + }, + maskBase64: border, + params: outParams, + }), + }); + if (!outRes.ok) { + const err = (await outRes.json().catch(() => ({}))) as { error?: string }; + throw new Error(err.error || `AI request failed (${outRes.status}).`); + } + const outJson = (await outRes.json()) as { imageBase64: string; mimeType?: string }; + return base64ToBlob(outJson.imageBase64, outJson.mimeType || 'image/png'); + } + + const { data, mimeType, width, height } = imageToBase64(image, 1280); + // SD 3.5 masked ops (#9) carry an ImageData mask — rasterize it to a PNG + // matched to the (downscaled) image dims, and strip it from the op since + // ImageData is not JSON-serializable. + const maskBase64 = 'mask' in op ? maskToBase64(op.mask, width, height) : undefined; + const wireOp: Record = { type: op.type }; + if ('prompt' in op && typeof op.prompt === 'string') wireOp.prompt = op.prompt; + if ('factor' in op && typeof op.factor === 'number') wireOp.factor = op.factor; + + // Forward the "edit strength" slider (0..1) so the backend can scale the edit. + const params: Record = {}; + if ('strength' in op && typeof op.strength === 'number') params.strength = op.strength; + const res = await fetch('/api/ai/edit', { method: 'POST', headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ imageBase64: data, mimeType, op }), + body: JSON.stringify({ imageBase64: data, mimeType, op: wireOp, maskBase64, params }), }); if (!res.ok) { const err = (await res.json().catch(() => ({}))) as { error?: string }; @@ -53,27 +139,60 @@ export function createDemoAIProvider(): AIProvider { }; return base64ToBlob(imageBase64, outMime || 'image/png'); }, - }; -} -/** Draw an image to a canvas (downscaled) and return base64 JPEG (no data: prefix). */ -function imageToBase64( - image: ImageBitmap | HTMLImageElement, - maxDim: number, -): { data: string; mimeType: string } { - const w = (image as HTMLImageElement).naturalWidth || (image as ImageBitmap).width; - const h = (image as HTMLImageElement).naturalHeight || (image as ImageBitmap).height; - const scale = Math.min(1, maxDim / Math.max(w, h)); - const cw = Math.max(1, Math.round(w * scale)); - const ch = Math.max(1, Math.round(h * scale)); - const canvas = document.createElement('canvas'); - canvas.width = cw; - canvas.height = ch; - const ctx = canvas.getContext('2d'); - if (!ctx) throw new Error('Canvas not supported.'); - ctx.drawImage(image as CanvasImageSource, 0, 0, cw, ch); - const dataUrl = canvas.toDataURL('image/jpeg', 0.9); - return { data: dataUrl.split(',')[1] ?? '', mimeType: 'image/jpeg' }; + // Voice annotation: record → (optional denoise) → transcribe. Both proxy + // through server routes so the RunPod key stays server-side. + async transcribeAudio(audioBase64) { + const res = await fetch('/api/ai/transcribe', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ audio: audioBase64 }), + }); + if (!res.ok) { + const err = (await res.json().catch(() => ({}))) as { error?: string }; + throw new Error(err.error || `Transcription failed (${res.status}).`); + } + const { transcript } = (await res.json()) as { transcript?: string }; + return (transcript ?? '').trim(); + }, + + async denoiseAudio(audioBase64) { + const res = await fetch('/api/ai/denoise', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ audio: audioBase64 }), + }); + if (!res.ok) { + const err = (await res.json().catch(() => ({}))) as { error?: string }; + throw new Error(err.error || `Denoise failed (${res.status}).`); + } + const { audio } = (await res.json()) as { audio?: string }; + return audio ?? audioBase64; + }, + + // Camera-tilt estimation is opt-in (needs the RunPod tilt endpoint deployed); + // gate it so the editor's Auto-straighten button only appears when configured. + estimateTilt: + process.env.NEXT_PUBLIC_APG_RUNPOD_TILT === 'true' + ? async (_item, image) => { + const { data } = imageToBase64(image, 1024); + const res = await fetch('/api/ai/tilt', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ image: data }), + }); + if (!res.ok) { + const err = (await res.json().catch(() => ({}))) as { error?: string }; + throw new Error(err.error || `Tilt estimate failed (${res.status}).`); + } + return (await res.json()) as { + rollDegrees: number; + pitchDegrees: number; + fovDegrees: number; + }; + } + : undefined, + }; } /** Draw an image to a canvas (downscaled) and return a JPEG Blob. */ @@ -100,3 +219,55 @@ function base64ToBlob(base64: string, mime: string): Blob { for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i); return new Blob([bytes], { type: mime }); } + +/** + * Pad an image with a neutral border for outpaint and return {paddedImage, mask} + * as base64 PNG — the border is WHITE in the mask (regenerate), the original + * image area BLACK (keep). Capped at 1280px on the long side. + */ +function padForOutpaint( + image: ImageBitmap | HTMLImageElement, + factor: number, +): { imageBase64: string; maskBase64: string } { + const w = (image as HTMLImageElement).naturalWidth || (image as ImageBitmap).width; + const h = (image as HTMLImageElement).naturalHeight || (image as ImageBitmap).height; + const f = Math.max(1.1, Math.min(2, factor)); + const maxDim = 1280; + let pw = Math.round(w * f); + let ph = Math.round(h * f); + const scale = Math.min(1, maxDim / Math.max(pw, ph)); + pw = Math.max(16, Math.round(pw * scale)); + ph = Math.max(16, Math.round(ph * scale)); + const iw = Math.max(1, Math.round(w * scale)); + const ih = Math.max(1, Math.round(h * scale)); + const ox = Math.floor((pw - iw) / 2); + const oy = Math.floor((ph - ih) / 2); + + const imgCanvas = document.createElement('canvas'); + imgCanvas.width = pw; + imgCanvas.height = ph; + const ictx = imgCanvas.getContext('2d'); + if (!ictx) throw new Error('Canvas not supported.'); + // Fill the new border with a blurred, stretched copy of the photo so the model + // has real color/context to continue from — flat gray gives it nothing and it + // leaves the border gray. Then draw the original sharp on top, centered. + ictx.filter = 'blur(28px)'; + ictx.drawImage(image as CanvasImageSource, 0, 0, pw, ph); + ictx.filter = 'none'; + ictx.drawImage(image as CanvasImageSource, ox, oy, iw, ih); + + const maskCanvas = document.createElement('canvas'); + maskCanvas.width = pw; + maskCanvas.height = ph; + const mctx = maskCanvas.getContext('2d'); + if (!mctx) throw new Error('Canvas not supported.'); + mctx.fillStyle = '#ffffff'; + mctx.fillRect(0, 0, pw, ph); + mctx.fillStyle = '#000000'; + mctx.fillRect(ox, oy, iw, ih); + + return { + imageBase64: imgCanvas.toDataURL('image/png').split(',')[1] ?? '', + maskBase64: maskCanvas.toDataURL('image/png').split(',')[1] ?? '', + }; +} diff --git a/apps/web/src/lib/ai/imageEncode.ts b/apps/web/src/lib/ai/imageEncode.ts new file mode 100644 index 0000000..dc34ac0 --- /dev/null +++ b/apps/web/src/lib/ai/imageEncode.ts @@ -0,0 +1,56 @@ +/** + * Shared client-side image encoding helpers for the AI providers. Downscale an + * image to a JPEG base64 (keeping the exact output dims), and rasterize a mask + * to a PNG base64 matched to those same dims — SD 3.5 (#9) requires the image + * and mask to be identical pixel sizes. + */ + +export interface EncodedImage { + /** base64 JPEG (no data: prefix). */ + data: string; + mimeType: string; + /** Actual pixel dims of the encoded image (after downscale). */ + width: number; + height: number; +} + +/** Draw an image to a downscaled canvas and return base64 JPEG + its dims. */ +export function imageToBase64(image: ImageBitmap | HTMLImageElement, maxDim: number): EncodedImage { + const w = (image as HTMLImageElement).naturalWidth || (image as ImageBitmap).width; + const h = (image as HTMLImageElement).naturalHeight || (image as ImageBitmap).height; + const scale = Math.min(1, maxDim / Math.max(w, h)); + const cw = Math.max(1, Math.round(w * scale)); + const ch = Math.max(1, Math.round(h * scale)); + const canvas = document.createElement('canvas'); + canvas.width = cw; + canvas.height = ch; + const ctx = canvas.getContext('2d'); + if (!ctx) throw new Error('Canvas not supported.'); + ctx.drawImage(image as CanvasImageSource, 0, 0, cw, ch); + const dataUrl = canvas.toDataURL('image/jpeg', 0.9); + return { data: dataUrl.split(',')[1] ?? '', mimeType: 'image/jpeg', width: cw, height: ch }; +} + +/** + * Rasterize a mask (ImageData, white = region to regenerate) to a PNG base64 + * scaled to targetW×targetH so it matches the encoded image exactly. + */ +export function maskToBase64(mask: ImageData, targetW: number, targetH: number): string { + const tmp = document.createElement('canvas'); + tmp.width = mask.width; + tmp.height = mask.height; + const tctx = tmp.getContext('2d'); + if (!tctx) throw new Error('Canvas not supported.'); + tctx.putImageData(mask, 0, 0); + + const out = document.createElement('canvas'); + out.width = targetW; + out.height = targetH; + const octx = out.getContext('2d'); + if (!octx) throw new Error('Canvas not supported.'); + // Nearest-neighbour, not bilinear — keep the mask strictly binary so SD 3.5 gets + // crisp white(regenerate)/black(keep) edges instead of an anti-aliased grey halo. + octx.imageSmoothingEnabled = false; + octx.drawImage(tmp, 0, 0, targetW, targetH); + return out.toDataURL('image/png').split(',')[1] ?? ''; +} diff --git a/apps/web/src/lib/ai/runpodYoloProvider.ts b/apps/web/src/lib/ai/runpodYoloProvider.ts new file mode 100644 index 0000000..689adc1 --- /dev/null +++ b/apps/web/src/lib/ai/runpodYoloProvider.ts @@ -0,0 +1,32 @@ +import type { AIProvider, DetectedObject, MediaItem } from '@photo-gallery/sdk'; + +import { imageToBase64 } from './imageEncode'; + +/** + * Object detection backed by the RunPod YOLO construction-material classifier + * (#1), proxied through /api/ai/classify so the RunPod key stays server-side. + * + * Opt-in via `NEXT_PUBLIC_APG_RUNPOD_DETECT=true` (see createDemoAIProvider). If + * the server call fails (endpoint down / not configured) it falls back to the + * given in-browser provider (COCO-SSD), so detection never hard-fails. + */ +export function createRunpodYoloProvider(fallback: AIProvider): Pick { + return { + async detectObjects(item: MediaItem, image): Promise { + try { + const { data, mimeType, width, height } = imageToBase64(image, 1280); + const res = await fetch('/api/ai/classify', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ imageBase64: data, mimeType, width, height }), + }); + if (!res.ok) throw new Error(`classify failed (${res.status})`); + const { objects } = (await res.json()) as { objects?: DetectedObject[] }; + if (Array.isArray(objects)) return objects; + throw new Error('classify returned no objects'); + } catch { + return fallback.detectObjects ? fallback.detectObjects(item, image) : []; + } + }, + }; +} diff --git a/apps/web/src/lib/runpod/base64.ts b/apps/web/src/lib/runpod/base64.ts new file mode 100644 index 0000000..7677ca5 --- /dev/null +++ b/apps/web/src/lib/runpod/base64.ts @@ -0,0 +1,80 @@ +/** + * base64 / data-URI helpers shared by the RunPod endpoint wrappers. + * Different endpoints name their image field differently and some prefix a + * `data:` URI — these helpers normalize both. + */ + +/** "data:image/png;base64,XXXX" -> "XXXX" (leaves a bare base64 string untouched). */ +export function stripDataUri(s: string): string { + if (!s.startsWith('data:')) return s; + const i = s.indexOf(','); + return i === -1 ? s : s.slice(i + 1); +} + +/** Wrap a bare base64 string in a data: URI. */ +export function toDataUri(b64: string, mime: string): string { + return `data:${mime};base64,${stripDataUri(b64)}`; +} + +/** + * Pull the first image-like base64 string out of a RunPod endpoint's `output`, + * regardless of which field name it used. Handles the common shapes: + * "iVBOR..." (raw string) + * { image: "..." } / { image_png } / { image_base64 } + * { images: ["..."] } (array) + * { output: { image: "..." } } (nested) + * Mirrors the A1111 normalization already used for the local SD backend. + */ +export function pickOutputImage(output: unknown): string { + const s = findImageString(output, false); + if (!s) throw new Error('RunPod output did not contain an image.'); + return stripDataUri(s); +} + +/** Keys whose name implies the value IS the image — any non-empty string is accepted. */ +const IMAGE_KEYS = ['image_png', 'image', 'image_base64', 'images'] as const; +/** Generic wrapper keys — a string here must actually look like image data. */ +const CONTAINER_KEYS = ['output', 'result', 'data'] as const; + +/** A base64 image payload is long; a status/id string ("success", "job-abc") is short. */ +function looksLikeImageData(s: string): boolean { + if (s.startsWith('data:image/')) return true; + return s.length >= 256 && /^[A-Za-z0-9+/=\s]+$/.test(s.slice(0, 256)); +} + +/** + * `strict` is true when we descended through a generic wrapper key (output/result/ + * data), where a bare string could be a status/id rather than an image — so it must + * pass `looksLikeImageData`. Under an explicit image key (or at top level) any + * non-empty string is taken as the image. + */ +function findImageString(value: unknown, strict: boolean, depth = 0): string | undefined { + if (depth > 5) return undefined; + if (typeof value === 'string') { + if (!value) return undefined; + return !strict || looksLikeImageData(value) ? value : undefined; + } + if (Array.isArray(value)) { + for (const el of value) { + const s = findImageString(el, strict, depth + 1); + if (s) return s; + } + return undefined; + } + if (value && typeof value === 'object') { + const o = value as Record; + for (const key of IMAGE_KEYS) { + if (key in o) { + const s = findImageString(o[key], false, depth + 1); + if (s) return s; + } + } + for (const key of CONTAINER_KEYS) { + if (key in o) { + const s = findImageString(o[key], true, depth + 1); + if (s) return s; + } + } + } + return undefined; +} diff --git a/apps/web/src/lib/runpod/client.ts b/apps/web/src/lib/runpod/client.ts new file mode 100644 index 0000000..802a897 --- /dev/null +++ b/apps/web/src/lib/runpod/client.ts @@ -0,0 +1,121 @@ +/** + * Low-level RunPod transport. Every model in the spec runs as its own RunPod + * endpoint; this handles the common request envelope, bearer auth, and both + * response modes: + * + * - `/runsync` (preferred): the job runs synchronously and the body already + * contains `output` (or IS the output for a custom handler). + * - `/run`: returns `{ id, status }`; we poll `/status/{id}` until the job + * reaches a terminal state or the time budget (kept under Vercel's 60s + * function cap) is exhausted. + * + * RUNPOD_API_KEY is read here so callers never handle the secret directly. + */ + +export class RunpodError extends Error { + status: number; + constructor(message: string, status = 502) { + super(message); + this.name = 'RunpodError'; + this.status = status; + } +} + +export interface RunpodCallOpts { + /** Human label used in error messages, e.g. "sd-inpaint". */ + name: string; + /** Full endpoint URL from the per-model env var (…/runsync or …/run). */ + url: string; + input: Record; + /** Total budget for the whole call incl. polling. Default 55s (< Vercel 60s). */ + timeoutMs?: number; + pollIntervalMs?: number; +} + +interface RunpodEnvelope { + id?: string; + status?: string; + output?: unknown; + error?: unknown; +} + +export async function runpodCall(opts: RunpodCallOpts): Promise { + const { name, url, input } = opts; + const timeoutMs = opts.timeoutMs ?? 55_000; + const pollIntervalMs = opts.pollIntervalMs ?? 1500; + + const key = process.env.RUNPOD_API_KEY; + if (!key) throw new RunpodError('RUNPOD_API_KEY is not set.', 500); + + const deadline = Date.now() + timeoutMs; + const authHeaders = { authorization: `Bearer ${key}` }; + + let res: Response; + try { + res = await fetch(url, { + method: 'POST', + headers: { 'content-type': 'application/json', ...authHeaders }, + body: JSON.stringify({ input }), + signal: AbortSignal.timeout(timeoutMs), + }); + } catch { + throw new RunpodError(`Could not reach RunPod (${name}).`, 502); + } + if (!res.ok) { + const detail = (await res.text().catch(() => '')).slice(0, 160); + throw new RunpodError(`RunPod ${name} error (${res.status}). ${detail}`.trim(), 502); + } + + const data = (await res.json().catch(() => null)) as RunpodEnvelope | null; + if (!data || typeof data !== 'object') { + throw new RunpodError(`RunPod ${name} returned an invalid response.`, 502); + } + + // Terminal failure reported in a job envelope. + if (data.status === 'FAILED' || data.status === 'CANCELLED') { + throw new RunpodError(`RunPod ${name} job ${data.status.toLowerCase()}.`, 502); + } + // No job id → this is not an async envelope; the body itself is the output. + // Covers custom /runsync handlers that return their result directly, even when + // it carries a `status` field (e.g. "success"). + if (data.id === undefined) { + return (data.output !== undefined ? data.output : data) as TOut; + } + // Job envelope that already carries a completed/inline output. + if (data.output !== undefined && (data.status === undefined || data.status === 'COMPLETED')) { + return data.output as TOut; + } + + // Async: poll /status/{id} until COMPLETED / FAILED / the time budget runs out. + // Each wait + fetch is clamped to the remaining budget so the whole call stays + // under `timeoutMs` (kept below Vercel's 60s function cap). + const statusUrl = url.replace(/\/run(sync)?(\/?)$/, `/status/${data.id}`); + for (;;) { + const remaining = deadline - Date.now(); + if (remaining < 500) break; // not enough budget for another round + await sleep(Math.min(pollIntervalMs, remaining)); + const left = deadline - Date.now(); + if (left <= 0) break; + let sres: Response; + try { + sres = await fetch(statusUrl, { + headers: authHeaders, + signal: AbortSignal.timeout(Math.min(10_000, Math.max(1000, left))), + }); + } catch { + continue; // transient — keep polling until the deadline + } + if (!sres.ok) continue; + const sdata = (await sres.json().catch(() => null)) as RunpodEnvelope | null; + if (!sdata) continue; + if (sdata.status === 'COMPLETED') return sdata.output as TOut; + if (sdata.status === 'FAILED' || sdata.status === 'CANCELLED') { + throw new RunpodError(`RunPod ${name} job ${sdata.status.toLowerCase()}.`, 502); + } + } + throw new RunpodError(`RunPod ${name} timed out (raise the endpoint's speed or use /runsync).`, 504); +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/apps/web/src/lib/runpod/endpoints.ts b/apps/web/src/lib/runpod/endpoints.ts new file mode 100644 index 0000000..fc90d11 --- /dev/null +++ b/apps/web/src/lib/runpod/endpoints.ts @@ -0,0 +1,283 @@ +/** + * Typed wrappers for each RunPod endpoint in `model-api-spec.docx`. + * Every function reads its own `RUNPOD_*_URL` env var, sends the exact `input` + * contract the spec documents, and normalizes the response. + * + * Server-side only. Missing/invalid URLs throw a RunpodError(500) so an + * unconfigured op surfaces as a clear message in the editor rather than a crash. + */ + +import { stripDataUri, pickOutputImage } from './base64'; +import { RunpodError, runpodCall } from './client'; +import type { + Img2ImgReq, + InpaintReq, + RunpodDetection, + RunpodImageResult, + TiltResult, + TranscriptResult, +} from './types'; + +function envNum(v: string | undefined, fallback: number): number { + // Treat a blank/whitespace env var as unset — Number('') is 0 (finite), which + // would otherwise send e.g. strength:0 for `RUNPOD_SD_STRENGTH=`. + if (v == null || v.trim() === '') return fallback; + const n = Number(v); + return Number.isFinite(n) ? n : fallback; +} + +function endpointUrl(envVar: string): string { + const url = process.env[envVar]; + if (!url || !/^https?:\/\//i.test(url)) { + throw new RunpodError(`${envVar} is not set (or is not an http(s) URL).`, 500); + } + return url; +} + +/** Run an image-in/image-out endpoint and normalize the result to base64 PNG. */ +async function imageOp( + name: string, + envVar: string, + input: Record, +): Promise { + const output = await runpodCall({ name, url: endpointUrl(envVar), input }); + return { imageBase64: pickOutputImage(output), mimeType: 'image/png' }; +} + +// --------------------------------------------------------------------------- +// Image endpoints +// --------------------------------------------------------------------------- + +/** #6 Background removal (U²-Net via rembg). model: u2net | u2netp | u2net_human_seg */ +export function rpRemoveBackground(imageB64: string, model?: string): Promise { + return imageOp('background-removal', 'RUNPOD_BG_REMOVE_URL', { + task: 'remove-bg', + image: imageB64, + ...(model ? { model_name: model } : {}), + }); +} + +/** #7 Real-ESRGAN enhance/upscale (RealESRGAN_x4plus). The caller's scale (from the + * op / restore pass) is authoritative — it is not overridden by any env default. */ +export function rpUpscale(imageB64: string, scale: 2 | 4, faceEnhance = false): Promise { + return imageOp('upscale', 'RUNPOD_UPSCALE_URL', { + task: 'upscale', + image: imageB64, + scale, + face_enhance: faceEnhance, + }); +} + +/** #8 DDColor B&W → colorize. */ +export function rpColorize(imageB64: string, inputSize?: number): Promise { + return imageOp('colorize', 'RUNPOD_COLORIZE_URL', { + image: imageB64, + ...(inputSize ? { input_size: inputSize } : {}), + }); +} + +/** #9 SD 3.5 masked inpainting (sky fix / eraser / fill). Also #11 outpaint (pre-padded). */ +export function rpInpaint(p: InpaintReq): Promise { + const input: Record = { + task: 'inpaint', + image: p.imageB64, + mask: p.maskB64, + prompt: p.prompt, + strength: p.strength ?? envNum(process.env.RUNPOD_SD_STRENGTH, 0.8), + guidance_scale: p.guidanceScale ?? envNum(process.env.RUNPOD_SD_GUIDANCE, 7), + num_inference_steps: p.steps ?? envNum(process.env.RUNPOD_SD_STEPS, 35), + }; + const negative = p.negativePrompt ?? process.env.RUNPOD_SD_NEGATIVE_PROMPT; + if (negative) input.negative_prompt = negative; + if (p.seed != null) input.seed = p.seed; + return imageOp('sd-inpaint', 'RUNPOD_SD_INPAINT_URL', input); +} + +/** #10 SD 3.5 general prompt edit (img2img, no mask). */ +export function rpImg2Img(p: Img2ImgReq): Promise { + const input: Record = { + task: 'img2img', + image: p.imageB64, + prompt: p.prompt, + strength: p.strength ?? envNum(process.env.RUNPOD_SD_STRENGTH, 0.6), + guidance_scale: p.guidanceScale ?? envNum(process.env.RUNPOD_SD_GUIDANCE, 7), + num_inference_steps: p.steps ?? envNum(process.env.RUNPOD_SD_STEPS, 35), + }; + const negative = p.negativePrompt ?? process.env.RUNPOD_SD_NEGATIVE_PROMPT; + if (negative) input.negative_prompt = negative; + if (p.seed != null) input.seed = p.seed; + return imageOp('sd-img2img', 'RUNPOD_SD_IMG2IMG_URL', input); +} + +// --------------------------------------------------------------------------- +// #1 YOLO detection → SDK DetectedObject shape (box as fractions 0..1) +// --------------------------------------------------------------------------- + +export async function rpDetect( + imageB64: string, + width: number, + height: number, +): Promise { + const output = await runpodCall({ + name: 'yolo-detect', + url: endpointUrl('RUNPOD_YOLO_URL'), + input: { image: imageB64, task: 'detect' }, + }); + return normalizeDetections(output, width, height); +} + +function extractDetectionArray(output: unknown): unknown[] { + if (Array.isArray(output)) return output; + if (output && typeof output === 'object') { + const o = output as Record; + for (const key of ['detections', 'predictions', 'objects', 'results', 'boxes']) { + if (Array.isArray(o[key])) return o[key] as unknown[]; + } + } + return []; +} + +/** First non-empty STRING among the args (numbers ignored — a numeric `class` is an index, not a name). */ +function firstLabel(...vals: unknown[]): string | undefined { + for (const v of vals) { + if (typeof v === 'string' && v.trim()) return v.trim(); + } + return undefined; +} + +function normalizeDetections(output: unknown, width: number, height: number): RunpodDetection[] { + const out: RunpodDetection[] = []; + for (const raw of extractDetectionArray(output)) { + if (!raw || typeof raw !== 'object') continue; + const o = raw as Record; + // Prefer a human-readable name (ultralytics tojson puts the string in `name` + // and a numeric index in `class`); fall back to class_. Using firstLabel + // (not `??`) also means an explicit empty-string label doesn't get kept + dropped. + const classId = o.class_id ?? (typeof o.class === 'number' ? o.class : undefined); + const label = ( + firstLabel(o.label, o.name, o.class_name, typeof o.class === 'string' ? o.class : undefined) ?? + (classId != null ? `class_${classId}` : 'object') + ).toLowerCase(); + const confidence = Number(o.confidence ?? o.score ?? o.conf ?? 0) || 0; + const box = normalizeBox(o, width, height); + if (box && label) out.push({ label, confidence, box }); + } + return out; +} + +function num(v: unknown): number | null { + const n = Number(v); + return Number.isFinite(n) ? n : null; +} + +function asNum4(v: unknown): [number, number, number, number] | null { + if (!Array.isArray(v) || v.length < 4) return null; + const a = num(v[0]); + const b = num(v[1]); + const c = num(v[2]); + const d = num(v[3]); + return a === null || b === null || c === null || d === null ? null : [a, b, c, d]; +} + +/** + * Normalize a detection box to {x, y, width, height} as fractions 0..1 of the + * image, from whatever shape the endpoint emits: + * - `xyxy: [x1,y1,x2,y2]` (ultralytics) and generic `box: [...]` → corner form + * - `xywh: [...]` and COCO `bbox: [x,y,w,h]` → x/y/width/height form + * - object `{x1,y1,x2,y2}` / `{left,top,right,bottom}` (ultralytics tojson) → corners + * - object `{x,y,width,height}` → x/y/width/height + * Pixel values (any component > 1) are divided by the image dims; already- + * normalized fractions pass through. VERIFY the shape against your live #1 endpoint. + */ +function normalizeBox(o: Record, width: number, height: number): RunpodDetection['box'] | null { + const W = width || 1; + const H = height || 1; + const clamp01 = (n: number) => Math.max(0, Math.min(1, n)); + const frac = (x: number, y: number, w: number, h: number): RunpodDetection['box'] => { + if (Math.max(Math.abs(x), Math.abs(y), Math.abs(w), Math.abs(h)) > 1) { + x /= W; + y /= H; + w /= W; + h /= H; + } + return { x: clamp01(x), y: clamp01(y), width: clamp01(w), height: clamp01(h) }; + }; + const fromXyxy = (x1: number, y1: number, x2: number, y2: number) => frac(x1, y1, x2 - x1, y2 - y1); + + // 1. Array boxes, interpreted by which key holds them. + const xyxyArr = asNum4(o.xyxy); + if (xyxyArr) return fromXyxy(xyxyArr[0], xyxyArr[1], xyxyArr[2], xyxyArr[3]); + const xywhArr = asNum4(o.xywh) ?? asNum4(o.bbox); // COCO `bbox` is [x,y,w,h] + if (xywhArr) return frac(xywhArr[0], xywhArr[1], xywhArr[2], xywhArr[3]); + const boxArr = asNum4(o.box); // generic array box → assume corner form + if (boxArr) return fromXyxy(boxArr[0], boxArr[1], boxArr[2], boxArr[3]); + + // 2. Object boxes (either nested under `box` or directly on the detection). + const src = o.box && typeof o.box === 'object' && !Array.isArray(o.box) ? (o.box as Record) : o; + const x1 = num(src.x1 ?? src.left); + const y1 = num(src.y1 ?? src.top); + const x2 = num(src.x2 ?? src.right); + const y2 = num(src.y2 ?? src.bottom); + if (x1 !== null && y1 !== null && x2 !== null && y2 !== null) return fromXyxy(x1, y1, x2, y2); + + const x = num(src.x); + const y = num(src.y); + const w = num(src.width); + const h = num(src.height); + if (x !== null && y !== null && w !== null && h !== null) return frac(x, y, w, h); + + return null; +} + +// --------------------------------------------------------------------------- +// Scaffold-only endpoints (typed + wired to env, but no route/UI yet). +// See docs/AI-SETUP.md → "Not yet wired". Kept so the client covers all 15 +// endpoints and a future feature can call them without new plumbing. +// --------------------------------------------------------------------------- + +/** #2 Camera tilt (DeepSingleImageCalibration). */ +export async function rpTilt(imageB64: string): Promise { + const o = await runpodCall>({ + name: 'tilt', + url: endpointUrl('RUNPOD_TILT_URL'), + input: { image: imageB64 }, + }); + return { + rollDegrees: Number(o.roll_degrees ?? 0) || 0, + pitchDegrees: Number(o.pitch_degrees ?? 0) || 0, + fovDegrees: Number(o.fov_degrees ?? 0) || 0, + }; +} + +/** #3 Voice-to-text (Parakeet). Audio must be WAV 16kHz mono PCM16. */ +export async function rpTranscribe( + audioB64: string, + opts?: { language?: string; timestamps?: boolean; punctuation?: boolean }, +): Promise { + const o = await runpodCall>({ + name: 'transcribe', + url: endpointUrl('RUNPOD_STT_URL'), + input: { audio: audioB64, task: 'transcribe', ...(opts ?? {}) }, + }); + const rawSegments = Array.isArray(o.segments) ? o.segments : []; + const segments = rawSegments.map((s) => { + const seg = (s ?? {}) as Record; + return { + text: String(seg.text ?? ''), + startSec: Number(seg.start_sec ?? 0) || 0, + endSec: Number(seg.end_sec ?? 0) || 0, + }; + }); + return { transcript: String(o.transcript ?? ''), segments: segments.length ? segments : undefined }; +} + +/** #12 Audio noise removal (RNNoise). Audio must be WAV 48kHz mono 16-bit PCM. */ +export async function rpDenoiseAudio(audioB64: string): Promise<{ audioB64: string }> { + const o = await runpodCall>({ + name: 'audio-denoise', + url: endpointUrl('RUNPOD_AUDIO_DENOISE_URL'), + input: { audio: audioB64, task: 'denoise' }, + }); + const a = o.audio ?? o.output ?? ''; + return { audioB64: stripDataUri(String(a)) }; +} diff --git a/apps/web/src/lib/runpod/types.ts b/apps/web/src/lib/runpod/types.ts new file mode 100644 index 0000000..2c8273e --- /dev/null +++ b/apps/web/src/lib/runpod/types.ts @@ -0,0 +1,62 @@ +/** + * Request/response types for the RunPod endpoints described in + * `model-api-spec.docx`. Server-side only — imported by the /api/ai/* routes, + * never by client/SDK code (which must not see a RunPod URL or key). + */ + +/** Normalized image result returned by every image endpoint (raw base64, no data: prefix). */ +export interface RunpodImageResult { + imageBase64: string; + mimeType: string; +} + +/** #9 SD 3.5 masked inpainting (also #11 outpaint, pre-padded). white in mask = regenerate. */ +export interface InpaintReq { + imageB64: string; + maskB64: string; + prompt: string; + negativePrompt?: string; + strength?: number; + guidanceScale?: number; + steps?: number; + seed?: number; +} + +/** #10 SD 3.5 general prompt edit (img2img, no mask). */ +export interface Img2ImgReq { + imageB64: string; + prompt: string; + negativePrompt?: string; + strength?: number; + guidanceScale?: number; + steps?: number; + seed?: number; +} + +/** + * #1 YOLO construction-material classifier, normalized to the SDK's + * `DetectedObject` shape (box as fractions 0..1 of the image). + */ +export interface RunpodDetection { + label: string; + confidence: number; + box: { x: number; y: number; width: number; height: number }; +} + +/** #2 DeepSingleImageCalibration — camera tilt (scaffold; no route/UI yet). */ +export interface TiltResult { + rollDegrees: number; + pitchDegrees: number; + fovDegrees: number; +} + +/** #3 Parakeet voice-to-text (scaffold; no route/UI yet). */ +export interface TranscriptSegment { + text: string; + startSec: number; + endSec: number; +} +export interface TranscriptResult { + transcript: string; + segments?: TranscriptSegment[]; +} diff --git a/apps/web/src/lib/seed.ts b/apps/web/src/lib/seed.ts index 2a3ff64..3815738 100644 --- a/apps/web/src/lib/seed.ts +++ b/apps/web/src/lib/seed.ts @@ -137,5 +137,8 @@ export function buildSeedPhotos(now = Date.now()): MediaInput[] { }); } - return items; + // TESTING: seed only the demo video(s) — start with an EMPTY photo library so you + // can upload 1–2 images and watch them get analyzed. To restore the full demo, + // change this back to `return items;`. + return items.filter((it) => it.kind === 'video'); } diff --git a/apps/web/src/middleware.ts b/apps/web/src/middleware.ts index be12ec9..0a6e502 100644 --- a/apps/web/src/middleware.ts +++ b/apps/web/src/middleware.ts @@ -10,7 +10,6 @@ import { type NextRequest, NextResponse } from 'next/server'; */ export function middleware(request: NextRequest) { const nonce = Buffer.from(crypto.randomUUID()).toString('base64'); - const isDev = process.env.NODE_ENV === 'development'; const csp = [ "default-src 'self'", @@ -18,7 +17,10 @@ export function middleware(request: NextRequest) { // cdn.jsdelivr.net = defense for tesseract.js' worker bootstrap (the load-bearing // path is the blob: worker + connect-src fetch; under 'strict-dynamic' this host is // ignored, but it covers non-strict-dynamic fallback browsers). - `script-src 'self' 'nonce-${nonce}' 'strict-dynamic' 'wasm-unsafe-eval' https://cdn.jsdelivr.net${isDev ? " 'unsafe-eval'" : ''}`, + // 'unsafe-eval' is required by onnxruntime-web (Remove Background @imgly, CLIP + // semantic search) which evals a JS glue string — 'wasm-unsafe-eval' alone + // doesn't cover it. Trade-off: needed for the in-browser ML to run in production. + `script-src 'self' 'nonce-${nonce}' 'strict-dynamic' 'wasm-unsafe-eval' 'unsafe-eval' https://cdn.jsdelivr.net`, // Leaflet & Framer Motion inject inline styles; keep style inline allowed. "style-src 'self' 'unsafe-inline'", // *.supabase.co serves uploaded media from Supabase Storage public URLs. @@ -30,7 +32,7 @@ export function middleware(request: NextRequest) { // cdn.jsdelivr.net = face-api models + tesseract.js worker/WASM-core/traineddata (OCR) + onnxruntime WASM; // huggingface.co + *.hf.co = transformers.js CLIP model config + weights, which // redirect to HF's Xet CDN (e.g. us.aws.cdn.hf.co) — semantic search. - "connect-src 'self' blob: data: https://*.tile.openstreetmap.org https://server.arcgisonline.com https://picsum.photos https://fastly.picsum.photos https://storage.googleapis.com https://*.supabase.co https://staticimgly.com https://cdn.jsdelivr.net https://huggingface.co https://*.huggingface.co https://*.hf.co", + "connect-src 'self' blob: data: https://*.tile.openstreetmap.org https://nominatim.openstreetmap.org https://server.arcgisonline.com https://picsum.photos https://fastly.picsum.photos https://storage.googleapis.com https://*.supabase.co https://staticimgly.com https://cdn.jsdelivr.net https://huggingface.co https://*.huggingface.co https://*.hf.co", "worker-src 'self' blob:", "frame-ancestors 'none'", "base-uri 'self'", diff --git a/chat-pod/README.md b/chat-pod/README.md new file mode 100644 index 0000000..d655f86 --- /dev/null +++ b/chat-pod/README.md @@ -0,0 +1,69 @@ +# CRM Chat + Translate Pod (self-bootstrapping) + +One RunPod **Pod** running **Qwen2.5-14B** (chat/assistant, GPU) + **NLLB-200-1.3B** +(translation, CPU). It **auto-starts on every boot/resume** — you only Stop/Resume; +the CRM team's URLs never change and nothing needs editing. + +## One-time: create the Pod +RunPod → **Pods → Deploy**: + +| Setting | Value | +|---|---| +| **GPU** | **RTX 3090 (24 GB)** — On-Demand / Secure Cloud (not Spot) | +| **Template** | RunPod PyTorch (has python3 + CUDA) | +| **Network Volume** | create one, **~50 GB**, mount at **`/workspace`** ← holds models + deps so Stop/Resume keeps them | +| **Container Disk** | **40 GB** | +| **Expose HTTP Ports** (Edit Template) | `8000,11434` | +| **Docker / Start Command** | see below | + +**Start Command** (paste exactly — this is the whole setup): +``` +bash -c "curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/start.sh | bash" +``` + +First boot downloads Qwen (~16 GB) + converts NLLB → **~15–20 min**. Every Resume +after that: models already on the volume → **ready in ~1–2 min**, same URLs. + +## The two URLs to share with the CRM team +After deploy, each exposed port has a stable proxy URL (stable as long as you +**Stop/Resume**, never Terminate): +``` +Chat (Qwen, OpenAI-compatible): https://-11434.proxy.runpod.net/v1/chat/completions +Translate (NLLB): https://-8000.proxy.runpod.net/translate +``` + +## API for the CRM +**Chat / summarize** (OpenAI format, supports `stream:true`): +```json +POST /v1/chat/completions +{ "model": "qwen2.5:14b-instruct-q8_0", "stream": true, + "messages": [{ "role": "user", "content": "Summarize this thread in English: ..." }] } +``` +**Translate before send / on receive:** +```json +POST /translate +{ "text": "When can you deliver the cement?", "source": "eng_Latn", "target": "hin_Deva" } +-> { "translation": "..." } +``` + +## FLORES language codes (NLLB) +`eng_Latn` English · `hin_Deva` Hindi · `spa_Latn` Spanish · `fra_Latn` French · +`deu_Latn` German · `arb_Arab` Arabic · `zho_Hans` Chinese · `rus_Cyrl` Russian · +`por_Latn` Portuguese · `ben_Beng` Bengali · `tam_Taml` Tamil · `tel_Telu` Telugu · +`mar_Deva` Marathi · `guj_Gujr` Gujarati · `pan_Guru` Punjabi · `jpn_Jpan` Japanese · +`kor_Hang` Korean · `vie_Latn` Vietnamese · `ind_Latn` Indonesian +(full list: NLLB-200 FLORES-200 codes.) Your CRM maps each user's language → its code. + +## Security (do this) +- **Only call these URLs from your CRM backend**, never the browser (the proxy URLs are public). +- Optional: set env **`CHAT_POD_KEY`** on the pod → the translate API then requires + `Authorization: Bearer `. (Ollama has no built-in auth — keep it backend-only or front it with a gateway.) + +## Daily operation (what you actually do) +- **Start working:** Pod → **Resume** → wait ~1–2 min → team uses the same URLs. +- **Done:** Pod → **Stop** (stops GPU billing; keeps the volume + URL). +- **Never Terminate** — that deletes the pod and changes the URL. + +## Tuning +- Lighter/faster model: set env `QWEN_MODEL=qwen2.5:14b` (Q4, ~9 GB) or `qwen2.5:7b`. +- The script reads `QWEN_MODEL`, `NLLB_HF`, `CHAT_POD_KEY` from env if you want to override. diff --git a/chat-pod/start.sh b/chat-pod/start.sh new file mode 100644 index 0000000..bc143df --- /dev/null +++ b/chat-pod/start.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# ============================================================================= +# CRM chat + translate pod — self-bootstrapping start script. +# +# Set this as the Pod's Docker/Start Command (one line): +# bash -c "curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/start.sh | bash" +# +# It is IDEMPOTENT: safe to run on first boot AND on every Resume. Everything +# heavy (models + python deps) lives on the /workspace VOLUME, so Stop/Resume +# keeps it — the CRM team just needs the two proxy URLs, nothing to edit. +# +# Qwen2.5-14B (chat/assistant) -> GPU, Ollama, port 11434 (OpenAI-compatible) +# NLLB-200-1.3B (translation) -> CPU, CTranslate2 + FastAPI, port 8000 +# ============================================================================= +set -uo pipefail + +WORK=/workspace +VENV="$WORK/venv" +export OLLAMA_MODELS="$WORK/ollama" +export OLLAMA_HOST=0.0.0.0 +export NLLB_HF="facebook/nllb-200-1.3B" +export NLLB_CT2="$WORK/nllb-ct2" +QWEN_MODEL="${QWEN_MODEL:-qwen2.5:14b-instruct-q8_0}" # 8-bit ~16GB; override via env + +mkdir -p "$WORK" "$OLLAMA_MODELS" +echo "[chat-pod] boot — models/deps on $WORK (persist across Stop/Resume)" + +# --- 0. system deps: lspci lets Ollama auto-detect the GPU (else it may use CPU) --- +if ! command -v lspci >/dev/null 2>&1; then + echo "[chat-pod] installing pciutils (GPU detection)..." + apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq pciutils >/dev/null 2>&1 || true +fi + +# --- 1. Ollama (installs to container disk; reinstalled cheaply if missing) ---- +if ! command -v ollama >/dev/null 2>&1; then + echo "[chat-pod] installing Ollama..." + curl -fsSL https://ollama.com/install.sh | sh +fi + +# --- 2. Python deps on the VOLUME (persist) ----------------------------------- +if [ ! -x "$VENV/bin/uvicorn" ]; then + echo "[chat-pod] creating venv + installing NLLB deps (one time)..." + python3 -m venv "$VENV" + "$VENV/bin/pip" install -q --upgrade pip + "$VENV/bin/pip" install -q ctranslate2 transformers sentencepiece fastapi uvicorn +fi +# torch (CPU) is required to CONVERT the NLLB model; ensure it exists on any venv. +"$VENV/bin/python" -c "import torch" >/dev/null 2>&1 || { + echo "[chat-pod] installing CPU torch (needed to convert NLLB)..." + "$VENV/bin/pip" install -q torch --index-url https://download.pytorch.org/whl/cpu +} + +# --- 3. Fetch the translate server (always latest from repo) ------------------- +curl -fsSL https://raw.githubusercontent.com/Sumit-Pluto/photo_gallery/main/chat-pod/translate.py -o "$WORK/translate.py" \ + || echo "[chat-pod] WARN: could not refresh translate.py (using existing)" + +# --- 4. Start Ollama, wait for it, ensure Qwen is pulled (idempotent) --------- +echo "[chat-pod] starting Ollama (GPU)..." +ollama serve & +for i in $(seq 1 30); do curl -s http://localhost:11434/api/tags >/dev/null 2>&1 && break; sleep 2; done +echo "[chat-pod] ensuring model $QWEN_MODEL (first time downloads ~16GB)..." +ollama pull "$QWEN_MODEL" || echo "[chat-pod] WARN: pull failed (will use cached if present)" + +# --- 5. Convert NLLB once (cached on the volume after) ------------------------ +if [ ! -f "$NLLB_CT2/model.bin" ]; then + echo "[chat-pod] converting NLLB -> CTranslate2 int8 (one time)..." + rm -rf "$NLLB_CT2" # clear any partial/failed conversion + "$VENV/bin/ct2-transformers-converter" --model "$NLLB_HF" --output_dir "$NLLB_CT2" --quantization int8 +fi + +# --- 6. Start the NLLB translate API (CPU) ----------------------------------- +echo "[chat-pod] starting NLLB translate API (CPU) on :8000..." +cd "$WORK" +"$VENV/bin/uvicorn" translate:app --host 0.0.0.0 --port 8000 & + +echo "[chat-pod] READY -> Qwen :11434 (chat) | NLLB :8000 (translate)" +wait diff --git a/chat-pod/translate.py b/chat-pod/translate.py new file mode 100644 index 0000000..0473fbd --- /dev/null +++ b/chat-pod/translate.py @@ -0,0 +1,56 @@ +""" +NLLB-200 translation API (CPU, CTranslate2 int8). Part of the CRM chat pod. +Runs on port 8000 alongside Ollama/Qwen (port 11434) on the same pod. + +POST /translate {"text": "...", "source": "eng_Latn", "target": "hin_Deva"} + -> {"translation": "..."} +GET /health -> {"ok": true} + +Optional auth: set env CHAT_POD_KEY, then callers must send +`Authorization: Bearer `. +""" + +import os + +import ctranslate2 +import transformers +from fastapi import FastAPI, Header, HTTPException +from pydantic import BaseModel + +CT2 = os.environ.get("NLLB_CT2", "/workspace/nllb-ct2") +HF = os.environ.get("NLLB_HF", "facebook/nllb-200-1.3B") +API_KEY = os.environ.get("CHAT_POD_KEY", "") + +_translator = ctranslate2.Translator(CT2, device="cpu", compute_type="int8") +_tok = transformers.AutoTokenizer.from_pretrained(HF) + +app = FastAPI(title="CRM Translate (NLLB-200)") + + +class Req(BaseModel): + text: str + source: str # FLORES code, e.g. "eng_Latn" + target: str # FLORES code, e.g. "hin_Deva" + + +def _check(authorization: str) -> None: + if API_KEY and authorization != f"Bearer {API_KEY}": + raise HTTPException(status_code=401, detail="unauthorized") + + +@app.post("/translate") +def translate(r: Req, authorization: str = Header(default="")): + _check(authorization) + text = (r.text or "").strip() + if not text: + return {"translation": ""} + _tok.src_lang = r.source + source = _tok.convert_ids_to_tokens(_tok.encode(text)) + results = _translator.translate_batch([source], target_prefix=[[r.target]]) + hyp = results[0].hypotheses[0][1:] # drop the leading target-language token + return {"translation": _tok.decode(_tok.convert_tokens_to_ids(hyp))} + + +@app.get("/health") +def health(): + return {"ok": True} diff --git a/docs/AI-SETUP.md b/docs/AI-SETUP.md index 3a0f539..4df02dc 100644 --- a/docs/AI-SETUP.md +++ b/docs/AI-SETUP.md @@ -30,7 +30,65 @@ These edits *generate new pixels* from a prompt: **Restore, Colorize, Replace Sk Prompt**. Pick a backend with the `AI_EDIT_PROVIDER` env var (default `auto`). The server route (`/api/ai/edit`) proxies to it so no key ever reaches the browser. -`auto` chooses the first configured of: **local → huggingface → gemini**. +`auto` chooses the first configured of: **runpod → local → huggingface → gemini**. + +### Option 0 — RunPod serverless GPU endpoints (each model is its own endpoint) 🚀 + +This is the production backend for the models in `model-api-spec.docx`. Each model runs as a +separate RunPod endpoint; the app proxies to them **server-side** (`/api/ai/edit`, +`/api/ai/classify`) so `RUNPOD_API_KEY` and the URLs are **never** exposed to the browser. + +```env +AI_EDIT_PROVIDER=runpod +RUNPOD_API_KEY=rpa_xxxxxxxx +RUNPOD_SD_IMG2IMG_URL=https://api.runpod.ai/v2//runsync # #10 +RUNPOD_SD_INPAINT_URL=https://api.runpod.ai/v2//runsync # #9 +RUNPOD_UPSCALE_URL=https://api.runpod.ai/v2//runsync # #7 +RUNPOD_COLORIZE_URL=https://api.runpod.ai/v2//runsync # #8 +RUNPOD_BG_REMOVE_URL=https://api.runpod.ai/v2//runsync # #6 (optional; default is in-browser @imgly) +RUNPOD_YOLO_URL=https://api.runpod.ai/v2//runsync # #1 (also set NEXT_PUBLIC_APG_RUNPOD_DETECT=true) +``` + +**Which editor action hits which endpoint** (the app maps them for you): + +| Editor action | `op.type` | RunPod endpoint | Notes | +|---|---|---|---| +| Restore & Enhance | `restore` | #7 Real-ESRGAN (`RUNPOD_UPSCALE_URL`) | ×4 + face-enhance | +| Upscale | `upscale` | #7 Real-ESRGAN | ×2 / ×4 (`op.factor`) | +| Colorize | `colorize` | #8 DDColor (`RUNPOD_COLORIZE_URL`) | | +| Apply Prompt | `prompt` | #10 SD 3.5 img2img (`RUNPOD_SD_IMG2IMG_URL`) | whole-image, no mask | +| Replace Sky | `replace-sky` | #9 SD 3.5 inpaint **if a mask is sent**, else #10 img2img (low strength) | see masking below | +| Magic Eraser | `magic-eraser` | #9 SD 3.5 inpaint (`RUNPOD_SD_INPAINT_URL`) | **mask required** | +| Generative Fill | `generative-fill` | #9 SD 3.5 inpaint | **mask + prompt required** | + +**Endpoint I/O contract:** the app POSTs `{ input: { image, mask?, prompt?, strength?, guidance_scale?, +num_inference_steps?, seed?, … } }` and reads the image back from `output` (any of `image`, +`image_png`, `images[0]`). Deploy your handlers to accept that `input` shape (or adjust +`apps/web/src/lib/runpod/endpoints.ts`). `/runsync` is preferred; `/run` + `/status/{id}` polling is +supported as a fallback but must finish inside Vercel's 60 s function limit. + +**Masking (SD 3.5 #9):** masked ops send a `maskBase64` PNG **the same pixel size as the image** +(white = regenerate, black = keep). The client rasterizes the editor's `ImageData` mask to match the +downscaled image automatically (`apps/web/src/lib/ai/imageEncode.ts` → `maskToBase64`). **Magic +Eraser / Generative Fill therefore need a selection** — until a brush-mask UI is added to the editor +they return a clear "needs a mask/selection" error; **Replace Sky** works today by degrading to a +low-strength img2img (#10) when no mask is present (add sky-segmentation or a drawn mask to get true +masked #9). + +**Detection (#1):** set `NEXT_PUBLIC_APG_RUNPOD_DETECT=true` to route object detection to the YOLO +construction-material classifier via `/api/ai/classify`; it falls back to in-browser COCO-SSD if the +endpoint is unreachable. Verify the box format in `endpoints.ts` (`normalizeBox`) against your model — +it assumes ultralytics `xyxy` pixel coordinates. + +**Already local (no RunPod needed):** screenshot detection (#4), EXIF geolocation (#15), and the tag +rename lookup (#5) run in the app/browser — see `lib/classify.ts`, `lib/media.ts`. Background removal +(#6) also runs free in-browser by default. + +**Not yet wired (typed client + env only):** camera tilt (#2), voice-to-text (#3), audio denoise +(#12), and video frame-rate/resolution (#13/#14) have `rp*` functions in `endpoints.ts` and env vars, +but **no route or UI** — they have no consuming feature yet. #13/#14 in particular need an out-of-band +pipeline (presigned upload → async job → webhook), because per-frame video can't fit Vercel's 4.5 MB +body / 60 s limits or the in-browser video export path. Add these when the corresponding feature is scheduled. ### Option 1 — Your own local Stable Diffusion (free + private, recommended) 🏆 diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md index b362ad2..728de83 100644 --- a/docs/DEPLOY.md +++ b/docs/DEPLOY.md @@ -60,12 +60,21 @@ install the whole workspace). Do not commit `.env.local` (it's gitignored). **Server-only (optional — do NOT prefix with `NEXT_PUBLIC`):** | Name | Value | |---|---| - | `GEMINI_API_KEY` | a Gemini key *(only for the AI generative-edit button)* | + | `AI_EDIT_PROVIDER` | `runpod` *(or `local` / `huggingface` / `gemini` / `auto`)* | + | `RUNPOD_API_KEY` | your RunPod API key *(shared by every RunPod endpoint)* | + | `RUNPOD_SD_IMG2IMG_URL` | #10 prompt endpoint URL *(…/runsync)* | + | `RUNPOD_SD_INPAINT_URL` | #9 masked sky/eraser/fill endpoint URL | + | `RUNPOD_UPSCALE_URL` | #7 restore/upscale endpoint URL | + | `RUNPOD_COLORIZE_URL` | #8 colorize endpoint URL | + | `RUNPOD_YOLO_URL` | #1 detection endpoint URL *(+ `NEXT_PUBLIC_APG_RUNPOD_DETECT=true`)* | + | `GEMINI_API_KEY` | a Gemini key *(alternative generative-edit backend)* | | `GEMINI_IMAGE_MODEL` | `gemini-2.5-flash-image` *(optional)* | - Without the Supabase vars the app still runs, but on browser-local storage only (data won't sync - across devices). Without `GEMINI_API_KEY` the generative-edit route returns 503 while all the free - in-browser AI keeps working. + The RunPod URLs and `RUNPOD_API_KEY` are read **only** in the `/api/ai/*` route handlers (server + side) — never send them to the browser (no `NEXT_PUBLIC_` prefix). Full endpoint map + the op → + endpoint table is in [`docs/AI-SETUP.md`](./AI-SETUP.md). Without the Supabase vars the app still + runs on browser-local storage; without any AI-edit provider the generative-edit route returns 503 + while all the free in-browser AI (detection, faces, OCR, background removal) keeps working. Optional theming/feature flags (`NEXT_PUBLIC_APG_*`) are documented in [`docs/ENV.md`](./ENV.md). @@ -90,9 +99,12 @@ they're intentionally not stored. ## Gotchas / limits (free tier) -- **Gemini AI edit + Vercel body limit:** Vercel Hobby caps request bodies at ~4.5 MB. The AI-edit - route is capped accordingly and large images are downscaled client-side first; very large images - may still be rejected. The free in-browser AI is unaffected. +- **AI edit + Vercel body limit:** Vercel Hobby caps request bodies at ~4.5 MB. The AI-edit route is + capped accordingly and large images are downscaled client-side first (≤1280 px). For masked SD 3.5 + ops the **image + mask share one body**, so they're budgeted together; a mask PNG is mostly flat + black/white and compresses small, so this holds. Very large images may still be rejected. Prefer + RunPod `/runsync` endpoints so a job returns within the 60 s function limit. The free in-browser AI + is unaffected. - **Single-user demo:** the Supabase adapter does a full-state sync (no auth). For multi-user, add Supabase Auth and scope rows per user before going to production (the RLS policies are open for the demo — tighten them with `auth.uid()`). diff --git a/packages/photo-sdk/src/adapters/localStorage.ts b/packages/photo-sdk/src/adapters/localStorage.ts index 51a0460..b29580c 100644 --- a/packages/photo-sdk/src/adapters/localStorage.ts +++ b/packages/photo-sdk/src/adapters/localStorage.ts @@ -43,10 +43,24 @@ export function createLocalStorageAdapter(key: string = STORAGE_KEY): StorageAda : [], })) : []; + // Keep only string→string entries — never trust persisted data blindly. + const labelAliases: Record = + parsed.labelAliases && typeof parsed.labelAliases === 'object' + ? Object.fromEntries( + Object.entries(parsed.labelAliases).filter( + ([k, v]) => typeof k === 'string' && typeof v === 'string', + ), + ) + : {}; + const deletedLabels: string[] = Array.isArray(parsed.deletedLabels) + ? (parsed.deletedLabels as unknown[]).filter((l): l is string => typeof l === 'string') + : []; return { media, albums, people: Array.isArray(parsed.people) ? parsed.people : [], + labelAliases, + deletedLabels, version: typeof parsed.version === 'number' ? parsed.version : CURRENT_VERSION, }; } catch { diff --git a/packages/photo-sdk/src/adapters/types.ts b/packages/photo-sdk/src/adapters/types.ts index fc5beae..e39bd8b 100644 --- a/packages/photo-sdk/src/adapters/types.ts +++ b/packages/photo-sdk/src/adapters/types.ts @@ -4,6 +4,12 @@ export interface PersistedState { media: MediaItem[]; albums: Album[]; people: Person[]; + /** + * User's permanent object-tag renames (canonical lowercased detector label → + * chosen label). Optional for backward compatibility with pre-rename data. + */ + labelAliases?: Record; + deletedLabels?: string[]; version: number; } diff --git a/packages/photo-sdk/src/ai/types.ts b/packages/photo-sdk/src/ai/types.ts index 72a4f6c..0f5e757 100644 --- a/packages/photo-sdk/src/ai/types.ts +++ b/packages/photo-sdk/src/ai/types.ts @@ -33,18 +33,46 @@ export interface AIProvider { image: ImageBitmap | HTMLImageElement, op: GenerativeEditOp, ): Promise; + + /** + * Transcribe recorded speech (base64 WAV, 16 kHz mono PCM16) to text. + * Powers voice input for photo annotations / comments. + */ + transcribeAudio?(audioBase64: string): Promise; + + /** + * Denoise recorded audio (base64 WAV) → cleaned base64 WAV. Useful before + * transcription when the recording was made on a noisy site. + */ + denoiseAudio?(audioBase64: string): Promise; + + /** Estimate camera tilt (roll/pitch/fov, degrees) for auto-straightening. */ + estimateTilt?(item: MediaItem, image: ImageBitmap | HTMLImageElement): Promise; +} + +/** Camera-tilt estimate (degrees). `rollDegrees` = in-plane rotation to correct. */ +export interface TiltEstimate { + rollDegrees: number; + pitchDegrees: number; + fovDegrees: number; } export type GenerativeEditOp = | { type: 'remove-background' } - | { type: 'replace-sky'; prompt?: string } - | { type: 'magic-eraser'; mask: ImageData } - | { type: 'generative-fill'; prompt: string; mask: ImageData } + | { type: 'replace-sky'; prompt?: string; strength?: number } + | { type: 'magic-eraser'; mask: ImageData; strength?: number } + | { type: 'generative-fill'; prompt: string; mask: ImageData; strength?: number } | { type: 'restore' } | { type: 'upscale'; factor: 2 | 4 } | { type: 'colorize' } - /** Free-form natural-language edit instruction. */ - | { type: 'prompt'; prompt: string }; + /** Outpaint / expand-canvas: pad the image and generatively fill the new border. */ + | { type: 'outpaint'; prompt?: string; factor?: number; strength?: number } + /** + * Free-form natural-language edit instruction. `strength` (0..1) controls how + * strongly the edit is applied (subtle → strong); the backend maps it to the + * appropriate model knob. + */ + | { type: 'prompt'; prompt: string; strength?: number }; /** Cosine similarity between two equal-length vectors. */ export function cosineSimilarity(a: number[], b: number[]): number { diff --git a/packages/photo-sdk/src/components/AIAnalyzer.tsx b/packages/photo-sdk/src/components/AIAnalyzer.tsx index b212e2e..aa07e64 100644 --- a/packages/photo-sdk/src/components/AIAnalyzer.tsx +++ b/packages/photo-sdk/src/components/AIAnalyzer.tsx @@ -4,12 +4,13 @@ import { useEffect, useRef } from 'react'; import type { AIProvider } from '../ai/types'; import { PET_LABELS } from '../constants'; +import { resolveLabel } from '../lib/smartAlbums'; import { sanitizeOcrText } from '../lib/text'; import { useGallery, useGalleryStoreApi } from '../store/context'; import type { MediaItem } from '../types'; const CONCURRENCY = 2; -const CONFIDENCE = 0.5; +const CONFIDENCE = 0.15; const START_DELAY_MS = 1200; /** @@ -33,14 +34,16 @@ export function AIAnalyzer({ provider }: { provider: AIProvider }) { provider.detectObjects || provider.detectFaces || provider.ocr || provider.embedImage; if (!ready || !canDetect || runningRef.current) return; - // Each capability has its OWN backfill gate (not the shared analyzedAt), so a - // capability added in a later session re-processes items analyzed before it - // existed: objects key off analyzedAt; faces off `faces === undefined`; OCR - // off `ocrText === undefined`. + // Analyze each image EXACTLY ONCE: every capability is gated on `!analyzedAt`, + // so once an item has been analyzed (and analyzedAt persisted) it is never + // re-processed on later reloads — even if an individual result field didn't + // round-trip through storage. Fresh uploads (no analyzedAt) run everything. const needsObjects = (m: MediaItem) => !!provider.detectObjects && !m.analyzedAt; - const needsFaces = (m: MediaItem) => !!provider.detectFaces && m.faces === undefined; - const needsOcr = (m: MediaItem) => !!provider.ocr && m.ocrText === undefined; - const needsEmbedding = (m: MediaItem) => !!provider.embedImage && m.embedding === undefined; + const needsFaces = (m: MediaItem) => + !!provider.detectFaces && !m.analyzedAt && m.faces === undefined; + const needsOcr = (m: MediaItem) => !!provider.ocr && !m.analyzedAt && m.ocrText === undefined; + const needsEmbedding = (m: MediaItem) => + !!provider.embedImage && !m.analyzedAt && m.embedding === undefined; // Collect items still needing analysis, NEWEST FIRST — so a freshly uploaded or // captured photo is tagged immediately instead of waiting behind the whole library. const collectPending = () => @@ -99,8 +102,17 @@ export function AIAnalyzer({ provider }: { provider: AIProvider }) { const patch: Partial = { analyzedAt: Date.now() }; if (objects) { patch.objects = objects; + // Map each detected label through the user's rename map so future uploads + // are stored/grouped under the renamed label instead of a fresh class. + const aliases = api.getState().labelAliases; + const deleted = api.getState().deletedLabels; patch.objectLabels = [ - ...new Set(objects.filter((o) => o.confidence >= CONFIDENCE).map((o) => o.label)), + ...new Set( + objects + .filter((o) => o.confidence >= CONFIDENCE) + .map((o) => resolveLabel(o.label, aliases)) + .filter((l) => !deleted.includes(l)), + ), ]; } if (faces) patch.faces = faces; diff --git a/packages/photo-sdk/src/components/InfoPanel.tsx b/packages/photo-sdk/src/components/InfoPanel.tsx index 581bbce..abacfa5 100644 --- a/packages/photo-sdk/src/components/InfoPanel.tsx +++ b/packages/photo-sdk/src/components/InfoPanel.tsx @@ -5,7 +5,9 @@ import { useEffect, useRef, useState } from 'react'; import { sourceLabel } from '../lib/classify'; import { editFilterCss, editTransformCss } from '../lib/edits'; import { formatBytes, formatDate, formatDuration, formatTime } from '../lib/format'; +import { blobToWavBase64, startRecording, wavBase64ToBlob, type Recorder } from '../lib/audioCapture'; import { Icon } from '../icons'; +import { useAIProvider } from './aiContext'; import { useGallery, useGalleryStoreApi } from '../store/context'; import type { MediaItem } from '../types'; @@ -49,6 +51,13 @@ export function InfoPanel() { {formatDate(item.takenAt)} · {formatTime(item.takenAt)} + {item.kind === 'image' && !item.analyzedAt ? ( +
+ + Analyzing image… +
+ ) : null} + @@ -70,6 +79,10 @@ export function InfoPanel() { .join(' · ')} /> ) : null} + {item.exif?.LensModel ? : null} + {item.exif?.Orientation ? ( + + ) : null} {item.objectLabels.length ? ( api.getState().focusMap({ lat: item.location!.lat, lng: item.location!.lng })} /> + api.getState().focusMap({ lat: item.location!.lat, lng: item.location!.lng })} + />
- Click the map to open it in full. + Click the map or address to open it in full.
) : null} @@ -221,6 +239,49 @@ function Comments({ item }: { item: MediaItem }) { return window.localStorage.getItem('apg:comment-author') || 'You'; }); + const provider = useAIProvider(); + const canVoice = Boolean(provider?.transcribeAudio); + const canDenoise = Boolean(provider?.denoiseAudio); + const [recording, setRecording] = useState(false); + const [denoise, setDenoise] = useState(false); + const [voiceStatus, setVoiceStatus] = useState(null); + const recorderRef = useRef(null); + + const startVoice = async () => { + setVoiceStatus(null); + try { + recorderRef.current = await startRecording(); + setRecording(true); + } catch (e) { + setVoiceStatus(e instanceof Error ? e.message : 'Microphone unavailable.'); + } + }; + + const stopVoice = async () => { + const rec = recorderRef.current; + recorderRef.current = null; + setRecording(false); + if (!rec || !provider?.transcribeAudio) return; + try { + const blob = await rec.stop(); + let wav16: string; + if (denoise && provider.denoiseAudio) { + setVoiceStatus('Reducing noise…'); + const wav48 = await blobToWavBase64(blob, 48000); + const cleaned = await provider.denoiseAudio(wav48); + wav16 = await blobToWavBase64(wavBase64ToBlob(cleaned), 16000); + } else { + wav16 = await blobToWavBase64(blob, 16000); + } + setVoiceStatus('Transcribing…'); + const spoken = (await provider.transcribeAudio(wav16)).trim(); + if (spoken) setText((prev) => (prev ? `${prev} ${spoken}` : spoken)); + setVoiceStatus(null); + } catch (e) { + setVoiceStatus(e instanceof Error ? e.message : 'Could not transcribe audio.'); + } + }; + const post = () => { const t = text.trim(); if (!t) return; @@ -294,6 +355,36 @@ function Comments({ item }: { item: MediaItem }) { rows={2} maxLength={2000} /> + {canVoice ? ( +
+ + {canDenoise ? ( + + ) : null} + + {recording ? '● Listening…' : (voiceStatus ?? '')} + +
+ ) : null} + ); + } + + return ( + + ); +} + /** Small non-interactive Leaflet map for the location preview (click to open full Map). */ function MiniMap({ lat, lng, onOpen }: { lat: number; lng: number; onOpen?: () => void }) { const ref = useRef(null); diff --git a/packages/photo-sdk/src/components/Sidebar.tsx b/packages/photo-sdk/src/components/Sidebar.tsx index 0b1e03c..0ad36ac 100644 --- a/packages/photo-sdk/src/components/Sidebar.tsx +++ b/packages/photo-sdk/src/components/Sidebar.tsx @@ -7,7 +7,7 @@ import { Icon, type IconName } from '../icons'; import { useGallery, useGalleryStoreApi } from '../store/context'; import type { Album, ViewId } from '../types'; import { closeContextMenu, openContextMenu } from './ContextMenu'; -import { openSecuritySettings, promptAlbumName } from './modals'; +import { confirmAction, openSecuritySettings, promptAlbumName } from './modals'; interface RowProps { icon: IconName; @@ -161,6 +161,38 @@ export function Sidebar() { ]); }; + // Right-click an auto object album → permanently rename its tag (e.g. car → excavator). + const objectMenu = (album: Album) => (e: React.MouseEvent) => { + e.preventDefault(); + const label = album.id.slice('sys:obj:'.length); + openContextMenu(e.clientX, e.clientY, [ + { + label: 'Rename Tag', + icon: 'tag', + onClick: () => + promptAlbumName( + 'Rename Tag', + album.name, + (name) => api.getState().renameLabel(label, name), + { placeholder: 'Tag name' }, + ), + }, + { + label: 'Delete Tag', + icon: 'trash', + danger: true, + onClick: () => + confirmAction({ + title: 'Delete Tag', + message: `Remove the "${album.name}" tag? It's deleted from all photos and won't be created again.`, + confirmLabel: 'Delete', + danger: true, + onConfirm: () => api.getState().deleteLabel(label), + }), + }, + ]); + }; + const newAlbum = () => promptAlbumName('New Album', '', (name) => { const id = api.getState().createAlbum(name); @@ -182,7 +214,7 @@ export function Sidebar() { - + )) : null} diff --git a/packages/photo-sdk/src/components/editor/MaskBrush.tsx b/packages/photo-sdk/src/components/editor/MaskBrush.tsx new file mode 100644 index 0000000..ef602c4 --- /dev/null +++ b/packages/photo-sdk/src/components/editor/MaskBrush.tsx @@ -0,0 +1,171 @@ +'use client'; + +import { type PointerEvent as ReactPointerEvent, useEffect, useRef, useState } from 'react'; + +import { Icon } from '../../icons'; + +/** + * Full-screen brush overlay for masked AI edits (Magic Eraser / Generative Fill). + * The user paints over a region; on Apply we emit a binary ImageData mask where + * painted pixels are WHITE (regenerate) and everything else is BLACK (keep) — + * exactly what the inpaint backend expects (see maskToBase64 / rpInpaint). + * + * The paint canvas is sized to the image's aspect ratio (long side = BASE), then + * scaled with CSS to fit the viewport, so the returned mask lines up with the + * photo regardless of screen size. + */ +const BASE = 640; + +export function MaskBrush({ + src, + aspect, + title, + onCancel, + onApply, +}: { + src: string; + aspect: number; // width / height + title: string; + onCancel: () => void; + onApply: (mask: ImageData) => void; +}) { + const canvasRef = useRef(null); + const paintingRef = useRef(false); + const lastRef = useRef<{ x: number; y: number } | null>(null); + const [brush, setBrush] = useState(48); + const [dirty, setDirty] = useState(false); + + const cw = aspect >= 1 ? BASE : Math.round(BASE * aspect); + const ch = aspect >= 1 ? Math.round(BASE / aspect) : BASE; + + // Escape cancels. + useEffect(() => { + const onKey = (e: KeyboardEvent) => { + if (e.key === 'Escape') onCancel(); + }; + window.addEventListener('keydown', onKey); + return () => window.removeEventListener('keydown', onKey); + }, [onCancel]); + + const ctx = () => canvasRef.current?.getContext('2d') ?? null; + + const toCanvas = (e: ReactPointerEvent) => { + const c = canvasRef.current; + if (!c) return { x: 0, y: 0 }; + const r = c.getBoundingClientRect(); + return { + x: ((e.clientX - r.left) / r.width) * c.width, + y: ((e.clientY - r.top) / r.height) * c.height, + }; + }; + + const paintTo = (x: number, y: number) => { + const c = ctx(); + if (!c) return; + c.fillStyle = 'rgba(255,60,60,0.55)'; + c.strokeStyle = 'rgba(255,60,60,0.55)'; + c.lineWidth = brush; + c.lineCap = 'round'; + const last = lastRef.current; + if (last) { + c.beginPath(); + c.moveTo(last.x, last.y); + c.lineTo(x, y); + c.stroke(); + } + c.beginPath(); + c.arc(x, y, brush / 2, 0, Math.PI * 2); + c.fill(); + lastRef.current = { x, y }; + }; + + const onDown = (e: ReactPointerEvent) => { + e.preventDefault(); + (e.target as HTMLElement).setPointerCapture?.(e.pointerId); + paintingRef.current = true; + lastRef.current = null; + const p = toCanvas(e); + paintTo(p.x, p.y); + setDirty(true); + }; + const onMove = (e: ReactPointerEvent) => { + if (!paintingRef.current) return; + const p = toCanvas(e); + paintTo(p.x, p.y); + }; + const onUp = () => { + paintingRef.current = false; + lastRef.current = null; + }; + + const clear = () => { + const c = ctx(); + if (c && canvasRef.current) c.clearRect(0, 0, canvasRef.current.width, canvasRef.current.height); + setDirty(false); + }; + + const apply = () => { + const c = ctx(); + const canvas = canvasRef.current; + if (!c || !canvas) return; + const painted = c.getImageData(0, 0, canvas.width, canvas.height); + // Binarize: any painted (alpha) pixel → opaque white, else opaque black. + const out = new ImageData(canvas.width, canvas.height); + for (let i = 0; i < painted.data.length; i += 4) { + const on = painted.data[i + 3]! > 10; + const v = on ? 255 : 0; + out.data[i] = v; + out.data[i + 1] = v; + out.data[i + 2] = v; + out.data[i + 3] = 255; + } + onApply(out); + }; + + return ( +
+
{title}
+
+ + +
+
+ + +
+ + +
+
+ ); +} diff --git a/packages/photo-sdk/src/components/editor/PhotoEditor.tsx b/packages/photo-sdk/src/components/editor/PhotoEditor.tsx index 6aeef16..cc10d74 100644 --- a/packages/photo-sdk/src/components/editor/PhotoEditor.tsx +++ b/packages/photo-sdk/src/components/editor/PhotoEditor.tsx @@ -17,6 +17,8 @@ import { addToAlbumPicker, confirmAction } from '../modals'; import { useAIProvider } from '../aiContext'; import { Annotations, type AnnotationTool } from './Annotations'; import { CropBox, type CropRect } from './CropBox'; +import { MaskBrush } from './MaskBrush'; +import { VoiceButton } from './VoiceButton'; const RATIOS: Array<{ label: string; value: number | null }> = [ { label: 'Free', value: null }, @@ -77,6 +79,14 @@ const AI_OPS: Array<{ label: string; op: GenerativeEditOp; icon: 'wand' | 'image { label: 'Replace Sky', op: { type: 'replace-sky' }, icon: 'image' }, ]; +/** Ops whose result varies with the "edit strength" slider (the backend maps it per model). */ +const STRENGTH_OPS = new Set([ + 'prompt', + 'replace-sky', + 'magic-eraser', + 'generative-fill', +]); + export function PhotoEditor() { const api = useGalleryStoreApi(); const editorId = useGallery((s) => s.editorId); @@ -91,6 +101,10 @@ export function PhotoEditor() { const [aiError, setAiError] = useState(null); const [aiResultUrl, setAiResultUrl] = useState(null); const [aiPrompt, setAiPrompt] = useState(''); + const [aiStrength, setAiStrength] = useState(0.5); + const [maskMode, setMaskMode] = useState<'magic-eraser' | 'generative-fill' | null>(null); + const [tiltBusy, setTiltBusy] = useState(false); + const [tiltError, setTiltError] = useState(null); const [annTool, setAnnTool] = useState('rect'); const [annColor, setAnnColor] = useState('#ff3b30'); const [cropRatio, setCropRatio] = useState(null); @@ -146,7 +160,11 @@ export function PhotoEditor() { setAiError(null); try { const img = await loadCrossOriginImage(item.src); - const blob = await provider.generativeEdit(item, img, op); + // Attach the current "edit strength" to ops that support it. + const opToRun = STRENGTH_OPS.has(op.type) + ? ({ ...op, strength: aiStrength } as GenerativeEditOp) + : op; + const blob = await provider.generativeEdit(item, img, opToRun); aiBlobRef.current = blob; setAiResultUrl((prev) => { if (prev) URL.revokeObjectURL(prev); @@ -159,6 +177,22 @@ export function PhotoEditor() { } }; + const autoStraighten = async () => { + if (!provider?.estimateTilt) return; + setTiltBusy(true); + setTiltError(null); + try { + const img = await loadCrossOriginImage(item.src); + const t = await provider.estimateTilt(item, img); + const roll = Math.max(-45, Math.min(45, Math.round(t.rollDegrees))); + setEdits((e) => ({ ...e, straighten: roll })); + } catch (err) { + setTiltError(err instanceof Error ? err.message : 'Could not estimate tilt.'); + } finally { + setTiltBusy(false); + } + }; + const clearAI = () => { aiBlobRef.current = null; setAiError(null); @@ -379,7 +413,7 @@ export function PhotoEditor() { }} > - Generating with Gemini… + Generating…
) : null} @@ -526,6 +560,20 @@ export function PhotoEditor() { Reset Straighten ) : null} + {provider?.estimateTilt ? ( + + ) : null} + {tiltError ? ( +

{tiltError}

+ ) : null} ) : null} @@ -547,6 +595,28 @@ export function PhotoEditor() { ))} + {provider?.transcribeAudio ? ( + { + setAnnotations([ + ...annotations, + { + id: nanoid(8), + shape: 'text', + color: annColor, + strokeWidth: 2, + x1: 0.08, + y1: 0.08, + x2: 0.55, + y2: 0.17, + text: t, + }, + ]); + setAnnTool('select'); + }} + /> + ) : null}
Color
@@ -596,7 +666,7 @@ export function PhotoEditor() { {tab === 'ai' ? (

- Generative edits powered by Gemini. Results replace the photo when you press Save. + Generative edits run through your configured AI backend. Results replace the photo when you press Save.

{AI_OPS.map((a) => ( + + + {aiResultUrl ? (
+ + {maskMode ? ( + setMaskMode(null)} + onApply={(mask) => { + const m = maskMode; + setMaskMode(null); + if (m === 'generative-fill') { + void runAI({ type: 'generative-fill', prompt: aiPrompt.trim() || 'fill naturally', mask }); + } else { + void runAI({ type: 'magic-eraser', mask }); + } + }} + /> + ) : null} ); diff --git a/packages/photo-sdk/src/components/editor/VideoEditor.tsx b/packages/photo-sdk/src/components/editor/VideoEditor.tsx index bbbd8be..a2b5c9b 100644 --- a/packages/photo-sdk/src/components/editor/VideoEditor.tsx +++ b/packages/photo-sdk/src/components/editor/VideoEditor.tsx @@ -9,6 +9,9 @@ import { useFocusTrap } from '../../hooks/useFocusTrap'; import { editFilterCss } from '../../lib/edits'; import { summarizeEdits } from '../../lib/versions'; import { bakeVideo } from '../../lib/videoBake'; +import { blobToWavBase64, wavBase64ToBlob } from '../../lib/audioCapture'; +import { useAIProvider } from '../aiContext'; +import { VoiceButton } from './VoiceButton'; import { normalizeSegments, outputDuration, @@ -91,6 +94,9 @@ export function VideoEditor() { const [duration, setDuration] = useState(0); const [playhead, setPlayhead] = useState(0); const [selOverlay, setSelOverlay] = useState(null); + const provider = useAIProvider(); + const [denoiseBusy, setDenoiseBusy] = useState(false); + const [denoiseErr, setDenoiseErr] = useState(null); const [previewH, setPreviewH] = useState(360); const [baking, setBaking] = useState(false); const [progress, setProgress] = useState(0); @@ -209,6 +215,26 @@ export function VideoEditor() { opacity: 1, rotation: 0, }); + const runVideoDenoise = async () => { + if (!provider?.denoiseAudio) return; + setDenoiseBusy(true); + setDenoiseErr(null); + try { + const resp = await fetch(item.src); + const blob = await resp.blob(); + // Decode the video's audio track → 48 kHz mono WAV → RunPod denoise → clean WAV. + const wav48 = await blobToWavBase64(blob, 48000); + const cleaned = await provider.denoiseAudio(wav48); + const url = URL.createObjectURL(wavBase64ToBlob(cleaned)); + update({ audio: { ...edits.audio, denoisedSrc: url } }); + } catch (e) { + setDenoiseErr( + e instanceof Error ? e.message : 'Could not clean the audio (keep clips under ~30s).', + ); + } finally { + setDenoiseBusy(false); + } + }; const addKeyframe = (id: string) => { const o = overlays.find((x) => x.id === id); if (!o) return; @@ -314,6 +340,21 @@ export function VideoEditor() { }; const filterCss = editFilterCss(edits); + // Live preview for Crop & Rotate. videoBake applies these on export; without + // mirroring them on the preview the Rotate/Flip/Crop buttons look like they do + // nothing. Quarter-turn rotations are scaled to fit the frame. + const pvRot = (((edits.rotation ?? 0) % 360) + 360) % 360; + const pvQuarter = pvRot === 90 || pvRot === 270; + const pvAspect = (item.width || 16) / (item.height || 9); + const pvFit = pvQuarter ? Math.min(pvAspect, 1 / pvAspect) : 1; + const previewTransform = + pvRot || edits.flipH || edits.flipV + ? `rotate(${pvRot}deg) scale(${(edits.flipH ? -1 : 1) * pvFit}, ${(edits.flipV ? -1 : 1) * pvFit})` + : undefined; + const pvCrop = edits.crop; + const previewClip = pvCrop + ? `inset(${(pvCrop.y * 100).toFixed(3)}% ${((1 - pvCrop.x - pvCrop.width) * 100).toFixed(3)}% ${((1 - pvCrop.y - pvCrop.height) * 100).toFixed(3)}% ${(pvCrop.x * 100).toFixed(3)}%)` + : undefined; const sel = overlays.find((o) => o.id === selOverlay) ?? null; return ( @@ -360,7 +401,15 @@ export function VideoEditor() { controls playsInline crossOrigin="anonymous" - style={{ maxWidth: '100%', maxHeight: '70vh', display: 'block', filter: filterCss || undefined }} + style={{ + maxWidth: '100%', + maxHeight: '70vh', + display: 'block', + filter: filterCss || undefined, + transform: previewTransform, + clipPath: previewClip, + transition: 'transform 0.15s ease', + }} onLoadedMetadata={(e) => { const d = e.currentTarget.duration || 0; setDuration(d); @@ -651,6 +700,13 @@ export function VideoEditor() { onChange={(e) => patchOverlay(sel.id, { text: e.target.value })} /> + { + const cur = sel.text && sel.text !== 'Your text' ? sel.text : ''; + patchOverlay(sel.id, { text: cur ? `${cur} ${t}` : t }); + }} + />
{COLORS.map((c) => ( +
+ ) : ( + + ) + ) : null} + {denoiseErr ? ( +

{denoiseErr}

+ ) : null} {!edits.audio?.muted ? (