lynkeduppro-crm/docs/SMART_GALLERY.md

Smart Gallery — AI route surface

The Smart Gallery embeds @photo-gallery/sdk into the CRM. The SDK never talks to a model directly: it calls a pluggable AIProvider, which the CRM supplies via createCrmAIProvider() in src/lib/gallery-ai.ts.

Roughly half the intelligence runs in the browser for free, and the other half is proxied through five server routes under /api/gallery/ai/* so the RunPod API key never reaches the client.

---

1. Where each capability runs

Capability	Where	Backend	Needs a key?
Object detection (default)	Browser	TensorFlow.js COCO-SSD (80 COCO classes)	No
Object detection (opt-in)	Server	RunPod YOLO → /classify	Yes
Face detection + recognition	Browser	@vladmandic/face-api (128-D descriptors → People)	No
OCR / document search	Browser	tesseract.js	No
Semantic ("beach photos") search	Browser	CLIP via @huggingface/transformers	No
Background removal (default)	Browser	@imgly/background-removal (WASM)	No
Background removal (opt-in)	Server	RunPod U²-Net → /edit	Yes
Generative edits, restore, upscale, outpaint	Server	/edit	Yes
Speech-to-text	Server	/transcribe	Yes
Audio denoise	Server	/denoise	Yes
Camera tilt / auto-straighten	Server	/tilt	Yes

Every in-browser model is loaded with await import(...) on first use, so none of it lands in the initial bundle. Every capability degrades gracefully — a model that fails to load returns [] / '' / null with a console.warn rather than throwing, so a blocked CDN never breaks the gallery UI.

---

2. Auth model

All five routes share one gate: requireGallerySession() in src/lib/server/session.ts.

These routes proxy a paid, rate-limited GPU backend using a secret held only on the server. An unauthenticated route here is not just an information leak — it is an open invitation to spend the operator's GPU budget. (The upstream SDK demo's routes are completely unauthenticated; that is the single biggest thing this port fixes.)

When the Shell is configured (NEXT_PUBLIC_SUPABASE_URL is set): the incoming cookie header is forwarded to ${BFF_ORIGIN}/api/session/context and only a 200 is accepted.

• 401 upstream → 401 { error: "Not signed in" }
• any other non-200, or a network/timeout failure → 503 { error: "Session service unavailable" } (fail closed — if we cannot prove a session, we do not spend GPU budget)
• Nothing is cached. A cached "yes" would keep a revoked session alive, so every AI request costs one BFF round trip. Correctness over latency for a spend gate.
• The BFF is trusted absolutely — BFF_ORIGIN must only ever point at an origin the operator controls.

When the Shell is NOT configured (local demo mode): the request is allowed and a warning is logged once. Never deploy to a public origin with the Shell unconfigured and a real RUNPOD_API_KEY present — that combination is an open, billable endpoint.

This is authentication only, not authorization. Per-resource gallery policy lives in be-crm behind the crm.gallery resource.

---

3. Rate limits

src/lib/server/rate-limit.ts — a fixed-window counter keyed by the authenticated principal when known, otherwise the first hop of x-forwarded-for. Each route has its own namespace, so spending your edit budget does not consume your classify budget. Exceeding it returns 429 { error: "Too many requests — slow down." } with a Retry-After header.

Route	Limit
/classify	30 / min
/tilt	30 / min
/transcribe	20 / min
/denoise	20 / min
/edit	12 / min (most expensive)

This limiter is per-instance and in-memory. With N instances behind a load balancer a caller gets up to N x the budget, and a restart clears all counters. It is a cost guard, not a security boundary. Replace it with Redis before running more than one instance. The x-forwarded-for key is also client-controlled unless a trusted proxy overwrites it — the session gate, not this, is the security boundary.

---

4. The routes

All five are POST only, and all declare runtime = "nodejs", maxDuration = 60, dynamic = "force-dynamic". Failures always return { error: string }.

Shared status codes: 400 invalid body/params · 401 not signed in · 413 payload too large · 429 rate limited · 500 server misconfigured (e.g. missing RUNPOD_API_KEY) · 502 upstream failed · 503 not configured / session service unavailable · 504 upstream timed out (the RunPod client's budget is 55s, under the 60s cap).

Upstream error bodies are truncated to a 160-character excerpt; the RunPod key is never included in any response.

POST /api/gallery/ai/classify — object detection

Backed by RUNPOD_YOLO_URL. Max ~4 MB of base64. Returns boxes as fractions 0..1 of the image, matching the SDK's DetectedObject.

// request
{ "imageBase64": "…", "width": 1280, "height": 853 }
// response
{ "objects": [ { "label": "excavator", "confidence": 0.91,
                 "box": { "x": 0.12, "y": 0.30, "width": 0.25, "height": 0.40 } } ] }

POST /api/gallery/ai/edit — generative image editing

The backend is pluggable via AI_EDIT_PROVIDER (auto | runpod | local | huggingface | gemini | none). Image + mask are budgeted together against the ~4 MB cap. Prompts come from an allow-listed op set — arbitrary server-side prompts are never accepted, and free-text is clamped to 500 chars.

// request
{ "imageBase64": "…", "mimeType": "image/jpeg",
  "op": { "type": "restore" },            // or prompt | colorize | replace-sky |
                                          // magic-eraser | generative-fill | upscale |
                                          // remove-background
  "maskBase64": "…",                      // required for magic-eraser / generative-fill
  "params": { "strength": 0.8 } }
// response
{ "imageBase64": "…", "mimeType": "image/png" }

Op → RunPod endpoint: restore/upscale → RUNPOD_UPSCALE_URL · prompt/colorize → RUNPOD_SD_IMG2IMG_URL · replace-sky/magic-eraser/ generative-fill → RUNPOD_SD_INPAINT_URL · remove-background → RUNPOD_BG_REMOVE_URL. upscale, magic-eraser and generative-fill are RunPod-only and return 400 under another backend. Outpaint is client-side padding plus a generative-fill call — it needs no separate route.

POST /api/gallery/ai/tilt — camera tilt

Backed by RUNPOD_TILT_URL. Max ~4 MB (413 over). Only reachable when NEXT_PUBLIC_APG_RUNPOD_TILT=true, which also reveals the editor's Auto-straighten button.

{ "image": "…" }  →  { "rollDegrees": -2.4, "pitchDegrees": 1.1, "fovDegrees": 68.2 }

POST /api/gallery/ai/transcribe — speech to text

Backed by RUNPOD_STT_URL. Expects base64 WAV 16 kHz mono PCM16. Max ~8 MB (413).

{ "audio": "…", "language": "en" }
→ { "transcript": "…", "segments": [ { "text": "…", "startSec": 0, "endSec": 1.8 } ] }

POST /api/gallery/ai/denoise — audio noise removal

Backed by RUNPOD_AUDIO_DENOISE_URL. Expects base64 WAV 48 kHz mono PCM16. Max ~12 MB (413). Used before transcription on noisy sites.

{ "audio": "…" }  →  { "audio": "…" }

---

5. Environment variables

See .env.local.example for the fully commented template.

Var	Backs
RUNPOD_API_KEY	all five routes (server-only — never NEXT_PUBLIC_)
RUNPOD_YOLO_URL	/classify
RUNPOD_SD_IMG2IMG_URL	/edit — prompt, colorize, maskless replace-sky
RUNPOD_SD_INPAINT_URL	/edit — magic-eraser, generative-fill, outpaint, masked replace-sky
RUNPOD_UPSCALE_URL	/edit — restore, upscale
RUNPOD_BG_REMOVE_URL	/edit — remove-background
RUNPOD_STT_URL	/transcribe
RUNPOD_AUDIO_DENOISE_URL	/denoise
RUNPOD_TILT_URL	/tilt
AI_EDIT_PROVIDER	which backend /edit uses
BFF_ORIGIN	the session gate

Client switches (NEXT_PUBLIC_, inlined at build time — a change needs a rebuild, and each must be the literal string "true"): NEXT_PUBLIC_APG_RUNPOD_DETECT, NEXT_PUBLIC_APG_RUNPOD_BG, NEXT_PUBLIC_APG_RUNPOD_TILT.

The SDK's standalone demo also reads a family of NEXT_PUBLIC_APG_* theming vars (_THEME, _ACCENT, _RADIUS, _BG_DARK, _SIDEBAR_BG_*, …). Those are not used here — the CRM passes themeTokens to the gallery directly so it inherits the dashboard's design tokens. Setting them in this app has no effect.

---

6. CSP / CDN hosts

The CRM ships no Content-Security-Policy today, so the in-browser models load without any allow-listing. This section is what would need to be permitted if a CSP is ever added — miss any of these and the affected capability silently degrades (returns empty + console.warn) rather than erroring visibly, which makes it easy to misdiagnose.

Host	Needed by	Directive
https://cdn.jsdelivr.net	face-api models; tesseract worker, WASM core and language data	script-src, connect-src, worker-src
https://huggingface.co, https://cdn-lfs.huggingface.co	CLIP model weights (transformers.js)	connect-src
https://storage.googleapis.com	TensorFlow.js COCO-SSD model	connect-src
https://staticimgly.com	@imgly/background-removal WASM + assets	connect-src, worker-src

Also required by the ML runtimes themselves:

• wasm-unsafe-eval in script-src — ONNX Runtime (CLIP), tesseract-core and the @imgly remover are all WebAssembly. Without it, semantic search, OCR and in-browser background removal all fail.
• blob: in worker-src/child_src — tesseract and the WASM runtimes spawn workers from blob URLs.
• data: and blob: in img-src — canvas round-trips and generated results.

Notes:

• All three tesseract assets (worker, core, tessdata) are pinned to jsDelivr on purpose. Left at its defaults, tesseract fetches language data from tessdata.projectnaptha.com, which would be a second host to allow-list.
• tesseract.js is pinned to exactly 5.1.1 (no caret) in package.json because the worker CDN URL embeds the version — a floating range would let the worker drift out of sync with the installed main-thread code.
• The tfjs providers prefer the WebGL backend, which avoids eval and is therefore CSP-friendly; they fall back to the default backend if WebGL is unavailable.
• The SDK's own demo app runs a strict per-request nonce CSP with these hosts already allow-listed — see that repo's middleware for a working reference.