lynkeduppro-crm/docs/SMART_GALLERY.md

# Smart Gallery — AI route surface

The Smart Gallery embeds `@photo-gallery/sdk` into the CRM. The SDK never talks to a
model directly: it calls a pluggable `AIProvider`, which the CRM supplies via
`createCrmAIProvider()` in [`src/lib/gallery-ai.ts`](../src/lib/gallery-ai.ts).

Roughly half the intelligence runs **in the browser** for free, and the other half is
proxied through **five server routes** under `/api/gallery/ai/*` so the RunPod API key
never reaches the client.

---

## 1. Where each capability runs

| Capability | Where | Backend | Needs a key? |
| --- | --- | --- | --- |
| Object detection (default) | Browser | TensorFlow.js COCO-SSD (80 COCO classes) | No |
| Object detection (opt-in) | **Server** | RunPod YOLO → `/classify` | Yes |
| Face detection + recognition | Browser | `@vladmandic/face-api` (128-D descriptors → People) | No |
| OCR / document search | Browser | `tesseract.js` | No |
| Semantic ("beach photos") search | Browser | CLIP via `@huggingface/transformers` | No |
| Background removal (default) | Browser | `@imgly/background-removal` (WASM) | No |
| Background removal (opt-in) | **Server** | RunPod U²-Net → `/edit` | Yes |
| Generative edits, restore, upscale, outpaint | **Server** | `/edit` | Yes |
| Speech-to-text | **Server** | `/transcribe` | Yes |
| Audio denoise | **Server** | `/denoise` | Yes |
| Camera tilt / auto-straighten | **Server** | `/tilt` | Yes |

Every in-browser model is loaded with `await import(...)` on first use, so none of it
lands in the initial bundle. **Every capability degrades gracefully** — a model that
fails to load returns `[]` / `''` / `null` with a `console.warn` rather than throwing,
so a blocked CDN never breaks the gallery UI.

---

## 2. Auth model

All five routes share one gate: `requireGallerySession()` in
[`src/lib/server/session.ts`](../src/lib/server/session.ts).

These routes proxy a **paid, rate-limited GPU backend** using a secret held only on the
server. An unauthenticated route here is not just an information leak — it is an open
invitation to spend the operator's GPU budget. (The upstream SDK demo's routes are
completely unauthenticated; that is the single biggest thing this port fixes.)

**When the Shell is configured** (`NEXT_PUBLIC_SUPABASE_URL` is set): the incoming
`cookie` header is forwarded to `${BFF_ORIGIN}/api/session/context` and only a `200`
is accepted.

- `401` upstream → `401 { error: "Not signed in" }`
- any other non-200, or a network/timeout failure → `503 { error: "Session service unavailable" }`
  (**fail closed** — if we cannot prove a session, we do not spend GPU budget)
- **Nothing is cached.** A cached "yes" would keep a revoked session alive, so every AI
  request costs one BFF round trip. Correctness over latency for a spend gate.
- The BFF is trusted absolutely — `BFF_ORIGIN` must only ever point at an origin the
  operator controls.

**When the Shell is NOT configured** (local demo mode): the request is allowed and a
warning is logged once. **Never deploy to a public origin with the Shell unconfigured
and a real `RUNPOD_API_KEY` present** — that combination is an open, billable endpoint.

This is authentication only, not authorization. Per-resource gallery policy lives in
be-crm behind the `crm.gallery` resource.

---

## 3. Rate limits

[`src/lib/server/rate-limit.ts`](../src/lib/server/rate-limit.ts) — a fixed-window
counter keyed by the authenticated principal when known, otherwise the first hop of
`x-forwarded-for`. Each route has its own namespace, so spending your `edit` budget
does not consume your `classify` budget. Exceeding it returns
`429 { error: "Too many requests — slow down." }` with a `Retry-After` header.

| Route | Limit |
| --- | --- |
| `/classify` | 30 / min |
| `/tilt` | 30 / min |
| `/transcribe` | 20 / min |
| `/denoise` | 20 / min |
| `/edit` | 12 / min (most expensive) |

> **This limiter is per-instance and in-memory.** With N instances behind a load
> balancer a caller gets up to N x the budget, and a restart clears all counters. It is
> a cost guard, not a security boundary. **Replace it with Redis before running more
> than one instance.** The `x-forwarded-for` key is also client-controlled unless a
> trusted proxy overwrites it — the session gate, not this, is the security boundary.

---

## 4. The routes

All five are `POST` only, and all declare `runtime = "nodejs"`, `maxDuration = 60`,
`dynamic = "force-dynamic"`. Failures always return `{ error: string }`.

Shared status codes: `400` invalid body/params · `401` not signed in · `413` payload too
large · `429` rate limited · `500` server misconfigured (e.g. missing `RUNPOD_API_KEY`) ·
`502` upstream failed · `503` not configured / session service unavailable ·
`504` upstream timed out (the RunPod client's budget is 55s, under the 60s cap).

Upstream error bodies are truncated to a **160-character excerpt**; the RunPod key is
never included in any response.

### `POST /api/gallery/ai/classify` — object detection
Backed by `RUNPOD_YOLO_URL`. Max ~4 MB of base64. Returns boxes as **fractions 0..1**
of the image, matching the SDK's `DetectedObject`.

```jsonc
// request
{ "imageBase64": "…", "width": 1280, "height": 853 }
// response
{ "objects": [ { "label": "excavator", "confidence": 0.91,
                 "box": { "x": 0.12, "y": 0.30, "width": 0.25, "height": 0.40 } } ] }
```

### `POST /api/gallery/ai/edit` — generative image editing
The backend is pluggable via `AI_EDIT_PROVIDER` (`auto` | `runpod` | `local` |
`huggingface` | `gemini` | `none`). Image + mask are budgeted **together** against the
~4 MB cap. Prompts come from an allow-listed op set — arbitrary server-side prompts are
never accepted, and free-text is clamped to 500 chars.

```jsonc
// request
{ "imageBase64": "…", "mimeType": "image/jpeg",
  "op": { "type": "restore" },            // or prompt | colorize | replace-sky |
                                          // magic-eraser | generative-fill | upscale |
                                          // remove-background
  "maskBase64": "…",                      // required for magic-eraser / generative-fill
  "params": { "strength": 0.8 } }
// response
{ "imageBase64": "…", "mimeType": "image/png" }
```

Op → RunPod endpoint: `restore`/`upscale` → `RUNPOD_UPSCALE_URL` ·
`prompt`/`colorize` → `RUNPOD_SD_IMG2IMG_URL` · `replace-sky`/`magic-eraser`/
`generative-fill` → `RUNPOD_SD_INPAINT_URL` · `remove-background` →
`RUNPOD_BG_REMOVE_URL`. `upscale`, `magic-eraser` and `generative-fill` are RunPod-only
and return `400` under another backend. Outpaint is client-side padding plus a
`generative-fill` call — it needs no separate route.

### `POST /api/gallery/ai/tilt` — camera tilt
Backed by `RUNPOD_TILT_URL`. Max ~4 MB (`413` over). Only reachable when
`NEXT_PUBLIC_APG_RUNPOD_TILT=true`, which also reveals the editor's Auto-straighten button.

```jsonc
{ "image": "…" }  →  { "rollDegrees": -2.4, "pitchDegrees": 1.1, "fovDegrees": 68.2 }
```

### `POST /api/gallery/ai/transcribe` — speech to text
Backed by `RUNPOD_STT_URL`. Expects base64 **WAV 16 kHz mono PCM16**. Max ~8 MB (`413`).

```jsonc
{ "audio": "…", "language": "en" }
→ { "transcript": "…", "segments": [ { "text": "…", "startSec": 0, "endSec": 1.8 } ] }
```

### `POST /api/gallery/ai/denoise` — audio noise removal
Backed by `RUNPOD_AUDIO_DENOISE_URL`. Expects base64 **WAV 48 kHz mono PCM16**. Max
~12 MB (`413`). Used before transcription on noisy sites.

```jsonc
{ "audio": "…" }  →  { "audio": "…" }
```

---

## 5. Environment variables

See [`.env.local.example`](../.env.local.example) for the fully commented template.

| Var | Backs |
| --- | --- |
| `RUNPOD_API_KEY` | all five routes (server-only — never `NEXT_PUBLIC_`) |
| `RUNPOD_YOLO_URL` | `/classify` |
| `RUNPOD_SD_IMG2IMG_URL` | `/edit` — prompt, colorize, maskless replace-sky |
| `RUNPOD_SD_INPAINT_URL` | `/edit` — magic-eraser, generative-fill, outpaint, masked replace-sky |
| `RUNPOD_UPSCALE_URL` | `/edit` — restore, upscale |
| `RUNPOD_BG_REMOVE_URL` | `/edit` — remove-background |
| `RUNPOD_STT_URL` | `/transcribe` |
| `RUNPOD_AUDIO_DENOISE_URL` | `/denoise` |
| `RUNPOD_TILT_URL` | `/tilt` |
| `AI_EDIT_PROVIDER` | which backend `/edit` uses |
| `BFF_ORIGIN` | the session gate |

Client switches (`NEXT_PUBLIC_`, **inlined at build time** — a change needs a rebuild,
and each must be the literal string `"true"`):
`NEXT_PUBLIC_APG_RUNPOD_DETECT`, `NEXT_PUBLIC_APG_RUNPOD_BG`, `NEXT_PUBLIC_APG_RUNPOD_TILT`.

> The SDK's standalone demo also reads a family of `NEXT_PUBLIC_APG_*` **theming** vars
> (`_THEME`, `_ACCENT`, `_RADIUS`, `_BG_DARK`, `_SIDEBAR_BG_*`, …). Those are **not used
> here** — the CRM passes `themeTokens` to the gallery directly so it inherits the
> dashboard's design tokens. Setting them in this app has no effect.

---

## 6. CSP / CDN hosts

**The CRM ships no Content-Security-Policy today**, so the in-browser models load
without any allow-listing. This section is what would need to be permitted **if a CSP
is ever added** — miss any of these and the affected capability silently degrades
(returns empty + `console.warn`) rather than erroring visibly, which makes it easy to
misdiagnose.

| Host | Needed by | Directive |
| --- | --- | --- |
| `https://cdn.jsdelivr.net` | face-api models; tesseract worker, WASM core **and** language data | `script-src`, `connect-src`, `worker-src` |
| `https://huggingface.co`, `https://cdn-lfs.huggingface.co` | CLIP model weights (transformers.js) | `connect-src` |
| `https://storage.googleapis.com` | TensorFlow.js COCO-SSD model | `connect-src` |
| `https://staticimgly.com` | `@imgly/background-removal` WASM + assets | `connect-src`, `worker-src` |

Also required by the ML runtimes themselves:

- **`wasm-unsafe-eval` in `script-src`** — ONNX Runtime (CLIP), tesseract-core and the
  `@imgly` remover are all WebAssembly. Without it, semantic search, OCR and in-browser
  background removal all fail.
- **`blob:` in `worker-src`/`child_src`** — tesseract and the WASM runtimes spawn
  workers from blob URLs.
- **`data:` and `blob:` in `img-src`** — canvas round-trips and generated results.

Notes:
- All three tesseract assets (worker, core, tessdata) are pinned to **jsDelivr** on
  purpose. Left at its defaults, tesseract fetches language data from
  `tessdata.projectnaptha.com`, which would be a second host to allow-list.
- `tesseract.js` is pinned to **exactly `5.1.1`** (no caret) in `package.json` because
  the worker CDN URL embeds the version — a floating range would let the worker drift
  out of sync with the installed main-thread code.
- The tfjs providers prefer the **WebGL** backend, which avoids `eval` and is therefore
  CSP-friendly; they fall back to the default backend if WebGL is unavailable.
- The SDK's own demo app runs a strict per-request nonce CSP with these hosts already
  allow-listed — see that repo's middleware for a working reference.