feat: add core domain types for Photo Gallery SDK including media items, albums, and annotations
This commit is contained in:
@@ -0,0 +1,230 @@
|
||||
# Smart Gallery — AI route surface
|
||||
|
||||
The Smart Gallery embeds `@photo-gallery/sdk` into the CRM. The SDK never talks to a
|
||||
model directly: it calls a pluggable `AIProvider`, which the CRM supplies via
|
||||
`createCrmAIProvider()` in [`src/lib/gallery-ai.ts`](../src/lib/gallery-ai.ts).
|
||||
|
||||
Roughly half the intelligence runs **in the browser** for free, and the other half is
|
||||
proxied through **five server routes** under `/api/gallery/ai/*` so the RunPod API key
|
||||
never reaches the client.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where each capability runs
|
||||
|
||||
| Capability | Where | Backend | Needs a key? |
|
||||
| --- | --- | --- | --- |
|
||||
| Object detection (default) | Browser | TensorFlow.js COCO-SSD (80 COCO classes) | No |
|
||||
| Object detection (opt-in) | **Server** | RunPod YOLO → `/classify` | Yes |
|
||||
| Face detection + recognition | Browser | `@vladmandic/face-api` (128-D descriptors → People) | No |
|
||||
| OCR / document search | Browser | `tesseract.js` | No |
|
||||
| Semantic ("beach photos") search | Browser | CLIP via `@huggingface/transformers` | No |
|
||||
| Background removal (default) | Browser | `@imgly/background-removal` (WASM) | No |
|
||||
| Background removal (opt-in) | **Server** | RunPod U²-Net → `/edit` | Yes |
|
||||
| Generative edits, restore, upscale, outpaint | **Server** | `/edit` | Yes |
|
||||
| Speech-to-text | **Server** | `/transcribe` | Yes |
|
||||
| Audio denoise | **Server** | `/denoise` | Yes |
|
||||
| Camera tilt / auto-straighten | **Server** | `/tilt` | Yes |
|
||||
|
||||
Every in-browser model is loaded with `await import(...)` on first use, so none of it
|
||||
lands in the initial bundle. **Every capability degrades gracefully** — a model that
|
||||
fails to load returns `[]` / `''` / `null` with a `console.warn` rather than throwing,
|
||||
so a blocked CDN never breaks the gallery UI.
|
||||
|
||||
---
|
||||
|
||||
## 2. Auth model
|
||||
|
||||
All five routes share one gate: `requireGallerySession()` in
|
||||
[`src/lib/server/session.ts`](../src/lib/server/session.ts).
|
||||
|
||||
These routes proxy a **paid, rate-limited GPU backend** using a secret held only on the
|
||||
server. An unauthenticated route here is not just an information leak — it is an open
|
||||
invitation to spend the operator's GPU budget. (The upstream SDK demo's routes are
|
||||
completely unauthenticated; that is the single biggest thing this port fixes.)
|
||||
|
||||
**When the Shell is configured** (`NEXT_PUBLIC_SUPABASE_URL` is set): the incoming
|
||||
`cookie` header is forwarded to `${BFF_ORIGIN}/api/session/context` and only a `200`
|
||||
is accepted.
|
||||
|
||||
- `401` upstream → `401 { error: "Not signed in" }`
|
||||
- any other non-200, or a network/timeout failure → `503 { error: "Session service unavailable" }`
|
||||
(**fail closed** — if we cannot prove a session, we do not spend GPU budget)
|
||||
- **Nothing is cached.** A cached "yes" would keep a revoked session alive, so every AI
|
||||
request costs one BFF round trip. Correctness over latency for a spend gate.
|
||||
- The BFF is trusted absolutely — `BFF_ORIGIN` must only ever point at an origin the
|
||||
operator controls.
|
||||
|
||||
**When the Shell is NOT configured** (local demo mode): the request is allowed and a
|
||||
warning is logged once. **Never deploy to a public origin with the Shell unconfigured
|
||||
and a real `RUNPOD_API_KEY` present** — that combination is an open, billable endpoint.
|
||||
|
||||
This is authentication only, not authorization. Per-resource gallery policy lives in
|
||||
be-crm behind the `crm.gallery` resource.
|
||||
|
||||
---
|
||||
|
||||
## 3. Rate limits
|
||||
|
||||
[`src/lib/server/rate-limit.ts`](../src/lib/server/rate-limit.ts) — a fixed-window
|
||||
counter keyed by the authenticated principal when known, otherwise the first hop of
|
||||
`x-forwarded-for`. Each route has its own namespace, so spending your `edit` budget
|
||||
does not consume your `classify` budget. Exceeding it returns
|
||||
`429 { error: "Too many requests — slow down." }` with a `Retry-After` header.
|
||||
|
||||
| Route | Limit |
|
||||
| --- | --- |
|
||||
| `/classify` | 30 / min |
|
||||
| `/tilt` | 30 / min |
|
||||
| `/transcribe` | 20 / min |
|
||||
| `/denoise` | 20 / min |
|
||||
| `/edit` | 12 / min (most expensive) |
|
||||
|
||||
> **This limiter is per-instance and in-memory.** With N instances behind a load
|
||||
> balancer a caller gets up to N x the budget, and a restart clears all counters. It is
|
||||
> a cost guard, not a security boundary. **Replace it with Redis before running more
|
||||
> than one instance.** The `x-forwarded-for` key is also client-controlled unless a
|
||||
> trusted proxy overwrites it — the session gate, not this, is the security boundary.
|
||||
|
||||
---
|
||||
|
||||
## 4. The routes
|
||||
|
||||
All five are `POST` only, and all declare `runtime = "nodejs"`, `maxDuration = 60`,
|
||||
`dynamic = "force-dynamic"`. Failures always return `{ error: string }`.
|
||||
|
||||
Shared status codes: `400` invalid body/params · `401` not signed in · `413` payload too
|
||||
large · `429` rate limited · `500` server misconfigured (e.g. missing `RUNPOD_API_KEY`) ·
|
||||
`502` upstream failed · `503` not configured / session service unavailable ·
|
||||
`504` upstream timed out (the RunPod client's budget is 55s, under the 60s cap).
|
||||
|
||||
Upstream error bodies are truncated to a **160-character excerpt**; the RunPod key is
|
||||
never included in any response.
|
||||
|
||||
### `POST /api/gallery/ai/classify` — object detection
|
||||
Backed by `RUNPOD_YOLO_URL`. Max ~4 MB of base64. Returns boxes as **fractions 0..1**
|
||||
of the image, matching the SDK's `DetectedObject`.
|
||||
|
||||
```jsonc
|
||||
// request
|
||||
{ "imageBase64": "…", "width": 1280, "height": 853 }
|
||||
// response
|
||||
{ "objects": [ { "label": "excavator", "confidence": 0.91,
|
||||
"box": { "x": 0.12, "y": 0.30, "width": 0.25, "height": 0.40 } } ] }
|
||||
```
|
||||
|
||||
### `POST /api/gallery/ai/edit` — generative image editing
|
||||
The backend is pluggable via `AI_EDIT_PROVIDER` (`auto` | `runpod` | `local` |
|
||||
`huggingface` | `gemini` | `none`). Image + mask are budgeted **together** against the
|
||||
~4 MB cap. Prompts come from an allow-listed op set — arbitrary server-side prompts are
|
||||
never accepted, and free-text is clamped to 500 chars.
|
||||
|
||||
```jsonc
|
||||
// request
|
||||
{ "imageBase64": "…", "mimeType": "image/jpeg",
|
||||
"op": { "type": "restore" }, // or prompt | colorize | replace-sky |
|
||||
// magic-eraser | generative-fill | upscale |
|
||||
// remove-background
|
||||
"maskBase64": "…", // required for magic-eraser / generative-fill
|
||||
"params": { "strength": 0.8 } }
|
||||
// response
|
||||
{ "imageBase64": "…", "mimeType": "image/png" }
|
||||
```
|
||||
|
||||
Op → RunPod endpoint: `restore`/`upscale` → `RUNPOD_UPSCALE_URL` ·
|
||||
`prompt`/`colorize` → `RUNPOD_SD_IMG2IMG_URL` · `replace-sky`/`magic-eraser`/
|
||||
`generative-fill` → `RUNPOD_SD_INPAINT_URL` · `remove-background` →
|
||||
`RUNPOD_BG_REMOVE_URL`. `upscale`, `magic-eraser` and `generative-fill` are RunPod-only
|
||||
and return `400` under another backend. Outpaint is client-side padding plus a
|
||||
`generative-fill` call — it needs no separate route.
|
||||
|
||||
### `POST /api/gallery/ai/tilt` — camera tilt
|
||||
Backed by `RUNPOD_TILT_URL`. Max ~4 MB (`413` over). Only reachable when
|
||||
`NEXT_PUBLIC_APG_RUNPOD_TILT=true`, which also reveals the editor's Auto-straighten button.
|
||||
|
||||
```jsonc
|
||||
{ "image": "…" } → { "rollDegrees": -2.4, "pitchDegrees": 1.1, "fovDegrees": 68.2 }
|
||||
```
|
||||
|
||||
### `POST /api/gallery/ai/transcribe` — speech to text
|
||||
Backed by `RUNPOD_STT_URL`. Expects base64 **WAV 16 kHz mono PCM16**. Max ~8 MB (`413`).
|
||||
|
||||
```jsonc
|
||||
{ "audio": "…", "language": "en" }
|
||||
→ { "transcript": "…", "segments": [ { "text": "…", "startSec": 0, "endSec": 1.8 } ] }
|
||||
```
|
||||
|
||||
### `POST /api/gallery/ai/denoise` — audio noise removal
|
||||
Backed by `RUNPOD_AUDIO_DENOISE_URL`. Expects base64 **WAV 48 kHz mono PCM16**. Max
|
||||
~12 MB (`413`). Used before transcription on noisy sites.
|
||||
|
||||
```jsonc
|
||||
{ "audio": "…" } → { "audio": "…" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Environment variables
|
||||
|
||||
See [`.env.local.example`](../.env.local.example) for the fully commented template.
|
||||
|
||||
| Var | Backs |
|
||||
| --- | --- |
|
||||
| `RUNPOD_API_KEY` | all five routes (server-only — never `NEXT_PUBLIC_`) |
|
||||
| `RUNPOD_YOLO_URL` | `/classify` |
|
||||
| `RUNPOD_SD_IMG2IMG_URL` | `/edit` — prompt, colorize, maskless replace-sky |
|
||||
| `RUNPOD_SD_INPAINT_URL` | `/edit` — magic-eraser, generative-fill, outpaint, masked replace-sky |
|
||||
| `RUNPOD_UPSCALE_URL` | `/edit` — restore, upscale |
|
||||
| `RUNPOD_BG_REMOVE_URL` | `/edit` — remove-background |
|
||||
| `RUNPOD_STT_URL` | `/transcribe` |
|
||||
| `RUNPOD_AUDIO_DENOISE_URL` | `/denoise` |
|
||||
| `RUNPOD_TILT_URL` | `/tilt` |
|
||||
| `AI_EDIT_PROVIDER` | which backend `/edit` uses |
|
||||
| `BFF_ORIGIN` | the session gate |
|
||||
|
||||
Client switches (`NEXT_PUBLIC_`, **inlined at build time** — a change needs a rebuild,
|
||||
and each must be the literal string `"true"`):
|
||||
`NEXT_PUBLIC_APG_RUNPOD_DETECT`, `NEXT_PUBLIC_APG_RUNPOD_BG`, `NEXT_PUBLIC_APG_RUNPOD_TILT`.
|
||||
|
||||
> The SDK's standalone demo also reads a family of `NEXT_PUBLIC_APG_*` **theming** vars
|
||||
> (`_THEME`, `_ACCENT`, `_RADIUS`, `_BG_DARK`, `_SIDEBAR_BG_*`, …). Those are **not used
|
||||
> here** — the CRM passes `themeTokens` to the gallery directly so it inherits the
|
||||
> dashboard's design tokens. Setting them in this app has no effect.
|
||||
|
||||
---
|
||||
|
||||
## 6. CSP / CDN hosts
|
||||
|
||||
**The CRM ships no Content-Security-Policy today**, so the in-browser models load
|
||||
without any allow-listing. This section is what would need to be permitted **if a CSP
|
||||
is ever added** — miss any of these and the affected capability silently degrades
|
||||
(returns empty + `console.warn`) rather than erroring visibly, which makes it easy to
|
||||
misdiagnose.
|
||||
|
||||
| Host | Needed by | Directive |
|
||||
| --- | --- | --- |
|
||||
| `https://cdn.jsdelivr.net` | face-api models; tesseract worker, WASM core **and** language data | `script-src`, `connect-src`, `worker-src` |
|
||||
| `https://huggingface.co`, `https://cdn-lfs.huggingface.co` | CLIP model weights (transformers.js) | `connect-src` |
|
||||
| `https://storage.googleapis.com` | TensorFlow.js COCO-SSD model | `connect-src` |
|
||||
| `https://staticimgly.com` | `@imgly/background-removal` WASM + assets | `connect-src`, `worker-src` |
|
||||
|
||||
Also required by the ML runtimes themselves:
|
||||
|
||||
- **`wasm-unsafe-eval` in `script-src`** — ONNX Runtime (CLIP), tesseract-core and the
|
||||
`@imgly` remover are all WebAssembly. Without it, semantic search, OCR and in-browser
|
||||
background removal all fail.
|
||||
- **`blob:` in `worker-src`/`child_src`** — tesseract and the WASM runtimes spawn
|
||||
workers from blob URLs.
|
||||
- **`data:` and `blob:` in `img-src`** — canvas round-trips and generated results.
|
||||
|
||||
Notes:
|
||||
- All three tesseract assets (worker, core, tessdata) are pinned to **jsDelivr** on
|
||||
purpose. Left at its defaults, tesseract fetches language data from
|
||||
`tessdata.projectnaptha.com`, which would be a second host to allow-list.
|
||||
- `tesseract.js` is pinned to **exactly `5.1.1`** (no caret) in `package.json` because
|
||||
the worker CDN URL embeds the version — a floating range would let the worker drift
|
||||
out of sync with the installed main-thread code.
|
||||
- The tfjs providers prefer the **WebGL** backend, which avoids `eval` and is therefore
|
||||
CSP-friendly; they fall back to the default backend if WebGL is unavailable.
|
||||
- The SDK's own demo app runs a strict per-request nonce CSP with these hosts already
|
||||
allow-listed — see that repo's middleware for a working reference.
|
||||
Reference in New Issue
Block a user