127 lines
5.3 KiB
Markdown
127 lines
5.3 KiB
Markdown
# AI setup — free & local (no paid Gemini required)
|
|
|
|
The gallery's AI splits into two groups. **Group 1 already runs for free, on-device, with no API key.**
|
|
Only **Group 2** (generative pixel editing) ever needed Gemini — and that backend is now swappable to
|
|
a free or fully-local model.
|
|
|
|
---
|
|
|
|
## Group 1 — Analysis & background removal — FREE, LOCAL, no key ✅
|
|
|
|
These run entirely in the user's browser (models download once, then cached). Nothing is sent to any
|
|
server; no key or config is required. This is the bulk of "AI" in the app:
|
|
|
|
| Capability | Runs on | Library |
|
|
|---|---|---|
|
|
| Object detection → tags, Objects browser, auto object albums | in-browser (WebGL/WASM) | TensorFlow.js COCO-SSD |
|
|
| Face detection + recognition → People clustering | in-browser | face-api.js |
|
|
| OCR (text in images) → search + Documents album | in-browser | tesseract.js |
|
|
| Semantic search / captions embeddings | in-browser | transformers.js (CLIP) |
|
|
| **Remove Background** (editor) | in-browser (WASM) | @imgly/background-removal |
|
|
|
|
So imported photos get the **same analysis as captured ones**, and Remove Background works, **without
|
|
Gemini or any key**. If you deploy with no AI keys at all, everything here still works.
|
|
|
|
---
|
|
|
|
## Group 2 — Generative pixel editing — pick a free or local backend
|
|
|
|
These edits *generate new pixels* from a prompt: **Restore, Colorize, Replace Sky, and free-form
|
|
Prompt**. Pick a backend with the `AI_EDIT_PROVIDER` env var (default `auto`). The server route
|
|
(`/api/ai/edit`) proxies to it so no key ever reaches the browser.
|
|
|
|
`auto` chooses the first configured of: **local → huggingface → gemini**.
|
|
|
|
### Option 1 — Your own local Stable Diffusion (free + private, recommended) 🏆
|
|
|
|
Run Stable Diffusion locally with any of **AUTOMATIC1111 WebUI**, **Forge**, or **SD.Next** — they all
|
|
expose the same `img2img` HTTP API. Start it with the API enabled:
|
|
|
|
```bash
|
|
# AUTOMATIC1111 example
|
|
./webui.sh --api --listen # serves http://127.0.0.1:7860
|
|
```
|
|
|
|
Then set on the app server:
|
|
|
|
```env
|
|
AI_EDIT_PROVIDER=local
|
|
LOCAL_SD_URL=http://127.0.0.1:7860
|
|
# optional tuning:
|
|
LOCAL_SD_DENOISE=0.55 # 0=keep original … 1=fully reimagine
|
|
LOCAL_SD_STEPS=25
|
|
LOCAL_SD_SAMPLER=Euler a
|
|
```
|
|
|
|
100% free, unlimited, private (nothing leaves your machine), and works for every generative op. Needs
|
|
a machine with a GPU (or a slow CPU). This is the best "own model, everything free" path.
|
|
|
|
> Deploying to Vercel? A serverless function can't reach your `localhost`. Either run the whole app on
|
|
> the same box as SD (Option B in [DEPLOY-MANUAL.md](./DEPLOY-MANUAL.md)), or expose your SD box over a
|
|
> tunnel (e.g. `cloudflared`, `ngrok`) and point `LOCAL_SD_URL` at the public URL.
|
|
|
|
### Option 2 — Hugging Face Inference API (free hosted, zero GPU)
|
|
|
|
Free token, no GPU needed. Uses an instruction image-editing model.
|
|
|
|
1. Create a free token at <https://huggingface.co/settings/tokens> (read scope is fine).
|
|
2. Set:
|
|
```env
|
|
AI_EDIT_PROVIDER=huggingface
|
|
HF_API_TOKEN=hf_xxxxxxxx
|
|
HF_IMAGE_MODEL=timbrooks/instruct-pix2pix # optional; instruction-based image edit
|
|
```
|
|
Caveats of the free tier: the first call may return **503 "model loading"** (retry in ~20s), and
|
|
there are rate limits. Great for a demo; for heavy use prefer Option 1. If a model becomes gated, pick
|
|
another image-to-image model with `HF_IMAGE_MODEL`.
|
|
|
|
### Option 3 — Google Gemini (needs a *billed* key for images)
|
|
|
|
The **free** Gemini tier only returns text — image output (`gemini-2.5-flash-image` / "Nano Banana")
|
|
requires billing enabled, which is why it fails for you. If you enable billing:
|
|
|
|
```env
|
|
AI_EDIT_PROVIDER=gemini
|
|
GEMINI_API_KEY=...
|
|
GEMINI_IMAGE_MODEL=gemini-2.5-flash-image # optional
|
|
```
|
|
|
|
### Option 4 — none
|
|
|
|
Leave all three unset (or `AI_EDIT_PROVIDER=none`). Generative edits show a friendly "not configured"
|
|
message; **all of Group 1 (analysis + Remove Background) still works**, plus every non-AI editor tool
|
|
(crop, rotate, filters, adjust, annotations, and the whole video editor).
|
|
|
|
---
|
|
|
|
## Bring your own provider (SDK-level)
|
|
|
|
The SDK takes a pluggable `AIProvider` (`ai` prop on `<PhotoGallery>`). Every method is optional:
|
|
|
|
```ts
|
|
interface AIProvider {
|
|
detectObjects?(item, image): DetectedObject[] | Promise<…>;
|
|
detectFaces?(item, image): DetectedFace[] | Promise<…>;
|
|
ocr?(item, image): string | Promise<string>;
|
|
embedImage?(item, image): number[] | Promise<number[]>;
|
|
embedText?(text): number[] | Promise<number[]>;
|
|
generativeEdit?(item, image, op): Blob | Promise<Blob>; // return the edited image
|
|
}
|
|
```
|
|
|
|
The demo's provider (`apps/web/src/lib/ai/createDemoAIProvider.ts`) wires the in-browser models for
|
|
analysis and background-removal, and routes the other generative ops to `/api/ai/edit`. To use a
|
|
different service (local Ollama+vision, ComfyUI, Replicate, your own model server, etc.), implement
|
|
`generativeEdit` (and/or the analysis methods) and pass your provider — no changes elsewhere.
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| Task | Free? | How |
|
|
|---|---|---|
|
|
| Object/face/OCR/embedding analysis | ✅ always free | in-browser, no key |
|
|
| Remove Background | ✅ always free | in-browser (@imgly) |
|
|
| Filters / crop / rotate / adjust / annotate / **video editor** | ✅ always free | no model at all |
|
|
| Restore / Colorize / Replace-Sky / Prompt edit | ✅ free via **local SD** or **HF free token** | `AI_EDIT_PROVIDER` |
|