Adds OTel auto-instrumentation (http, express, prisma/pg, redis, ...) exporting
OTLP to Tempo, so iios-service requests appear as distributed traces in Grafana.
observability/tracing.ts is imported FIRST in main.ts — auto-instrumentation
patches modules as they are require()d, so anything loaded earlier is never
traced. Env-driven (OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_ENDPOINT); with the
endpoint unset the SDK never starts, so dev and tests carry zero overhead.
Health/metrics probes are excluded — they would otherwise swamp Tempo and bury
real request traces.
The existing P9 trace middleware now adopts the ACTIVE OTel trace id, so an
http.request log line and its distributed trace share one id (Loki -> Tempo
pivot in Grafana), falling back to x-trace-id / traceparent / generated.
pnpm-lock.yaml regenerated — CI and the Dockerfile both use
'pnpm install --frozen-lockfile', which would hard-fail on an unchanged lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dind-builder runner is host-mode/persistent, so /tmp/kp from a prior run
survives and 'git clone ... /tmp/kp' fails with 'destination path already exists
and is not an empty directory'. Clear it first.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generic notification engine: reacts to message.sent and runs three gates before
delivering — policy (DM always; group only on @mention or reply-to-you), presence
(skip if the recipient is focused on that thread), mute (per-thread). Delivery via a
swappable NotificationPort (Web Push/VAPID adapter); a 'gone' (404/410) prunes the sub.
- PresenceService + gateway `focus_thread` signal + disconnect cleanup (room membership
!= viewing, since the sidebar joins every thread room).
- IiosNotificationSubscription table + `muted` on IiosThreadParticipant (migration).
- Endpoints: GET vapid-public-key, POST/DELETE subscribe, POST threads/:id/mute|unmute;
listThreads returns the caller's `muted`.
- DM-vs-group read from the opaque `membership` attribute in the notification POLICY only
— kernel stays generic (grep-verified).
Tests: 13 new (3 gates + reply-to-you + prune + web-push adapter states); full suite 205
green. Verified live: module boots, subscribe stored, mute/unmute round-trips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
API & SDK guide:
- §5.3 Threads & Messaging fully filled in: GET/POST /v1/threads, participants,
my-annotations; the Message shape (senderId, attachment, annotations); the full
socket event set (open_thread/send_message/add_participant/annotate/read/typing +
server message/receipt/typing/annotation); reactions/pins/saves as the generic
interaction-annotation primitive; mentions → MENTION inbox item.
- §5.4 inbox MENTION kind; SDK reactions/pins/saves + listThreads/createThread;
vocab adds MENTION, message-part kinds, attachment kind, annotation types.
DEPLOYMENT:
- Real-IdP auth env (SUPABASE_URL/AUTH_ISSUERS via JWKS, no secret) + MEDIA_SECRET.
- Media StoragePort in topology: dev local disk → prod object storage; ⚠ local disk is
single-instance/non-durable; bytes flow client↔storage direct (scales independently).
CEO overview: refreshed counts (192 tests, 53 tables / 19 migrations) and noted P9 has
begun — real Supabase auth (first stubbed port turned real) + generic reactions/pins/
saves/mentions/media on the kernel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- New §5.10 Media: presign-upload/download + upload/blob endpoints, attachment shape,
governance notes (25MB + type allowlist via OPA, signed tokens, tenant fence), and a
sequence diagram of the presign→upload→send→signed-download flow with the OPA gate.
- SDK: uploadMedia/mediaUrl; send() attachment/mentions.
- Auth §3: real OIDC (Supabase/AUTH_ISSUERS) JWKS verification alongside dev HS256.
- Env: MEDIA_DIR/MEDIA_SECRET/PUBLIC_URL/SUPABASE_URL/AUTH_ISSUERS; test count 192.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Media the industry way: the DB stores a reference, bytes live behind a storage port.
- StoragePort + LocalDiskStorage (dev). MediaService presigns short-lived signed
upload/download URLs (HS256 tokens) pointing at IIOS's own endpoints; the bytes
never touch the kernel. Prod swaps STORAGE_PORT to S3/Supabase — same as auth.
- MediaController: presign-upload / PUT upload/:token (raw stream) / GET blob/:token /
presign-download. Uploads are OPA-governed (iios.media.upload: 25 MB cap +
image/video/audio/pdf/office allowlist); downloads are tenant-fenced by object key.
- send() + MessageDto carry an attachment (contentRef/mimeType/sizeBytes → generic
MEDIA_REF/VOICE_REF/FILE_REF part; DTO exposes kind image|video|audio|file).
Tests: media.service.spec (6) — round-trip, oversize/type denied, oversized PUT
refused, tampered token rejected, tenant fence. Full suite 192 green. Verified live:
presign→upload→send→history→signed download round-trips the exact bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The app needs a reliable "is this message mine?" signal. senderActorId worked only
with a lazily-learned actorId, which Supabase token-refresh wipes. senderId is the
sender's externalId (email/username) — always present and comparable to the session
user — so message alignment is deterministic. Verified: senderId round-trips = email.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SessionVerifier now holds a REGISTRY of trusted OIDC issuers instead of a single
Supabase project. A token is routed by its `iss` claim to that issuer's entry,
verified against that issuer's JWKS (ES256, no secret), and stamped with that
entry's appId/orgId — so two Supabase projects / IdPs map to two isolated app
scopes on one IIOS (chat vs a future support app). App A's tokens can't reach B.
- Config: `AUTH_ISSUERS` (JSON array of { url, appId, orgId? }); the single
`SUPABASE_URL` (+ SUPABASE_APP_ID) still works as a one-entry shorthand.
- Per-issuer JWKS cache; untrusted issuer → reject; forgery (right issuer claim,
wrong key) → reject.
- HS256 app-token path (dev/tests) unchanged.
Tests: new session.verifier.spec (4) — routes to correct appId, second issuer →
different appId, untrusted issuer rejected, cross-issuer forgery rejected.
Note: outbox `replay.spec` has a pre-existing clock-skew flake (nextAttemptAt vs
Postgres now()) that surfaces under certain suite orderings — unrelated to this
change (fails in isolation on a clean tree; passes when another spec runs first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SessionVerifier gains a Supabase mode (set SUPABASE_URL): user SATs are ES256,
verified against the project's public JWKS — no shared secret. Keys are fetched at
startup (onModuleInit) and cached as PEM so verify() stays synchronous; a rotated
kid triggers a background refresh. Claims map to MessagePrincipal with userId=email
(stable, human-readable → mentions/directory keep working; RealMDM canonicalises
later). The legacy HS256 app-token path is unchanged (dev/tests untouched).
senderName now prefers the actor display name over the raw handle, so real names
show on bubbles when identity is an email.
Verified end-to-end against a live project: signup → real ES256 SAT → GET /v1/threads
200 → thread created + owned by the resolved email principal. Full suite 182 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mentions ride the existing event-driven inbox — no chat parsing in the kernel:
- send() carries an OPAQUE mentions[] (userIds) into the message event; the kernel
never parses "@". The app supplies the notify-list.
- InboxProjector fans out a MENTION inbox item (new generic inbox kind) to each
mentioned *participant* (never the sender), idempotent per source message; reading
the thread resolves the reader's MENTION + NEEDS_REPLY items to DONE.
- New MENTION value in the generic IiosInboxItemKind taxonomy (migration).
Pins/saves reuse the annotation primitive; new generic query powers a Saved list:
- MessageService.listMyAnnotated(principal, type) + GET /v1/threads/my-annotations
returns the caller's annotated messages (type "save") with thread context.
Tests: 182 pass (+4: mention fan-out, resolve-on-read, saved query, opaque mentions).
Verified live: mention→inbox delivery, read→resolve, pin/save persistence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds IiosInteractionAnnotation — an actor attaches an OPAQUE (annotationType, value)
label to an interaction. The kernel stores + aggregates them but never interprets the
strings (the chat app writes type "reaction" / value = emoji); the same primitive backs
pins/saves/flags/tags later. No chat vocabulary in kernel code — verified by grep.
- MessageService.toggleAnnotation(): governed (participant-only via new OPA rule
iios.interaction.annotate), idempotent toggle keyed on
(scope, interaction, actor, type, value); history DTO carries aggregated
{ type, value, users[] } groups.
- Gateway: `annotate` event → broadcasts `annotation` (refreshed user list) to the room.
- DevOpaPort: annotate allowed only for thread members (real OPA can tighten later).
- Migration add_interaction_annotations (additive; dev data untouched).
Tests: 178 pass (+3: toggle/aggregate, coexisting values, non-member denied).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- openThread/createThread accept a generic `subject` (group name) — stored on the
thread, echoed via listThreads; kernel never interprets it.
- Gateway open_thread now returns an ACK'd { error } on failure instead of throwing
(which never acked → clients hung on "loading"). Non-member/missing-thread opens
fail cleanly.
- Tests run against an isolated `iios_test` database (vitest globalSetup creates +
migrates it; test.env overrides DATABASE_URL) so `pnpm test` can never TRUNCATE the
dev database again. Verified: full suite green, dev DB row counts unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Messages now carry senderName (resolved actorId → sourceHandle.externalId) so
clients render usernames instead of raw actor cuids.
- GET /v1/dev/users exposes the dev directory (DEV_USERS keys) so a host app can
validate add-by-username; real apps validate against their user store / MDM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /v1/threads now returns participants[] (member usernames) so a host app can
render conversation titles / member lists without a second call. Still generic —
it only lists members of threads the caller belongs to.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enriches the platform PLANE, not the kernel:
- DevOpaPort: the dev OPA stub now evaluates a small policy table (behind the
same opa.decide) — DM capped at 2, add-participant requires member/group-admin,
governed self-join for membership threads. Real OPA swaps in unchanged.
- Dev IdP login (POST /v1/dev/login) issues the same JWT claims a real IdP would.
- PolicyDeniedFilter maps fail-closed denials to HTTP 403.
Generic kernel additions (no chat vocabulary — 'dm'/'group' live only as OPA
policy + an opaque thread attribute):
- MessageService.addParticipant (governed membership by userId), governed
openThread self-join (scoped to threads with a membership attribute),
parentInteractionId on send (reply link), and a generic listThreads.
- REST: GET /v1/threads, POST /v1/threads, POST /v1/threads/:id/participants;
socket add_participant + membership/parentInteractionId. ensureParticipant
gains a role.
Tests: dev-opa.port.spec + message.spec (DM cap / group admin / governed join /
listThreads / reply). smoke-membership.mjs; realtime smokes updated for governed
join. 175 unit tests + all smokes green; kernel free of dm/group literals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The declarative services: Postgres was never reachable at postgres:5432 (P1001)
on this Gitea runner even with a valid healthcheck — service-name networking to
the job isn't wired here. Switch to the mdm CI pattern: host-mode runner, start
pgvector via 'docker run -p 5434:5432', connect over localhost:5434 (also the
specs' default DATABASE_URL). Clean up the container in always().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A quoted '--health-cmd "pg_isready -U iios"' in the folded options scalar
splits on spaces, so docker received an invalid healthcheck command; the
service never went healthy and the job container couldn't resolve postgres:5432
(P1001). Match go-monorepo's proven unquoted '--health-cmd pg_isready'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The integration *.spec.ts suites need a live Postgres (schema applied) at
DATABASE_URL. Run on the dind-builder runner in a job container with a
pgvector/pgvector:pg16 service (proven pattern from go-monorepo CI), point
DATABASE_URL at the service, and 'prisma migrate deploy' before pnpm test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pnpm -r typecheck ran before pnpm -r build, but the @insignia/* packages
publish their types only via built dist/*.d.ts. So consumers failed with
TS2307 'Cannot find module @insignia/iios-contracts' before any dist existed.
Also generate the iios-service Prisma client so its schema-derived types
resolve. Build now precedes typecheck; prisma:generate precedes both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds adapter-owned externalEventId extraction to the ChannelAdapter contract
(WebhookAdapter→eventId, EmailAdapter→Message-ID) and uses it in InboundService,
so email replays dedup on Message-ID (previously only payload.eventId worked).
Adds an email-dedup spec + smoke-email.mjs proving inbound → normalized, a reply
joins the same email thread, bad signature rejected, and outbound EMAIL SENT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AdapterRegistry now backs the EMAIL channel with the email-native EmailAdapter
(RFC threading) instead of the generic WebhookAdapter. Adds EmailProvider — an
email-envelope egress provider ({to,subject,text,html,inReplyTo}) registered for
EMAIL when IIOS_PROVIDER_URL_EMAIL is set (sandbox stays the default). Spec:
signed email → normalized interaction on the email thread; a reply joins the
same thread (one conversation); bad signature → rejected (fail-closed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a canonical EmailInboundPayload + EmailAdapter implementing the
ChannelAdapter contract: HMAC signature verify + normalize with email-native
threading (whole reply chain → one IIOS thread via references/inReplyTo root),
messageId as providerEventId for dedup, html→text fallback, Re:/Fwd: stripping.
Plus emailInboundFixture/emailReplyFixture and unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an opt-in RedisIoAdapter (wired in main.ts when REDIS_URL is set) so
socket.io room emits fan out across instances via Redis pub/sub — a client on
replica B now receives messages emitted by replica A. Without REDIS_URL the
in-memory adapter is kept (single-instance dev unchanged). Adds redis to
docker-compose, REDIS_URL to the env contract, and smoke-realtime-cluster.mjs
which proves cross-instance delivery fails without Redis and passes with it.
Delivery is at-least-once at N>1; clients dedupe by message id (documented).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a multi-stage Dockerfile for iios-service (build → pnpm deploy prune →
slim non-root runtime), a docker-entrypoint that runs `prisma migrate deploy`
then starts the server as PID 1, a .dockerignore, a fully-commented
.env.example config contract, and docs/DEPLOYMENT.md (topology, scaling,
release strategy, prod-readiness gaps). Moves prisma to dependencies so the
CLI ships in the prod bundle. Image builds, migrates, and serves /health 200.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Establishes a docs/adr convention and records the naming-migration decision:
interaction-centric Iios-prefixed vocabulary as canonical, support as a
namespaced specialization (KG-16), @insignia/iios-* package scope, and a
greenfield-and-deprecate (strangler) migration off the legacy support-service.
Includes the legacy→IIOS naming map.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OutboundService.send is now opt-in idempotent: an explicit idempotencyKey routes
the send through IdempotencyService.run (Slice 6), so reusing a key with a
different target/payload returns a 409 conflict and a same-key/same-request retry
replays the cached command. The inner findUnique early-return stays as a backstop
so a FAILED-then-retried send never collides on the command's unique key. Keyless
sends keep the old behavior (no ledger row).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /v1/dev/retention/sweep runs the sweep on demand; /metrics gains a global
retention section (snapshot counts by status). smoke-retention.mjs (0-day
windows) proves an aged interaction is redacted in place and surfaces as a
REDACTED snapshot on /metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds IiosRetentionPolicySnapshot (per-interaction frozen lifecycle timestamps)
+ IiosInteraction.dataClass, and a RetentionService whose sweep captures
snapshots then archives (status change) and, past delete_after, redacts content
in place (reusing the Slice-8 tombstone) — never a hard delete. An active
compliance hold blocks both; every action is audited and tenant-scopeable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tenant-scoped /v1/dsr: POST /erase (self-service right-to-erasure), POST/GET
/holds + POST /holds/:id/release. smoke-dsr.mjs proves PII present → hold
blocks erase (409) → release → erase redacts the ticket subject in place →
re-erase idempotent → cross-tenant cannot see the hold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds IiosComplianceHold + IiosSourceHandle.redactedAt and a DsrService whose
eraseSubject redacts the caller's PII in place (message bodies, identity
display, raw payloads, support/inbox/transcript text) while preserving
externalId (source preservation), structure, and the audit trail. An active
compliance hold blocks erasure until released; every op is audited and
tenant-fenced. Erasure is idempotent (KG-03).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a global `projections` section to GET /metrics exposing each cursor's
position (last_event_id/last_offset), determinism checksum, and lagMs.
smoke-projection.mjs injects events and asserts the projection's ordered
high-water-mark advances with a 64-hex rolling checksum (KG-06).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All 5 projectors (route/ai/inbox/calendar/raw-event) now advance their
IiosProjectionCursor after a freshly-claimed event is applied — split into
claim → apply → advance so a duplicate never double-counts and a throw leaves
the cursor untouched (event replays via the DLQ). Inbox's two topics get two
cursor rows. Proves KG-06 replay reproduction at the projection level.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the doc-mandated IiosProjectionCursor (ordered high-water-mark +
rolling checksum per projection/topic/partition) and a reusable
ProjectionCursorService.advance(). The checksum is a deterministic fold over
the ordered event stream, so replaying the same events reproduces it — the
KG-06 proof that replay reproduces projection outcomes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
smoke-idempotency.mjs: dup POST /v1/support/tickets with the same
Idempotency-Key replays the same ticket, same key + different body → 409,
and a second tenant reusing the key gets its own ticket (isolation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SupportService.createTicket + requestCallback now accept an optional
idempotencyKey (from the Idempotency-Key header). With a key, a same-body
retry replays the same row and a different-body reuse is a 409; without one,
they run unwrapped as before. Tenant-scoped via the command ledger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the doc-mandated IiosIdempotencyCommand table + a reusable
IdempotencyService.run() that writes IN_FLIGHT at command start, replays the
cached response on a same-key/same-body retry, rejects a same-key/different-body
reuse (409), and retries a FAILED command. Tenant-scoped via
@@unique([scopeId, commandName, idempotencyKey]) (KG-02/KG-06).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Projectors register a replay handler and route bus errors to
dlq.onConsumerFailure (dead-letter + clear the claim-then-fail claim)
instead of swallowing them. Relay dead-letters an outbox event after
IIOS_OUTBOX_MAX_ATTEMPTS instead of retrying forever. Chaos test proves
poison → DLQ → replay resolves while sibling events keep flowing (KG-13/KG-06).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task O.4: GET /metrics (session-auth) returns the caller tenant's counters
(interactions by kind/status, outbound by status, AI jobs + cost, tickets,
meetings, inbox, audit-link count) via findScope + Prisma groupBy — no cross-tenant
totals — plus a global relay-lag section. smoke-observability proves the mandated
gate end to end: x-trace-id response header, event correlationId == request trace,
audit link in DB, /metrics counters. Slice-4 verification: 122 tests green, builds,
boundary clean, observability smoke + all prior smokes pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task O.3: IiosAuditLink (traceId/correlationId/scopeId/actorRefId/action/
resourceType/resourceId) + recordAudit() helper (reads the current trace from ALS,
best-effort). Written at the decision/mutation points — route approve/deny, AI
accept/reject, meeting summarize, outbound send — so the trace appears in DB and a
decision can be replayed / traced across services (mandated iios_audit_link).
Migration audit-link; resetDb truncates it. 2 tests bind approve/accept to a trace.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task O.2: ai/calendar/route/ingest emit sites now stamp traceparent + insignia.
correlationId from interaction.traceId ?? currentTraceId() ?? generated (mirroring
message.send; inbox/inbound already carried correlationId). toTraceparent is hex-safe
(UUID used as-is, other ids hashed). OutboxRelay logs a structured relay.publish per
event (eventType/aggregateId/correlationId); MessageGateway logs ws.deliver. Tests:
an event's correlationId equals the active request trace, with a generated fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task O.1: AsyncLocalStorage trace context (currentTraceId/runWithTrace/toTraceparent)
read by singleton services with no DI churn; Express traceMiddleware derives a trace
id per request (x-trace-id → traceparent → generated), echoes x-trace-id on the
response, runs the request in-context, and logs a structured http.request line on
finish. Minimal JSON structured logger stamps traceId on every line. Wired via
app.use in main.ts. 4 tests: scope, concurrency isolation, traceparent shape/parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full as-built reference: run/auth (dev tokens), conventions (idempotency, tenancy,
error codes), every REST endpoint across health/dev/interactions/threads/inbox/
support/adapters/routes/ai/calendar with request/response shapes, the kernel-client
+ 6 React hook SDKs, a curl QA quickstart + the 8 smoke scripts, env var reference,
and enum vocabularies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude
maaz519