# CLAUDE.md — Project rules

Working rules for **Agent Engine**. This file is about *how we work*.

| Question | File |
|----------|------|
| What is this product, what is shipped, what is planned? | `PROJECT.md` |
| How is it built, what talks to what? | `ARCHITECTURE.md` |
| What is still being decided? | `THREADS.md` |
| How do we work? | **`CLAUDE.md`** (this file) |

`AGENTS.md` is a pointer to this file. Rules live here only, so there is one place to change them.

---

## 1. Repository layout

```
agent-engine/
├─ apps/
│  ├─ web/                                     # Next.js — dashboard + internal API + public API + widget host
│  ├─ agents/rag-agent/                        # FastAPI — the RAG/DB agent runtime
│  └─ workers/async/embedding-pipeline-worker/ # Python — document → chunks → embeddings
├─ docs/
│  ├─ db-schema/migrations/                    # hand-applied SQL migrations (the real migration history)
│  ├─ architecture/                            # flow diagrams, SSE contract, latency playbook
│  └─ …
├─ supabase/full-schema.sql                    # consolidated dump of the live schema (gitignored)
├─ CLAUDE.md  PROJECT.md  ARCHITECTURE.md  THREADS.md
```

Everything under `data/`, `design/`, `supabase/`, `tests/` is gitignored and local-only.

---

## 2. Web app conventions (`apps/web`)

Group by **feature name**, not by file type. A feature gets the same name in all three places:

| Area | Path | Purpose |
|------|------|---------|
| UI | `src/components/<name>/` | React components and views |
| Logic | `src/lib/<name>/` | Business logic, helpers, clients, server-safe modules |
| HTTP API | `src/app/api/<name>/` | Next.js Route Handlers (`route.ts`) |

**Logic vs rendering:**

- **`.ts`** — pure functions, data shaping, API clients, Supabase calls, validation. All non-UI code.
- **`.tsx`** — layout, markup, props, event wiring only. Call into `src/lib/…`; reach the backend through `src/app/api/…`. Do not inline business logic in a component.

**API routes:** server-only work (secrets, privileged DB access, orchestration) goes behind `src/app/api/<name>/`, with reusable pieces in `src/lib/<name>/`. Do not add endpoints without a clear need.

---

## 3. Git and GitHub

- Remote: `https://github.com/jaimin-webbrains/agent-engine`.
- **`main`** is production. **`dev`** is the integration branch.
- **Our work ends at the merge into `main`.** Deployment is DevOps' responsibility — we do not deploy, configure hosting, or run release infrastructure. What we own is: implement on `dev`, commit, push, open the PR, review it, merge it.

### The only route to `main` is a reviewed PR from `dev`

**Never push to `main` directly.** Every change follows this sequence, in this order:

1. **Implement on `dev`** — or on a feature branch cut from `dev` that merges back into `dev`.
2. **Commit** the changes on `dev`.
3. **Push** `dev` to the remote.
4. **Open a PR** from `dev` → `main`.
5. **Review** the PR.
6. **Merge** the PR into `main`.

No step is skipped and no step is reordered, however small the change. Anything that lands on `main` without a reviewed PR is a process failure, and a hotfix is not an exception — it is a fast trip through the same six steps.

### Other git rules

- **Never `git push`, open a PR, or merge a PR without being asked explicitly.** The six steps describe *what* the sequence is; each one still waits for the go-ahead.
- **Never force-push.** Never rewrite history on `dev` or `main`.
- Commit messages: imperative mood, one concern per commit, and name the surface — e.g. `Add per-device avatar size slider to Connected Agents card`.
- Before any commit, run `git status` and confirm nothing staged is an env file (§4), a key, a database dump, or a client dataset.

---

## 4. Environment and secrets

- Each service reads its own `.env` from its own directory:
  - `apps/web/.env`
  - `apps/agents/rag-agent/.env`
  - `apps/workers/async/embedding-pipeline-worker/.env`
- **Never commit an env file, in any form.** This covers `.env` and every variant and backup of it — `.env.local`, `.env.production`, `.env.bak`, `.env.bak2`, `.env.backup`, `.env.old`, `.env.save`, `env.bak`, and anything else carrying real values. If a file holds a real secret, it does not go in git, whatever it is called.
- The **only** env file that belongs in git is **`env.example`**: placeholder values, never real ones.
- Before committing, check for stray backups — an editor or a quick `cp .env .env.bak` is the usual way a secret escapes:
  ```bash
  git status --short && git ls-files | grep -iE '(^|/)\.?env(\.|$)'
  ```
  The second command must return only `env.example` paths.
- **When you add an env var, add it to that service's `env.example` in the same change**, with a placeholder value and a one-line description. A missing `env.example` entry is a broken handoff to DevOps.
- Never print a secret value into terminal output, logs, a commit, or a doc. Refer to variables by name.
- `SUPABASE_SERVICE_ROLE_KEY`, `RAG_AGENT_INTERNAL_SECRET`, and every LLM provider key are **server-only**. Only `NEXT_PUBLIC_*` may reach the browser.
- **There is no platform LLM or embedding key, and none may be added to any `.env`.** Each project supplies its own in the Connected Agents card; they live in `project_agent_llm_secrets` (RLS on, no policies — service role only). Never return a stored key to the browser (the last four characters are the most it may see), never log one, and never fall back to a shared key when a project has none — the agent must refuse instead.
- Production values live in the DigitalOcean App Platform secret store, never in chat, email, or a file.

---

## 5. Service commands (Makefile)

The root `Makefile` drives all three services — **`web`**, **`rag-agent`**, **`embedding-worker`** — through docker compose. Every service runs in its own container. `make help` lists every command; `make help <command>` explains one command and its parameters.

| Target | Meaning |
|--------|---------|
| `make up` · `make up-<svc>` | Start every container, or one (builds the image if missing) |
| `make down` · `make down-<svc>` | Stop and remove every container, or one |
| `make restart` · `make restart-<svc>` | Recreate every container, or restart one in place |
| `make logs` · `make logs-<svc>` | Follow all container logs interleaved, or one |
| `make status` | Every container at a glance — state, health, ports |
| `make ps` · `make ps-<svc>` | The running container per service, in detail |
| `make build` · `make build-<svc>` | Build every image, or one |
| `make help` · `make help <command>` | Command list, or detail for one command |

Parameters: `N=<lines>` and `FOLLOW=0|1` (logs), `NOCACHE=0|1` (build). Host ports: web **3001** → container 3000, rag-agent **8080** → 8080; override with `WEB_HOST_PORT` / `RAG_AGENT_HOST_PORT`.

**`make up` does not rebuild.** After changing code, `make build-<svc> && make up-<svc>`.

### Rules when extending the Makefile

- **Every service supports every target.** No per-service special cases beyond the one-line forwarder.
- **The help list is self-documenting** — the same `##` convention the Rancho project uses. Every target carries its own description on its rule line:
  ```make
  up-web: ## Start the web container
  ```
  `make help` generates the list by grepping the Makefile for `^target: ## description`, so help can never drift from the targets. A target without a `## ` comment is invisible and effectively does not exist.
- **Because the list is generated from file text, per-service targets are written out explicitly** rather than produced by pattern rules — a generated target has no `##` line to scan. Each is a single line forwarding to compose or to an internal `_logs` / `_ps` helper.
- **Do not list pattern-rule targets in `.PHONY`.** GNU Make does not apply pattern rules to phony targets, and the target silently becomes a no-op.
- **A command with parameters also gets a `HELP_<name>` block** for `make help <command>`, naming each parameter, its default, and an example.
- **Compose is always invoked with `--env-file apps/web/.env`**, because the web image needs `NEXT_PUBLIC_*` as build args. Each service still loads its own `.env` at runtime through `env_file:`.

### Still to add

`setup`, `lint`, `test`, `clean`. Until they land:

```bash
cd apps/web && npm install && npm run build && npm run lint
cd apps/agents/rag-agent && python3 environment/setup.py
cd apps/workers/async/embedding-pipeline-worker && python3 environment/setup.py
```

---

## 6. Dockerising the services

Each service runs in **its own container**, built from a `Dockerfile` beside its own code, wired together by the root `docker-compose.yml`.

| Service | Dockerfile | Base | Port | Health |
|---------|-----------|------|------|--------|
| `web` | `apps/web/Dockerfile` | `node:20-slim` | 3000 (host 3001) | HTTP `/` |
| `rag-agent` | `apps/agents/rag-agent/Dockerfile` | `python:3.12-slim` | 8080 | HTTP `/health` |
| `embedding-worker` | `apps/workers/async/embedding-pipeline-worker/Dockerfile` | `python:3.12-slim` | none | none — it serves nothing |

Rules:

- **Multi-stage builds.** Build dependencies never reach the runtime image; Python services build into a venv and copy `/opt/venv` across.
- **Pinned base images** — `node:20-slim`, `python:3.12-slim`.
- **Never bake a secret into an image.** Runtime env arrives through compose `env_file:`, and in production through the platform secret store. The only build args are `NEXT_PUBLIC_*`, which are publishable by definition.
- **Non-root runtime user** — `node` for web, `appuser` (uid 10001) for both Python services.
- **`HEALTHCHECK` on every service that serves HTTP.** `web` waits for `rag-agent` to be healthy before starting (`depends_on: condition: service_healthy`).
- **Each service has its own `.dockerignore`**, and the build context is the service directory — not the repo root — so one service's files can never leak into another's image. Every `.dockerignore` excludes `.env` and its backups, `node_modules`, `.venv`, `.next`, and `.git`.
- **Compose must reproduce production wiring locally**: web → rag-agent by service name on the internal bridge network, all three → Supabase. `apps/web/.env` points `RAG_AGENT_BASE_URL` at localhost for host-side runs, so compose overrides it to `http://rag-agent:8080`.

Two things to know before editing the web image:

- **`next start` transpiles `next.config.ts` on boot**, so `typescript` is required at runtime even though it is a devDependency. The runner stage installs it with `--no-save`. Removing that line breaks the container with a confusing "Failed to load next.config.ts".
- **Fonts are self-hosted. Never add `next/font/google`.** The app loads Plus Jakarta Sans from `apps/web/public/fonts/` through `next/font/local`, so builds are hermetic and no request ever reaches Google. `<body>` carries `font-plus-jakarta-sans`, which makes it the app-wide default that everything inherits; `font-mono` and other element-level font classes still win over it. A new font is added by committing its files and extending the `localFont` call in `src/app/layout.tsx`.

---

## 7. Database and migrations

- The live database is **Supabase project `Agent Engine`**, ref `xbkcqtqrgrnbepunsayc` (Oceania / Sydney, Postgres 17.6). The repo is linked via the Supabase CLI.
- **`docs/db-schema/migrations/` is the migration history.** Supabase CLI migrations are not used.
- **New migrations are numbered:** `NNNN-YYYY-MM-DD-short-description.sql`, zero-padded to four digits, **starting at `0035`**.
  ```
  0035-2026-09-08-add-monthly-partitions.sql
  ```
  The number is the apply order and it is authoritative. The date is documentation. This exists because the 34 legacy files sort wrongly — `2026-06-29-…-avatar-size-per-device.sql` sorts *before* `2026-06-29-…-avatar-size.sql` but was applied *after* it.
- The 34 legacy migrations keep their original names. Their true order is recorded in `ARCHITECTURE.md`.
- **Never edit a migration that has been applied.** Write a new one.
- Every migration must be re-runnable: `if not exists`, `if exists`, `on conflict do nothing`.
- Every migration states in a header comment what it changes and why.
- **Apply migrations with `supabase db query --linked`**, one statement per invocation. The command does not accept a multi-statement script — passing the whole file fails partway with a confusing error and leaves the migration half-applied. Split it:
  ```bash
  supabase db query --linked "alter table ... add column ...;"
  supabase db query --linked "comment on column ... is '...';"
  supabase db query --linked "create index ...;"
  ```
- **After applying a migration**, refresh `supabase/full-schema.sql` and confirm live matches:
  ```bash
  supabase gen types typescript --linked > /tmp/live-types.ts
  ```
- A schema change is not done until RLS policies and indexes are considered. Every tenant-scoped table needs RLS.

---

## 8. The embeddable widget

`apps/web/public/scripts/rag-agent-widget.js` runs on customer websites. Treat it as a published artefact, not an ordinary source file.

- **It stays in `apps/web/public/`.** It is served by the web app. No separate CDN, no separate deployment.
- **It must be built and content-hashed** — `rag-agent-widget.<hash>.js` — so a cached old copy can never be served against a new API.
- The stable embed URL customers paste must keep working and resolve to the current hashed build.
- Cache headers: immutable + long max-age on the hashed file, short max-age on the stable entry point.
- **The widget's font is inlined, not fetched.** Plus Jakarta Sans ships inside `rag-agent-widget.js` as a base64 `data:font/woff2` URI (latin subset, variable 200–800, upright only — 27 KB of woff2, 36 KB of base64). It costs no extra request on a customer's page, sends no visitor IP to Google, and has no CORS header that can silently break. Regenerate with `python3 apps/web/scripts/build-widget-font.py`; the generated block is marked in the file and must not be hand-edited.
- **Register widget font faces through `document.fonts.add(new FontFace(...))`, never an injected `@font-face` rule.** Font faces are document-scoped: a rule placed inside a shadow root is silently ignored and the widget renders in the fallback while the stylesheet looks correct. The widget is light-DOM today, but this keeps working if it is ever isolated in a shadow root.
- **Widget chrome is translated through `t(key)`, never inline literals.** `STRINGS` holds `en` / `gu` / `hi`; language resolves from `data-language`, then the browser locale, then the visitor's script once they type. Two traps: entries inside the `STRINGS` table must stay literal strings — a `t()` call in there recurses before `STRINGS` exists and throws on load for every customer — and the table is linted, so use `catch {` rather than an unused `catch (e)`.
- **Never make a breaking change to `POST /api/public/rag/chat` or the SSE events in `docs/architecture/rag-agent-sse-protocol.md`.** Adding a key is safe; renaming or removing one breaks every live customer site. Version the endpoint instead.

The build step does not exist yet — the file is currently hand-edited and served raw. Tracked in `THREADS.md`.

---

## 9. Testing

Five rules. All five apply.

1. **Contract tests on the public API.** Lock the request/response shape of `/api/public/rag/chat`, `/history`, `/contact`, `/widget-config`, and every SSE event in `docs/architecture/rag-agent-sse-protocol.md`. Customer widgets in the wild depend on these; a silent rename is a production break.
2. **Unit tests on `apps/web/src/lib/**/*.ts`.** The `.ts`/`.tsx` split in §2 exists precisely so this logic is testable with no browser and no database. New logic in `lib/` ships with tests.
3. **pytest on the RAG agent's pure helpers.** `_extract_sql_placeholders`, `_extract_mongo_placeholders`, `_coerce_param_value`, `_bind_sql_template_params`, `_apply_mongo_template_params`, and the SQL/Mongo verifiers. No LLM calls, no network — and this is exactly where a bug becomes a SQL injection.
4. **Security regression tests.** SELECT-only enforcement, `Origin` vs `projects.domain`, `is_domain_verified`, project API-key auth, the `X-RAG-Agent-Secret` check, the temporary `template_only` guard, and RLS. Each of these must have a test that fails if the protection is removed.
5. **One scripted end-to-end smoke test.** Replace the manual six-step procedure in `tests/sample-rag-widget-site/README.md` with a script: widget → `/api/public/rag/chat` → RAG agent → Supabase, asserting a reply arrives and a `messages` row is written.

Not in scope: LLM answer-quality evals and UI snapshot tests. High maintenance, low return at this stage.

No test infrastructure exists yet. Tracked in `THREADS.md`.

---

## 10. Keeping the docs true

- A change that alters *what the product does* updates `PROJECT.md`.
- A change that alters *how services fit together* updates `ARCHITECTURE.md`.
- A decision that closes an open question moves it out of `THREADS.md`.
- A new working rule goes in `CLAUDE.md`, not in a commit message or a code comment.
- `README.md` is currently inaccurate and is being narrowed to a short public overview. Do not treat it as a source of truth; `PROJECT.md` is.
