Groundnote — architecture and contract (Phase 1)#

Status: binding. This file is the authoritative English contract of the project: data model, HTTP API, MCP surface and module layout. It replaces the German draft ("Wissensdienst", datenmodell.md / api-vertrag.md) for this repository. German terms map to English identifiers in glossary.md; every place where Groundnote departs from the German draft is in abweichungen.md. Change this file first, then the code. The knowledge graph and agent guidance (built, slices G1–G8) are specified in graph.md; see section 14. Approval modes A/B/C with their guardrails (slice A, built) are specified in approval-modes.md; see section 15.1. How agents use the MCP surface: agents.md. What is still open: section 15.

1. Invariants#

1. Nothing is overwritten. rule, rule_version, decision, source, source_attachment, source_segment and (migration 0004) classification, source_auth, secret_finding are append-only by trigger (RAISE(ABORT, 'append-only')). Every change is a new version, every decision a new row.
2. Only the decision path confirms, and it has two entrances. Status confirmed, superseded, rejected is written only by src/decision/**. Person: decide() in src/decision/decide.ts, reachable only from POST /v1/rules/:id/decision (approval channel, person). Policy: adopt() and retract() in src/decision/policy/**; adopt() runs only inside intake() after classify() and knows only proposed → confirmed for a checked quote, without supersedes, without a mandatory exception and under the daily limit; retract() only confirmed → superseded while the newest confirm/review decision is the policy's. Triggers enforce both (approval-modes.md §7). src/mcp/** reaches src/decision/policy/** (propose, retract) but never decide.ts (AT1). No MCP tool confirms, rejects, revokes or reviews; the mode is set only in groundnote.json.
3. The quote is evidence. For jamie and mail the quote is checked in code against the stored source (section 6.3). Not found → 422 quote-not-found, nothing stored. The quote never changes after version 1 (trigger).
4. Superseding is one transaction (section 6.4).
5. Validity is checked by the schema (CHECKs on rule_version). valid_from/valid_until are calendar days, said_at/changed_at are UTC instants ending in Z.
6. A file knows who it is. A scope file opens only if meta.scope equals its file name.
7. Loss locks, it does not answer. A scope file with fewer rule versions than ops.sqlite remembers is locked (503 scope-locked).
8. Rejected never leaks. Outside the approval channel a rejected rule does not exist (404), also with include_proposals or as_of.
9. Secrets only from the environment, never in groundnote.json, never logged.
10. Agents shape structure, persons confirm it (graph.md). Entities and links written via mcp or api take effect at once as unconfirmed versions or proposed links; only src/structure/**, reachable only from the approval channel, confirms, reverts or archives.
11. Foreign text reaches a model only framed. The text content of every MCP result is rendered by src/security/model-output.ts: quotes, excerpts, subjects, names and every other field not on its server-field list stay inside untrusted blocks with a random boundary; a statement counts as human-confirmed only while its record is confirmed by a person; an auto-adopted statement stays framed and is marked auto-adopted (section 9, approval-modes.md §9).
12. Guardrails are code, not configuration. The mandatory exceptions of approval-modes.md §5.4 apply in every mode; no config key turns one off, the validator refuses a config that names one, and a secret in a proposal is refused (422 secret-detected) before anything is stored.

2. Module layout#

createApp(deps) receives config, env, stores, a clock (now(): Date) and the JWT key resolver, so tests run in-process (local RSA key + createLocalJWKSet).

Architecture tests (test/arch/*.test.ts, TypeScript compiler API, no extra dependency):

• AT1 the transitive relative-import closure of src/mcp/** contains no file in src/decision/** outside src/decision/policy/** (slice A; AT1b keeps policy/** itself away from decide.ts).
• AT2 src/decision/** outside policy/ is imported only by src/http/routes/decision.ts and src/approval/**; AT2c/AT2d bound the importers of policy/** and of propose().
• AT3 the strings INSERT INTO rule_version occur only in src/domain/proposals.ts (with the literal 'proposed'), src/decision/decide.ts and src/decision/policy/{adopt,retract}.ts; INSERT INTO decision only there and in src/db/migrations/**; rejected_fts only in decide.ts, rejected-similar.ts and migrations.
• AT4 no registered MCP tool name matches /confirm|reject|revoke|review|decide|approve/.
• AT1b, AT2c, AT2d, AT6, AT7 and the exact wording of AT1–AT4 after slice A: approval-modes.md §14.
• AT1 also covers src/structure/**; AT3 also covers the graph writers (graph.md §4); AT4 also covers prompt names and the words archive and revert.
• AT2b src/structure/** is imported only by src/http/routes/structure.ts and src/approval/**.
• AT5 entity_closure and entity_fts are changed only by their writers and migrations (graph.md §13).
• AT6 the static import closure of src/ops/cli*.ts and src/ops/doctor/** contains no file of src/decision/**, src/structure/**, src/approval/**, src/redaction/**, src/mcp/**, src/http/**, src/auth/** and none of the writer modules src/domain/proposals.ts, src/domain/sources.ts, src/graph/entities.ts, src/graph/links.ts, src/db/fts.ts; the only dynamic import() in src/ops/** is ../main.js in src/ops/cli.ts; and the CLI's route table (src/ops/cli-routes.ts, the only way the CLI builds an HTTP request) holds no approval-channel route except GET /v1/export/:scope. Prerequisite (one refactor commit, no behaviour change): SourceRequest, AttachmentIn, Segment move to domain/source-schemas.ts, ProposalRequest, ProposalSource to domain/proposal-schemas.ts, LINK_TYPES, PROPOSABLE_TYPES to graph/link-types.ts; callers import from there (no re-exports).

3. Configuration and environment#

groundnote.json (path GROUNDNOTE_CONFIG, default ./groundnote.json), validated strictly (unknown keys fail start).

const ScopeName = z.string().regex(/^[a-z][a-z0-9-]{1,39}$/);
const Provider  = z.enum(["local", "voyage", "openai", "jina", "typesafe"]);
const Off       = z.object({ enabled: z.literal(false) }).strict();
const Semantic  = z.discriminatedUnion("enabled", [Off, z.object({    // stage b (search/stages-b-c.md §8)
  enabled: z.literal(true), model: z.enum(["bge-m3", "qwen3-embedding-0.6b"]).default("bge-m3"),
  model_dir: z.string().min(1).optional() }).strict()]);                // default <data dir>/models
const Reranker  = z.discriminatedUnion("enabled", [Off, z.object({    // stage c; key TYPESAFE_API_KEY from the environment
  enabled: z.literal(true), provider: z.literal("typesafe").default("typesafe"),
  model: z.enum(["jev-1.14.0"]).default("jev-1.14.0"), monthly_cap_usd: z.number().min(0).max(1000).default(5),
  timeout_ms: z.number().int().min(100).max(5000).default(1000), threshold: z.number().min(0).max(1).default(0.5) }).strict()]);

const Config = z.object({
  instance: ScopeName,
  zone: z.string().refine(isIanaZone),                  // e.g. "Europe/Berlin"
  public_url: z.string().url(),                         // no trailing slash; https in cloudflare-access mode
  scopes: z.array(z.object({ name: ScopeName, providers_allowed: z.array(Provider),
                         reranker: z.boolean().default(false) }).strict())   // c needs reranker + "typesafe"
           .min(1).refine(uniqueNames),                 // what MAY exist; files come from `groundnote scope create`
  access: z.object({                                    // required in cloudflare-access mode
    team_domain: z.string().url(),                      // https://<team>.cloudflareaccess.com; JWKS = <team_domain>/cdn-cgi/access/certs
    aud_mcp: z.string().min(1), aud_approval: z.string().min(1), aud_api: z.string().min(1),
  }).strict().refine(distinctAuds).optional(),
  search: z.object({
    synonyms: z.array(z.array(z.string().min(1)).min(2)).default([]),  // groups, case-insensitive
    languages: z.array(z.enum(["de", "en"])).min(1).default(["de", "en"]),
    lexical: LexicalConfig.prefault({}),                // stage a switches and weights (below); every key optional
    min_coverage: z.number().min(0).max(1).default(0),  // 7.6a; 0 = off
    semantic: Semantic.prefault({ enabled: false }), reranker: Reranker.prefault({ enabled: false }),
  }).strict(),
  backup: z.discriminatedUnion("kind", [
    z.object({ kind: z.literal("litestream"), target: z.string().url(), prefix: z.string().min(1),
               deadman_url: z.string().url(),
               export_directory: AbsolutePath.optional() }).strict(),  // default for single operators
    z.object({ kind: z.literal("copy"), directory: z.string().min(1),
               deadman_url: z.string().url(),
               export_directory: AbsolutePath.optional() }).strict(),  // consistent copy for an existing run (11.1)
  ]),                                                   // AbsolutePath = z.string().refine(isAbsolute)
  at_rest: z.discriminatedUnion("kind", [               // optional until redaction R6 decides "required"
    z.object({ kind: z.literal("volume"), attested_at: Day.optional() }).strict(),
  ]).optional(),
  doctor: z.object({                                    // every key optional (cli.md §6)
    max_backup_age_hours: z.number().int().min(1).max(24 * 14).default(26),
    max_probe_age_hours: z.number().int().min(1).max(24 * 14).default(26),
  }).strict().prefault({}),
  recheck_after_days: z.number().int().min(30).max(730).default(180),
  limits: z.object({                                    // in-process abuse limits (F1); every key optional
    requests_per_minute: PerMinute(60),                 // per grant on /mcp and /v1 (except decisions)
    decisions_per_minute: PerMinute(60),                // per grant on decisions and structure reviews (approval)
    login_attempts_per_minute: PerMinute(5),            // per client address on POST /approval/login
    exports_per_hour: z.number().int().min(1).max(10_000).default(6),  // per grant on GET /v1/export/:scope
    trusted_proxy: z.boolean().default(false),          // true: the last X-Forwarded-For entry is the client
  }).strict().prefault({}),                             // PerMinute(d) = z.number().int().min(1).max(100_000).default(d)
  graph: z.object({                                     // graph.md §7; every key optional
    entity_types: z.array(EntityType).min(1).default(["organisation", "department", "project", "customer",
                    "object", "person", "case", "topic"]),   // unique; ^[a-z][a-z0-9_]{1,29}$
    max_depth: z.number().int().min(2).max(64).default(16),
    proposed_min_confidence: z.number().min(0).max(1).nullable().default(null),  // null: only confirmed proposed links count
  }).strict().prefault({}),
  approval: ApprovalConfig.optional(),                  // approval-modes.md §3; absent: every scope `manual`
}).strict();

The service never schedules anything: an OS timer calls groundnote backup nightly (cli.md §8), which asks the service over the ops socket to copy (copy only, 11.1) and to probe (11.2). Litestream supervision stays the operator's (section 12). backup tells backup run, backup probe, backup verify and restore where the newest backup is: copy → <directory>/scopes/<scope>.sqlite and <directory>/ops.sqlite (the data dir layout); litestream → litestream restore -o <tmp> <live path>, which finds the replica in Litestream's own config (LITESTREAM_CONFIG or /etc/litestream.yml); target and prefix are not read by the code. directory and export_directory must already exist (a missing mount must not turn into a local directory) and must not lie inside the data dir. export_directory set → the probe writes the nightly Markdown export there (11.2), plaintext until slice R5; unset → no nightly export. deadman_url is read by the CLI only (backup nightly, D95).

at_rest is the operator's statement that the data dir lies on an encrypted volume, for hosts where doctor cannot detect it (container, network file system); attested_at is the day the operator last checked, older than 365 days is a fail (cli.md §6). doctor holds the two age limits of groundnote doctor (26 h: a daily timer with jitter and DST may slip by up to two hours). The calendar-day schema of attested_at lives in config/ (domain imports config, not the reverse). search.semantic and search.reranker leave Off with the search-stages slice (15.4), which owns their schema (snake_case, see cli.md §6 "Stage checks"); this slice does not change them.

search.lexical (src/config/lexical.ts) configures the lexical stage a (src/search/lexical/, measures in search/lexical.md); a missing key takes the module default:

const LexicalConfig = z.object({
  features: z.object({                                  // default on: stem, decompound, normalize, fuzzy,
    stem, decompound, normalize, trigram, fuzzy,        //   synonyms, validity, keyterms; off: trigram, prf
    synonyms, validity, keyterms, prf,                  // each z.boolean(), all optional
  }).partial().strict().prefault({}),
  field_weights: z.tuple([w, w, w, w]).optional(),      // bm25 keywords/statement/relations/quote, default 4/3/2/1
  fusion: z.enum(["score", "rrf"]).optional(),          // default "score"
  rrf_k: z.number().int().min(1).max(1000).optional(),  // default 60
  sub_weights: z.object({ word, stem, tri, fuzzy, syn, prf }).partial().strict().prefault({}),
  max_key_terms: z.number().int().min(1).max(12).optional(),      // default 6
  long_text_tokens: z.number().int().min(1).max(10_000).optional(), // default 25
  max_fuzzy_terms: z.number().int().min(0).max(12).optional(),    // default 8
}).strict();                                            // w = z.number().min(0).max(10)

search.synonyms and search.languages feed the lexical stage too. Pool size (200) and the term cap (12) are contract values, not keys. Changing stem, decompound, normalize or trigram changes what is stored: the next search or write rebuilds the lex_* tables of the scope (4.1). search.min_coverage is the only relevance threshold (7.6a). The tree limits and resolve thresholds of the graph are code constants (graph.md §7).

limits are token buckets in process memory (src/http/rate-limit.ts, clock from the app deps): each key holds n tokens and refills n per minute (exports_per_hour: per hour) continuously; they start fresh on restart. Requests, decisions and exports are counted per grant after authentication, so a request without a valid credential never counts. Decisions default to 60 a minute: confirming one hierarchy took 16 structure reviews in G9, which 10 a minute blocked; a bulk confirm is open (15.5). An export is the whole scope, so it has its own budget of 6 an hour. Login attempts are counted per socket peer address (@hono/node-server); X-Forwarded-For is read only with trusted_proxy: true, and then only its last entry. Over the limit → 429 rate-limited with Retry-After in whole seconds (6.6).

Refuse start when: mode missing/unknown; any required secret missing/short; any two secrets equal; cloudflare-access mode without access, with a non-https public_url, or with any GROUNDNOTE_DEV_* set; dev-token mode with a non-loopback host (127.0.0.1, ::1, localhost) unless the allow flag is true; GROUNDNOTE_OPS_PORT set (removed, the ops listener is a socket); the data dir, scopes/ or a *.sqlite, *.sqlite-wal, *.sqlite-shm or *.sqlite-journal file in them group- or world-accessible (data-dir-permissions, the message names the path and chmod 700/chmod 600); ops.sock exists as a non-socket or is served by another process (ops-socket-unavailable; a stale socket from a crash is removed); the ops.sock path, as given and resolved, longer than sun_path (104 bytes on macOS and the BSDs, 108 on Linux, null byte included; ops-socket-path-too-long, the message names length, limit and "choose a shorter GROUNDNOTE_DATA_DIR"). Logs are JSON lines to stdout, UTC, never containing tokens, JWTs, cookies, request bodies or source text.

4. Storage#

data/scopes/<scope>.sqlite per scope, data/ops.sqlite per instance. Each connection: journal_mode=WAL, busy_timeout=5000, synchronous=NORMAL, foreign_keys=ON, recursive_triggers=ON (otherwise INSERT OR REPLACE deletes a row without firing the append-only DELETE trigger). No manual checkpoints, no VACUUM (Litestream-friendly, see backup.md). The service is the only writer of every file. Migrations are numbered TS modules, applied in one transaction each; meta.schema_version records the last one. Unknown higher schema_version → refuse to open. Files in data/scopes/ whose name is not a configured scope are ignored with a warning.

4.1 Scope file (migrations/scope/0001-init.ts)#

CREATE TABLE meta (
  singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
  scope TEXT NOT NULL, schema_version INTEGER NOT NULL CHECK (schema_version >= 1),
  created_at TEXT NOT NULL);
CREATE TRIGGER meta_fixed BEFORE UPDATE OF scope, created_at ON meta BEGIN SELECT RAISE(ABORT, 'append-only'); END;
CREATE TRIGGER meta_no_delete BEFORE DELETE ON meta BEGIN SELECT RAISE(ABORT, 'append-only'); END;

CREATE TABLE rule (
  seq INTEGER PRIMARY KEY,                 -- explicit max+1; = rowid of rule_fts / rejected_fts
  id TEXT NOT NULL UNIQUE,                 -- <scope>-<nnnn>, never reused
  created_at TEXT NOT NULL, created_by TEXT NOT NULL);
CREATE TRIGGER rule_id_in_scope BEFORE INSERT ON rule
WHEN substr(NEW.id, 1, length((SELECT scope FROM meta)) + 1) <> (SELECT scope FROM meta) || '-'
  OR length(NEW.id) - length((SELECT scope FROM meta)) - 1 < 4
  OR substr(NEW.id, length((SELECT scope FROM meta)) + 2) GLOB '*[^0-9]*'
BEGIN SELECT RAISE(ABORT, 'rule-id-not-in-scope'); END;

CREATE TABLE source (
  id TEXT PRIMARY KEY,                     -- jamie:<id> | mail:<message-id> | manual:<grant-id>
  kind TEXT NOT NULL CHECK (kind IN ('jamie','mail','manual')),
  title TEXT, occurred_at TEXT, sender TEXT,
  thread TEXT,                             -- mail only: mail:<root message-id>
  link TEXT,
  excerpt TEXT,                            -- mail: text as received; jamie: NULL (segments); manual: NULL
  fulltext_sha256 TEXT,
  fetched_at TEXT NOT NULL,
  feeder TEXT NOT NULL,                    -- Phase 1: grant id that stored it
  CHECK (substr(id, 1, length(kind) + 1) = kind || ':'),
  CHECK (kind = 'mail' OR thread IS NULL),
  CHECK (kind <> 'mail' OR excerpt IS NOT NULL),
  CHECK (kind = 'mail' OR excerpt IS NULL),
  CHECK ((kind = 'manual') = (fulltext_sha256 IS NULL)),
  CHECK (kind = 'manual' OR (title IS NOT NULL AND occurred_at GLOB '????-??-??T??:??:??*Z')));
CREATE INDEX source_thread ON source(thread) WHERE thread IS NOT NULL;

CREATE TABLE source_segment (              -- jamie transcript segments as received
  source_id TEXT NOT NULL REFERENCES source(id), n INTEGER NOT NULL CHECK (n >= 1),
  start_s INTEGER NOT NULL CHECK (start_s >= 0), speaker TEXT,
  text TEXT NOT NULL CHECK (length(text) > 0),
  PRIMARY KEY (source_id, n)) WITHOUT ROWID;

CREATE TABLE source_attachment (
  source_id TEXT NOT NULL REFERENCES source(id), n INTEGER NOT NULL CHECK (n >= 1),
  filename TEXT NOT NULL, media_type TEXT NOT NULL,
  size INTEGER CHECK (size IS NULL OR size >= 0),
  external TEXT,                           -- paperless:123, provider attachment id
  text TEXT,
  text_origin TEXT NOT NULL CHECK (text_origin IN ('ocr','description','none')),
  CHECK ((text_origin = 'none') = (text IS NULL)),
  PRIMARY KEY (source_id, n)) WITHOUT ROWID;
CREATE TRIGGER attachment_contiguous BEFORE INSERT ON source_attachment
WHEN NEW.n <> 1 + coalesce((SELECT max(n) FROM source_attachment WHERE source_id = NEW.source_id), 0)
BEGIN SELECT RAISE(ABORT, 'attachment-gap'); END;

CREATE TABLE rule_version (
  rule_id TEXT NOT NULL REFERENCES rule(id),
  version INTEGER NOT NULL CHECK (version >= 1),
  kind TEXT NOT NULL CHECK (kind IN ('rule','note')),
  statement TEXT NOT NULL CHECK (length(statement) BETWEEN 10 AND 500),
  source_id TEXT NOT NULL REFERENCES source(id),
  quote TEXT NOT NULL,
  timestamp_s INTEGER CHECK (timestamp_s IS NULL OR timestamp_s >= 0),
  source_link TEXT,
  said_at TEXT NOT NULL CHECK (said_at GLOB '????-??-??T??:??:??*Z'),
  speaker TEXT,
  relations TEXT NOT NULL CHECK (json_valid(relations) AND json_type(relations) = 'array'),
  keywords TEXT NOT NULL CHECK (json_valid(keywords) AND json_type(keywords) = 'array'
                                AND json_array_length(keywords) <= 30),
  valid_from TEXT NOT NULL CHECK (valid_from GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'),
  valid_until TEXT CHECK (valid_until IS NULL OR valid_until GLOB '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'),
  supersedes TEXT REFERENCES rule(id),
  superseded_by TEXT REFERENCES rule(id),
  status TEXT NOT NULL CHECK (status IN ('proposed','confirmed','superseded','rejected')),
  reason TEXT NOT NULL CHECK (length(trim(reason)) > 0),
  changed_by TEXT NOT NULL,                -- grant id
  changed_at TEXT NOT NULL CHECK (changed_at GLOB '????-??-??T??:??:??*Z'),
  PRIMARY KEY (rule_id, version),
  CHECK ((status = 'superseded') = (valid_until IS NOT NULL)),
  CHECK (superseded_by IS NULL OR status = 'superseded'),
  CHECK (valid_until IS NULL OR valid_until >= valid_from),
  CHECK (supersedes IS NULL OR supersedes <> rule_id),
  CHECK (superseded_by IS NULL OR superseded_by <> rule_id),
  CHECK ((source_id GLOB 'jamie:*') = (timestamp_s IS NOT NULL)),
  CHECK (length(quote) >= CASE WHEN source_id GLOB 'manual:*' THEN 1 ELSE 20 END));

CREATE TRIGGER rule_version_transition BEFORE INSERT ON rule_version BEGIN
  SELECT RAISE(ABORT, 'version-gap') WHERE NEW.version <>
    1 + coalesce((SELECT max(version) FROM rule_version WHERE rule_id = NEW.rule_id), 0);
  SELECT RAISE(ABORT, 'terminal-state') WHERE (SELECT status FROM rule_version
    WHERE rule_id = NEW.rule_id ORDER BY version DESC LIMIT 1) IN ('rejected','superseded');
  SELECT RAISE(ABORT, 'invalid-transition') WHERE
       (NEW.version = 1 AND NEW.status <> 'proposed')
    OR (NEW.version > 1 AND NEW.status = 'proposed')
    OR (NEW.status = 'rejected' AND (SELECT status FROM rule_version WHERE rule_id = NEW.rule_id
          ORDER BY version DESC LIMIT 1) <> 'proposed')
    OR (NEW.status = 'superseded' AND (SELECT status FROM rule_version WHERE rule_id = NEW.rule_id
          ORDER BY version DESC LIMIT 1) <> 'confirmed');
  SELECT RAISE(ABORT, 'source-immutable') WHERE NEW.version > 1 AND EXISTS (
    SELECT 1 FROM rule_version f WHERE f.rule_id = NEW.rule_id AND f.version = 1 AND (
      f.source_id IS NOT NEW.source_id OR f.quote IS NOT NEW.quote
      OR f.timestamp_s IS NOT NEW.timestamp_s OR f.source_link IS NOT NEW.source_link));
END;

CREATE VIEW current_version AS SELECT v.* FROM rule_version v
WHERE v.version = (SELECT max(w.version) FROM rule_version w WHERE w.rule_id = v.rule_id);

CREATE TABLE decision (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  rule_id TEXT NOT NULL, version INTEGER NOT NULL,          -- the version the person saw
  kind TEXT NOT NULL CHECK (kind IN ('confirm','reject','revoke','review')),
  decided_by TEXT NOT NULL CHECK (decided_by GLOB 'person:*'),   -- grant id
  decided_by_identity TEXT NOT NULL,                       -- email from the verified credential
  decided_at TEXT NOT NULL CHECK (decided_at GLOB '????-??-??T??:??:??*Z'),
  reason TEXT,
  new_version INTEGER,                                     -- NULL only for review without change
  FOREIGN KEY (rule_id, version) REFERENCES rule_version(rule_id, version),
  FOREIGN KEY (rule_id, new_version) REFERENCES rule_version(rule_id, version),
  CHECK (kind NOT IN ('reject','revoke') OR length(trim(coalesce(reason, ''))) > 0),
  CHECK (kind = 'review' OR new_version IS NOT NULL),
  CHECK (new_version IS NULL OR new_version > version));

-- append-only: one pair per table (rule, rule_version, decision, source, source_attachment, source_segment)
CREATE TRIGGER rule_no_update BEFORE UPDATE ON rule BEGIN SELECT RAISE(ABORT, 'append-only'); END;
CREATE TRIGGER rule_no_delete BEFORE DELETE ON rule BEGIN SELECT RAISE(ABORT, 'append-only'); END;
-- … same two triggers for the other five tables …

CREATE VIRTUAL TABLE rule_fts USING fts5(keywords, statement, relations, quote,
  tokenize = 'unicode61 remove_diacritics 2');            -- rowid = rule.seq; never contains rejected rules
CREATE VIRTUAL TABLE rejected_fts USING fts5(keywords, statement, relations, quote,
  tokenize = 'unicode61 remove_diacritics 2');            -- approval only: "previously rejected similar"

CREATE TABLE search_log (                -- every scoped access, not only searches
  id TEXT PRIMARY KEY,                   -- search_id (UUID) shared by all scopes of one request
  at TEXT NOT NULL, grant_id TEXT NOT NULL,
  channel TEXT NOT NULL CHECK (channel IN ('mcp','api','approval','proactive','task')),
  operation TEXT NOT NULL CHECK (operation IN ('search','read_rule','list_rules','provenance',
    'propose','list_proposals','list_rejected','decide','store_source','feedback','list_scopes','status')),
  outcome INTEGER NOT NULL,              -- HTTP-equivalent status
  cross_scope INTEGER NOT NULL CHECK (cross_scope IN (0,1)),
  refs TEXT NOT NULL CHECK (json_valid(refs)),             -- heads / touched refs with version
  terms TEXT, variants TEXT, stages TEXT,                  -- JSON, search only
  input_sha256 TEXT,                                       -- of the request text; the text is never stored
  CHECK (operation <> 'search' OR (terms IS NOT NULL AND variants IS NOT NULL AND stages IS NOT NULL)));

CREATE TABLE feedback (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  search_id TEXT NOT NULL REFERENCES search_log(id),
  ref TEXT NOT NULL,                     -- rule/<id>@<n> for a verdict, rule/<id> for missing
  verdict TEXT NOT NULL CHECK (verdict IN ('relevant','irrelevant','missing')),
  given_by TEXT NOT NULL, given_at TEXT NOT NULL);

FTS maintenance (db/fts.ts, no triggers): after every write to a rule, in the same transaction, DELETE FROM rule_fts WHERE rowid = seq, then — unless the current status is rejected — insert the current version: keywords joined by space, statement, relations as "<kind> <name>" joined by space, quote. On reject, decision/ inserts the same row into rejected_fts instead.

Approval tables (migrations/scope/0004-approval.ts, approval-modes.md §7): classification, source_auth, secret_finding; decision is rebuilt with origin (grant·auto), mode, classification_id and kind retract; search_log/feedback are rebuilt for the operations retract, list_exceptions, list_auto.

Search log operations. Migration 0002-graph rebuilds search_log and feedback (SQLite cannot alter a CHECK) and extends operation with resolve_entity, read_entity, create_entity, move_entity, update_entity, propose_link, list_links, list_structure, review_structure, archive_entity and export. The graph tables entity, entity_version, entity_closure, entity_fts, link and review are in graph.md §3.

Lexical tables (migrations/scope/0003-lexical.ts, the same SQL as search/lexical/schema.ts; rowid = rule.seq): lex_word (FTS5, the four columns, words and canonical code tokens), lex_stem (FTS5, Snowball stems de+en and compound parts), lex_tri (FTS5 trigram, only filled with features.trigram), lex_word_vocab / lex_stem_vocab (fts5vocab over the first two, for IDF, key terms and typo correction), lex_doc (status, valid_from, valid_until of the indexed version) and lex_meta (build = analyzer version and the content-relevant features). Derived data only. reindexRule(db, ruleId, lexical) maintains them next to rule_fts in the same transaction (proposal create; confirm incl. the superseded predecessor; reject; revoke; review with changed keywords); a rejected rule is removed and never written. When lex_meta.build is missing (a file migrated to 0003 with existing rules) or differs from the configured options, the first search or write on the connection rebuilds the tables from current_version (not rejected), in one transaction.

4.2 Ops file (migrations/ops/0001-init.ts)#

CREATE TABLE meta (singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
  schema_version INTEGER NOT NULL, created_at TEXT NOT NULL);

CREATE TABLE grant (
  seq INTEGER PRIMARY KEY,
  id TEXT NOT NULL UNIQUE,                         -- <kind>:<seq>, e.g. person:2, service:3
  kind TEXT NOT NULL CHECK (kind IN ('person','service')),
  auth TEXT NOT NULL CHECK (auth IN ('email','access-service','bearer')),
  identity TEXT NOT NULL,                          -- lowercased email | Access common_name | sha256 hex of bearer
  scopes TEXT NOT NULL CHECK (json_valid(scopes) AND json_array_length(scopes) >= 1),
  rights TEXT NOT NULL CHECK (json_valid(rights) AND json_array_length(rights) >= 1),
  label TEXT, active INTEGER NOT NULL CHECK (active IN (0,1)),
  created_at TEXT NOT NULL, disabled_at TEXT,
  CHECK ((kind = 'person') = (auth = 'email')),
  CHECK (kind = 'person' OR instr(rights, '"decide"') = 0),   -- rights stored as canonical JSON array
  CHECK ((active = 1) = (disabled_at IS NULL)));
CREATE UNIQUE INDEX grant_active_identity ON grant(auth, identity) WHERE active = 1;
CREATE TRIGGER grant_disable_only BEFORE UPDATE ON grant WHEN NOT (OLD.active = 1 AND NEW.active = 0
  AND NEW.disabled_at IS NOT NULL AND NEW.id IS OLD.id AND NEW.kind IS OLD.kind AND NEW.auth IS OLD.auth
  AND NEW.identity IS OLD.identity AND NEW.scopes IS OLD.scopes AND NEW.rights IS OLD.rights
  AND NEW.label IS OLD.label AND NEW.created_at IS OLD.created_at)
BEGIN SELECT RAISE(ABORT, 'grant-disable-only'); END;
CREATE TRIGGER grant_no_delete BEFORE DELETE ON grant BEGIN SELECT RAISE(ABORT, 'append-only'); END;

CREATE TABLE backup_state (                      -- later phases add backup/export/probe columns
  scope TEXT PRIMARY KEY, created_at TEXT NOT NULL,
  version_count INTEGER NOT NULL CHECK (version_count >= 0), counted_at TEXT NOT NULL);
CREATE TRIGGER backup_state_monotonic BEFORE UPDATE OF version_count ON backup_state
WHEN NEW.version_count < OLD.version_count BEGIN SELECT RAISE(ABORT, 'count-decreased'); END;
CREATE TRIGGER backup_state_no_delete BEFORE DELETE ON backup_state BEGIN SELECT RAISE(ABORT, 'append-only'); END;

-- 0002-backup-verify: result of the last `groundnote backup verify`
ALTER TABLE backup_state ADD COLUMN verified_at TEXT;        -- UTC instant
ALTER TABLE backup_state ADD COLUMN verify_ok INTEGER;       -- 0/1
ALTER TABLE backup_state ADD COLUMN verify_result TEXT;      -- JSON report (checks, counts, exceptions; no content)
ALTER TABLE backup_state ADD COLUMN verified_counts TEXT;    -- JSON append-only row counts of the last passed backup

-- 0003-backup-run: backup run and restore probe (section 11.1, 11.2); writer: src/db/ops-store.ts
ALTER TABLE backup_state ADD COLUMN last_backup_at TEXT;     -- UTC instant of the last successful copy of this scope (copy only)
ALTER TABLE backup_state ADD COLUMN probed_at TEXT;          -- UTC instant of the last probe of this scope
ALTER TABLE backup_state ADD COLUMN probe_ok INTEGER;        -- 0/1
ALTER TABLE backup_state ADD COLUMN probe_result TEXT;       -- JSON report (checks, counts, exceptions; no content)
ALTER TABLE backup_state ADD COLUMN exported_at TEXT;        -- UTC instant of the last nightly export file
ALTER TABLE backup_state ADD COLUMN export_sha256 TEXT;      -- 64 lowercase hex of that archive
CREATE TABLE backup_run (                                    -- one row per `backup run` and per `backup probe`
  seq INTEGER PRIMARY KEY,
  kind TEXT NOT NULL CHECK (kind IN ('run','probe')),
  backup_kind TEXT NOT NULL CHECK (backup_kind IN ('copy','litestream')),
  started_at TEXT NOT NULL, finished_at TEXT NOT NULL,       -- UTC instants, finished_at >= started_at
  ok INTEGER NOT NULL CHECK (ok IN (0,1)),
  result TEXT NOT NULL CHECK (json_valid(result)));          -- the route's answer; paths, counts, hashes, no content
CREATE TRIGGER backup_run_no_update BEFORE UPDATE ON backup_run BEGIN SELECT RAISE(ABORT, 'append-only'); END;
CREATE TRIGGER backup_run_no_delete BEFORE DELETE ON backup_run BEGIN SELECT RAISE(ABORT, 'append-only'); END;

Every new instant column has the GLOB '????-??-??T??:??:??*Z' CHECK of 0002, probe_ok IN (0,1), probe_result json_valid, export_sha256 length = 64 AND NOT GLOB '*[^0-9a-f]*'. verified_counts only moves on a passed verification or probe; it is the floor for the next one. The time of the last copy of ops.sqlite itself is finished_at of the newest backup_run row with kind = 'run' whose result.ops.ok is true (ops_backup_at in GET /ops/status).

4.3 Ids, refs, locks#

• Rule id <scope>-<nnnn>: ^[a-z][a-z0-9-]{1,39}-\d{4,}$; number = rule.seq, zero-padded to 4. Scope of an id = id without the trailing -\d{4,}. Refs: rule/<id>, rule/<id>@<n>, source/<source-id>, attachment/<source-id>#<n>. Hits and bundles always carry @<n>.
• Source ids: jamie: + [A-Za-z0-9._-]{1,200}; mail:<message-id> incl. angle brackets (^mail:<[^<>\s]+>$), thread is the root message id in the same form; manual:<grant-id>, set by the service from the caller's grant (one source row per grant, created on first use).
• Open check (scope-store, on first use and cached): file exists, PRAGMA quick_check is ok, meta.scope equals the file name, schema_version known, and count(rule_version) ≥ backup_state.version_count. Any failure → scope locked until restart, 503 scope-locked, error log line. After every committed write the service raises backup_state.version_count.

4.4 Markdown export (src/export/)#

buildExport(db, {zone, now}) reads one scope file inside one read transaction (one snapshot) and returns the files below; exportArchive packs them as <scope>/<file> into a POSIX ustar .tar.gz built from node built-ins (zlib), so no archive dependency is audited or updated. GET /v1/export/:scope and groundnote backup verify use the same function. The export is never read back by the service.

The export contains rejected rules and the e-mail of every deciding person: it is the complete artifact of the scope. It is therefore served only on the approval channel to persons with decide on the scope, who see all of it there anyway; services get 403 channel-not-allowed. Machine backups use the ops socket (backup verify) and the backup copies (D71). Foreign text is normalized (CRLF, CR, NEL, U+2028, U+2029 → LF) and its Markdown/HTML characters (\ ` | [ ] < >) are backslash-escaped, so a statement cannot form a link, tag, heading or cell.

5. Access#

5.1 Channels and identities#

JWT check (jose): createRemoteJWKSet(<team_domain>/cdn-cgi/access/certs), jwtVerify with algorithms: ['RS256'], issuer: team_domain, audience: <aud of this channel>, clockTolerance: 30, exp required, nbf honoured, an iat more than 30 s in the future → 401. The key set has explicit fetch limits: cooldownDuration 60 s (no refetch for an unknown kid sooner), cacheMaxAge 10 min, timeoutDuration 5 s. Identity: string email → person (printable ASCII only, else 401; lowercased, auth email); no email but common_name → service (auth access-service); else 401. Service bearer: Authorization: Bearer gn_<43 base64url chars>, looked up by SHA-256 hex (auth bearer). Both a JWT header and a bearer on one request → 401. Dev tokens are compared with timingSafeEqual on SHA-256 digests.

5.2 Scope-check pipeline (every call, every channel; auth/scope-check.ts)#

1. Identity → active grant (grant row with matching auth+identity, active = 1). Else 401 unauthorized; on /mcp and /v1/* with header WWW-Authenticate: Bearer resource_metadata="<public_url>/.well-known/oauth-protected-resource". Wrong identity kind for the channel (service on mcp/approval) → 403 channel-not-allowed.
2. Parse the body/query (Zod; 400 validation-failed). Every requested scope — from parameters and from the id of every ref — must be in grant.scopes. Else 403 scope-not-allowed, identical whether the scope exists or not.
3. More than one scope only with cross_scope: true, else 400 multiple-scopes-without-flag. No scope given → 400 validation-failed; there is no default scope. Exception: the grant-wide listings scopes (MCP), GET /approval and GET /v1/status name no scope and cover exactly the grant's scopes.
4. The operation's right (read: search/read/list/provenance/status/feedback; propose: propose/store_source; decide: decide) must be in the channel's max rights and in grant.rights. Else 403 channel-not-allowed.
5. Open every touched scope (4.3; 503 scope-locked). Execute.
6. Log one search_log row in every touched scope (same id for all), including outcomes 404/409/422. Denials in steps 1–4 go to the process log only (grant id, channel, code).

5.3 Approval session and CSRF#

• Dev-token mode: GET /approval/login (form), POST /approval/login (token compared to GROUNDNOTE_DEV_APPROVAL_TOKEN; at most limits.login_attempts_per_minute per client address) → cookie __Host-gn_session (public_url https: Secure, no Domain) or gn_session (http loopback dev only, no Secure) = base64url({sub: identity, exp}) + . + HMAC-SHA256(GROUNDNOTE_SESSION_SECRET), HttpOnly; SameSite=Strict; Path=/, lifetime 8 h; POST /approval/logout. In cloudflare-access mode these routes are 404.
• Every state-changing approval request (decision, logout) needs both: same origin (Origin equals the origin of public_url; without Origin, Sec-Fetch-Site: same-origin; neither → fail) and form field / header csrf equal to base64url(HMAC-SHA256(secret, "csrf\0" + identity)), compared in constant time. Else 403 csrf-failed.
• Pages send Content-Security-Policy: default-src 'none'; style-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'none', X-Content-Type-Options: nosniff. All output is escaped (hono/html).

6. HTTP API#

Body limits: 10 MiB on /v1/sources, 256 KiB elsewhere (413). relation query = <kind>:<name>; status ∈ proposed|confirmed|superseded (rejected is not addressable); limit 1–200 (default 100), after = rule id (order by seq).

6.1 Shared schemas (domain/schemas.ts)#

const Day      = z.string().regex(/^\d{4}-\d{2}-\d{2}$/).refine(isCalendarDay);
const Instant  = z.iso.datetime();                         // UTC, ends with Z, no offset
const RuleId   = z.string().regex(/^[a-z][a-z0-9-]{1,39}-\d{4,}$/);
const RuleRef  = z.string().regex(/^rule\/[a-z][a-z0-9-]{1,39}-\d{4,}(@[1-9]\d*)?$/);
const Relation = z.object({ kind: z.enum(["customer", "product", "supplier"]),
                            name: z.string().min(1).max(200), ref: z.string().min(1).max(200).nullable() }).strict();
const Keywords = z.array(z.string().min(2).max(100)).max(30);   // including synonyms
const Status   = z.enum(["proposed", "confirmed", "superseded", "rejected"]);
const ErrorBody = z.object({ error: z.string(), message: z.string().optional(),
                             current: RuleRef.optional(), issues: z.array(z.unknown()).optional(),
                             remedy: z.string().optional() });

6.2 Sources#

const AttachmentIn = z.object({
  filename: z.string().min(1).max(500), media_type: z.string().min(3).max(200),
  size: z.number().int().nonnegative().nullable(), external: z.string().max(500).nullable(),
  text: z.string().max(2_000_000).nullable(),
  text_origin: z.enum(["ocr", "description"]).nullable(),
}).strict().refine(a => (a.text === null) === (a.text_origin === null));   // stored as 'none' when null

const Segment = z.object({ start_s: z.number().int().nonnegative(),
  speaker: z.string().min(1).max(200).nullable(), text: z.string().min(1).max(10_000) }).strict();

const SourceRequest = z.object({
  scope: ScopeName,
  source: z.discriminatedUnion("kind", [
    z.object({ kind: z.literal("mail"), id: MailId, title: z.string().min(1).max(500),
               occurred_at: Instant, sender: z.string().max(500).nullable(),
               thread: MailId.nullable(), link: Link.nullable() }).strict(),
    z.object({ kind: z.literal("jamie"), id: JamieId, title: z.string().min(1).max(500),
               occurred_at: Instant, sender: z.string().max(500).nullable(),
               link: Link }).strict(),
  ]),
  text: z.string().min(1).max(5_000_000).optional(),               // mail: required
  segments: z.array(Segment).min(1).max(20_000).optional(),         // jamie: required, start_s non-decreasing
  attachments: z.array(AttachmentIn).max(200),                      // complete, incl. unreadable ones
}).strict();   // superRefine: mail ⇔ text present and no segments; jamie ⇔ segments present and no text

const SourceStored = z.object({ created: z.boolean(), id: z.string() });   // id = source id (D66)

Stored synchronously. First call → 201 (source, segments n = array position, attachments, fulltext_sha256 = SHA-256 of text or of the canonical JSON of segments). A mail source may carry auth {from, authentication_results} (approval-modes.md §4), stored in source_auth and part of the identity check below. Same source.id in the same scope again, from the same grant and identical (source fields, full-text hash, the complete attachment list by position) → 200, created: false; nothing is written. Attachments are never added later (D66).

6.2a Reading a stored source back (GET /v1/sources/:id, D92)#

The feeder's read-back of its own write (groundnote sources get), not a knowledge read: persons and agents read sources through provenance. Same checks as POST: scope (query, required) passes steps 1–4 of 5.2 with the right propose; then the row must exist in that scope and source.feeder must equal the caller's grant id. An unknown id, an id stored by another grant and an id of another scope all answer the same 404 not-found with the same body, so the route tells nobody whether a source exists or which feeder stored it (threat model R3, D66). The :id is the URL-encoded source id (mail:<…> contains <, >, @).

const SourceQuery = z.object({ scope: ScopeName,
  include: z.string().optional() }).strict();            // comma list of "text" | "attachments", each at most once
const SourceOut = z.object({
  id: z.string(), scope: ScopeName, kind: z.enum(["mail", "jamie", "manual"]),
  title: z.string().nullable(), occurred_at: Instant.nullable(), sender: z.string().nullable(),
  thread: z.string().nullable(), link: z.string().nullable(),
  fulltext_sha256: z.string().nullable(), fetched_at: Instant,
  text_chars: z.number().int().nullable(),               // mail: length of the stored text, else null
  segment_count: z.number().int().nullable(),            // jamie, else null
  attachments: z.array(z.object({ ref: z.string(), n: z.number().int(), filename: z.string(),
    media_type: z.string(), size: z.number().int().nullable(), external: z.string().nullable(),
    text_origin: z.enum(["ocr", "description", "none"]),
    text: z.string().nullable().optional() }).strict()), // only with include=attachments
  text: z.string().optional(),                           // include=text, mail
  segments: z.array(Segment).optional(),                 // include=text, jamie (start_s, speaker, text)
}).strict();                                             // no `feeder`, no `excerpt` beyond `text`

Default answer: metadata, hash and the attachment list, no content; content only on request (include). The access goes to the process log (source-read: grant, scope, outcome), not to search_log, whose operation CHECK has no read_source yet (as for export, D71); the next scope migration adds both. A repeat from another grant, with different source fields or a different attachment list (also a longer one) → 409 source-conflict, always with the same generic message and nothing about the stored source; the reason (other-feeder, source-differs, attachments-differ) goes only to the process log. link is Link: http:/https: only.

6.3 Proposals and quote check#

const ProposalSource = z.discriminatedUnion("kind", [
  z.object({ kind: z.literal("jamie"), id: JamieId, quote: z.string().min(20).max(2000),
             timestamp_s: z.number().int().nonnegative(), link: Link }).strict(),
  z.object({ kind: z.literal("mail"), id: MailId, quote: z.string().min(20).max(2000),
             link: Link.nullable() }).strict(),
  z.object({ kind: z.literal("manual"), quote: z.string().min(1).max(2000),       // own words
             link: Link.nullable() }).strict(),
]);
const ProposalRequest = z.object({
  scope: ScopeName, kind: z.enum(["rule", "note"]), statement: z.string().min(10).max(500),
  source: ProposalSource, said_at: Instant, speaker: z.string().min(1).max(200).nullable(),
  relations: z.array(Relation).max(20), keywords: Keywords, valid_from: Day,
  supersedes: RuleId.nullable(),
}).strict();
const ProposalCreated = z.object({ ref: RuleRef, status: z.literal("proposed"),
  quote_check: z.object({ checked: z.boolean() }), possible_duplicates: z.array(RuleRef).max(3),
  approval_url: z.string().url() });                   // <public_url>/approval/rules/<id>
const ProposalSummary = z.object({ ref: RuleRef, kind: z.enum(["rule", "note"]), statement: z.string(),
  said_at: Instant, source: z.object({ kind: z.string(), id: z.string() }), supersedes: RuleId.nullable(),
  created_at: Instant, approval_url: z.string().url() });

Order: jamie/mail source must exist in the scope (422 source-unknown); quote check (422 quote-not-found); supersedes must name a confirmed rule of the same scope (422 reference-unknown); then in one transaction: rule (seq = max + 1), rule_version v1 proposed with reason = "proposed via <channel>", changed_by = grant id, FTS row. possible_duplicates: stage a over the same scope with text = statement, terms = keywords, proposals included, the new rule excluded, top 3. Failed checks are counted by their search_log row (operation = 'propose', outcome = 422). Slice A: both routes call intake() (approval-modes.md §6): secret scan first (422 secret-detected), then this order, then classification and, when the outcome is auto, adoption in the same transaction; ProposalCreated gains adoption, valid, marking, reasons and status widens to proposed | confirmed.

Quote check (domain/quote-check.ts): normalize(s) = NFKC → lowercase → every \p{P} and quote character to a space → collapse whitespace → trim. Normalized quote shorter than 20 chars → not found. mail: normalized quote is a substring of normalized source.excerpt. jamie: the window is all segments with start_s ∈ [timestamp_s − 120, timestamp_s + 120], joined in n order with a space (speakers excluded); normalized quote must be a substring of the normalized window. manual: no check, quote_check.checked = false.

6.4 Decision (decision/decide.ts)#

const Change = z.object({ statement: z.string().min(10).max(500), kind: z.enum(["rule", "note"]),
  keywords: Keywords, relations: z.array(Relation).max(20), valid_from: Day, said_at: Instant,
  speaker: z.string().min(1).max(200).nullable(), supersedes: RuleId.nullable() }).partial().strict();
const DecisionRequest = z.discriminatedUnion("decision", [
  z.object({ decision: z.literal("confirm"), version: z.number().int().positive(), change: Change.optional(), reason: z.string().min(1).max(1000).optional() }).strict(),
  z.object({ decision: z.literal("reject"),  version: z.number().int().positive(), reason: z.string().min(1).max(1000) }).strict(),
  z.object({ decision: z.literal("revoke"),  version: z.number().int().positive(), reason: z.string().min(1).max(1000), valid_until: Day }).strict(),
  z.object({ decision: z.literal("review"),  version: z.number().int().positive(), change: z.object({ keywords: Keywords }).strict().optional(), reason: z.string().min(1).max(1000).optional() }).strict(),
]);

Checks in this order: change has any of source, quote, timestamp_s, source_link, link → 400 source-immutable; reject/revoke without non-blank reason → 400 reason-missing; Zod → 400 validation-failed; rule unknown or not in scope → 404; current status rejected or superseded → 409 already-decided; version ≠ current → 409 version-outdated; confirm/reject on confirmed → 409 already-decided; revoke/review on proposed → 400 validation-failed. Every 409 carries current: rule/<id>@<n>.

One transaction per decision: version row(s), one decision row (decided_by = grant id, decided_by_identity = email, new_version), FTS, search_log. last_reviewed_at of a rule = latest decided_at of its confirm/review decisions (the policy's included), else rule.created_at. Since slice A every decision row carries origin (grant for a grant's decision, auto for the policy, decided_by = policy@<ruleset>); DecisionOut gains origin and mode.

6.5 Search request and response#

const SearchRequest = z.object({
  scopes: z.array(ScopeName).min(1).max(20), cross_scope: z.boolean().default(false),
  text: z.string().min(1).max(20_000).optional(),
  terms: z.array(z.string().min(1).max(100)).max(20).optional(),
  relation: z.object({ kind: Relation.shape.kind, name: z.string().min(1) }).strict().optional(),
  as_of: Day.nullable().default(null), include_proposals: z.boolean().default(false),
  limit: z.number().int().min(1).max(50).default(10),     // counts bundles
  entity: z.string().regex(ENTITY_ID).optional(),         // heads with a counted link to this entity or an ancestor
}).strict().refine(r => r.text !== undefined || (r.terms?.length ?? 0) > 0)
  .refine(r => r.entity === undefined || (r.scopes.length === 1 && scopeOfEntityId(r.entity) === r.scopes[0]));  // else 400

const StageState = z.object({ state: z.enum(["ran", "off", "failed", "budget", "not-allowed"]),
  duration_ms: z.number().int().nullable(), reason: z.string().nullable() });

const EntryKind = z.enum(["rule", "note", "meeting", "mail", "attachment", "entity"]);
const EntryBase = z.object({ ref: z.string(), kind: EntryKind, date: Instant, title: z.string(),
  excerpt: z.string().nullable(), link: z.string().nullable() });
const Entry = EntryBase.extend({                       // a group entry
  origin: z.enum(["deterministic", "proposed", "human"]),   // of the link it hangs on
  proposed: z.boolean() });                            // proposed and not yet confirmed (graph.md G5)
const PathItem = z.object({ ref: z.string(), name: z.string(), type: z.string() });
const HeadEntity = PathItem.extend({ path: z.array(PathItem), proposed: z.boolean() });  // path root first

const Head = EntryBase.extend({
  kind: z.enum(["rule", "note"]), scope: ScopeName,
  status: z.enum(["confirmed", "proposed", "superseded"]), unconfirmed: z.boolean(),
  valid_from: Day, valid_until: Day.nullable(), superseded_by: RuleId.nullable(),
  last_reviewed_at: Instant, age_days: z.number().int(),
  recheck: z.literal("still valid?").nullable(),
  source: z.object({ kind: z.enum(["jamie", "mail", "manual"]), id: z.string(), quote: z.string(),
                     timestamp: z.string().nullable(), link: z.string().nullable() }),  // timestamp "m:ss" / "h:mm:ss"
  speaker: z.string().nullable(), relations: z.array(Relation),
  rank: z.object({ a: z.number().nullable(), b: z.number().nullable(),
                   rrf: z.number().nullable(), reranker: z.number().min(0).max(1).nullable() }),
  found_via: z.array(z.string()),                        // matched query words, parts, corrections, synonym phrases (sorted, no stems), later "semantic"
  coverage: z.number().min(0).max(1).nullable(),         // 7.6a; null for a read (GET /v1/rules/:id)
  matched_terms: z.number().int().min(0).nullable(),     // 7.6a; null for a read
  entities: z.array(HeadEntity).max(10),                 // section 8; empty without graph data
});
const Group = z.object({
  kind: z.enum(["source", "thread", "attachment", "entity", "predecessor", "successor", "contradicts", "same-relation"]),
  count: z.number().int(), entries: z.array(Entry).max(5), more: z.number().int() });
const Bundle = z.object({ head: Head, groups: z.array(Group), line: z.string() });

const SearchResponse = z.object({
  search_id: z.string().uuid(), scopes: z.array(ScopeName),
  stages: z.object({ a: StageState, b: StageState, c: StageState }),
  query: z.object({ terms: z.array(z.string()), variants: z.record(z.string(), z.array(z.string())) }),
  bundles: z.array(Bundle),
  abstained: z.object({                                  // null iff bundles is non-empty (7.6a)
    reason: z.enum(["below-min-coverage", "below-reranker-threshold", "no-candidate"]),
    best_coverage: z.number().min(0).max(1).nullable(),  // null for no-candidate
    min_coverage: z.number().min(0).max(1),              // search.min_coverage
  }).nullable(),
});
const FeedbackRequest = z.object({
  bundles: z.array(z.object({ ref: RuleRef, verdict: z.enum(["relevant", "irrelevant"]) }).strict()).max(50),
  missing: z.array(RuleRef).max(50) }).strict().refine(f => f.bundles.length + f.missing.length > 0);

stages is in every response; b or c not running is never an error (200). An empty bundles is always explained by abstained (still 200): no-candidate when stage a found nothing that passes the filters, below-min-coverage when candidates existed but none reached a configured search.min_coverage > 0 (7.6a); the weak candidates are not returned, only the best coverage (deviation D67). With the default min_coverage: 0 only no-candidate occurs. below-reranker-threshold: stage c ran and no head reached search.reranker.threshold; where stage c ran in a scope, its threshold replaces min_coverage there (stages-b-c.md §14.2, §14.3). Feedback: every ref's scope passes steps 2–4 and must have a search_log row with this id, operation = 'search' and the caller's grant id (else 404, the same answer as for an unknown id); one feedback row per item in that scope.

Compact form (MCP search_rules only). The MCP tool takes SearchRequest plus detail: "compact" | "full" (default compact) and answers detail: "full" with the SearchResponse above. POST /v1/search has no detail and always answers the full SearchResponse: API clients are programs that read fields, not a model with an output cap, and one HTTP contract stays one schema. The compact form is built from the full response in search/compact.ts, after search() wrote its search_log row:

const CompactGroup = z.object({ kind: Group.shape.kind, count: z.number().int(),
  refs: z.array(z.string()).max(5),                         // the refs of the group's newest 5 entries
  proposed_refs: z.array(z.string()).min(1).max(5).optional() });  // refs whose link is proposed (G5); absent when none
const CompactHead = z.object({ ref, kind, status, scope, title, quote,   // quote = head.source.quote
  date, valid_from, valid_until, recheck,
  source: z.object({ kind, id, timestamp, link }), coverage });          // types as in Head
const CompactBundle = z.object({ head: CompactHead, groups: z.array(CompactGroup), line: z.string() });
const RelatedEntry = EntryBase.pick({ ref: true, kind: true, date: true, title: true });
const CompactSearchResponse = z.object({ search_id, scopes, stages,      // stages complete, as in SearchResponse
  query: z.object({ terms: z.array(z.string()) }),                       // no variants
  bundles: z.array(CompactBundle), abstained,
  related: z.array(RelatedEntry),                                        // every entry the bundles name, once
  truncated: z.object({ omitted: z.number().int().min(0),               // whole bundles dropped from the end
    omitted_refs: z.number().int().min(1).optional(),                    // group refs of the first bundle dropped
    quote_cut: z.literal(true).optional() }).nullable() });              // first head's quote cut with "…"

The source group is left out (it is head.source); every other group keeps kind and count and names its entries by ref; an entry that several bundles name (typically the same-relation rules) appears once in related. line is unchanged, so counts and the "(of which p proposed)" part stay. Not in the compact head: excerpt (= quote), link (= source.link), unconfirmed (= status), superseded_by, last_reviewed_at, age_days (both in line), speaker, relations, rank, found_via, matched_terms, entities (the entity group refs and their path in related). Size cap SEARCH_COMPACT_MAX_CHARS = 30 000 characters of the larger of the tool result's model text and its structured JSON: the practice test saw agents fail on answers of 51 000 characters and more; ten bundles with 300-character quotes and a shared relation need about 24 000 (full form about 115 000). Titles in related (mail subjects, attachment names, entity paths: foreign text whose every sentence the model text frames) are cut to RELATED_TITLE_CHARS = 120 characters ending in "…". Over the cap, whole bundles are dropped from the end until it fits and truncated.omitted says how many; if the first bundle alone is over, its group refs are dropped from the end (truncated.omitted_refs, count stays), then its quote is cut with "…" (truncated.quote_cut). Each step is a binary search over the rendered size (about 25 renders at limit: 50). The first head with statement and line always stays, so a search with hits never answers empty and abstained keeps its meaning. search_log.refs lists every bundle the search produced, also the omitted ones. For a full bundle the agent calls read_rule, or repeats the search with detail: "full" or a narrower query.

RuleVersionOut mirrors rule_version (relations/keywords as arrays, source as {kind, id, quote, timestamp_s, link}). DecisionOut = {kind, version, decided_by, decided_at, reason, new_version} (the identity email is shown only in the approval UI). Provenance = {ref, source: {id, kind, title, occurred_at, sender, thread, link, fulltext_sha256, fetched_at}, quote, timestamp_s, excerpt, segments, thread: Entry[], attachments: [{ref, filename, media_type, size, external, text_origin, text}]}; excerpt is the mail text or the jamie window ([m:ss] speaker: text lines), segments the window for jamie, else null.

6.6 Errors#

Body ErrorBody. Codes are stable API.

In MCP, scope errors are tool results with isError: true and the same error code; input rejected by the SDK's schema check is an isError result with the text "Input validation error" (deviation D45). A request to /mcp with a foreign Origin → 403 csrf-failed. Failures of the JWKS fetch are 500 internal, not 401. A configured scope without a file → 503 scope-locked.

Startup refusals (process exits ≠ 0, no HTTP): team-domain-not-https, public-url-not-https and the other reasons of section 3.

7. Search stage a (search/)#

Stage a is the lexical module search/lexical/ (LexicalIndex, options from search.lexical, section 3), wired in search/stage-a.ts; its measures and numbers are in search/lexical.md. Steps 1–4 are the module with the default switches; every switch off reproduces the original single-index contract (D70).

1. Terms. Text → NFKC, lowercase, split on [^\p{L}\p{N}]+, drop de/en stopwords (per search.languages), dedupe, keep the first 12. With normalize, canonical code and unit tokens of the text (RAL 9005 → ral9005) come first. Key terms (keyterms): a text with more than long_text_tokens (25) content words keeps only its max_key_terms (6) most characteristic terms, weighted by rarity in the scope (IDF), repetition, capitalization and codes, without the mail frame; unknown words count only as compounds of known parts, so a long mail is not dominated by small talk. Given terms come first, each as the phrase of its letter/digit tokens; total ≤ 20. Multi-word terms stay phrases. Terms depend on the scope's vocabulary; a cross-scope response lists the terms of all scopes in first-seen order.
2. Variants per term (query.variants[term]): the term; spelling ä↔ae, ö↔oe, ü↔ue, ß↔ss; one inflection strip (de -ern -en -er -es -n -e -s, en -ing -ed -es -s, stem ≥ 3 chars); with decompound the compound parts; with fuzzy typo corrections from the vocabulary. Synonym groups (synonyms: base list plus search.synonyms) are listed under ~<matched words>.
3. Sub-queries. word (prefix variants on lex_word), stem (stems and compound parts on lex_stem), syn, fuzzy, optional tri and prf; every token a quoted FTS5 literal (inner " doubled), user text never reaches MATCH unquoted. No terms → stage a ran, zero bundles.
4. Pool and fusion. Each sub-query ranks by bm25(…, 4, 3, 2, 1) over status IN ('confirmed','superseded'[, 'proposed']) (lex_doc); fusion: "score" adds, per sub-query, sub_weights[name] × (−bm25 / best −bm25 of that sub-query); a single sub-query keeps −bm25. validity multiplies by the version's standing on the target day: superseded or expired ×0.5, not yet valid ×0.8, proposed ×0.95, small bonus for recent valid_from. Top 200 by fused value, ties by seq. rank.a = the fusion value, rounded to 4 decimals (higher is better); it is comparable within one search, not across searches or scopes.
5. Filter. Validity is by dates, not only by status. Target day = as_of if given, else today (date in config.zone from the injected clock). A confirmed or superseded version passes if it is in force on the target day: valid_from ≤ day < coalesce(valid_until, '9999-12-31'). Superseding sets the predecessor's valid_until to the successor's valid_from, so a superseded rule stays the head (status superseded, successor in its successor group) until its successor starts. A hit on a superseded version no longer in force (valid_until ≤ day) is replaced by following superseded_by (≤ 20 hops) to a rule in force on that day, else dropped. include_proposals: proposals pass with unconfirmed: true. rejected never. relation: head has a relation with equal kind and case-insensitive name. entity: head has a counted link to the entity or one of its ancestors (section 8).
6. Dedupe by head rule id (best fusion value); found_via = union of what matched the indexed text of the hit (same normalization as the index), sorted: a query word whose word, stem or spelling variant matched is named as the query spells it (never the bare stem), a compound part or typo correction only when the whole query word did not match, synonym expansions as phrases. 6a. Coverage, no default threshold. For the first 50 candidates the module computes coverage = share of the query's term weight (IDF in the scope; a phrase weighs as its rarest word) that the head matches, 0–1, 3 decimals, and matched_terms = how many query terms it matches (any variant). Both are on every search head (null past the first 50 candidates and on reads). Stage a does not drop heads by default: a fixed threshold cost 38–50 % of the answerable queries in eval/lexical (D67). Only with search.min_coverage > 0 are heads with coverage < min_coverage (or no coverage) dropped before limit, and an empty result says below-min-coverage with the best coverage. Whether nothing suitable was found is decided by the caller on coverage / matched_terms, or later by stages b and c together; proactive hints (Phase 2) will rely on a+b+c or a configured minimum coverage, not on stage a alone. Take limit bundles.
7. Cross-scope: each scope searched separately; results interleaved by per-scope position, ties in request order. Bundles never mix scopes.
8. Stages b/c: interface SearchStage { name: "b" | "c"; availability(scope): StageState | null; start({ query, deadline, signal }): StageRequest }, with StageRequest.run({ scope, db, candidates }): Promise<StageOutcome> (ran with candidates, or failed / budget with a reason). search() is async; stage a runs while b embeds the query; a and b are fused by RRF, c reranks the top 50 heads per scope. Switched off, a stage reports {state: "off", duration_ms: null, reason: "disabled in config"}; a stage that throws is failed (internal error) and the result is that of a (or a+b). Contract and deviations of the build: search/stages-b-c.md §4, §15.
9. Log per scope: terms, variants, stages, head refs, input_sha256 of text — never the text.

8. Bundles (search/bundle.ts, search/line.ts)#

Head = matched rule version: ref rule/<id>@<n>, date = said_at, title = statement, excerpt = quote, link = source link. age_days = calendar days (zone) since last_reviewed_at; recheck = "still valid?" iff status confirmed and age_days > recheck_after_days. An entry excerpt of the source, thread and attachment groups is the start of the text, at most 300 characters, ending in … when shortened; the head quote stays full. head.entities (≤ 10) = the entities of the scope, not archived, that a counted concerns, belongs_to or mentions link ties to the head rule, each with path root first (ref entity/<id>, name, type) and proposed (graph.md §11). Groups in this fixed order, empty groups omitted, entries = newest 5, count = all, more = count − entries.length; every entry carries origin (of the link it hangs on) and proposed (proposed, not yet confirmed):

Source, thread, attachment, predecessor, successor and relation entries are computed (origin: "deterministic", proposed: false). Counted links (G5): stored deterministic/human links always; proposed links only when a person confirmed them, or when graph.proposed_min_confidence is set and the confidence reaches it without a reject; rejected never; only links open on the target day (as_of or today) and only ends in the bundle's scope. The search filter entity (one scope, the entity's own; else 400) keeps heads with a counted link to the entity or one of its ancestors.

line is built in code (English, dates in config.zone, ISO): <Label> from <YYYY-MM-DD>: <statement>[ — with <parts>]; <review>. Label Rule/Note, prefixed Unconfirmed for proposals or Superseded for superseded heads. Parts in group order: 1 meeting/1 mail, <n> mail(s) in thread, <n> attachment(s) (+ (<u> unread) if u > 0), <n> entit(y|ies), replaces <id>, superseded by <id>, <n> contradicting rule(s), <n> related rule(s); when shown entries are proposed, the parts end with (of which <p> proposed). Bundles without graph data are unchanged. Review: last reviewed <d> day(s) ago (+ , still valid? when recheck), for proposals proposed <d> day(s) ago, not confirmed. Example: Rule from 2026-03-12: Ask the supplier for the delivery time before ordering black hook rails — with 1 meeting, 2 attachments; last reviewed 220 days ago, still valid?

9. MCP (src/mcp/)#

SDK @modelcontextprotocol/sdk, Streamable HTTP server transport in stateless mode (sessionIdGenerator: undefined, JSON responses), a fresh McpServer({name: "groundnote"}) per request bound to the authenticated context, created with the instructions text of graph.md §9.1; auth runs in Hono before the transport. Each tool calls the same domain/search/graph functions as the API with channel mcp.

Output. Every successful result has structuredContent and content[0].text. The structured content is the API object plus untrusted: true on every object that holds foreign text and signals (injection signals, warning only) on every record with a ref (annotateForeign, src/security/model-output.ts). The text is not the JSON: it is the same result rendered for a model (renderResultForModel). Server-generated fields (refs, ids, scope, status, dates, enums, approval_url, recheck, the bundle line) are written as is; a statement is marked human-confirmed only while its record is confirmed and its adoption.kind is person (an auto-adopted one stays in a block marked auto-adopted, approval-modes.md §9); every other string (quote, excerpt, title, subject, speaker, entity name, alias, attachment name, keyword, query term) is foreign and appears only inside a block ⟪gn:<id> untrusted …⟫ … ⟪/gn:<id>⟫ with a random per-response boundary, escaped so it cannot close the block, after a fixed preamble that says the content is data, not instructions. The rule is fail-closed: a field not on the server-field list is foreign. Expected tool errors are plain JSON {error, …} (codes and request ids only); graph refusals (remedy, next, candidates, paths) are framed like results because candidate names are foreign text. The CLI prints through the same renderer.

That is 15 tools (14 before slice A); no name matches /confirm|reject|revoke|review|decide|approve|archive|revert/ (AT4). search_rules description, verbatim five sentences: "Cite every bundle with its date and source. If a head carries recheck, ask the person "still valid?" before applying it and read provenance first. If a stage did not run and nothing was found, say which stages ran. Prefer rules that state a restriction for the action you are planning. Groups list refs, each entry once in related; for a full bundle call read_rule or pass detail: \"full\", and if truncated is set, search narrower for the omitted bundles." Asked to confirm, the model answers with the approval_url. The verbatim descriptions of the graph tools are in graph.md §9.2.

Server instructions (src/mcp/instructions.ts, verbatim in graph.md §9.1) tell the model to name the scope, call entity_resolve before entity_create, tell the person the full path, say that created or moved structure is unconfirmed and give the approval_url, never confirm anything, read trees step by step, follow next and remedy, and cite date and source of every rule.

Prompts (src/mcp/prompts.ts, five, one user message each; texts in graph.md §9.4): create_level, file_incoming_mail, assign_document, restructure, where_does_this_belong. Guide and example dialogues: agents.md.

10. Approval UI (src/approval/)#

Forms post to POST /v1/rules/:id/decision (form-encoded; keywords comma-separated). Relations are edited through JSON on the same route.

11. Ops listener and CLI (src/ops/)#

Separate Hono app on the Unix socket $GROUNDNOTE_DATA_DIR/ops.sock, mode 0600, Authorization: Bearer $GROUNDNOTE_OPS_TOKEN (constant-time). There is no TCP listener for ops: only the user that owns the data directory can connect, and the public listener has no ops routes.

Run, probe and verify share one in-process lock: a second call while one runs → 409 backup-running. They read the live files through a read-only connection in a worker thread, so the public listener keeps answering during a long VACUUM INTO (better-sqlite3 is synchronous); writes stay with the service's own handles (section 4: the service is the only writer). Paths in ops answers are fine (the socket is the data dir owner's); they never reach /v1/*.

const Finding = z.object({ id: z.string(), status: z.enum(["ok", "warn", "fail", "skip"]),
  message: z.string(), remedy: z.string().nullable() });               // cli.md §6
const BackupRunOut = z.object({ seq: z.number().int(), kind: z.enum(["run", "probe"]),
  started_at: Instant, finished_at: Instant, ok: z.boolean() });
const OpsStatus = z.object({
  service: z.literal("groundnote"), instance: ScopeName, version: z.string(),
  started_at: Instant, now: Instant, data_dir: z.string(),            // real path
  backup: z.object({ kind: z.enum(["copy", "litestream"]), directory: z.string().nullable(),   // real path, absolute
    export_directory: z.string().nullable(),   // against the service's cwd while missing, like data_dir
    running: z.enum(["run", "probe", "verify"]).nullable(),
    ops_backup_at: Instant.nullable(), last_run: BackupRunOut.nullable(), last_probe: BackupRunOut.nullable() }),
  scopes: z.array(z.object({ scope: ScopeName, created: z.boolean(), locked: z.boolean(),
    lock_reason: z.string().nullable(), version_count: z.number().int().nullable(),
    last_backup_at: Instant.nullable(), verified_at: Instant.nullable(), verify_ok: z.boolean().nullable(),
    probed_at: Instant.nullable(), probe_ok: z.boolean().nullable(),
    exported_at: Instant.nullable(), export_sha256: z.string().nullable() })),   // config order
  usage: z.object({ reranker_month_usd: z.number() }).optional(),   // spentThisMonthUsd in the instance zone; read by doctor (cli.md §6)
});
const GrantOut = z.object({ id: z.string(), kind: z.enum(["person", "service"]),
  auth: z.enum(["email", "access-service", "bearer"]),
  identity: z.string().nullable(),                    // e-mail or Access common name; null for bearer
  fingerprint: z.string().nullable(),                 // bearer only: "sha256:" + first 12 hex of the stored hash
  scopes: z.array(ScopeName), rights: z.array(z.enum(["read", "propose", "decide"])),
  label: z.string().nullable(), active: z.boolean(), created_at: Instant, disabled_at: Instant.nullable() });

11.1 Backup run (src/backup/run.ts, POST /ops/backup/run)#

litestream: nothing is copied; 200 {ok: true, kind: "litestream", ran: false, message: "Litestream replicates continuously; nothing to copy"} and a backup_run row. copy:

1. backup.directory must be an existing directory owned by the service user with mode 0700, not inside the data dir (real paths); else 409 backup-directory-invalid naming the reason. <directory>/scopes/ is created 0700 if missing.
2. ops.sqlite first, then every created scope in config order. The order matters for invariant 7: every scope copy is then at least as new as the version count the ops.sqlite copy remembers, so restoring the pair never locks a scope.
3. Per file: VACUUM INTO <target dir>/.<file>.tmp-<uuid> (a consistent, compact snapshot; the live file is not vacuumed, section 4), inspectFile on the result (integrity, meta-scope; ops: integrity and a known schema_version), then no regression: every append-only count ≥ the larger of the count of the copy it replaces (read before) and, for a scope, backup_state.verified_counts (the floor of the last passed verify or probe) — a shrunken live file never overwrites the last good copy. A previous copy that exists but cannot be read (SQLITE_CORRUPT, SQLITE_CANTOPEN, SQLITE_BUSY …) fails the item (no-regression: previous copy unreadable: <code>) and stays untouched. A created scope without a backup_state row is not copied (backup-state-missing). Then fsync, rename over <directory>/scopes/<scope>.sqlite or <directory>/ops.sqlite, fsync of the directory, mode 0600. Any failure removes the temporary file and keeps the previous copy.
4. A locked scope (4.3) is not copied (its item fails, the previous copy stays).
5. backup_state.last_backup_at per copied scope; one backup_run row with the answer.

const RunAnswer = z.object({ ok: z.boolean(), kind: z.enum(["copy", "litestream"]), ran: z.boolean(),
  started_at: Instant, finished_at: Instant, message: z.string().optional(),
  ops: z.object({ ok: z.boolean(), file: z.string(), bytes: z.number().int().nullable(), detail: z.string().optional() }).nullable(),
  scopes: z.array(z.object({ scope: ScopeName, ok: z.boolean(), file: z.string(),
    bytes: z.number().int().nullable(), detail: z.string().optional() })) });

11.2 Restore probe (src/backup/probe.ts, POST /ops/backup/probe)#

The daily restore drill (backup.md "Restore drill"), through the real restore path:

1. A throw-away directory <data dir>/.probe-<random> (0700), removed in every case afterwards.
2. ops.sqlite: fetch the newest backup (copy: <directory>/ops.sqlite; litestream: litestream restore -o <tmp>/ops.sqlite <data dir>/ops.sqlite), check integrity and schema_version.
3. Per created scope (or the named one): restoreScope into <tmp>/<scope>.sqlite (the checks of section 11 "Restore" incl. the count floor), then pair-consistent (the restored file has at least the backup_state.version_count of the restored ops.sqlite, else restoring both would lock the scope) and, against a VACUUM INTO snapshot of live, not-ahead-of-live and export-hashes with exceptions exactly as in verify step 2.
4. If every check of the scope passed and backup.export_directory is set: buildExport of the restored file (so the archive is the proven backup state) → exportArchive → <export_directory>/<scope>/<scope>-export-<UTC stamp with milliseconds>-<8 hex>.tar.gz plus <file>.sha256 (two probes in one second never collide) (sha256sum format), each written to a temporary name and hard-linked (no overwrite), mode 0600, directories 0700. The directory must exist, be the service user's, not be writable by group or others unless sticky, and not lie in the data dir; <scope>/ (and <directory>/scopes/ of backup run) is a real directory of the service user with mode exactly 0700, no symlink (409 export-directory-invalid / backup-directory-invalid). Groundnote never deletes old archives (retention is the operator's backup tool).
5. backup_state.probed_at, probe_ok, probe_result, on success verified_counts (the floor) and exported_at, export_sha256; one backup_run row.

const ProbeAnswer = z.object({ ok: z.boolean(), kind: z.enum(["copy", "litestream"]), checked_at: Instant,
  ops: z.object({ ok: z.boolean(), checks: z.array(Check) }),           // Check = {check, ok, detail?}
  scopes: z.array(z.object({ scope: ScopeName, ok: z.boolean(), checks: z.array(Check),
    exceptions: z.array(z.string()),
    export: z.object({ file: z.string(), bytes: z.number().int(), sha256: z.string() }).nullable() })) });

The archive is plaintext like every export today: it holds rejected rules and deciding persons' e-mails (4.4). It is written for the holder of the ops token, who owns the data dir and can read every SQLite file anyway, so it widens nobody's access (D94); keep export_directory on the encrypted volume. Encryption to the scope key follows with slice R5 (redaction.md §9).

CLI groundnote (full reference: cli.md; reads GROUNDNOTE_DATA_DIR, GROUNDNOTE_OPS_TOKEN; sends the token only after resolving the data dir to its real path: the dir must be no symlink, owned by the calling user with mode 0700, every parent owned by root or the user and not writable by others unless sticky, and ops.sock a real socket of the user, else ops-socket-untrusted; a missing dir or socket is ops-unreachable, exit 1; the connect uses the resolved path). Commands:

There is no command that confirms, rejects, revokes, reviews, archives, reverts or redacts (invariants 2 and 10, AT6). Every api and approval command is a thin client with the caller's credential and the server's own request schemas (cli.md §7). export is a thin client of GET /v1/export/:scope with the same rights model, so it needs a person's Access token: credentials from GROUNDNOTE_URL + GROUNDNOTE_ACCESS_JWT (sent as cf-access-token and cf-access-jwt-assertion; api commands use GROUNDNOTE_API_TOKEN or a person's GROUNDNOTE_API_ACCESS_JWT, exactly one) or ~/.config/groundnote/cli.json ({url, token?, api_access_jwt?, access_jwt?}, $XDG_CONFIG_HOME honoured; refused unless a regular file of the user with mode 0600, credentials-file-permissions); https only, http only for loopback; the file is written only by config init (secret from stdin, never argv), with mode 0600, and never overwritten. Exit codes: 0 ok, 1 error (code on stderr), 2 usage, 3 a check failed (D98). stdout carries only the --json answer (also an error body), a new service token once, or the bare value of version, config path and approvals open; tokens are never printed otherwise. Human output goes to stderr, foreign text in it through renderResultForModel.

Backup verify (src/backup/verify.ts), per scope, temporary files in a .verify-* directory inside the private data dir, removed afterwards:

1. live-snapshot: VACUUM INTO from the live handle (reads only; the source file is not vacuumed, so section 4 "no VACUUM" holds), then integrity, meta-scope, sample-query (the newest rule with a search row is found by a word of its statement through rule_fts; a scope without one says so) and versions-remembered (the D63 count ≥ backup_state.version_count).
2. copy or litestream: fetch the newest backup into a temporary file (fetch-copy / fetch-litestream; a non-zero exit, or exit 0 without a file, fails), the same three checks, not-older-than-last-verified (every append-only count ≥ verified_counts), not-ahead-of-live (no append-only table has more rows than live) and export-hashes: both files are exported and the SHA-256 of every <id>.md compared. A rule may differ only if its newest changed_at / decided_at is after the backup's newest one; it is named in exceptions, everything else fails. Append-only tables are those with a BEFORE DELETE … 'append-only' trigger, read from the schema.
3. The report goes to backup_state (verify_result), the floor moves only on success.

Restore (src/backup/restore.ts, runs in the CLI, needs no running service): groundnote restore --to <path> --scope <s> [--from litestream|copy] (default backup.kind) resolves the target's directory to its real path and refuses the live file (target-is-live), anything inside data/scopes/ (target-in-scopes-dir, also through symlinks), an existing target or -wal/-shm/-journal next to it (target-exists) and a missing directory. It fetches into a temporary file next to the target, runs the checks of step 2 that need no live file and the count floor from ops.sqlite, and only then hard-links the file to the target (no overwrite, also not by a race). A failed check writes nothing (exit 3). Warnings: no backup_state row (restore ops.sqlite first), or fewer rule versions than ops.sqlite remembers (the service would lock the file, invariant 7). --from export is a later phase (section 12).

12. Reserved for later phases#

The items below are not built. Scope, files, constraints and spec of the next slices: section 15.

13. Slice plan#

Each slice ends green on pnpm lint && pnpm typecheck && pnpm test and is one commit.

Slices G1–G9 of the knowledge graph (graph.md §14) follow slice 12.

14. Knowledge graph and agent guidance#

Specified in graph.md and built (slices G1–G8; G9 is agents.md), from the German contract's section "Wissensgraph" (rules G1–G8) and API section H. In short: migration 0002-graph adds entity, entity_version (append-only history; a move is a new version), the derived entity_closure (maintained in the same transaction), entity_fts (trigram), link (stored links only; thread, attachment, source and supersedes links are computed) and review (person verdicts). Every graph answer carries the full path and next steps; every error an error code with remedy. Structure written by agents takes effect at once and appears in the approval section "Structure changes" until a person confirms, reverts or archives it (invariant 10). Bundles gain head.entities with paths and the groups entity and contradicts; proposed links count only when confirmed or above graph.proposed_min_confidence, always marked proposed.

15. Next slices#

Nothing here is built. Each entry names scope, what it touches, constraints and the spec to read first; a slice ends green on pnpm lint && pnpm typecheck && pnpm test, is one commit per reviewable step, and changes the contract (this file, then abweichungen.md) before the code.

15.1 Approval modes A, B, C (specified: approval-modes.md, slices A1–A6)#

Scope. Per scope and optionally per sensitive class: autonomous (A), classified (B, project default) and manual (C, today's behaviour and the default without an approval block). A deterministic, versioned ruleset classifies every proposal; the policy writer src/decision/policy/** adopts what passes, mandatory exceptions (payment, secret, contradiction, external-sensitive, injection, unchecked quote, doubt, large subtree) always wait for a person, auto-adopted rules are marked in every answer, retractable over MCP, API and CLI, limited per day and logged in classification + decision. Changes invariants 1, 2, 11 and adds 12; AT1–AT4 as in approval-modes.md §14. Not built: digest, model classifier, structure adoption (D81–D83). The operator decided the open questions on 2026-11-06 (approval-modes.md §16); structure adoption in A/B is the next slice.

15.2 Redaction and encryption (docs/redaction.md, slices R1–R7)#

Scope. Erasure of personal content by tombstone (rows stay, content goes), secure delete, per-scope age keys for backups and exports, volume encryption as a mandatory operating step, groundnote doctor. This also covers threat F6 of the threat model.

Order. R1 storage hygiene (secure_delete, temp_store, FTS5 secure-delete) → R2 tombstone schema (one trigger exemption, property-tested) → R3 redaction core (src/redaction/**) → R4 approval flow (preview, then confirmation, person with decide) → R5 keys and encrypted export/copy (src/keys/**, groundnote key …, groundnote backup run) → R6 doctor → R7 docs. R5 depends only on R1.

Touches. src/db/connection.ts, new scope and ops migrations, src/approval/**, src/backup/**, src/export/**, invariants 1, 3 and 9. Constraint. Redaction is reachable only from the approval channel; no MCP tool, ops route or CLI command.

Seven open questions for the operator (docs/redaction.md §16): doctor backup age 24 h or 26 h; exports encrypted to a reader's own recipient; a separate redact right; refusing plaintext copies once R5 lands; at_rest required in groundnote.json; message ids stay as identifiers; Litestream reseed or refusal when the backup kind is litestream.

15.3 Full CLI and groundnote doctor#

Status: built on feat/cli (steps 1–11 below; of step 12 cli.md, this section and backup.md are updated, README and glossary follow). Specified in cli.md, section 11 (ops routes, 11.1 backup run, 11.2 restore probe), 6.2a (GET /v1/sources/:id), 4.2 (ops migration 0003), section 3 (at_rest, backup.export_directory, doctor), AT6 and D92–D99. The stage checks of doctor are wired since the merge with search-stages (cli.md §6).

Steps (one commit each, green on pnpm lint && pnpm typecheck && pnpm test): (1) schema modules moved, AT6; (2) CLI core: route table, api client, --json, credentials incl. api_access_jwt and credentials-ambiguous, config show|path|init; (3) read commands; (4) write commands with --body and client-side validation; (5) GET /v1/sources/:id, sources get; (6) ops: GET /ops/status, GET /ops/grants, grants rename; (7) migration 0003, backup run; (8) backup probe with nightly export; (9) backup nightly with dead-man ping, examples; (10) doctor (pure checks first, stage checks wired after the search-stages merge); (11) /v1/status checks; (12) docs (README, glossary, backup.md "Phase 1 status").

Constraints. Exit codes stay 0 ok, 1 error, 2 usage, 3 a check failed (D98); credentials from the environment or ~/.config/groundnote/cli.json with mode 0600, secrets never in argv; stdout carries only the answer, an error body under --json, or a new token once; no command that confirms, rejects, revokes, reviews, archives, reverts or redacts, ever (invariants 2 and 10, AT6). Human output goes through src/security/model-output.ts. Touches src/ops/**, src/backup/ (new run.ts, probe.ts), src/db/ops-store.ts, src/db/migrations/ops/0003-*, src/config/file.ts, src/http/routes/sources.ts, src/http/routes/status.ts, the three schema moves. Agreed with siblings: search-stages owns the semantic/reranker schema and spentThisMonthUsd; redaction R6 adds its doctor checks to the same table and decides whether at_rest becomes required.

15.4 Search stages b and c (done)#

Built on feat/search-stages; kept here as the record of the cut. Current behaviour and deviations: search/stages-b-c.md §15; measurement §16.

Scope. Stage b (local embeddings; vectors in the scope file, table vec_rule keyed by seq and version, one model id and dimension per row, D101; fusion with stage a by RRF) and stage c (JEV reranker; provider and configuration are decided in providers.md). Both register through SearchStage (7.8) and report ran | off | failed | budget | not-allowed. Module cut, schemas and flow: search/stages-b-c.md. Touches src/search/stage.ts, src/search/search.ts, src/config/file.ts (semantic, reranker leave the Off schema), rank.b, rank.rrf, rank.reranker. Constraints. providers_allowed per scope is checked before any text leaves the host; a scope never reaches stage c without the provider listed; stage a stays the recall floor. The coverage signal (7.6a) is the base for "nothing suitable"; the relevance threshold is re-measured on real data once b or c runs (see 15.5).

15.5 Smaller open items#