Practice test findings (2026-11-06)#
Practice test of feat/core @ 65b696e (Phase 1, search stage a only) with one operator's real
meetings, mails and chats in three scopes (private, company, holding). The data stayed on the
operator's machine; nothing of it is in this file. Setup: cloudflare-access mode with a local
stand-in issuer (one grant per scope and agent), real Claude Code sessions (claude -p, Opus,
--strict-mcp-config) against /mcp, human decisions through the approval channel. Approval
modes A/B/C are not built; everything ran in mode C.
Known and not counted: approval from a real browser fails with csrf-failed (Origin null);
the fix is planned with approval modes.
Numbers#
- 100 sources stored via
POST /v1/sources; 159 rule proposals from 32 extraction sessions (1,175 tool calls, ~$32); 154 confirmed, 5 rejected; ~120 entities and ~430 links. - 15 later real messages, prompt "prepare a reply", no hint at a rule: the agent queried Groundnote unprompted in 12 of 15. Of 11 messages with an applicable rule: 8 full hits, 2 partial, 1 missed (not queried). 0 wrongly applied rules, 3 marginal citations.
- Scope separation: 0 leaks in 22 sessions (cross-scope search correctly refused).
- Supersedes and validity: correct in agent answers and with
as_of(old rule shown on its day, new rule today, revoked rule gone aftervalid_until). - Per answer: median 3 Groundnote calls, ~166k input tokens (mostly cached), ~$0.33.
Findings#
| # | Severity | Finding | Proposal |
|---|---|---|---|
| P1 | high | A 429 rate-limited during the MCP handshake makes the client treat the server as failed; 3 of 13 parallel sessions ran without any Groundnote tool. The limit is per grant (60/min), so a few parallel agents on one grant exhaust it; the MCP error carries no retry hint. |
Exempt initialize/tools/list from the bucket or rate-limit per session; return Retry-After and a remedy in the MCP error; document the per-grant budget for multi-agent setups. |
| P2 | high | search_rules answers are too large for an agent: 24 sessions got at least one answer above the client's output cap (51k–196k chars, default limit 10), mostly from same-relation groups repeated in every bundle. Agents then re-search blindly. |
Size budget per response; list related rules once per response (or by ref only) instead of per bundle; a compact mode (head + line + refs) as the default for MCP. |
| P3 | high | Human workload in mode C: ~45 sources from three weeks produced 159 rule decisions and ~550 structure reviews (one per entity and link), all bounded by decisions_per_minute. Not sustainable daily. |
Prioritise approval modes A/B and bulk confirm of a proposed hierarchy (15.5); count links of a confirmed rule as confirmed by default. |
| P4 | medium | possible_duplicates is mostly noise (topic-unrelated hits, refs only); agents learnt to ignore it, and one real intra-session duplicate slipped through. |
Return statement and a reason (matched terms, score) per candidate; only report above a similarity floor; flag same-source near-duplicates explicitly. |
| P5 | medium | quote-not-found gives no diagnosis. Quotes spanning two segments, with "[…]", from numbered lists or with line breaks fail; agents lose evidence by shortening. |
Add a remedy (closest match, which part failed); allow ellipsis-joined fragments that each pass the check inside the same window. |
| P6 | medium | Redundant inputs on propose_rule: the jamie link is required although the stored source has it (an agent without the link fell back to manual, so the quote went unchecked); said_at must be computed by the agent from meeting start + timestamp_s. |
Take link from the stored source; derive said_at from occurred_at + timestamp_s when omitted. |
| P7 | medium | The server instructions say to pass on "still valid?" unconditionally; agents ask it for nearly every rule (7 of 22 answers), also for rules confirmed today. | Say "only when the head carries recheck". |
| P8 | medium | No instruction tells the agent to consult Groundnote before replying to a message; it was skipped for chat transcripts (3 of 15). | Add one sentence to the instructions, or a prepare_reply prompt that starts with search_rules. |
| P9 | medium | A feeder can store the same source in a wrong scope without any signal, and it cannot be removed (append-only, no redaction yet). | Feeder-side scope routing guidance; an operator path to tombstone a misrouted source (redaction R2/R3). |
| P10 | medium | People: agents create first-name-only persons; entity_resolve does not match a full name against a stored first name ("A B" vs "A"); one confirmed duplicate pair, no merge path. |
Token-subset matching in resolve; a same_as/merge review on the approval page; ask for full names in the tool description. |
| P11 | low | Only mail and jamie source kinds; chats are stored as mails dated by their last message, without per-message time. |
A chat kind with per-message timestamps (like segments). |
| P12 | low | Inconsistent refs: read_rule needs rule/<id>, supersedes takes the bare id; entity_tree "exactly one of id and scope" is not in the description. |
Accept both forms; state the constraint in the schema description. |
| P13 | low | supersedes works only on confirmed rules, so a replacement found in the same batch cannot be linked; no "refines/related" link and no expiry for commitments. |
Allow supersedes on a pending proposal (resolved at confirm); a valid_until or due date on notes. |
| P14 | low | Relation kinds are only customer/product/supplier; agents mislabel brokers, authorities, partners as supplier. Entity types lack supplier/system. |
Configurable relation kinds; add partner/authority, and a system entity type to the defaults. |
| P15 | low | Decompounding splits domain words ("Vollmacht" → "voll" + "macht"), causing unrelated hits. | Do not decompound into function words; a stop list for split parts. |
| P16 | low | counts_in_bundles: false on proposed links without the threshold that would make them count. |
Return the effective proposed_min_confidence with the link. |
| P17 | low | Reported by agents, not verified: include_proposals: true did not show open proposals that open_proposals lists. |
Add a test for search with include_proposals right after propose_rule. |
What worked#
- Rules with verbatim quotes, source, minute and date; bundle line and paths are understood.
- Agents never tried to confirm anything and always pointed to the approval page.
- Supersedes and
as_ofvalidity behave as specified; scope separation held in every case. - Graph questions ("what applies to project X", "what did person Z promise") were answered with paths, dates and sources in 5–17 calls.