Lexical search (stage a)#

Stage a is the search stage that always runs: local, offline, deterministic and free. Stages b (vectors) and c (reranker) only reorder or add to what it finds, so its recall bounds the whole search. This module (src/search/lexical/) replaces steps 1–4 of the stage-a contract (architecture section 7) with measures that can each be switched on and off. With all switches off it reproduces the contract (golden test).

Recommendation#

Defaults are DEFAULT_FEATURES in options.ts; CONTRACT_FEATURES is the contract behaviour.

Results#

Run pnpm eval:lexical (all invented sets) and pnpm eval:lexical -- --scale 20000 (latency and size). Last output: eval/lexical/results/latest.md. A local real-data set can be added with -- --real <file>; nothing is then written to disk.

Test sets#

Metrics: R@k = share of relevant rules (grade ≥ 1) in the first k; MRR of the first relevant; nDCG@5 with grades 1–2; false alarms = share of no-answer queries whose first hit has coverage ≥ 0.5; stale first = share of validity queries where a superseded rule ranks above the best relevant one. Proposals are not in the pool (contract default).

Per measure — held-out set (not used for tuning), 300 rules, 137 queries#

Held-out by category (R@10 / MRR), baseline → default: code 0.93/0.94 → 0.96/0.94; compound 0.87/0.86 → 0.87/1.00; cross-lingual 0.29/0.21 → 0.36/0.31; direct 0.79/0.96 → 0.83/0.96; English 0.85/0.88 → 0.92/1.00; inflection 0.75/0.81 → 0.80/0.76; long mail 0.19/0.40 → 0.56/0.76; paraphrase 0.50/0.42 → 0.54/0.61; typo 0.77/0.72 → 0.90/0.88; validity 0.88/0.56 → 0.88/0.93.

All sets, baseline → default#

Real data, R@5: 0.59 → 0.70 (keyterms alone 0.73). Forbidden rules in the top 5 (20 queries name rules that must not appear): 0.30 → 0.40 — a better pool brings more near neighbours; stage c or the filter has to sort them out. JEV pair verdicts (paired rule in the top 10 of 72): tricky German pairs 0.14 → 0.29, tricky English 0 → 0, unrelated German 1.00 → 0.93; the paraphrase pairs stay stage c's job.

Abstention (asked for by hardening: no-answer queries always returned hits)#

Stage a would return nothing when its first hit is too weak. Two signals were measured: coverage (IDF share of the query terms the hit matches) and matchedTerms (count of matched terms).

A lexical threshold cannot tell "no rule" from "rule in other words": long texts match few of their terms even when the rule is right. Recommendation: no abstention in stage a by default; return coverage and matchedTerms with every candidate (done) so core can label weak results, and let stage c abstain. For a stage-a-only setup that must not answer wrongly, matchedTerms ≥ 3 is the best measured rule.

Latency, size, memory at 20,000 rules#

Synthetic rules built from the 300 (halves of two statements plus invented compounds, names and article numbers, so the vocabulary grows to ~98,000 words), file database in WAL mode, all 180 own queries including 20 long mails, fastest of three runs per query. Apple M5 Pro shared with other work (load average 6–16): performance cores, and efficiency cores (taskpolicy -b) as a stand-in for a small server.

• p95 < 50 ms holds for the default on both core types; long mails on efficiency cores are at the limit in wall time (other load on the machine) and at 33 ms in CPU time.
• Memory: the vocabulary cache (word and stem counts, bigram index for typo candidates) is the JS heap above: about 37 MB at 98,000 words. Disk: 33 MB for the lex_* tables (1.7 KB per rule), plus core's own rule_fts.
• Build: ~10 s for 20,000 rules in one transaction (performance core); per rule well under 1 ms.
• Writes do not reload the vocabulary: only the counts of the touched terms are re-read (also correct after a rollback), so the first search after a write costs 7 ms instead of ~50 ms.

How it works#

text / terms ──► terms (7.1)  ──┬─► word   : contract query, prefix variants (unicode61)        ─┐
  key terms for long texts      ├─► stem   : Snowball de+en stems, compound parts                │
  code/unit tokens              ├─► fuzzy  : edit-distance corrections against the vocabulary    ├─► score fusion ─► validity ─► pool (200)
                                ├─► syn    : other members of matching synonym groups            │                   weighting    found_via, coverage
                                ├─► tri    : substrings of rare words, trigram index (off)       │
                                └─► prf    : terms of the first hits (off)                      ─┘

Every sub-query is an FTS5 MATCH built only from quoted literals ("…" with inner quotes doubled, optional *); control characters are replaced first (NUL would end an FTS5 string). FTS syntax in user text (", NEAR, *, :, AND, ^) is inert (tested for text and terms). Each sub-query returns its bm25 top 200 with field weights keywords 4, statement 3, relations 2, quote 1, restricted to confirmed/superseded (plus proposed with includeProposals). Score fusion adds, per sub-query, weight × bm25 / best bm25 (weights word 1, stem 1, syn 0.6, fuzzy 0.5, tri 0.5, prf 0.3); ties break by seq, so results are deterministic.

Each candidate carries ranks (rank per sub-query), foundVia (contract step 6, see below), coverage and matchedTerms (see abstention).

foundVia names words, never bare stems, so it reads as an explanation and not as a typo:

• a query word that matched by itself, its inflection, stem, ä/ae spelling or a substring appears as the query spells it (frame → frame, not fram; Zählerschränke → zählerschränke);
• a compound part appears as the query word spells it (Wärmepumpenförderantrag → antrag, förder, wärmepumpen), but only when the whole word did not match the rule;
• a typo correction appears as the corrected index word (Walbox → wallbox), again only when the query word did not match by itself; the index stores folded text, so a correction carries no umlauts (zahlerschrank);
• synonym expansions and configured synonyms stay as the phrase (Ladestation → wallbox);
• given multi-word terms stay phrases (wallbox installation).

The list is sorted and free of duplicates; it is display only and does not change ranking or coverage.

Learned synonyms (suggestions only)#

suggestSynonyms(db) compares quote and statement of every confirmed rule: words of the quote (how people say it) that the statement expresses differently (how the rule says it). Pairs seen in at least two rules with NPMI ≥ 0.3 are returned with support and example rules; inflections, participles and compounds of the same word are filtered out. Nothing is applied: a person adds a pair to the scope's search.synonyms. On the 300-rule set about one in six of the top 25 suggestions is useful (e.g. "extra" → "Aufpreis", "Propan" → "Kältemittel"); the method needs more rules than that to be precise, so it stays a suggestion list for review.

Interface for core#

import { LexicalIndex } from "./search/lexical/index.js";

const lexical = new LexicalIndex(db, { synonyms: config.search.synonyms, languages: config.search.languages });
// in the transaction that writes a rule, next to reindexRule():
lexical.upsert({ seq, ...ftsRow(current), status, validFrom, validUntil }); // rejected → lexical.remove(seq)
// stage a, per scope:
const { terms, variants, candidates } = lexical.search({ text, terms, today, includeProposals });
// candidates: [{ seq, score, coverage, matchedTerms, foundVia, ranks }], best first, ≤ 200

• One LexicalIndex per connection. The tables carry a fingerprint of the content-relevant features and the analyzer version; opening with other options throws LexicalIndexMismatchError, and rebuild(docs) re-indexes in one transaction.
• Core keeps steps 5–9 (filter, successor replacement, dedupe, cross-scope interleave, log).

Contract changes this needs (in architecture.md first, by core):

• 4.1: tables lex_word, lex_stem, lex_tri, lex_doc, lex_meta and two fts5vocab views next to rule_fts (derived data, rebuildable from current_version).
• 7.1: long texts use key terms (at most 6) instead of the first 12 words; code/unit tokens.
• 7.4: rank.a is the fused score (higher is better) instead of −bm25.
• 7.6: found_via also lists compound parts, corrections and synonym phrases; stem and spelling matches name the query word, not the stem.
• 3: the base synonym list ships with the code; search.synonyms adds to it.

Licenses#

Not used, and why: OpenThesaurus (database dump LGPL 2.1, not for bundling in an Apache project), Open German WordNet / OdeNet (CC BY-SA 4.0, share-alike), GermaNet (research licence), fastText vectors (CC BY-SA 3.0), hunspell de_DE / igerman98 (GPL/LGPL). Princeton WordNet's licence is permissive, but it is English only and the measured gaps are German compounds and long texts. Small word vectors and co-occurrence expansion were not built: pseudo-relevance feedback (the cheap co-occurrence variant) gave nothing, and every vector set with a fitting licence is English-only or share-alike.

Limits#

• Invented data, written by agents; the defaults were tuned on the own set. The held-out set and the 52 real queries are the honest numbers.
• Paraphrases without a shared word and cross-lingual questions outside the synonym list stay weak (held-out cross-lingual R@10 0.36): that is stage b/c's job.
• Latency comes from one shared machine; efficiency cores are a stand-in, not a measured server.