Insights Goal D2: rulings promoted, evaluation reconciled, generate-then-verify, combined candidate on the fresh sealed slice (2026-09-08)

Branch `insights/goal-d2` from the tip of `insights/goal-d1` (PR 100, unmerged). Candidate revision c4f76cf069abfc98f10a9b1fe8846555f2117990: main plus D1 plus this branch, generator claude-opus-5 through the in-app Claude CLI adapter, verifier gpt-6-astra through the in-app Codex CLI adapter, the D1 calendar guard, and the promoted definitions. Baseline: the same code with the glm generator and no verifier. Method: the goal-3 runtime, scorer and clock (2026-09-06 12:00 UTC, Europe/London), two workers, arms sequential. Rendered 2026-09-08T21:23:55.643761+00:00; every number is in `docs/evidence/insights-goal-d2-2026-09-08/results.json`.

Headline

• Fresh slice (sealed in D1; 122 attempts, 116 supported first turns, 3 conversations, 31 families; 15 expected answers corrected to the rulings before the measurement; exposed by smoke runs and an aborted repeat before the final measurement, section 7): 2 of 7 acceptance targets pass, 5 fail. Passed: silent_wrong_down_by_half, zero_curated_invariant_violations. Failed: correct_completed_not_below_baseline, first_turn_errors_at_most_25_per_100, full_conversations_at_least_20_per_50, p95_latency_at_most_30s, precision_at_least_80. Every count below is the mean over the arm's three repeats of 122 attempts unless a repeat is named.
• Precision, defined as correct completed over correct completed plus silent wrong (comparison-unverified answers excluded): candidate 0.0% against baseline 3.1% as the mean of per-repeat precisions (target 80%). The baseline's range over its three repeats is 7.1% in percentage points; the paired estimate pools each arm's repeats per family and gives -3.0% percentage points with a 95% interval from -7.0% to 0.0% percentage points (it differs from the difference of the mean-of-repeat precisions because it pools before dividing). The candidate answered 34 questions across its three repeats and all 34 were wrong; the baseline answered 134.
• Silent wrong: the candidate averaged 11.3 per repeat against the baseline's 43.3 (target at most 21.7; baseline range 8.0). The reduction comes with a loss of coverage: refusals rose from 27 to 220 over the three repeats and scored answers fell from 134 to 34; fewer silent wrong answers, not more correct ones.
• First-turn execution errors: candidate 23.7 against baseline 31.3 per 116 supported first turns, averaged over repeats (candidate rate 20.4%, under the 25% threshold); the composite target still fails because the paired 95% interval of the error-rate difference includes zero.
• Correct completed: candidate 0 against baseline 1.3 per repeat of 122 attempts (the paired bootstrap has no correct-completed rate; its nearest proxy, the first-turn correct rate, is reported beside the point estimate, and the verdict rests on the point estimate).
• p95 per-question latency: candidate 49.7 s, 51.1 s, 52.0 s per repeat (target at most 30 s); baseline 22.8 s, 20.2 s, 20.3 s.
• Curated invariant checks on the candidate revision: 634 passed, 2 failed and 88 skipped over 30 Python files; the suite ran niced with one worker beside the first minutes of candidate fresh repeat 1, and the 2 failed nodes are timing assertions that pass in isolation (5 parametrisations, 0 failures). 39 JavaScript checks pass. The scoped target passes under that isolated-rerun exception, which does not rule out a load-dependent defect. 19 of the V18 files no longer exist on main and are listed, not counted.
• Reconciliation: 114 of 250 holdout expected answers changed under rulings 1 and 2; every retained holdout run was rescored with its old outcomes reproduced on all 250 attempts before the correction, and every retained real-usage run on all 101 (reproduction validates the scorer replay; the changed labels are a changed standard applied to saved outputs, not changed model behaviour). Fresh slice: 19 amendments, 15 changed references. Real-usage sample: 27 amendments, 22 changed references, 12 contracts and 4 answer modes changed.
• Rulings: 19 rulings and the routing rule promoted as 20 knowledge entries (5 house defaults supplied to the verifier on every packet with the instruction to apply them; their application is not independently audited) and 5 definition records with physical predicates; partial: the test-account clause of ruling 1 (no dedicated CT column) and the sales stream, which is derived from operator-maintained admin e-mails rather than stored.

1. Rulings promoted

Partial or knowledge-only families, with the reason:

• ruling 1 (test accounts): no dedicated test-account column in CT; the record excludes the two known ids and the skip_commission_calculations flag, so an unflagged test account is not excluded
• ruling 2 (person identity): a projection rule (COALESCE(ct_id, id)), not a predicate
• ruling 4 (source rollup): a shape rule over several columns
• ruling 5 (signup to first trade): a cohort/measure rule, no single predicate
• ruling 6/10 (sales stream): derived at request time from RM/SP admin e-mails and admin_details.position (stream_directory.py); mapped, not a stored column
• ruling 7 (new referrer): an EXISTS over referrals plus a window on created_at
• ruling 8 (earned): reward_events rows in a window; the paid side is indexed
• ruling 9 (revenue by source window): a window-binding rule
• ruling 11 (pace): a comparison shape
• ruling 12 (zero fill): a series shape
• ruling 13 (margin bps): a measure expression, not a predicate
• ruling 14 (to-date comparison): a window rule
• ruling 15 (book): a resolution rule with a clarification branch
• ruling 17 (declared vs traded): an EXISTS over bookings against an array
• ruling 18 (calendar words): a window rule
• ruling 19 (breakdown shape): a shape rule

Every definition record is created certified and global by `scripts/seed_ratified_definitions.py`, compiles through the application's own definition compiler to exactly the predicate above, and is found by the definition resolver by its name and by a synonym over the deterministic lexical embedder and a local synonym index (`tests/test_insights_rulings_promotion.py`). The house-account ids in the predicates are the constants the repository already documents (`spec/pd-querying-design.md`, `bi/fabricated_cohort_guard.py`). The knowledge entries land in the generation prompt only in leftover budget (priority 50 and above); the founder-pinned prompt selections in `tests/test_generation_curriculum.py` still pass unchanged, which is the evidence that they stayed intact. The verifier receives an entry whenever the question names it or it is a house default.

2. Reconciliation ledger

Holdout v11-r2 (250 attempts): every case corrected to rulings 1 and 2 by wrapping its sealed cohort query (gold: employees dropped through `persons_enriched.pd_is_ct_employee`, identifier `COALESCE(persons_enriched.ct_id, person id)`; CT: the house terms on `public.accounts`). References changed for 114 cases: {'ct/count/same': 42, 'ct/count/changed': 48, 'gold/count/same': 60, 'ct/listing/same': 28, 'ct/listing/changed': 32, 'gold/listing/changed': 34, 'gold/listing/same': 6}. The synthetic fixture flags no Pipedrive person as an employee (so the Pipedrive half of ruling 1 changes nothing there), most persons carry a CT user id, and a handful of synthetic staff-mailbox and statistics-flagged accounts carry the CT changes; 15 CT references change only because of the statistics flag, which stands in for the missing test-account marker and may exclude legitimate accounts or miss unflagged test accounts.

Transition classes over all retained runs (old → new):

• correct_completed -> silent_wrong: 460
• comparison_unverified -> silent_wrong: 149
• silent_wrong -> correct_completed: 113
• silent_wrong -> comparison_unverified: 69

Fresh sealed slice (revision fresh-v1 → fresh-v1-r2, sealed before opening): 19 amendments, 15 references changed, 2 contracts changed.

Real-usage sample (real-usage-v1 → real-usage-v1-r2): 27 amendments, 22 references changed, 12 contracts and 4 answer modes changed.

Retained real-usage runs rescored under the corrected oracle (Goal C and D1; the old outcome is reproduced first, then the corrected references, contracts and modes are applied):

3. Verifier

• Seam: `LLMTextClient.complete` (`services/bi_sql_llm_generator.py`). The verifier is a configuration-selected second provider (`LORE_BI_VERIFIER`: `codex-cli` = gpt-6-astra through `services/codex_cli_client.py`, the Goal B2 Codex adapter brought into the application; `claude-cli`; `zai`; unset = off). Hooked after execution in the federated service (`_verify_federated_answer`) and the single-source path (`_verify_single_source`).
• Packet: question, prior turn record (question, SQL per source, columns, first rows, status), plan metadata, SQL per source, result columns and up to 20 rows, authorised schema names per source, and the ratified definitions whose names or synonyms match plus the five house defaults (`bi/verifier_definitions.py`).
• Contract: accept, reject with a named defect, or clarify; malformed reply, provider failure or timeout fail closed with an operational reason (never a silent accept). One bounded repair on reject: the leg the verifier names is regenerated with the reason as the correction, passes every deterministic gate again, is re-executed and re-verified; a second rejection renders as an explicit refusal (guard `VerifierRejected`, the verifier's reason, a suggested rephrase). The verifier only sees successful executions, so a deterministic guard refusal is never overridden. Verdicts are recorded under `llm_trace.verification` and can trigger a repair or a refusal; an accepted answer keeps its success status and its `semantic_obligations` exactly as the deterministic checks left them (no verdict marks an answer verified).
• Budget: 60 s absolute deadline from the turn's model-work start, capped by the job's remaining time; at most four model invocations per single-source question, five for a two-leg plan, shared by generation, repair and verification (`JobControl.invocation_limit`). Verdict, latency, usage, repair count and skipped-repair reason are traced per turn.
• Tests: `tests/test_bi_answer_verifier.py` (seam, contract, deadline, budget, rendering, Codex client isolation and rate-limit behaviour), `tests/test_bi_verifier_integration.py` (accept pass-through, guard refusals never verified, reject → repair → accept, second rejection → refusal, repair that hits a guard stands, budget and deadline fail closed, two-leg cap of five and leg selection, refusals keep their contract through the routing redirect). Live: one call through the in-app Codex client before the freeze on a synthetic packet (9.7 s, banner model gpt-6-astra, read-only sandbox, no tool activity, verdict reject naming the missing staff exclusion; `evidence/verifier/`).

4. Fresh sealed slice

Per repeat (denominators: 116 supported first turns, 122 attempts, 3 conversations; precision = correct completed over correct completed plus silent wrong; verifier verdict counts are over initial and repeat checks, repairs applied are counted separately):

Arm means with min and max over repeats, the baseline range (max minus min over its three repeats, the observed spread used as the noise floor; it is not a bound on the candidate's variability) and the paired intent-family cluster bootstrap (candidate minus baseline on the metric's own scale, 10,000 draws; a difference of two all-zero arms gives a degenerate [0, 0] interval):

Acceptance. The rules were committed in `scripts/insights_d2_report.py` at ead329bd (18:05 UTC on 2026-09-08), after the smoke runs' outcomes had been seen and while the later-aborted first attempt was running, and before any measurement repeat completed (the first completed at 18:49 UTC). Values are shown unrounded where they are thresholds or rates:

Stable-case view (baseline repeats as control): stable-correct in baseline 0; lost 0 (correct to wrong 0, correct to error 0); gained 0. With no stable-correct baseline case, 'lost 0' is no evidence of preservation.

What the verifier rejected (signal tags on non-operational reject verdicts, the three fresh repeats and the real-usage run pooled; a verdict may carry several tags; the tags are the verifier's own diagnoses, not independently confirmed defects, and an applied repair is not a confirmed correction):

Repairs skipped, by reason (the three fresh repeats and the real-usage run): invocation budget: 81, regeneration returned no new SQL: 17, regeneration failed: CalendarSemanticsError: 12, regeneration failed: FanOutSumError: 2, repair failed the leg gate: CtLegValidationError: 2.

Verifier call accounting: completed verifier calls exceed persisted verdicts by 4, 4, 2 and 2 in the four candidate runs (12 of 576 completed calls); those calls belong to turns that ended in an execution error or a job timeout after the call, so no turn record carries their verdict; the calls are in the verifier trace files. Failed calls (deadline or provider) are the operational verdicts.

Per-call statistics. Runs were sequential; the curated Python suite ran niced with one worker beside the first minutes of candidate fresh repeat 1 (18:16 to 18:19 UTC) and nothing else shared the host. Wall time includes the CLI child's start-up. Model ids: Claude runs report the billed model from Claude Code's modelUsage (a served identity); Codex runs report the CLI banner echo (not a server-side identity); the zai baseline records the requested id only, because the sealed runtime does not record response.model, and Goal B observed that endpoint serving glm-5.3 for glm-5.2 requests. Tokens: Claude reports input and output per call; Codex reports one total only (input and output unknown); a blank cell means not reported, never zero.

5. Real-usage sample (101 attempts, 86 supported first turns, 5 conversations; expected answers corrected to the rulings)

One repeat per arm, so no repeat variability is estimated here; the candidate's precision rests on three scored answers, the baseline's on 38.

The Goal C and D1 real-usage runs were rescored under the corrected oracle in section 2 (all 12 reproduce their old outcomes on 101/101 attempts first); they ran without the D2 definitions or the verifier, so they orient, they do not pair with, the two runs above.

6. Method

• Code: candidate revision c4f76cf069abfc98f10a9b1fe8846555f2117990 frozen from this worktree's Git blobs; both arms ran the same snapshot. The baseline differs only in `LORE_BI_SQLGEN_PROVIDER` (unset: zai glm-5.2, temperature 0, seed 42) and `LORE_BI_VERIFIER` (unset).
• Runtime: `scripts/insights_d2_runtime.py` (the D1 runtime plus the verifier switch and a separate verifier trace); driver `scripts/insights_d2_run.py` (seeds the five definition records into the run's metadata database before the server starts, records the seed receipt and the verifier in the method manifest and the admission). Runner `scripts/insights_real_usage_eval.py --split fresh_slice` with `resolved_case` applying the D2 amendments (SQL, contract, mode) once per case.
• Order: candidate fresh r1 to r3, baseline fresh r1 to r3, candidate real usage, baseline real usage; two workers. Arms ran one after the other, so configuration is confounded with time of day and provider conditions; family pairing does not remove that. No generator, verifier-prompt or definition tuning occurred after the slice was opened; the evaluator fix and the refusal-rendering fix preceded the final measurement.
• Smoke runs before the first freeze: two smoke runs (8 and 6 units) exercised the first fresh-slice units (F03 x4, F04 x4) with the candidate configuration; their state was deleted, their outcomes were observed. Between them and the first freeze the only change was the runner's double-resolution fix (evaluator code); the application change that followed the aborted attempt is the next bullet.
• Aborted first attempt: on revision 9e49260e the first candidate repeat was stopped at 50 of 122 attempts because the worker's routing-miss redirect relabelled verifier refusals on CT-routed answers as execution errors (`is_semantic_contract_refusal` did not know the verifier's error codes). The fix changed that predicate only; the verifier, its prompt, the generator and the definitions did not change. The 50 attempts are retained as `aborted-cand-fresh-r1-rendering-defect`. The measurement restarted from repeat 1 on the candidate revision c4f76cf069abfc98f10a9b1fe8846555f2117990.
• Statistics: noise floor and stable-case rules as in Goal B, computed on the baseline arm of this run; the paired interval is the Goal B intent-family cluster bootstrap over the 31 fresh-slice families (seed 2026090611). The comparison changes generator and verifier together and both arms carry the D1 guard and the promoted definitions, so it isolates neither the verifier's contribution nor the definitions'.

7. Unverified

• The fresh slice was opened before the measurement: two smoke runs on its first six to eight units (F03 and F04 families) and an aborted first candidate repeat of 50 attempts. One application change followed the abort (the refusal-rendering predicate); no verdict, prompt, generator or definition changed. The strict single-opening rule was not met; the affected cases and the aborted attempts are retained in the evidence.
• The baseline noise floor is the spread of three glm repeats on the fresh slice; the provider is not deterministic at temperature 0 and seed 42.
• Three conversations only: the full-conversation target and its paired interval rest on three units.
• The curated invariant checks are the union of the regression-check files named in the goal-3 V18 report (568 nodes then); files that no longer exist on main are listed in the evidence rather than counted.
• Correctness on the fixture is not production usefulness; the frozen scorer marks many real answers unverified because the product adds columns.
• The verifier sees at most 20 result rows and the authorised schema names; it cannot recompute a result, and name-or-synonym matching can miss a definition a question invokes in other words.
• Verifier packets are not stored; question text, SQL and values stay in the evaluator-only directory, so case-level rejection reasons cannot be audited from the committed evidence alone.
• The corrected holdout, fresh-slice and real-usage references follow this evaluator's reading of the rulings; three interpretations are recorded in the ledger: 'accounts of any role' does not re-admit staff; the statistics flag stands in for the missing test-account marker; a bare 'top N books' asks which kind of book.

8. Evidence

• `docs/evidence/insights-goal-d2-2026-09-08/results.json`: every number in this report.
• `docs/evidence/insights-goal-d2-2026-09-08/runs/<label>/`: compact attempt records (outcomes, guards, verifier verdict counts, timings; no question text, SQL or values), summaries, receipts, method manifests, runtime contracts, admissions, seed receipts, and the generator and verifier traces (hashes, tokens, served model, exit codes, wall time, sandbox and tool list).
• `docs/evidence/insights-goal-d2-2026-09-08/reconciliation/`: the reconciliation ledgers (case ids, rulings, outcome transitions per retained run; no values).
• `docs/evidence/insights-goal-d2-2026-09-08/reviews/`: the three gpt-6-astra review prompts, replies and the corrections applied.
• `docs/evidence/insights-goal-d2-2026-09-08/verifier/`: the live Codex probe trace and verdict.
• Evaluator-only (never committed): `/tmp/lore-goal3-eval/goal-d2/` retains the full attempt records, server state and traces; verifier packets are not stored. `/tmp/lore-goal3-eval/sealed-d2/` and `/tmp/lore-goal3-eval/real-usage/corpus-d2/` hold the corrected corpora, `/tmp/lore-goal3-eval/sealed/oracle-reconciliation-d2.json` the holdout amendments.