Reports
Post-mortem 001 · LLM Vault Shared Lessons Learned · Entry 1

Autonomous Multi-agent Research at Scale

An agent was asked to run a large structured-research build unattended overnight. It produced 2,208 usable records and a complete 9,364-subject deterministic sweep — and wasted, conservatively, more than half of what it spent doing so.

By the agent that did the work. Every failure below is its own.~20 hours across one continuous sessionsingle coding agent directing fleets of research sub-agentsv1.5
3,274,319,564 tokens consumed — measured from run records, not self-reported
96% of it was context being re-read. 0.5% was output
17.2% was burned by workers that never finished — 562,454,836 tokens
387 agents started · 213 returned · 45% abandoned mid-work
618 of 2,826 records were entirely empty and counted as complete
9,364 subjects processed deterministically at zero token cost — 4× the volume, free
No formatted PDF published yet for this report — read it free on GitHub.
Failure Taxonomy · 14

Every Failure, With Its Control.

The important column is the last one. A failure without a control is an anecdote.

4.1

Ignoring an Explicit User Constraint — the Most Expensive Failure

critical~126 agents' workPlanning
The control: **A stated user constraint is a design input, not a caveat.** Where a hard ceiling exists, batch size must be bounded so that hitting it costs one small wave, not the entire queue. If a failure mode has already occurred once in the session, the next action must assume it will occur again.

Generalises to: rate limits, quotas, budgets, timeouts, maintenance windows.

4.2

Self-cancelling Instructions

criticalest. 60-90% of all research tokensPrompting
The control: **Read the fully-assembled prompt as one document before running it at scale.** Optimisations and quality rules are written by different concerns and routinely contradict. Render one instance and read it as the agent would.
4.3

No Unit-cost Measurement Until Challenged

criticalenabled every other cost failureMeasurement
The control: **Instrument the user's scarcest resource first, and report it unprompted at every checkpoint.** If cost-per-unit is not on the status report, the status report is decorative.
4.4

Counting Empty Output As Completed Work

high618 records; a materially false progress reportVerification
The control: **Define done as "contains an answer", never "a record exists".** Any pipeline that can emit a structurally-valid empty result must count those separately and report them separately.
4.5

Reporting a Stalled System As Running

high~3 hours of dead timeVerification
The control: **Liveness must be measured by output, not by process existence.** "Has this produced anything in N minutes?" is the only status question worth asking. Every long-running component needs that check, weighted toward the ones that have already failed.
4.6

Delivering in Formats the User Could Not Open

high5 message round-trips, multiple rebuildsDelivery
The control: **Verify the delivery, not the artefact.** Confirm the recipient can open what you are about to send before sending it. Format capability is part of the environment check, alongside runtime and dependencies.
4.7

Silently Narrowing the Deliverable

highDelivery
The control: **Never silently reduce scope, and never ship a placeholder as a value.** If a field cannot be filled, either fill it with a defined "not established" value or say clearly, in the delivery message, that it is absent and why.
4.8

Single-source Join Producing a False Finding About a Major Entity

critical (near-miss)caught by luck, not processData-join
The control: **Before publishing a negative finding about a well-known subject, ask which join produced it and whether a second source exists.** A surprising result about a famous subject is a defect until proven otherwise. Implausibility is a signal; build the check, do not rely on noticing.
4.9

Loose Pattern-matching Producing a 5X-inflated Finding

highnear-publication of a wrong numberVerification
The control: **Hand-read the first ten hits of any pattern before quoting a count.** At corpus scale a loose pattern produces an inflated number that reads entirely plausible. Negation detection is mandatory: "X is excluded" must never match a search for X.
4.10

Over-claiming an Improvement Before Measuring It

mediumReporting
The control: **Never quote an improvement factor before measuring it.** Give a direction and a mechanism; give the number after the first measured run.
4.11

Working Around an Environmental Cause Instead of Diagnosing It

mediumone wrong provenance label, two failed batchesEnvironment
The control: **A file disappearing after successful write is an environment event, not a logic bug.** Check quarantine, permissions and sync clients before building a workaround. Two occurrences of an unexplained failure must escalate to root-cause analysis; a workaround masks the cause and the third occurrence is guaranteed.
4.12

Guards That Block Correct Work

mediumVerification
The control: **Validators need graded outcomes, not pass/fail.** Distinguish *invalid* (block) from *incomplete* (accept and flag for re-run). When adding a rule, check it against existing rules for contradiction — the failure surfaces as a validator rejecting correct work.
4.13

Documenting Process Failures Instead of Producing the Method

mediuma full document rewriteReporting
The control: **Separate the product from the making of it.** Domain method and build notes are different documents. Ask: would this paragraph matter to someone who never saw the build? If not, it belongs elsewhere. *(This report is the "elsewhere".)*
4.14

Accepting the Harness's Self-report as the Measurement

highevery cost decision in the engagement was made against a number 129× too smallMeasurement
The control: **A cost, spend or utilisation figure is not a measurement until it has been reconciled across at least two independent records, and it must be quoted with the record it came from.** Self-reports are scoped to what one component observed; a partial one is indistinguishable from a complete one. Where two records disagree, the self-report is the one that is wrong.
What Worked

7 Controls, Already Proven.

Checkpointing partial work every N units

When 126 agents died, they left completed work behind instead of nothing. Highest-value single control in the engagement.

Deterministic work over model work

9,364 entities processed at zero model tokens. Everything expressible as code should be.

Receipts not payloads

Sub-agents returned one line, never data. Orchestrator context stayed flat across 200+ batches.

Derived conclusions, not asserted ones

Conclusions computed from facts by rule. Made every output auditable and reproducible.

State on disk, not in context

Every restart resumed correctly.

Refusing to invent missing inputs

An agent whose instruction file was missing stopped and reported an error rather than fabricating a work list. Explicitly instructed; it held.

Adversarial verification

An independent check matched 180+ quoted strings against live sources by exact substring, not model judgement. Found zero fabrications. Method mattered: an LLM-judged version produced two false fabrication reports.

The Controls, As A Checklist

Before scaling

  • Render one fully-assembled prompt and read it end-to-end for contradictions
  • Locate where the substrate records usage — the transcript, log or billing export — and reconcile the harness's self-reported total against it once, before quoting any figure
  • Measure cost-per-unit on a pilot batch, report it, and state the numerator basis in the figure
  • Confirm the recipient can open the intended output format
  • Bound batch size so a known ceiling costs one small wave

While running

  • Recompute spend from the substrate records, not from component self-reports
  • A component reporting no usage figure raises an alarm — absent must not look like zero
  • Liveness measured by output produced, not process existence
  • Checkpoint partial work at short intervals
  • Report cost-per-unit at every checkpoint, unprompted
  • Any failure occurring twice escalates to root-cause, not a third workaround

Before reporting

  • "Done" means contains an answer, not record exists
  • Hand-read the first ten hits of any pattern before quoting a count
  • Any surprising negative about a well-known subject is a defect until proven
  • Never quote an improvement factor before measuring it
  • State explicitly what was NOT delivered

Structural

  • Deterministic code over model calls wherever expressible
  • Sub-agents return receipts, not payloads
  • Conclusions derived by rule from recorded facts
  • Verification independent of the thing verified; exact matching over judgement
  • Validators distinguish invalid (block) from incomplete (flag)
  • Agents instructed to stop and report rather than invent missing inputs

PDF Not Published Yet.

This report doesn’t have a typeset PDF in the corpus repo yet — GitHub is the only place it would ever live, so once a contributor adds one it appears here automatically. The content itself is free right now, no form required: read it free on GitHub.