Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/api/analyst-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,15 @@ the most recent run in the store (excluding `runId` itself); pass

Strategy for forwarding prior findings into `ctx.priorFindings`.

##### chainFindings?

> `optional` **chainFindings?**: `boolean`

Pass findings produced earlier in this registry run to each later analyst
through `ctx.upstreamFindings`.
Registration order becomes dependency order when enabled.
Disabled by default so independent analyst suites keep their current behavior.

##### knowledgeProposalSource?

> `optional` **knowledgeProposalSource?**: [`KnowledgeProposalSource`](#knowledgeproposalsource)\<`unknown`\>
Expand Down Expand Up @@ -419,6 +428,10 @@ readonly `object`[]

readonly `AnalystFinding`[] \| `Record`\<`string`, readonly `AnalystFinding`[]\>

###### chainFindings?

`boolean`

###### Returns

`Promise`\<`AnalystRunResult`\>
Expand Down Expand Up @@ -520,6 +533,10 @@ readonly `object`[]

readonly `AnalystFinding`[] \| `Record`\<`string`, readonly `AnalystFinding`[]\>

###### chainFindings?

`boolean`

###### Returns

`Promise`\<`AnalystRunResult`\>
Expand Down Expand Up @@ -548,6 +565,10 @@ readonly `AnalystFinding`[] \| `Record`\<`string`, readonly `AnalystFinding`[]\>

readonly `AnalystFinding`[] \| `Record`\<`string`, readonly `AnalystFinding`[]\>

###### chainFindings?

`boolean`

###### Returns

`AsyncIterable`\<`AnalystRunEvent`\>
Expand Down
6 changes: 5 additions & 1 deletion docs/api/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -3028,7 +3028,7 @@ bridge; it only runs the projected inputs and firewalls the merged findings.

> `readonly` `optional` **opts?**: `object`

Optional `run` opts (e.g. `priorFindings`) forwarded verbatim to the registry.
Optional `run` opts (e.g. `priorFindings`, `chainFindings`) forwarded verbatim to the registry.

###### Index Signature

Expand All @@ -3038,6 +3038,10 @@ Optional `run` opts (e.g. `priorFindings`) forwarded verbatim to the registry.

> `optional` **priorFindings?**: readonly `AnalystFinding`[] \| `Record`\<`string`, readonly `AnalystFinding`[]\>

###### chainFindings?

> `optional` **chainFindings?**: `boolean`

***

### Persona
Expand Down
1 change: 1 addition & 0 deletions docs/canonical-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ A general "loop" primitive is the single most common modelling error in this rep
| Render a **multi-profile × multi-axis benchmark leaderboard** (ranked board + score matrix + SVG/HTML charts) from an EXISTING fleet of matrix runs | `leaderboard(records)` + `renderLeaderboardMarkdown` / `renderLeaderboardSvg` / `renderLeaderboardHtml`: `/kernel` (feed it `runProfileMatrix().records`, any domain; `defineLeaderboard` calls these for you) | a per-benchmark report/chart renderer; hand-rolled SVG/markdown tables; a curated subset of axes |
| Attach N observers to a running loop | `composeRuntimeHooks(...)`: root export | a second event-bus or callback-prop zoo (there is ONE stream) |
| Ship traces to an OTLP collector | `createOtelExporter()` + `buildLoopOtelSpans()`: root export | your own OTLP serializer or pulling the OTEL SDK |
| Run an ordered analyst pass where later analysts use findings from earlier analysts | `runAnalystLoop({ chainFindings: true })`: `/analyst-loop`; registration order defines the dependency order, while omission keeps analysts independent | manually invoke each analyst and pipe findings between calls |
| Know **what got mounted into a run** / **why a candidate won** | `result.provenance.mounts` / `result.provenance.selectionReceipts` (`MountManifestEntry`/`SelectionReceipt`/`RunProvenance`); declare mounts via the `recordMount` recorder in `prepareBox`: root export | re-reading box contents to reconstruct what was mounted, or re-deriving which candidate the selector picked |
| State any benchmark/A-B claim | `pairedLift(...)` (bench) over `pairedBootstrap`/`heldoutSignificance` (substrate) | your own bootstrap loop/PRNG per gate; a point lift without `low/high/pairs` |
| Let an agent **delegate ONE generic INTENT** (no fixed coder/researcher type) and get the result + real spend SYNCHRONOUSLY | the **`delegate` tool**: `createDelegateHandler` via `createMcpServer({ delegateSupervisor })`; mount it over the `agent-runtime mcp` bin with `MCP_ENABLE_DELEGATE=1` (the bin authors a supervisor over a `sandbox` backend): `/mcp` | a hardcoded coder/researcher profile, or task-specific `delegate_code`/`delegate_research` verbs (RETIRED): `delegate` is the ONE delegation path and the only one with a cost channel |
Expand Down
1 change: 1 addition & 0 deletions src/analyst-loop/run-analyst-loop.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ async function runRegistry(
const reg = opts.registry as AnalystRegistryStreamingLike
const registryOptions = {
...(priorFindings ? { priorFindings } : {}),
...(opts.chainFindings !== undefined ? { chainFindings: opts.chainFindings } : {}),
...(opts.costLedger ? { costLedger: opts.costLedger } : {}),
...(opts.costPhase ? { costPhase: opts.costPhase } : {}),
...(opts.signal ? { signal: opts.signal } : {}),
Expand Down
9 changes: 9 additions & 0 deletions src/analyst-loop/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,13 @@ export interface RunAnalystLoopOpts {
baselineRunId?: string | null
/** Strategy for forwarding prior findings into `ctx.priorFindings`. */
priorFindingsStrategy?: 'per-kind' | 'wildcard' | 'none'
/**
* Pass findings produced earlier in this registry run to each later analyst
* through `ctx.upstreamFindings`.
* Registration order becomes dependency order when enabled.
* Disabled by default so independent analyst suites keep their current behavior.
*/
chainFindings?: boolean
/** Knowledge-side bridge — usually `agent-knowledge`'s `proposeFromFindings`. */
knowledgeProposalSource?: KnowledgeProposalSource
/** Agent-surface bridge — usually a prompt, skill, or tool diff producer. */
Expand Down Expand Up @@ -135,6 +142,7 @@ export interface AnalystRegistryLike {
inputs: AnalystRunInputs,
opts?: {
priorFindings?: ReadonlyArray<AnalystFinding> | Record<string, ReadonlyArray<AnalystFinding>>
chainFindings?: boolean
[k: string]: unknown
},
): Promise<AnalystRunResult>
Expand Down Expand Up @@ -164,6 +172,7 @@ export interface AnalystRegistryStreamingLike extends AnalystRegistryLike {
inputs: AnalystRunInputs,
opts?: {
priorFindings?: ReadonlyArray<AnalystFinding> | Record<string, ReadonlyArray<AnalystFinding>>
chainFindings?: boolean
[k: string]: unknown
},
): AsyncIterable<AnalystRunEvent>
Expand Down
2 changes: 1 addition & 1 deletion src/runtime/personify/analyst.ts
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ function readAnalystFindings<D>(settled: Settled<Outcome<D>>): ReadonlyArray<Ana
export interface RegistryAnalyzeProjection {
readonly runId: string
readonly inputs: AnalystRunInputs
/** Optional `run` opts (e.g. `priorFindings`) forwarded verbatim to the registry. */
/** Optional `run` opts (e.g. `priorFindings`, `chainFindings`) forwarded verbatim to the registry. */
readonly opts?: Parameters<AnalystRegistryLike['run']>[2]
}

Expand Down
95 changes: 90 additions & 5 deletions tests/analyst-loop.test.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import type {
AnalystFinding,
AnalystRunEvent,
AnalystRunInputs,
AnalystRunResult,
import {
type AnalystFinding,
AnalystRegistry,
type AnalystRunEvent,
type AnalystRunInputs,
type AnalystRunResult,
} from '@tangle-network/agent-eval'
import { describe, expect, it, vi } from 'vitest'
import type {
Expand Down Expand Up @@ -74,6 +75,57 @@ function stubRegistry(
return stub as AnalystRegistryLike & { lastOpts: unknown }
}

async function captureSameRunUpstreamFindings(
chainFindings?: boolean,
streaming = false,
): Promise<{
seen: ReadonlyArray<AnalystFinding> | undefined
emittedEvents: number
}> {
let seen: ReadonlyArray<AnalystFinding> | undefined
let emittedEvents = 0
const registry = new AnalystRegistry()
registry.register({
id: 'first',
description: 'Produces the first diagnosis.',
inputKind: 'custom',
cost: { kind: 'deterministic' },
version: '1',
async analyze() {
return [f('first-finding', 'first')]
},
})
registry.register({
id: 'second',
description: 'Consumes an earlier diagnosis when chaining is enabled.',
inputKind: 'custom',
cost: { kind: 'deterministic' },
version: '1',
async analyze(_input, context) {
seen = context.upstreamFindings
return []
},
})

await runAnalystLoop({
runId: 'run-cur',
registry,
inputs: { custom: { first: true, second: true } },
findingsStore: null,
chainFindings,
...(streaming
? {
onEvent: () => {
emittedEvents++
},
}
: {}),
log: () => {},
})

return { seen, emittedEvents }
}

describe('runAnalystLoop', () => {
it('returns the full loop duration', async () => {
const now = vi.spyOn(Date, 'now')
Expand Down Expand Up @@ -172,6 +224,39 @@ describe('runAnalystLoop', () => {
expect(opts.priorFindings).toBeUndefined()
})

it('passes earlier same-run findings to later analysts when chaining is enabled', async () => {
const { seen } = await captureSameRunUpstreamFindings(true)

expect(seen?.map((finding) => finding.finding_id)).toEqual(['first-finding'])
})

it('passes earlier same-run findings through the streaming registry path', async () => {
const { seen, emittedEvents } = await captureSameRunUpstreamFindings(true, true)

expect(emittedEvents).toBeGreaterThan(0)
expect(seen?.map((finding) => finding.finding_id)).toEqual(['first-finding'])
})

it('forwards an explicit false chain setting', async () => {
const registry = stubRegistry({ findings: [] }, [])

await runAnalystLoop({
runId: 'run-cur',
registry,
inputs: {},
findingsStore: null,
chainFindings: false,
log: () => {},
})

const opts = registry.lastOpts as { chainFindings?: boolean }
expect(opts.chainFindings).toBe(false)
})

it('keeps same-run findings isolated by default', async () => {
expect((await captureSameRunUpstreamFindings()).seen).toBeUndefined()
})

it('baselineRunId:null skips diff entirely', async () => {
const store = inMemoryStore()
await store.append('run-prev', [f('f-old', 'failure-mode')])
Expand Down