Skip to content

feat(claude-messages)!: emit invoke_agent, chat and execute_tool spans - #30

Draft
apucacao wants to merge 3 commits into
ag/py-telemetry-content-layerfrom
ag/py-telemetry-claude-messages
Draft

feat(claude-messages)!: emit invoke_agent, chat and execute_tool spans#30
apucacao wants to merge 3 commits into
ag/py-telemetry-content-layerfrom
ag/py-telemetry-claude-messages

Conversation

@apucacao

@apucacao apucacao commented Aug 11, 2026

Copy link
Copy Markdown

Replaces one flat span per call with the tree the TypeScript SDK emits, for claude-messages.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

A multi-turn run with tools was previously one span with one set of token counts, so per-turn cost and latency were not recoverable from a trace, and a tool call left no trace beyond its LD metric event.

What else changed

  • Span construction moved to spans.py beside the handler, so the tool loop reads as a tool loop rather than as span bookkeeping with a provider call in the middle. The five handler PRs above follow this shape.
  • Cache tokens now reach the span. A turn that read 19,971 tokens from cache and wrote 3,580 more previously reported an input of 3; it now reports 23,554, and there is a test with those numbers in it.
  • The handler still returns the cache fields unfolded, in Anthropic's own names, because parse_usage folds exactly once. RawRunUsage carries that shape and is named so it cannot be confused with the client's cache-inclusive RunUsage.
  • Finish reasons are mapped: end_turn becomes stop, tool_use becomes tool_calls.
  • A failed run reports what its completed turns cost, but only when a turn actually reported usage. All-zero attributes would assert the run cost nothing, which a run whose first call died mid-flight cannot claim.
  • The streaming path gets a finally. A consumer that breaks out of the iteration skips except entirely, because GeneratorExit inherits from BaseException, so the root span was never ended and never exported: the whole run vanished from AI Config Monitoring along with the feature_flag event it carries.

The telemetry tests are rewritten rather than extended, because they pinned the old flat span. The single shared mock span is replaced with a recorder that keeps one object per span, since the old approach could not tell a parent from a child.

Not changed: the tool catalog is still unfiltered, unlike the TypeScript SDK. That difference predates this work and changes what the model is offered rather than what the span reports.

Breaking change

The span is renamed from claude.messages to invoke_agent. Queries selecting on the old name will not match. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear.

Tests: 711 to 729.


Note

Overview
Replaces the flat claude.messages span with the TypeScript SDK's span tree for both blocking and streaming paths: an invoke_agent root, one chat {model} child per model turn, and sibling execute_tool {name} spans per tool call.

Span construction moves into a new spans.py. Cache tokens are now folded into span input totals (previously under-counted), finish reasons map to semconv (end_turnstop, tool_usetool_calls), and failed runs still report completed-turn spend on the root.

Breaking: prompt/completion content is off by default; pass capture_content=True to emit it. Streaming adds a finally so abandoned generators still end and export spans instead of vanishing from AI Config Monitoring.

Reviewed by Cursor Bugbot for commit 80c5a66. Bugbot is set up for automated code reviews on this repo. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

One flat span per call, named claude.messages, becomes the tree the
TypeScript SDK emits: an invoke_agent root, one `chat {model}` child per
model turn, one `execute_tool {name}` child per tool call. A five-turn run
with tools was previously one span with one set of token counts, so per-turn
cost and latency were not recoverable from a trace at all, and a tool call
left no trace beyond its LD metric event.

BREAKING CHANGE: the span this handler emits is renamed from
`claude.messages` and `claude.messages.stream` to `invoke_agent`. Queries
that select on the old names will not match. Prompt and completion content is
no longer on spans unless the caller passes capture_content=True.

Span construction moved to spans.py so the tool loop reads as a tool loop
rather than as span bookkeeping with a provider call in the middle.

Tool spans take the root's context, not the chat span's, so they are siblings
of chat rather than nested inside it. Both parents are passed explicitly:
these handlers open a plain span rather than an active one, so there is no
ambient span for a child to inherit, and a host app with its own tracer
provider would otherwise get a flat trace.

The root keeps what only it can carry: the launchdarkly.* identity, the
feature_flag event, and the run's token total. It is the span a config-scoped
query finds, and summing the children requires having already found them. A
test asserts children carry none of it.

Cache tokens now reach the span. Anthropic reports cache reads and writes
beside input_tokens rather than inside it, so a turn that read 19,971 tokens
from cache and wrote 3,580 more reported 3. The chat span now reports 23,554
for that turn, and there is a test with those numbers in it.

The handler's return value keeps the cache fields unfolded, in Anthropic's own
names, because parse_usage folds exactly once; a pre-folded figure returned
alongside the fields would count the cache twice downstream. RawRunUsage
carries that shape and is named so it cannot be confused with the client's
SpanUsage-based RunUsage, which is cache-inclusive.

Finish reasons are mapped rather than passed through: end_turn becomes stop,
tool_use becomes tool_calls. A consumer grouping by this attribute across
handlers previously saw two names for one outcome.

A failed run now reports what its completed turns cost, on the root, but only
when a turn actually reported usage. All-zero attributes would assert the run
cost nothing, which a run whose first call died mid-flight cannot claim.

The streaming path gets a `finally`. A consumer that breaks out of the
iteration makes the generator skip `except` entirely, because GeneratorExit
inherits from BaseException, so the root span was never ended and never
exported: the whole run vanished from AI Config Monitoring along with the
feature_flag event it carries. Every span now ends through end_span_once, and
an abandoned one is marked and left UNSET rather than ERROR, because
LaunchDarkly's own metrics record neither a success nor an error for
abandonment and ERROR would put two dashboards in disagreement about one run.

The success tail sets status without ending, so the `finally` owns every end.
Ending twice is ignored by the OTel SDK but recorded as a diagnostic error,
and would hide a genuine leak.

Tests: the telemetry classes are rewritten rather than extended, because they
pinned the old flat span. The single shared MagicMock span is replaced with a
recorder that keeps one object per span, since the old approach could not tell
a parent from a child. 82 tests here, up from 64. The fake usage object now
declares only the fields Anthropic sets, so a handler cannot read a cache
field the provider never reported.

Not changed: the tool catalog is still unfiltered, unlike the TypeScript SDK,
which offers the model only tools that have a registered handler. That
difference predates this work and changes what the model is offered rather
than what the span reports.
@apucacao
apucacao force-pushed the ag/py-telemetry-claude-messages branch from 0ccb68d to a4e837e Compare August 11, 2026 20:43
@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a4e837e. Configure here.

…apper

The wrapper never passed capture_content to the factory, so it stayed in kwargs
and reached config(), which takes no such argument. A caller asking for content
on spans got a TypeError rather than content.

Lifted out alongside variables, which was already handled the same way and for
the same reason: one configures the handler, the other belongs to the
invocation, and config() accepts neither.

Two tests, one per branch, asserting the flag reaches the factory and does not
reach config().

Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it;
each is fixed in its own layer.
… its span

The success-side content write and the span finish sat outside the try, so a
raise while recording the result skipped both the finish and the failure path.
The tool span was never ended, so the exporter never saw it: the run showed a
root marked ERROR and no sign the tool had been called.

Reachable rather than theoretical. Serialising a tool result raises TypeError
whenever capture_content is on and the result is not JSON-serialisable, which is
any object a handler happens to return.

The TypeScript handler has always done this inside the try. I put it outside when
porting, and the two handlers that copied this file's shape inherited it, so they
are fixed in their own layers.

Found by Bugbot on #34, against the handler that copied it rather than this one.
@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 80c5a66. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant