Skip to content

feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans - #35

Draft
apucacao wants to merge 5 commits into
ag/py-telemetry-langchain-messagesfrom
ag/py-telemetry-langchain-agents
Draft

feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans#35
apucacao wants to merge 5 commits into
ag/py-telemetry-langchain-messagesfrom
ag/py-telemetry-langchain-agents

Conversation

@apucacao

@apucacao apucacao commented Aug 11, 2026

Copy link
Copy Markdown

Replaces one flat span per call with the tree the TypeScript SDK emits, for langchain-agents. This is the last of the six handlers.

invoke_agent                     one per call, carries the LD identity and the run total
├── chat {model}                 one per model turn, with that turn's own tokens
└── execute_tool {name}          one per tool call, a sibling of chat

The per-turn data was already being summed from each message's usage_metadata; it now drives a span per turn.

The same two provider-attribute fixes as #34

gen_ai.system is now the literal string langchain rather than the configured provider name: the key names the instrumentation, and here that is the framework.

gen_ai.provider.name is new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated: anthropic when the config says so, openai for everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution.

Other changes

  • Cached tokens are read from usage_metadata.input_token_details and reported per turn, not added to the input figure, which LangChain already reports inclusive of them.
  • Finish reasons go through the shared LangChain helper rather than being dropped, so a turn that stopped to call a tool is distinguishable from one that finished.
  • The streaming path gets a finally, so a consumer that stops reading no longer leaves the root span unended and unexported.

The graph span is untouched. ld.ai.graph and its two attributes already matched the TypeScript SDK and are out of scope.

Breaking change

The span is renamed from langchain.agent to invoke_agent. Queries selecting on the old name will not match. gen_ai.system changes value, as above. Prompt and completion content is no longer on spans unless the caller passes capture_content=True.

Where this sits

Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because gh stack is linear. The oracle above (#36) needs all six.

Tests: 824 to 844.


Note

Overview
Replaces one flat langchain.agent span with the shared agent span tree: an invoke_agent root, one chat {model} child per model turn, and sibling execute_tool {name} spans per tool call. Span bookkeeping moves into a new spans.py module driven by LangChain's callback protocol.

Breaking changes: the root span is renamed from langchain.agent to invoke_agent; gen_ai.system is now always langchain (with a new binary gen_ai.provider.name of anthropic or openai); prompt/completion content is off by default and requires capture_content=True.

Also adds a streaming finally so abandoned consumers still end and export every open span (marked abandoned, not ERROR), reports per-turn finish reasons and cache token details, and rewrites telemetry tests against a real LangGraph agent.

Reviewed by Cursor Bugbot for commit 65fc5f8. Bugbot is set up for automated code reviews on this repo. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py Outdated
@apucacao
apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 082eaaf to 9ccef53 Compare August 11, 2026 20:43
@apucacao

Copy link
Copy Markdown
Author

bugbot run

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated
@apucacao
apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 9ccef53 to 326b8a6 Compare August 11, 2026 21:01
@apucacao

Copy link
Copy Markdown
Author

bugbot run

Comment thread packages/langchain-agents/src/launchdarkly_ai_langchain_agents/spans.py Outdated
One flat span named langchain.agent becomes the tree the TypeScript SDK emits:
an invoke_agent root, one `chat {model}` child per model turn, one
`execute_tool {name}` child per tool call. The per-turn data was already being
summed from each message's usage_metadata; it now drives a span per turn.

BREAKING CHANGE: the span this handler emits is renamed from `langchain.agent`
and `langchain.agent.stream` to `invoke_agent`. Queries selecting on the old
names will not match. Prompt and completion content is no longer on spans unless
the caller passes capture_content=True. gen_ai.system changes value; see below.

gen_ai.system is now the literal string langchain rather than the configured
provider name, matching the TypeScript SDK: the key names the instrumentation,
and here that is the framework.

gen_ai.provider.name is new and is a binary choice rather than a passthrough. It
names who served the model, so it follows the client actually instantiated:
anthropic when the config says so, openai for everything else, including
Bedrock, Azure and an unset value. That mirrors the handler's own model
resolution.

Cached tokens are now read from usage_metadata.input_token_details and reported
per turn. They are not added to the input figure, which LangChain already
reports inclusive of them.

Finish reasons go through the shared LangChain helper rather than being dropped,
so a turn that stopped to call a tool is distinguishable from one that finished.

The streaming path gets a finally, so a consumer that stops reading no longer
leaves the root span unended and unexported.

The graph span is untouched. ld.ai.graph and its two attributes already matched
the TypeScript SDK and are out of scope for this change.

Tests: 54 to 74.
…oes not fail them

The abandonment path reused close_open_spans, which records a synthetic
exception and sets ERROR on every span still open, so an early consumer stop was
indistinguishable from a provider failure in a trace. The comment three lines
above it already claimed the opposite.

Adds abandon_open_spans, mirroring the method openai-agents already had: every
open span ends through end_span_once, staying UNSET and carrying
launchdarkly.stream.abandoned. The failure path keeps its behaviour.

Tested on the callback handler directly rather than through the streaming path.
Reaching the state that matters, a chat or tool span still open at the break,
needs a fake model that yields mid-turn, and with the fixtures here LangGraph
has already run every callback by the time the first chunk reaches the consumer.
My first attempt went through stream() and passed whether or not the fix was
present, which is worse than no test. The test file says so, so the next person
does not repeat it.

Found by Bugbot on #35.
…rapper

The wrapper never passed capture_content to the factory, so it stayed in kwargs
and reached config(), which takes no such argument. A caller asking for content
on spans got a TypeError rather than content.

Lifted out alongside variables, which was already handled the same way and for
the same reason: one configures the handler, the other belongs to the
invocation, and config() accepts neither.

Two tests, one per branch, asserting the flag reaches the factory and does not
reach config().

Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it;
each is fixed in its own layer.
…nreachable

Two mirror-image leaks in the callback handler, both reachable through content
serialisation, which raises on any tool argument or result that is not
JSON-serialisable.

The end callbacks popped the span before doing that work. After the pop nothing
else can reach it, so close_open_spans could not recover it and the span was
never ended: the exporter never saw the turn or the tool call at all. Both now
end it on the way out.

on_tool_start had the reverse problem: it created the span, wrote the arguments,
and only then inserted it into the tracking dict. A raise in between left a span
no cleanup path knew about. It is now tracked first, so every later path can
still close it.

Two tests, each failing on the exact leak when the fix is reverted. The tracer
patch has to stay active while the callbacks run rather than only while they are
built, which is what my first attempt got wrong.

Found by Bugbot on #35.
…t can raise

_start_model created the span, wrote the conversation onto it, and only then
inserted it into the tracking dict. Serialising conversation content raises on
anything that is not JSON-serialisable, and a span created but never inserted is
unreachable by close_open_spans, abandon_open_spans and the end callbacks alike:
it never ends, so the exporter never sees it.

on_tool_start already had this fix. The model-start path is the mirror of it and
did not.

Found by Bugbot on #35.
@apucacao
apucacao force-pushed the ag/py-telemetry-langchain-agents branch from 326b8a6 to 65fc5f8 Compare August 11, 2026 21:18
@apucacao

Copy link
Copy Markdown
Author

bugbot run

"input_tokens": token_usage.get("prompt_tokens")
or token_usage.get("input_tokens"),
"output_tokens": token_usage.get("completion_tokens")
or token_usage.get("output_tokens"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zero tokens dropped in usage fallback

Low Severity

extract_llm_usage maps llm_output token fields with or, so a real 0 for prompt_tokens or completion_tokens is treated as missing. When both are zero, the result becomes all-None, lang_chain_span_usage returns None, and RunUsage never marks the turn as reported even though the provider sent an empty bag.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.

@apucacao

Copy link
Copy Markdown
Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.

)
except Exception as exc:
fail_span(span, exc)
raise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usage recorded after content capture

Medium Severity

In on_llm_end, output content is written before usage is extracted and added to run_usage. If content serialization raises with capture_content=True, the turn’s tokens never reach the run accumulator, so the failure path’s root span omits spend that was already billed.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant