feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans - #35
feat(langchain-agents)!: emit invoke_agent, chat and execute_tool spans#35apucacao wants to merge 5 commits into
Conversation
|
bugbot run |
082eaaf to
9ccef53
Compare
|
bugbot run |
9ccef53 to
326b8a6
Compare
|
bugbot run |
One flat span named langchain.agent becomes the tree the TypeScript SDK emits:
an invoke_agent root, one `chat {model}` child per model turn, one
`execute_tool {name}` child per tool call. The per-turn data was already being
summed from each message's usage_metadata; it now drives a span per turn.
BREAKING CHANGE: the span this handler emits is renamed from `langchain.agent`
and `langchain.agent.stream` to `invoke_agent`. Queries selecting on the old
names will not match. Prompt and completion content is no longer on spans unless
the caller passes capture_content=True. gen_ai.system changes value; see below.
gen_ai.system is now the literal string langchain rather than the configured
provider name, matching the TypeScript SDK: the key names the instrumentation,
and here that is the framework.
gen_ai.provider.name is new and is a binary choice rather than a passthrough. It
names who served the model, so it follows the client actually instantiated:
anthropic when the config says so, openai for everything else, including
Bedrock, Azure and an unset value. That mirrors the handler's own model
resolution.
Cached tokens are now read from usage_metadata.input_token_details and reported
per turn. They are not added to the input figure, which LangChain already
reports inclusive of them.
Finish reasons go through the shared LangChain helper rather than being dropped,
so a turn that stopped to call a tool is distinguishable from one that finished.
The streaming path gets a finally, so a consumer that stops reading no longer
leaves the root span unended and unexported.
The graph span is untouched. ld.ai.graph and its two attributes already matched
the TypeScript SDK and are out of scope for this change.
Tests: 54 to 74.
…oes not fail them The abandonment path reused close_open_spans, which records a synthetic exception and sets ERROR on every span still open, so an early consumer stop was indistinguishable from a provider failure in a trace. The comment three lines above it already claimed the opposite. Adds abandon_open_spans, mirroring the method openai-agents already had: every open span ends through end_span_once, staying UNSET and carrying launchdarkly.stream.abandoned. The failure path keeps its behaviour. Tested on the callback handler directly rather than through the streaming path. Reaching the state that matters, a chat or tool span still open at the break, needs a fake model that yields mid-turn, and with the fixtures here LangGraph has already run every callback by the time the first chunk reaches the consumer. My first attempt went through stream() and passed whether or not the fix was present, which is worse than no test. The test file says so, so the next person does not repeat it. Found by Bugbot on #35.
…rapper The wrapper never passed capture_content to the factory, so it stayed in kwargs and reached config(), which takes no such argument. A caller asking for content on spans got a TypeError rather than content. Lifted out alongside variables, which was already handled the same way and for the same reason: one configures the handler, the other belongs to the invocation, and config() accepts neither. Two tests, one per branch, asserting the flag reaches the factory and does not reach config(). Found by Bugbot on #33 against openai-agents. Five of the six wrappers had it; each is fixed in its own layer.
…nreachable Two mirror-image leaks in the callback handler, both reachable through content serialisation, which raises on any tool argument or result that is not JSON-serialisable. The end callbacks popped the span before doing that work. After the pop nothing else can reach it, so close_open_spans could not recover it and the span was never ended: the exporter never saw the turn or the tool call at all. Both now end it on the way out. on_tool_start had the reverse problem: it created the span, wrote the arguments, and only then inserted it into the tracking dict. A raise in between left a span no cleanup path knew about. It is now tracked first, so every later path can still close it. Two tests, each failing on the exact leak when the fix is reverted. The tracer patch has to stay active while the callbacks run rather than only while they are built, which is what my first attempt got wrong. Found by Bugbot on #35.
…t can raise _start_model created the span, wrote the conversation onto it, and only then inserted it into the tracking dict. Serialising conversation content raises on anything that is not JSON-serialisable, and a span created but never inserted is unreachable by close_open_spans, abandon_open_spans and the end callbacks alike: it never ends, so the exporter never sees it. on_tool_start already had this fix. The model-start path is the mirror of it and did not. Found by Bugbot on #35.
326b8a6 to
65fc5f8
Compare
|
bugbot run |
| "input_tokens": token_usage.get("prompt_tokens") | ||
| or token_usage.get("input_tokens"), | ||
| "output_tokens": token_usage.get("completion_tokens") | ||
| or token_usage.get("output_tokens"), |
There was a problem hiding this comment.
Zero tokens dropped in usage fallback
Low Severity
extract_llm_usage maps llm_output token fields with or, so a real 0 for prompt_tokens or completion_tokens is treated as missing. When both are zero, the result becomes all-None, lang_chain_span_usage returns None, and RunUsage never marks the turn as reported even though the provider sent an empty bag.
Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.
| ) | ||
| except Exception as exc: | ||
| fail_span(span, exc) | ||
| raise |
There was a problem hiding this comment.
Usage recorded after content capture
Medium Severity
In on_llm_end, output content is written before usage is extracted and added to run_usage. If content serialization raises with capture_content=True, the turn’s tokens never reach the run accumulator, so the failure path’s root span omits spend that was already billed.
Reviewed by Cursor Bugbot for commit 65fc5f8. Configure here.


Replaces one flat span per call with the tree the TypeScript SDK emits, for
langchain-agents. This is the last of the six handlers.The per-turn data was already being summed from each message's
usage_metadata; it now drives a span per turn.The same two provider-attribute fixes as #34
gen_ai.systemis now the literal stringlangchainrather than the configured provider name: the key names the instrumentation, and here that is the framework.gen_ai.provider.nameis new and is a binary choice rather than a passthrough. It names who served the model, so it follows the client actually instantiated:anthropicwhen the config says so,openaifor everything else, including Bedrock, Azure and an unset value. That mirrors the handler's own model resolution.Other changes
usage_metadata.input_token_detailsand reported per turn, not added to the input figure, which LangChain already reports inclusive of them.finally, so a consumer that stops reading no longer leaves the root span unended and unexported.The graph span is untouched.
ld.ai.graphand its two attributes already matched the TypeScript SDK and are out of scope.Breaking change
The span is renamed from
langchain.agenttoinvoke_agent. Queries selecting on the old name will not match.gen_ai.systemchanges value, as above. Prompt and completion content is no longer on spans unless the caller passescapture_content=True.Where this sits
Needs the usage layer (#28) and the content layer (#29). Independent of the other five handler PRs; the stack orders them only because
gh stackis linear. The oracle above (#36) needs all six.Tests: 824 to 844.
Note
Overview
Replaces one flat
langchain.agentspan with the shared agent span tree: aninvoke_agentroot, onechat {model}child per model turn, and siblingexecute_tool {name}spans per tool call. Span bookkeeping moves into a newspans.pymodule driven by LangChain's callback protocol.Breaking changes: the root span is renamed from
langchain.agenttoinvoke_agent;gen_ai.systemis now alwayslangchain(with a new binarygen_ai.provider.nameofanthropicoropenai); prompt/completion content is off by default and requirescapture_content=True.Also adds a streaming
finallyso abandoned consumers still end and export every open span (marked abandoned, not ERROR), reports per-turn finish reasons and cache token details, and rewrites telemetry tests against a real LangGraph agent.Reviewed by Cursor Bugbot for commit 65fc5f8. Bugbot is set up for automated code reviews on this repo. Configure here.