Skip to content

fix(langchain): add provider metadata, route tools to metadata, allowlist span fields#593

Merged
Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
fix/langchain-spec-alignment
Jul 17, 2026
Merged

fix(langchain): add provider metadata, route tools to metadata, allowlist span fields#593
Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
fix/langchain-spec-alignment

Conversation

@starfolkai

@starfolkai starfolkai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Align the LangChain callback handler with .agents/skills/sdk-integrations/SKILL.md. Audit turned up five issues:

  • Missing metadata.provider on every span — spec requires every llm span to carry both metadata.model and metadata.provider. Derive provider at start from serialized["id"] (strip langchain_ prefix + small alias table for google/aws/azure/mistral) and at end from response_metadata.model_provider as a fallback.
  • Tools in invocation_params instead of metadata.tools — spec says tool definitions live in metadata.tools. Lift tools / functions to metadata.tools and tool_choice / function_call to metadata.tool_choice via _split_tools, stripping them from the invocation params that flow into metadata.
  • Denylist → allowlist — dropped the exclude_metadata_props regex (dead code — was set but never applied) and stopped spreading **kwargs and the full serialized runnable dump into span metadata across every callback. Each callback now captures only LangChain user metadata (preserved as metadata.metadata so deepagents tests still resolve ls_integration / lc_agent_name / lc_versions), tags, and known-safe fields.
  • Excess serializationon_tool_start no longer duplicates the tool payload as inputs + input_str + input (parsed) + serialized; keeps only the parsed input. Removed the raw kwargs dump in every error callback.
  • on_agent_action span typeLLMTOOL (the callback fires when the agent picks a tool, not when it calls an LLM).

Also trimmed narrator comments per the skill's "default to no comments" guidance, and merged the two independent _walk_generations passes in on_llm_end into a single _model_and_provider_from_response walk.

The LangChain-native message / LLMResult shape in input / output is left as-is — the Braintrust backend already normalizes it.

Test plan

  • Extended existing VCR-backed tests to assert metadata.provider ("openai" for ChatOpenAI spans, "anthropic" for ChatAnthropic), and metadata.tools presence for the test_tool_usage case.
  • cd py && CI=1 BRAINTRUST_TEST_PACKAGE_VERSION=latest pytest src/braintrust/integrations/langchain/ — 18 passed, 1 skipped.
  • cd py && CI=1 BRAINTRUST_TEST_PACKAGE_VERSION=0.3.28 pytest src/braintrust/integrations/langchain/test_callbacks.py src/braintrust/integrations/langchain/test_anthropic.py — 13 passed, 1 skipped.
  • pylint --errors-only clean.
  • No cassettes re-recorded, no new mocks/fakes — used the existing VCR-backed suite.

🤖 Generated with Claude Code

Created by abhijeet

Slack thread

…list span fields

Align the LangChain callback handler with `.agents/skills/sdk-integrations/SKILL.md`.

- **Missing `metadata.provider`** — spec requires every `llm` span to carry
  both `metadata.model` and `metadata.provider`. Derive provider at start
  from `serialized["id"]` (strip `langchain_` prefix + small alias table for
  google/aws/azure/mistral) and at end from `response_metadata.model_provider`
  as a fallback.
- **Tools in `invocation_params` instead of `metadata.tools`** — lift `tools`
  / `functions` to `metadata.tools` and `tool_choice` / `function_call` to
  `metadata.tool_choice` via `_split_tools`.
- **Denylist → allowlist** — dropped the `exclude_metadata_props` regex (dead
  code — was set but never applied) and stopped spreading `**kwargs` and the
  full `serialized` runnable dump into span metadata across every callback.
  Each callback now captures only the LangChain user metadata (preserved as
  `metadata.metadata` so `deepagents` tests still resolve `ls_integration` /
  `lc_agent_name` / `lc_versions`), tags, and known-safe fields.
- **Excess serialization** — `on_tool_start` no longer duplicates the tool
  payload as `inputs` + `input_str` + `input` (parsed) + `serialized`; keeps
  only the parsed input. Removed the raw kwargs dump in every error callback.
- **`on_agent_action` span type** — `LLM` → `TOOL` (the callback fires when
  the agent picks a tool, not when it calls an LLM).
- Trimmed narrator comments per the skill guidance.

No new mocks, no re-recorded cassettes, no new test files. Existing VCR
suite (`test_callbacks.py`, `test_anthropic.py`, `test_deepagents.py`) covers
every changed path; added positive `metadata.provider` / `metadata.tools`
assertions on the tests whose cassettes exercise them.

The LangChain-native message / `LLMResult` shape in `input` / `output` is
left as-is — the Braintrust backend already normalizes it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 5c0b794 into main Jul 17, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the fix/langchain-spec-alignment branch July 17, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants