Skip to content

fix(instructor): add model/provider metadata, drop __future__ import#592

Merged
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
fix/instructor-provider-metadata
Jul 17, 2026
Merged

fix(instructor): add model/provider metadata, drop __future__ import#592
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
fix/instructor-provider-metadata

Conversation

@starfolkai

@starfolkai starfolkai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Audit of py/src/braintrust/integrations/instructor/ against .agents/skills/sdk-integrations/SKILL.md turned up two deviations. Everything else was already spec-compliant (task-typed parent, no token metrics on parent to avoid double-counting the OpenAI/Anthropic llm child, allowlist-only metadata, no eager model_dump()/bt_safe_deep_copy, no try/except around span.log, instrumentation.name = "instructor-auto", CompositeFunctionWrapperPatcher over the stable instructor.core.client surface, VCR-backed tests, subprocess auto_instrument test).

  • Missing metadata.model / metadata.provider on the parent task span. SKILL's "Framework llm-like spans that delegate to a separately instrumented client" paragraph says the framework span stays task but keeps metadata.model / metadata.provider for attribution. Instructor is exactly that shape — it calls into a wrapped OpenAI/Anthropic client which emits the llm leaf. Added:
    • _model_name(kwargs, instance) — reads kwargs["model"] (Instructor passes it straight through) with instance.default_model fallback.
    • _provider_name(instance) — reads instance.provider.value from Instructor's built-in Provider enum. Value is already the short string we want ("openai", "anthropic", "cohere", "bedrock", "mistral", …). Falls back to .name.lower() for older builds.
    • _build_metadata now emits model + provider alongside the existing 5 keys (response_model, mode, max_retries, retry_count, validation_errors). Still a strict allowlist — no raw kwargs / no denylist.
  • from __future__ import annotations at tracing.py:24 violates CLAUDE.md rule 8 ("Do not add … unless absolutely required"). Only integration doing this. Removed; requires-python = ">=3.10" means str | None and dict[str, Any] already work at runtime unchanged.

Explicitly not changed (verified against SKILL and left alone):

  • Parent span type stays task, not llm — provider integrations own the llm leaf and its token accounting.
  • input still allowlist {response_model, mode, messages}; no dump of raw kwargs.
  • Pydantic outputs still passed through raw — Braintrust serializes at send time; no model_dump() in tracing code.
  • Existing tuple/list unwrap in _extract_output is structural (unwraps create_with_completion's (model, raw_completion) tuple), not serialization.

Test plan

  • cd py && nox -s "test_instructor(latest)" — 8/8 passed (instructor 1.15.4, openai 2.45.0)
  • cd py && nox -s "test_instructor(1.11.0)" — 8/8 passed
  • Extended TestInstructorOpenAISpans.test_instructor_openai_single_success and test_instructor_openai_retries_then_succeeds to assert meta["model"] == "gpt-4o-mini" and meta["provider"] == "openai".
  • Existing VCR cassettes reused unchanged — attribution reads from client instance + kwargs, not the HTTP response.
  • Subprocess auto_instrument test still passes.

🤖 Generated with Claude Code

Created by abhijeet

Slack thread

…ions import

Audit against `.agents/skills/sdk-integrations/SKILL.md` found two deviations:

- Missing `metadata.model` / `metadata.provider` on the parent `task` span.
  Per SKILL "framework `llm`-like spans that delegate to a separately
  instrumented client" — Instructor calls into `wrap_openai`/`wrap_anthropic`
  which own the `llm` leaf — the framework span should stay `task` but
  carry `metadata.model` and `metadata.provider` for attribution. Read
  `model` from call kwargs (`instance.default_model` fallback) and
  `provider` from Instructor's `Provider` enum (`instance.provider.value`,
  which is already the short "openai"/"anthropic"/... string we want).
- `from __future__ import annotations` in `tracing.py` — violates CLAUDE.md
  rule 8. Only integration doing this. Removed; `requires-python = ">=3.10"`
  so `str | None` and `dict[str, Any]` work at runtime unchanged.

No new tests, no cassette re-recording — attribution reads from the
client instance and call kwargs, not the HTTP response. Extended the
two `TestInstructorOpenAISpans` assertions to cover the new keys.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit d3594dd into main Jul 17, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the fix/instructor-provider-metadata branch July 17, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants