fix(OPEN-11695): recognize LangChain Gemini provider/model and populate cost + usageDetails#658
Open
viniciusdsmello wants to merge 1 commit into
Conversation
7684bb5 to
eabfd4e
Compare
eabfd4e to
a103ae1
Compare
…sageDetails so LangChain cost estimation stops returning $0
Builds on the v1 callback modernization (OPEN-11315), which resolves the
provider from `metadata["ls_provider"]` (ChatGoogleGenerativeAI -> "Google").
Remaining gaps that still produced $0 / empty cost fields:
- Provider anchor: also map the `chat-google-generative-ai` _type to "Google"
in the legacy _type map, so the exact reported string still resolves when
`metadata["ls_provider"]` is absent (older langchain-google-genai, or callers
that don't forward it) -- not just via the ls_provider path.
- Model: strip the Gemini Developer API "models/" prefix (e.g.
"models/gemini-3.5-flash" -> "gemini-3.5-flash"), which the cost table (bare
names) otherwise misses -> $0. Runs before the LiteLLM prefix handling
("models" is never a provider, so it's safe).
- costDetails/usageDetails: the handler surfaced granular token_details only as
informational step metadata, never as a priced column. Emit a per-category
`usageDetails` map so the backend prices it into `costDetails`. LangChain
reports cache/audio categories as overlapping subsets of the input/output
totals, so they are re-partitioned to be non-overlapping and mapped to the
cost table's keys (cache_read -> cached_tokens, cache_creation ->
cache_creation_tokens, audio -> audio_input_tokens/audio_output_tokens);
reasoning stays folded into output_tokens. ChatCompletionStep gains an
optional usage_details field, serialized only when set.
Verified end-to-end against the live pipeline with the customer's inputs
(27131 prompt / 17739 completion): provider=Google, model=gemini-3.5-flash,
cost=0.2003475, costDetails populated; and with 10000 cached input tokens the
partition prices cached tokens at the cheaper rate (cost=0.1868475). Cross-
provider audit confirms the cost table uses these category keys uniformly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018t3JHc1kK6pPMdVBVwMbTs
a103ae1 to
fc3bfd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LangChain traces reported
$0cost with emptycostDetails/usageDetails. The customer (Globo, Python SDK +ChatGoogleGenerativeAI) sawprovider: chat-google-generative-ai,model: models/gemini-3.5-flash,costDetails: {}. The cost table keys on(provider, model), so an unrecognized provider or model silently yields$0.Closes OPEN-11695.
Context after rebase
Rebased onto
main, which now includes the v1 callback modernization (OPEN-11315). That work resolves the provider frommetadata["ls_provider"](soChatGoogleGenerativeAI→Googlewhenls_provideris present) and captures granulartoken_detailsinto step metadata. This PR closes the gaps that still produced$0/ empty cost fields on top of it.Fix
Provider anchor for the exact reported string. Map the
chat-google-generative-ai_type→Googlein the legacy_typemap, so the string named in this issue still resolves whenmetadata["ls_provider"]is absent (olderlangchain-google-genai, or callers that don't forward it) — not only via thels_providerpath.Strip the Gemini
models/prefix in_extract_model_info—ChatGoogleGenerativeAIreportsmodels/gemini-3.5-flash, which the cost table (bare names) doesn't match, so the lookup misses →$0. Runs before the existing LiteLLM prefix handling;modelsis never a provider, so it's safe.Emit a priced
usageDetailsmap so the backend populatescostDetails. Main only surfacedtoken_detailsas informational metadata; this adds the priced column. LangChain reports cache/audio categories as overlapping subsets of the input/output totals, so they're re-partitioned to be non-overlapping and mapped to the cost table's keys:cache_read→cached_tokens,cache_creation→cache_creation_tokensaudio→audio_input_tokens, outputaudio→audio_output_tokensreasoningstays folded intooutput_tokens(billed at the output rate; no separate price).ChatCompletionStepgains an optionalusage_detailsfield, serialized asusageDetailsonly when set — other integrations are unaffected.Verification
44 tests in
tests/lib/integrations/test_langchain_callback.py(main's + those added here), all passing;ruffclean. Guarded bypytest.importorskip("langchain_core")(langchain is an optional dep). Includes a regression test thatchat-google-generative-airesolves toGooglewithls_providerabsent.End-to-end against a live pipeline with the customer's inputs (27131 prompt / 17739 completion):
providerchat-google-generative-aiGooglemodelmodels/gemini-3.5-flashgemini-3.5-flashcost$00.2003475costDetails{}{input_tokens, output_tokens}populatedWith caching (10 000 cached input tokens):
usageDetails = {input_tokens: 17131, cached_tokens: 10000, output_tokens: 17739}(non-overlapping, sums to total),costDetailspricescached_tokensat the cheaper rate,cost = 0.1868475— correctly below the non-cached0.2003475. A cross-provider audit confirms the cost table uses these category keys uniformly (anthropic/openai/bedrock/azure/… all usecached_tokens/cache_creation_tokens/audio_*).Scope / follow-up
$0rows.google/gemini/vertex_ai;Google+ bare model resolves the reported case and most current Gemini models but not every variant — a general backend normalize-before-lookup layer would close that fully. The same$0symptom also appears on the Langfuse/OTLP ingestion path, untouched here. Both are candidate backend companion tickets.🤖 Generated with Claude Code