docs(ai): add LLM observability page - #4568
Conversation
The overview said an in-progress chat resumes on the new version after a redeploy, which contradicts the version-upgrades and backend pages. Chat agent runs are pinned to the version they started on; moving onto new code is an explicit version upgrade.
Document capturing Vercel AI SDK calls as spans in the run trace: turning it on per call, what each span inspector tab shows, linking a call to its prompt version, and querying usage across runs with TRQL.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (3)
WalkthroughAdded AI observability documentation for Vercel AI SDK telemetry, span inspection, prompt linking, custom metadata, and TRQL usage queries. Added the page to the Agents features navigation. Updated chat deployment documentation to describe version pinning during redeploys and explicit upgrades. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
AI SDK 7 auto-registration of @ai-sdk/otel only happens for chat agents, so plain tasks must register the OpenTelemetry integration themselves. Also note that chat agents auto-capture only when a prompt is set.
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
Summary
Adds a docs page for LLM observability: every opted-in Vercel AI SDK call inside a task becomes its own span in the run trace, carrying the model, provider, token counts, cost, and latency. The page covers turning it on per call with
experimental_telemetry: { isEnabled: true }, what each span inspector tab shows (Overview, Messages, Tools, and a Prompt tab when linked), linking a call to its prompt version withtoAISDKTelemetry(), and querying usage across runs with TRQL against thellm_metricstable.It sits in the AI dropdown under Features, next to Prompts, and cross-links the Query page.
It is explicit that capture is opt-in per call (not automatic) and only covers Vercel AI SDK calls, and notes the
@ai-sdk/otelrequirement on AI SDK 7. Every API name, span tab, and TRQL column was checked against the SDK and the live query schema.Also in this PR
Corrects one bullet in the AI Agents overview: it claimed an in-progress chat resumes on the new version after a redeploy, which contradicts the version-upgrades and backend pages. Chat agent runs are pinned to the version they started on; moving onto new code is an explicit version upgrade.