Skip to content

feat(ai): instrument Agent Inspector funnel telemetry - #9580

Open
Dooriya Li (dooriya) wants to merge 3 commits into
Azure:mainfrom
dooriya:dooriya-instrument-inspector-route-telemetry
Open

feat(ai): instrument Agent Inspector funnel telemetry#9580
Dooriya Li (dooriya) wants to merge 3 commits into
Azure:mainfrom
dooriya:dooriya-instrument-inspector-route-telemetry

Conversation

@dooriya

@dooriya Dooriya Li (dooriya) commented Aug 14, 2026

Copy link
Copy Markdown

Summary

  • report local_client.route.selected from azure.ai.agents after service/profile resolution, with ext.route=inspector|playground|suppressed
  • report inspector.funnel.stage from azure.ai.inspector when the SPA sends setViewReady, with ext.stage=ui_ready and ext.outcome=succeeded
  • emit UI readiness at most once per Inspector process and preserve the originating trace context
  • keep all telemetry best-effort; UI readiness does not imply agent connection or request success
  • update extension SDK requirements and shared telemetry documentation

Fixes #9585

Validation

  • go test ./... -count=1 in cli/azd/extensions/azure.ai.agents
  • go test ./... -count=1 in cli/azd/extensions/azure.ai.projects
  • go test ./... -count=1, go build ./..., go vet ./..., and go mod verify in cli/azd/extensions/azure.ai.inspector
  • targeted core TelemetryService tests in cli/azd/internal/grpcserver
  • local --trace-log-file verification for all three route values
  • real browser launch verified one accepted Inspector ext.usage span for ui_ready
  • a second setViewReady notification produced no duplicate span

Telemetry Change Checklist

Privacy review status: Required and pending approval before merge. The extension fields use fixed product-defined enums classified as SystemMetadata for FeatureInsight; they contain no user-derived or customer content.

New Fields

  • Core field defined in fields/fields.go; extension-supplied ext.* values are covered by the extension's own review instead — this PR uses ext.route, ext.stage, and ext.outcome
  • Fields have the correct classification and purpose — SystemMetadata / FeatureInsight
  • Fields documented in docs/specs/metrics-audit/telemetry-schema.md
  • Hashing applied where required — N/A; all values are fixed product-defined enums
  • Measurement fields use correct OTel type — N/A; these are string enums
  • Enum values documented with allowed value sets
  • Extension enum values contain no user-derived or customer content

New Events

  • Event constant defined in events/events.go — N/A for extension-owned events; constants are defined in their owning extensions
  • Events documented in docs/specs/metrics-audit/telemetry-schema.md
  • Events follow naming convention (prefix.noun.verb)

Privacy

  • Classification assigned using decision tree
  • No CustomerContent emitted in telemetry
  • No unhashed user-provided values
  • No PII in string attributes (names, emails, paths)
  • Privacy review triggered (if required per triggers above) — required and pending approval before merge

Testing

  • Unit tests verify event attributes, once-only UI-ready emission, non-fatal telemetry failure, and context forwarding
  • Integration test confirms end-to-end emission
  • Verified fields appear correctly in local telemetry output

Downstream

  • LENS job updated (if field is queried in dashboards) — N/A; no existing LENS consumer references these events
  • Kusto function updated (if field is used in cooked tables) — N/A; ext.usage already supports dynamic ext.* fields
  • Dashboard updated (if field powers a new metric) — N/A for this PR; downstream BI work is separate

Documentation

  • Feature-telemetry matrix updated (docs/specs/metrics-audit/feature-telemetry-matrix.md)
  • Telemetry schema updated (docs/specs/metrics-audit/telemetry-schema.md) with the new fields/events
  • Public telemetry reference and extension READMEs updated
  • Hashed-field table updated if needed — N/A; these fields are not hashed
  • This checklist is complete; privacy approval remains pending before merge

@dooriya
Dooriya Li (dooriya) force-pushed the dooriya-instrument-inspector-route-telemetry branch from d85f04c to cd09d23 Compare August 14, 2026 02:52
@dooriya Dooriya Li (dooriya) changed the title Instrument local client route selection feat(ai-agents): instrument local client route selection Aug 14, 2026
Report categorical Inspector, Playground, and suppressed routes through extension telemetry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds local-client route telemetry for the Azure AI Agents extension.

Changes:

  • Reports inspector, playground, or suppressed route selection.
  • Adds route-selection tests and updates the azd SDK dependency.
  • Documents the new telemetry contract.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
docs/specs/metrics-audit/telemetry-schema.md Defines the event contract.
docs/specs/metrics-audit/feature-telemetry-matrix.md Adds routing telemetry coverage.
docs/reference/telemetry-data.md Documents the public telemetry field.
cli/azd/extensions/azure.ai.agents/README.md Explains route telemetry semantics.
cli/azd/extensions/azure.ai.agents/internal/cmd/run.go Emits route-selection telemetry.
cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go Tests route values and error handling.
cli/azd/extensions/azure.ai.agents/go.mod Updates the azd SDK requirement.
cli/azd/extensions/azure.ai.agents/go.sum Updates dependency checksums.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cli/azd/extensions/azure.ai.agents/internal/cmd/run.go
Keep the Agents and Projects extensions on azd SDK v1.31.0 as required by their synthesis parity check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
6 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 14, 2026 08:09
@dooriya Dooriya Li (dooriya) changed the title feat(ai-agents): instrument local client route selection feat(ai): instrument Agent Inspector funnel telemetry Aug 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 18 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/specs/metrics-audit/telemetry-schema.md:255

  • Fixes #9585 is not satisfied by this contract. The issue also requires server/browser, agent-connection, first-request/response milestones and stable failure reasons, but this PR implements only route selection and ui_ready. Please either implement the remaining funnel stages and failures before closing #9585, or stop auto-closing it and track the remaining work separately. [azd-code-reviewer]
| `azure.ai.inspector` | `inspector.funnel.stage` | The Inspector SPA sends `setViewReady` after mounting | `ext.stage`: fixed enum `ui_ready`; `ext.outcome`: fixed enum `succeeded`; this does not indicate agent connection |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue] Complete Agent Inspector funnel telemetry and failure diagnostics

3 participants