Skip to content

[HDX-5090] Support creating alerts without saved searches or dashboard tiles (backend) - #3010

Open
wrn14897 wants to merge 2 commits into
mainfrom
warren/HDX-5090-detached-alerts-backend
Open

[HDX-5090] Support creating alerts without saved searches or dashboard tiles (backend)#3010
wrn14897 wants to merge 2 commits into
mainfrom
warren/HDX-5090-detached-alerts-backend

Conversation

@wrn14897

Copy link
Copy Markdown
Member

Summary

Backend foundation for detached alerts (HDX-5090): a new chart alert source that persists its own chart config directly on the alert document, so alerts no longer require a saved search (logs) or a dashboard tile (metrics). This unblocks customers (e.g. Epidemic Sound, migrating 1000+ Grafana alert rules) for whom creating a saved search or dashboard tile per alert does not scale.

The persisted config is the exact shape a dashboard tile stores (SavedChartConfig minus the embedded alert field), so chart alerts evaluate through the same battle-tested code path as tile alerts:

  • common-utils: AlertSource.CHART, exported AlertChartConfigSchema (builder + raw SQL, no PromQL), zChartAlert, new AlertSchema union member, and optional chartConfig on AlertsPageItemSchema.
  • Model: chartConfig (Mixed) on the Alert document; makeAlert persists it for chart alerts and clears it when the source changes (mirrors savedSearch/dashboard reference clearing). No migration needed.
  • Internal API: new internalAlertSchema accepts the chart source; validateAlertInput enforces supported display types (Line/Stacked Bar/Number), validates raw SQL templates, and checks team-scoped source/connection ownership. Alert responses include chartConfig.
  • check-alerts task: new AlertTaskType.CHART; the tile-alert config assembly is factored into a shared buildAlertChartConfigFromSavedConfig used by both tile and chart alerts, so group-by, multi-window, formulas, ratio mode, and raw SQL behavior are identical. Notifications link to the chart explorer seeded with the alert's config over the alerting window, and default their title to the config's name.

Deliberately out of scope (follow-ups):

  • Creation/edit UI (chart explorer create flow; full chart editor on the alert details page)
  • External API v2 + MCP write support — the v2 API keeps the narrower alertSchema and rejects source: 'chart' (guarded by a test) until its OpenAPI/Terraform contract is extended. v2 GETs echo chart alerts read-only.
  • Terraform/IaC: chart alerts are already excluded by isImportableAlert.

How to test on Vercel preview

N/A — non-UI change (backend only; no UI creates chart alerts yet).

Testing done:

  • make ci-lint, make ci-unit — pass
  • Integration: routers/api/alerts.int (60 tests, incl. new chart-alert CRUD/validation), full checkAlerts.int (176 tests, incl. new end-to-end chart alert evaluation + grouped notification + template link/title), external-api/alerts.int (54 tests, incl. the v2 rejection guard), checkAlerts/providers/default.int (33 tests)

References

  • Linear Issue: HDX-5090
  • Related PRs: N/A

Add a 'chart' alert source that persists its own chart config directly on
the alert document, so alerts no longer require a saved search (logs) or a
dashboard tile (metrics). Builder configs on log/trace/metric sources plus
raw SQL (Line/StackedBar/Number) are supported; PromQL is rejected.

- common-utils: AlertSource.CHART, AlertChartConfigSchema, zChartAlert,
  AlertSchema union member, chartConfig on AlertsPageItemSchema
- model: chartConfig (Mixed) on Alert; makeAlert persists/clears it like the
  other source references
- internal API: internalAlertSchema accepts the new source (external v2
  keeps the narrower alertSchema until its contract is extended);
  validateAlertInput checks display type, raw SQL template, and team-scoped
  source/connection ownership; responses include chartConfig
- check-alerts: new CHART task type evaluated through the same code path as
  tile alerts (shared buildAlertChartConfigFromSavedConfig), including
  group-by and multi-window behavior; notifications link to the chart
  explorer seeded with the alert's config and default their title to the
  config's name

Backend only; the creation/edit UI and external API v2 support land
separately.
@wrn14897 wrn14897 added the ai-generated AI-generated content; review carefully before merging. label Aug 26, 2026
@changeset-bot

changeset-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9bdd3e8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Minor
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview Aug 27, 2026 5:55am
hyperdx-storybook Ready Ready Preview Aug 27, 2026 5:55am

Request Review

@greptile-apps

greptile-apps Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds backend support for detached chart alerts, including persistence, internal API validation, evaluation, and notification links. The source/connection fix needs canonical ObjectId comparison so valid equivalent identifiers are not rejected.

  • Adds builder and raw-SQL chart alert schemas and model storage.
  • Reuses dashboard-chart evaluation behavior for detached alerts.
  • Validates formulas, source ownership, connection ownership, and raw-SQL templates.
  • Adds chart-explorer notification links and integration coverage.

Confidence Score: 4/5

The PR should not merge until valid equivalent ObjectId representations pass the new source/connection association check.

The new case-sensitive string comparison can reject a raw-SQL chart alert even after MongoDB resolves its source and connection to the same ObjectId.

Files Needing Attention: packages/api/src/controllers/alerts.ts

Important Files Changed

Filename Overview
packages/common-utils/src/types.ts Adds the chart alert source and persisted chart-config schemas while excluding PromQL.
packages/api/src/utils/zod.ts Adds the internal chart-alert union and write-time formula validation.
packages/api/src/controllers/alerts.ts Persists and validates detached chart configurations, but compares equivalent ObjectIds using their unnormalized string representations.
packages/api/src/tasks/checkAlerts/index.ts Shares chart-query assembly between tile and detached chart alerts and adds chart task handling.
packages/api/src/tasks/checkAlerts/providers/default.ts Loads detached chart alert sources and connections for worker evaluation.
packages/api/src/tasks/checkAlerts/template.ts Generates chart-explorer links and chart-config-based notification titles.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Client[Internal alerts API] --> Schema[Parse internal alert schema]
  Schema --> Validate[Validate chart config]
  Validate --> Persist[(Alert document)]
  Persist --> Worker[Check-alerts worker]
  Worker --> Query[Build and execute chart query]
  Query --> Notify[Send notification with explorer link]
Loading

Fix all with Greploop Fix All in Claude Code Fix All in Conductor Fix All in Cursor Fix All in Codex

Reviews (2): Last reviewed commit: "fix(api): validate chart alert formulas ..." | Re-trigger Greptile

Comment on lines +1764 to +1767
export const AlertChartConfigSchema = z.union([
BuilderSavedChartConfigWithoutAlertSchema,
RawSqlSavedChartConfigWithoutAlertSchema,
]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Formula validation is bypassed

When a detached builder alert contains a malformed formula or references a nonexistent series, AlertChartConfigSchema accepts it without applying validateChartConfigFormulas. The evaluator later throws while rendering the formula, repeatedly recording query errors while the alert never fires or resolves.

Knowledge Base Used:

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment thread packages/api/src/controllers/alerts.ts
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

E2E Test Results

All tests passed • 318 passed • 1 skipped • 1254s

Status Count
✅ Passed 318
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@wrn14897
wrn14897 marked this pull request as ready for review August 27, 2026 05:41
@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Aug 27, 2026
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 461 lines (bar: 30):
    • packages/api/src/tasks/checkAlerts/index.ts
    • packages/api/src/tasks/checkAlerts/providers/default.ts
    • packages/api/src/tasks/checkAlerts/providers/index.ts
    • packages/api/src/tasks/checkAlerts/template.ts
  • Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 10
  • Production lines changed: 728 (+ 750 in test files, excluded from tier calculation)
  • Critical-path lines changed: 461
  • Branch: warren/HDX-5090-detached-alerts-backend
  • Author: wrn14897

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Deep Review

Backend-only foundation for detached chart alerts. Ten reviewers ran; no P0/P1 issues survived re-grading. Both prior automated P1 comments — formula-validation bypass and raw-SQL source/connection divergence — are resolved by the latest commit (internalAlertSchema now runs validateChartAlertFormulas, and the raw-SQL branch rejects a source whose connection differs from chartConfig.connection). The alert-evaluation path has solid per-alert error isolation, and the external v2 write-rejection is tested.

✅ No critical issues found.

🟡 P2 — recommended

  • packages/api/src/routers/external-api/v2/alerts.ts:397 — v2 GET/list echoes chart alerts verbatim via translateAlertDocumentToExternalAlert, returning source: "chart" (with no chartConfig), a value outside the published OpenAPI AlertSource enum; the server does not 500, but strict response-validating clients (generated SDKs, Terraform provider) can fail to parse the payload once any chart alert exists.
    • Fix: filter chart-source alerts out of v2 read responses (or map them to a documented shape) until the v2 OpenAPI contract is extended to model the chart source.
    • api-contract, testing, correctness
  • packages/api/src/tasks/checkAlerts/providers/default.ts:498buildChartExplorerLink serializes the entire persisted chartConfig into the config query param via JSON.stringify, so a raw-SQL config with a long sqlTemplate or a builder config with many selects/filters can push the notification link past Slack button and common proxy URL limits, yielding a truncated or dropped "view chart" link.
    • Fix: bound the generated URL (e.g. server-resolved short id/token or compressed config) and cap or validate the resulting length.
    • adversarial
  • packages/api/src/tasks/checkAlerts/providers/default.ts:221 — the new getChartDetails failure branches (null chartConfig, deleted raw-SQL connection, deleted builder source, unsupported raw-SQL display type) have no test; the CHART integration tests build AlertDetails directly and bypass loadAlert/getChartDetails, so the eval-time skip path is unverified.
    • Fix: add provider-level tests asserting getAlertTasks() skips chart alerts with missing config/source/connection, mirroring the existing TILE/SAVED_SEARCH cases.
    • testing, reliability, adversarial, correctness
🔵 P3 nitpicks (3)
  • packages/api/src/tasks/checkAlerts/providers/default.ts:221getChartDetails duplicates ~90 lines of getTileDetails almost verbatim; the copies have already drifted in a log string (default.ts:131 still says "line/bar" while default.ts:238 says "line/bar/number"), though both branches gate on the same displayTypeSupportsRawSqlAlerts so behavior is identical.
    • Fix: extract the shared connection/source resolution into one helper used by both, matching the buildAlertChartConfigFromSavedConfig extraction already done in index.ts.
  • packages/api/src/tasks/checkAlerts/providers/default.ts:408 — when a chart alert's source or connection is deleted after creation, getChartDetails returns [] and the throw is caught at load time before processAlert runs, so no ERROR AlertHistory row is recorded and the alert silently stops evaluating with only a worker log; detached alerts make source deletion a more likely lifecycle event than for tile/saved-search alerts.
    • Fix: record an alert error/history row when a chart alert's source or connection cannot be resolved so the owner gets an in-product signal.
  • packages/api/src/tasks/checkAlerts/index.ts:866 — a supported CHART/TILE alert whose config cannot be built (PromQL or unsupported display type) falls through to the generic logger.error('Unsupported alert source'), which misleads operators since the source is supported and only the config variant is not.
    • Fix: log a config-specific message for the returns-undefined-on-supported-source case.

Reviewers (10): correctness, security, adversarial, api-contract, reliability, kieran-typescript, testing, maintainability, project-standards, previous-comments.

Testing gaps:

  • No end-to-end evaluation test for a raw-SQL chart alert (query-build + eval); only creation/validation is covered.
  • No test for a chart alert on a Metric source (metricTables/sample-weight branch) or for multi-window (numConsecutiveWindows > 1) chart alerts end-to-end.
  • No v2 GET/list test asserting the response shape for a chart alert created via the internal API.
  • Chart chartConfig is stored as Mongoose Schema.Types.Mixed; integrity depends entirely on internalAlertSchema validation at the API boundary, so any future write path that bypasses it (migration, admin tool, direct DB write) can persist an unvalidated config that only fails at evaluation time (noted as residual risk by security, reliability, typescript, maintainability, and project-standards — consistent with existing Mixed fields, not a new risk class).

… consistency (HDX-5090)

Two write-path gaps in the new chart alert source, both of which would
otherwise persist configs that fail on every evaluation tick:

- Builder configs skipped validateChartConfigFormulas (dashboards get it
  from the editor and the external tile refinement, but chart alerts are
  authored through this API directly). internalAlertSchema now rejects
  malformed formulas, references to nonexistent series, and formulas
  combined with seriesReturnType: 'ratio' (mapped onto the helper's
  external-shape asRatio).

- Raw-SQL configs accepted a team-owned source on a different team-owned
  connection. The worker executes through chartConfig.connection while
  expanding $__sourceTable/metricTables from the source, so a divergent
  pair yields wrong values or repeated query failures. validateAlertInput
  now requires the source to belong to the configured connection.
// source on a different (even team-owned) connection would query the
// wrong database — silently wrong values when the table also exists
// there, repeated query failures when it does not.
if (source.connection.toString() !== chartConfig.connection) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Equivalent ObjectIds rejected

When a raw-SQL chart alert supplies the source's connection ID in a valid non-canonical representation such as uppercase hexadecimal, the MongoDB lookups resolve both references but this case-sensitive comparison rejects them as different, preventing creation or update of an otherwise valid alert.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated AI-generated content; review carefully before merging. review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant