Skip to content

Record error.type on failed collections in PeriodicMetricReader - #8650

Merged
jkwatson merged 2 commits into
open-telemetry:mainfrom
TimurRakhmatullin86:fix/periodic-metric-reader-error-type
Jul 28, 2026
Merged

Record error.type on failed collections in PeriodicMetricReader#8650
jkwatson merged 2 commits into
open-telemetry:mainfrom
TimurRakhmatullin86:fix/periodic-metric-reader-error-type

Conversation

@TimurRakhmatullin86

Copy link
Copy Markdown
Contributor

Problem

In PeriodicMetricReader.Scheduled.doRun() the self-observability sample for
otel.sdk.metric_reader.collection.duration is always recorded with a null error:

long startNanoTime = CLOCK.nanoTime();
String error = null;
Collection<MetricData> metricData;
try {
  metricData = collectionRegistration.collectAllMetrics();
} finally {
  long durationNanos = CLOCK.nanoTime() - startNanoTime;
  instrumentation.recordCollection(durationNanos / 1_000_000_000.0, error);
}

error is declared, initialized to null, and never assigned anywhere in the method, so
MetricReaderInstrumentation.recordCollection's if (error != null) branch that attaches the
error.type attribute is unreachable from this caller. When collectAllMetrics() throws, the
finally still records the sample — as a successful collection with no error.type — and
the exception propagates to the outer catch (Throwable). The self-observability histogram
therefore reports error-free collections while the reader is exporting nothing, which is
exactly the signal an operator would use to detect the outage.

The sibling BatchSpanProcessor.exportCurrentBatch() uses the same String error = null; … finally { instrumentation…(…, error); } shape but assigns error on each failure path.

Fix

Set error to the throwable's class name before it propagates:

try {
  metricData = collectionRegistration.collectAllMetrics();
} catch (Throwable t) {
  error = t.getClass().getName();
  throw t;
} finally {
  ...
}

The rethrow keeps the existing control flow — the exception still reaches the same outer
catch (Throwable) — so the only change is that the recorded sample now carries error.type.
(ThrowableUtil.propagateIfFatal is intentionally not added: the outer catch already handles
all throwables, so it would be a no-op here.)

Tests

SdkMeterProviderMetricsTest.collectionFailureIsRecordedWithErrorType: registers a
MetricProducer that throws on the first produce() then succeeds, flushes twice, and asserts
the exported otel.sdk.metric_reader.collection.duration point carries
error.type = java.lang.IllegalStateException. It fails against the current code (the point has
no error.type) and passes with the fix. Full :sdk:metrics:test suite and
:sdk:metrics:spotlessCheck pass; errorprone/NullAway clean.

The self-observability sample for otel.sdk.metric_reader.collection.duration
was always recorded with a null error: the local `error` variable was declared
but never assigned, so a failed collectAllMetrics() was reported as a
successful collection (error.type never set) - hiding exactly the failures an
operator watches this signal for.

Set `error` to the throwable's class name before it propagates, mirroring
BatchSpanProcessor.exportCurrentBatch(), so the finally records the failure.

Signed-off-by: Timur Rakhmatullin <174210871+TimurRakhmatullin86@users.noreply.github.com>
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 20, 2026

Copy link
Copy Markdown

Pull request dashboard status

Merged · refreshed 2026-07-28 17:45 UTC

Status above doesn't look right?
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@jack-berg jack-berg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. One small comment but looks good otherwise.

Comment on lines +279 to +281
// Record the failure on the self-observability sample before it propagates to the
// outer handler; otherwise error.type is never set and collection failures are
// reported as successful collections.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Record the failure on the self-observability sample before it propagates to the
// outer handler; otherwise error.type is never set and collection failures are
// reported as successful collections.

The explanation in this comment will make no sense to a future reader since it explains code that is no longer problematic with this change. I would just drop the comment altogether since the code is self explanatory.

Per review: the comment explained behavior that is no longer a pitfall after this change, so
it adds noise rather than clarity. The code is self-explanatory.

Signed-off-by: Timur Rakhmatullin <174210871+TimurRakhmatullin86@users.noreply.github.com>
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.47%. Comparing base (09d6c17) to head (2c53e49).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8650      +/-   ##
============================================
- Coverage     91.64%   91.47%   -0.17%     
- Complexity    10348    10458     +110     
============================================
  Files          1013     1021       +8     
  Lines         27380    27650     +270     
  Branches       3218     3242      +24     
============================================
+ Hits          25092    25293     +201     
- Misses         1558     1615      +57     
- Partials        730      742      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jkwatson
jkwatson merged commit 6ffe557 into open-telemetry:main Jul 28, 2026
29 checks passed
@otelbot

otelbot Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution @TimurRakhmatullin86! 🎉 We would like to hear from you about your experience contributing to OpenTelemetry by taking a few minutes to fill out this survey.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants