Skip to content

perf: experiment with striped classic histogram accumulator#2329

Draft
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator
Draft

perf: experiment with striped classic histogram accumulator#2329
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator

Conversation

@zeitlinger

@zeitlinger zeitlinger commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

This is an experimental alternate implementation for benchmarking against main and the synchronized candidate in #2252. It is not being presented as production-ready yet.

For classic-only histograms, this replaces the existing Buffer/LongAdder recording path with an epoch-striped, double-buffered per-thread accumulator. Each recording thread owns a cell, observations update plain thread-confined bucket/count/sum fields, and collection advances a global epoch before draining the inactive buffers. This avoids a global synchronized (DataPoint) monitor on the shared four-thread observe path while preserving coherent bucket/count/sum snapshots. getCount() and getSum() use the same snapshot mechanism.

Native-only and hybrid histograms keep their existing implementation. The public API is unchanged.

Local evidence

  • mise run lint
  • ./mvnw -pl prometheus-metrics-core -Dtest=HistogramTest test — 35 tests passed
  • focused concurrent collect/lifecycle tests repeated 5 times
  • ./mvnw -pl prometheus-metrics-core -am verify — reactor passed; core ran 157 tests
  • git diff --check
  • representative local JMH smoke run (JDK 25.0.3, 4 threads, 1 fork, 3x1s warmup, 5x1s measurement):
    • HistogramBenchmark.prometheusClassic: 40,944.515 ± 302.666 ops/s

The local JMH number is a smoke result, not a controlled cross-branch conclusion. The benchmark workflow is being requested so this branch and main are measured on the same runner.

Added concurrency coverage

  • collection while eight shared observers record concurrently
  • monotonic and coherent bucket/count/sum snapshots
  • exact final getCount() / getSum() values
  • retention of observations from short-lived recording threads
  • clear() lifecycle behavior for old and newly-created data points

Remaining hardening before merge consideration

  • Review/validate the epoch handoff under the Java Memory Model, ideally with dedicated jcstress coverage rather than relying only on repeated JUnit stress tests.
  • Decide how to bound or reclaim retained per-thread cells for workloads with high thread churn; cells currently live for the data point lifetime so completed-thread observations cannot be lost.
  • Measure scrape latency and the spin-wait behavior when a recording thread is delayed after entering an epoch.
  • Measure the cost of getCount() / getSum(), which currently perform an epoch flip and cell scan to return exact accumulated values.
  • Run the full CI benchmark matrix and test additional supported JDKs/architectures before treating this as mergeable.

Related: #2252

@zeitlinger
perf: experiment with striped classic histogram accumulator
0e08d51 
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger zeitlinger mentioned this pull request Jul 23, 2026
Open
@zeitlinger
docs: update core API diff
6fe2995 
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger zeitlinger closed this Jul 23, 2026
@zeitlinger zeitlinger reopened this Jul 23, 2026
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

1 similar comment
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

@github-actions

github-actions Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark results

Benchmark run succeeded for 6fe299560f57865c3ca33d399163a0639f84b47e.

Prometheus Java Client Benchmarks

Run Information

  • Date: 2026-07-23T10:46:17Z
  • Commit: 6fe2995
  • JDK: 25.0.3 (OpenJDK 64-Bit Server VM)
  • Benchmark config: 3 fork(s), 3 warmup, 5 measurement, 4 threads
  • Hardware: AMD EPYC 9V74 80-Core Processor, 4 cores, 16 GB RAM
  • OS: Linux 6.17.0-1020-azure

Comparison with base

  • Head: 6fe2995
  • Base: 1ca2716
  • Change: positive means the PR is faster than base.
  • Note: Base and head are compared on the same runner within each topic.
Benchmark PR Base Change Result
CounterBenchmark.codahaleIncNoLabels 44.11K 44.37K -0.6% within noise
CounterBenchmark.openTelemetryAdd 3.25K 4.53K -28.2% slower
CounterBenchmark.openTelemetryInc 4.42K 4.49K -1.6% within noise
CounterBenchmark.openTelemetryIncNoLabels 3.59K 4.70K -23.7% within noise
CounterBenchmark.prometheusAdd 48.73K 48.49K +0.5% within noise
CounterBenchmark.prometheusInc 60.03K 59.25K +1.3% within noise
CounterBenchmark.prometheusNoLabelsInc 51.95K 51.05K +1.8% faster
CounterBenchmark.simpleclientAdd 6.12K 5.92K +3.4% faster
CounterBenchmark.simpleclientInc 6.16K 6.20K -0.5% within noise
CounterBenchmark.simpleclientNoLabelsInc 5.92K 5.92K +0.1% within noise
HistogramBenchmark.openTelemetryClassic 766.00 766.29 -0.0% within noise
HistogramBenchmark.openTelemetryExponential 628.37 704.07 -10.8% within noise
HistogramBenchmark.prometheusClassic 11.59K 7.18K +61.4% faster
HistogramBenchmark.prometheusClassicPerThread 12.23K 12.58K -2.8% slower
HistogramBenchmark.prometheusClassicSingleThread 4.40K 4.55K -3.2% slower
HistogramBenchmark.prometheusNative 2.81K 2.82K -0.3% within noise
HistogramBenchmark.simpleclient 4.41K 4.36K +1.2% within noise
HistogramTextFormatBenchmark.openMetricsWriteToNull 27.07K 27.24K -0.6% within noise
HistogramTextFormatBenchmark.prometheusWriteToNull 27.56K 27.63K -0.2% within noise
TextFormatUtilBenchmark.openMetricsWriteToByteArray 526.71K 533.80K -1.3% within noise
TextFormatUtilBenchmark.openMetricsWriteToNull 541.08K 546.14K -0.9% within noise
TextFormatUtilBenchmark.prometheusWriteToByteArray 568.30K 573.63K -0.9% within noise
TextFormatUtilBenchmark.prometheusWriteToNull 582.87K 570.66K +2.1% within noise

Results for PR head

CounterBenchmark

Benchmark Score Error Units Within run
prometheusInc 60.03K ± 926.53 ops/s fastest
prometheusNoLabelsInc 51.95K ± 18.60 ops/s 1.2x slower
prometheusAdd 48.73K ± 396.08 ops/s 1.2x slower
codahaleIncNoLabels 44.11K ± 115.85 ops/s 1.4x slower
simpleclientInc 6.16K ± 130.99 ops/s 9.7x slower
simpleclientAdd 6.12K ± 22.12 ops/s 9.8x slower
simpleclientNoLabelsInc 5.92K ± 18.44 ops/s 10x slower
openTelemetryInc 4.42K ± 1.19K ops/s 14x slower
openTelemetryIncNoLabels 3.59K ± 261.88 ops/s 17x slower
openTelemetryAdd 3.25K ± 151.58 ops/s 18x slower

HistogramBenchmark

Benchmark Score Error Units Within run
prometheusClassicPerThread 12.23K ± 292.11 ops/s fastest
prometheusClassic 11.59K ± 428.04 ops/s 1.1x slower
simpleclient 4.41K ± 76.08 ops/s 2.8x slower
prometheusClassicSingleThread 4.40K ± 19.23 ops/s 2.8x slower
prometheusNative 2.81K ± 252.77 ops/s 4.4x slower
openTelemetryClassic 766.00 ± 23.21 ops/s 16x slower
openTelemetryExponential 628.37 ± 82.07 ops/s 19x slower

HistogramTextFormatBenchmark

Benchmark Score Error Units Within run
prometheusWriteToNull 27.56K ± 317.22 ops/s fastest
openMetricsWriteToNull 27.07K ± 332.83 ops/s 1.0x slower

TextFormatUtilBenchmark

Benchmark Score Error Units Within run
prometheusWriteToNull 582.87K ± 12.95K ops/s fastest
prometheusWriteToByteArray 568.30K ± 5.31K ops/s 1.0x slower
openMetricsWriteToNull 541.08K ± 12.50K ops/s 1.1x slower
openMetricsWriteToByteArray 526.71K ± 15.29K ops/s 1.1x slower

Raw Results

Benchmark                                            Mode  Cnt          Score        Error  Units
CounterBenchmark.codahaleIncNoLabels                thrpt   15      44112.334    ± 115.847  ops/s
CounterBenchmark.openTelemetryAdd                   thrpt   15       3252.670    ± 151.579  ops/s
CounterBenchmark.openTelemetryInc                   thrpt   15       4416.392   ± 1185.330  ops/s
CounterBenchmark.openTelemetryIncNoLabels           thrpt   15       3586.093    ± 261.881  ops/s
CounterBenchmark.prometheusAdd                      thrpt   15      48729.646    ± 396.084  ops/s
CounterBenchmark.prometheusInc                      thrpt   15      60027.803    ± 926.533  ops/s
CounterBenchmark.prometheusNoLabelsInc              thrpt   15      51952.103     ± 18.603  ops/s
CounterBenchmark.simpleclientAdd                    thrpt   15       6118.802     ± 22.118  ops/s
CounterBenchmark.simpleclientInc                    thrpt   15       6163.636    ± 130.990  ops/s
CounterBenchmark.simpleclientNoLabelsInc            thrpt   15       5922.243     ± 18.435  ops/s
HistogramBenchmark.openTelemetryClassic             thrpt   15        765.996     ± 23.210  ops/s
HistogramBenchmark.openTelemetryExponential         thrpt   15        628.374     ± 82.069  ops/s
HistogramBenchmark.prometheusClassic                thrpt   15      11589.781    ± 428.043  ops/s
HistogramBenchmark.prometheusClassicPerThread       thrpt   15      12228.590    ± 292.114  ops/s
HistogramBenchmark.prometheusClassicSingleThread    thrpt   15       4398.687     ± 19.226  ops/s
HistogramBenchmark.prometheusNative                 thrpt   15       2808.686    ± 252.769  ops/s
HistogramBenchmark.simpleclient                     thrpt   15       4410.533     ± 76.084  ops/s
HistogramTextFormatBenchmark.openMetricsWriteToNull  thrpt   15      27071.116    ± 332.829  ops/s
HistogramTextFormatBenchmark.prometheusWriteToNull  thrpt   15      27561.877    ± 317.215  ops/s
TextFormatUtilBenchmark.openMetricsWriteToByteArray  thrpt   15     526713.833  ± 15292.911  ops/s
TextFormatUtilBenchmark.openMetricsWriteToNull      thrpt   15     541078.028  ± 12499.428  ops/s
TextFormatUtilBenchmark.prometheusWriteToByteArray  thrpt   15     568297.769   ± 5312.819  ops/s
TextFormatUtilBenchmark.prometheusWriteToNull       thrpt   15     582874.507  ± 12954.847  ops/s

Notes

  • Score = Throughput in operations per second (higher is better)
  • Error = 99.9% confidence interval
  • Comparison with base uses JMH confidence intervals when available; overlapping intervals are marked "within noise".
  • Within run compares benchmarks in the same result set, not against the base commit.

Benchmark Descriptions

Benchmark Description
CounterBenchmark Counter increment performance: Prometheus, OpenTelemetry, simpleclient, Codahale
HistogramBenchmark Histogram observation performance (classic vs native/exponential)
TextFormatUtilBenchmark Metric exposition format writing speed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fstab fstab Awaiting requested review from fstab fstab will be requested when the pull request is marked ready for review fstab is a code owner
@dhoard dhoard Awaiting requested review from dhoard dhoard will be requested when the pull request is marked ready for review dhoard is a code owner
@jaydeluca jaydeluca Awaiting requested review from jaydeluca jaydeluca will be requested when the pull request is marked ready for review jaydeluca is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

api-change benchmark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zeitlinger