feat(nativemem): categorized native-memory accounting — first cut#669
Open
rkennke wants to merge 17 commits into
Open
feat(nativemem): categorized native-memory accounting — first cut#669rkennke wants to merge 17 commits into
rkennke wants to merge 17 commits into
Conversation
Contributor
CI Test ResultsRun: #30007758532 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-23 12:54:39 UTC |
Contributor
Benchmark Results (commit e095203)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125289326 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10287 ms (21 iters) | ✅ 10260 ms (21 iters) | ≈ -0.3% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8867 ms (24 iters) | ✅ 8805 ms (24 iters) | ≈ -0.7% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5933 ms (33 iters) | ✅ 5935 ms (33 iters) | ≈ +0% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5475 ms (36 iters) | ✅ 5453 ms (36 iters) | ≈ -0.4% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2648 ms (71 iters) | ✅ 2700 ms (69 iters) | ≈ +2% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2825 ms (66 iters) | ✅ 2836 ms (66 iters) | ≈ +0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2121 ms (88 iters) | ✅ 2045 ms (90 iters) | 🟢 -3.6% | — / — |
| future-genetic | 25 | ✅ 1976 ms (94 iters) | ✅ 2086 ms (89 iters) | 🔴 +5.6% | — / — |
| naive-bayes | 21 | ✅ 1230 ms (139 iters) | ✅ 1259 ms (137 iters) | ≈ +2.4% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (168 iters) | ✅ 1011 ms (169 iters) | ≈ -0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 15811 ms (16 iters) | ✅ 16445 ms (15 iters) | ≈ +4% (±7.6%) | — / — |
| reactors | 25 | ✅ 18243 ms (15 iters) | ✅ 18764 ms (15 iters) | ≈ +2.9% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2094 / 1933 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2216 / 2195 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 8790 / 8777 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8165 / 8254 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1283 / 1240 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1281 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2935 / 2926 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2884 / 2927 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3486 / 3580 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / ✅ | 3448 / 3533 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1512 / 1844 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1866 / 1889 | ✅ / ✅ | ✅ / ✅ |
This comment has been minimized.
This comment has been minimized.
Contributor
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/126272097 |
Contributor
Benchmark Results (commit 62b6f2c)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125313150 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10303 ms (21 iters) | ✅ 10356 ms (21 iters) | ≈ +0.5% (±10.7%) | — / — |
| akka-uct | 25 | ✅ 9028 ms (24 iters) | ✅ 8895 ms (24 iters) | ≈ -1.5% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5921 ms (33 iters) | ✅ 5936 ms (33 iters) | ≈ +0.3% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5537 ms (36 iters) | ✅ 5535 ms (36 iters) | ≈ -0% (±24.4%) | |
| fj-kmeans | 21 | ✅ 2705 ms (70 iters) | ✅ 2667 ms (69 iters) | ≈ -1.4% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2829 ms (66 iters) | ✅ 2824 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (88 iters) | ✅ 2066 ms (90 iters) | 🟢 -2.6% | — / — |
| future-genetic | 25 | ✅ 2071 ms (90 iters) | ✅ 1960 ms (94 iters) | 🟢 -5.4% | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1012 ms (169 iters) | ≈ -1% (±31.6%) | — / — |
| reactors | 21 | ✅ 16191 ms (15 iters) | ✅ 15952 ms (15 iters) | ≈ -1.5% (±6.7%) | — / — |
| reactors | 25 | ✅ 18581 ms (15 iters) | ✅ 18426 ms (15 iters) | ≈ -0.8% (±3.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1975 / 2029 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2406 / 2283 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 7 | 8749 / 8443 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8539 / 8755 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 1305 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1284 / 1292 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3065 / 2961 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2938 / 2818 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 3483 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1495 / 1554 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1874 / 1984 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit bdf9f55)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125648678 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f1a84c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125658666 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10323 ms (21 iters) | ≈ +0.1% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8961 ms (24 iters) | ✅ 8884 ms (24 iters) | ≈ -0.9% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 6042 ms (33 iters) | ✅ 5955 ms (33 iters) | ≈ -1.4% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5468 ms (36 iters) | ✅ 5531 ms (36 iters) | ≈ +1.2% (±25%) | |
| fj-kmeans | 21 | ✅ 2649 ms (70 iters) | ✅ 2695 ms (70 iters) | ≈ +1.7% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2800 ms (67 iters) | ✅ 2807 ms (66 iters) | ≈ +0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2141 ms (87 iters) | ✅ 2039 ms (90 iters) | 🟢 -4.8% | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2127 ms (87 iters) | ≈ +1.9% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1227 ms (139 iters) | ✅ 1280 ms (134 iters) | ≈ +4.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1024 ms (167 iters) | ✅ 1022 ms (167 iters) | ≈ -0.2% (±32%) | — / — |
| reactors | 21 | ✅ 16285 ms (15 iters) | ✅ 16003 ms (15 iters) | ≈ -1.7% (±5.7%) | — / — |
| reactors | 25 | ✅ 18050 ms (15 iters) | ✅ 18143 ms (15 iters) | ≈ +0.5% (±6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 5 | 2017 / 2009 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2344 / 2264 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 5 | 8749 / 8648 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 4 | 8234 / 8352 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1239 / 1269 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1267 / 1274 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 3072 / 2901 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2974 / 2854 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3509 / 3545 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 3490 / 3444 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1756 / 1751 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1854 / 1753 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 277e2d8)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125665685 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10393 ms (21 iters) | ✅ 10229 ms (21 iters) | ≈ -1.6% (±11%) | — / — |
| akka-uct | 25 | ✅ 8931 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.8% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 6009 ms (33 iters) | ✅ 5981 ms (33 iters) | ≈ -0.5% (±24.6%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5496 ms (36 iters) | ≈ -0.2% (±24.7%) | |
| fj-kmeans | 21 | ✅ 2761 ms (68 iters) | ✅ 2694 ms (70 iters) | ≈ -2.4% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2757 ms (68 iters) | ✅ 2798 ms (67 iters) | ≈ +1.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2159 ms (86 iters) | ✅ 2039 ms (92 iters) | 🟢 -5.6% | — / — |
| future-genetic | 25 | ✅ 2030 ms (91 iters) | ✅ 2111 ms (88 iters) | 🔴 +4% | — / — |
| naive-bayes | 21 | ✅ 1283 ms (133 iters) | ✅ 1258 ms (136 iters) | ≈ -1.9% (±32.4%) | — / — |
| naive-bayes | 25 | ✅ 1019 ms (168 iters) | ✅ 1020 ms (168 iters) | ≈ +0.1% (±31.8%) | — / — |
| reactors | 21 | ✅ 16429 ms (15 iters) | ✅ 14845 ms (17 iters) | 🟢 -9.6% | — / — |
| reactors | 25 | ✅ 18219 ms (15 iters) | ✅ 18034 ms (15 iters) | ≈ -1% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 1 | 1988 / 2062 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 9 | 2080 / 2084 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8485 / 8415 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 8560 / 8615 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1282 / 1286 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 6 / 1 | 1278 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 3002 / 3031 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2944 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3504 / 3524 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 3485 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1536 / 1691 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1780 / 1837 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit ac8c128)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125736830 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10383 ms (21 iters) | ✅ 10256 ms (21 iters) | ≈ -1.2% (±11.4%) | — / — |
| akka-uct | 25 | ✅ 8923 ms (24 iters) | ✅ 8855 ms (24 iters) | ≈ -0.8% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 5980 ms (33 iters) | ✅ 6028 ms (33 iters) | ≈ +0.8% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5483 ms (36 iters) | ✅ 5498 ms (36 iters) | ≈ +0.3% (±24.1%) | |
| fj-kmeans | 21 | ✅ 2749 ms (68 iters) | ✅ 2717 ms (69 iters) | ≈ -1.2% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2826 ms (66 iters) | ✅ 2762 ms (67 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2042 ms (91 iters) | ✅ 2044 ms (90 iters) | ≈ +0.1% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2130 ms (87 iters) | ✅ 2051 ms (90 iters) | 🟢 -3.7% | — / — |
| naive-bayes | 25 | ✅ 1008 ms (170 iters) | ✅ 1009 ms (169 iters) | ≈ +0.1% (±31.7%) | — / — |
| reactors | 21 | ✅ 16229 ms (15 iters) | ✅ 16543 ms (15 iters) | ≈ +1.9% (±7.5%) | — / — |
| reactors | 25 | ✅ 18332 ms (15 iters) | ✅ 18379 ms (15 iters) | ≈ +0.3% (±3.4%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2019 / 2112 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2418 / 2216 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 8420 / 8855 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 8622 / 8796 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 1266 / 1280 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1307 / 1266 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3019 / 2888 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2890 / 2833 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 3505 / 3444 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1626 / 1497 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1806 / 1830 | ✅ / ✅ | ✅ / ✅ |
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new NativeMem facility to attribute the profiler’s own native memory usage into a small set of categories, tracking per-category live bytes, a moving-window average, and a precise per-category peak; totals are exported via existing counter/JFR pathways.
Changes:
- Added
NativeMemcore implementation (nativeMem.h/.cpp) with per-category live/avg/max tracking and tests. - Instrumented major native allocation sites (calltrace arena/buffers, dictionaries, thread-local data, thread filter, perf mmap, line tables, JFR recording buffers, native symbols gauge) to record alloc/free deltas.
- Emitted totals through the counters table and per-category metrics through
T_DATADOG_COUNTERevents on chunk finish.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| ddprof-lib/src/test/cpp/stringDictionary_ut.cpp | Adds lifecycle accounting-balance test for StringDictionaryBuffer NM_DICTIONARY instrumentation. |
| ddprof-lib/src/test/cpp/nativeMem_ut.cpp | New unit tests covering live/avg/max semantics, peaks between samples, clamping behavior, and category names. |
| ddprof-lib/src/test/cpp/dictionary_ut.cpp | Adds lifecycle accounting-balance test for Dictionary NM_DICTIONARY instrumentation. |
| ddprof-lib/src/main/cpp/threadLocalData.h | Records NM_THREAD_LOCAL on ProfiledThread creation via forTid. |
| ddprof-lib/src/main/cpp/threadLocalData.cpp | Records NM_THREAD_LOCAL decrements on TLS destruction/release paths. |
| ddprof-lib/src/main/cpp/threadFilter.cpp | Accounts for thread-filter chunk and freelist backing allocations under NM_THREAD_FILTER. |
| ddprof-lib/src/main/cpp/stringDictionary.h | Accounts arena chunks + SBTable allocations/frees under NM_DICTIONARY. |
| ddprof-lib/src/main/cpp/profiler.cpp | Accounts per-shard calltrace buffer allocations/frees under NM_CALLTRACE; mirrors native symbol gauge via setLive. |
| ddprof-lib/src/main/cpp/perfEvents_linux.cpp | Accounts perf ring mmap/unmap under NM_PERF. |
| ddprof-lib/src/main/cpp/nativeMem.h | Defines categories and NativeMem API (record/setLive/sample/live/avg/max/totals). |
| ddprof-lib/src/main/cpp/nativeMem.cpp | Implements totals, sampling window averaging, reset, and category naming. |
| ddprof-lib/src/main/cpp/linearAllocator.cpp | Accounts calltrace arena chunk alloc/free under NM_CALLTRACE. |
| ddprof-lib/src/main/cpp/flightRecorder.h | Declares Recording helpers to sample/export native-mem metrics. |
| ddprof-lib/src/main/cpp/flightRecorder.cpp | Samples native-mem each chunk and emits totals + per-category counter events. |
| ddprof-lib/src/main/cpp/dictionary.h | Accounts root table allocation under NM_DICTIONARY. |
| ddprof-lib/src/main/cpp/dictionary.cpp | Accounts key-string alloc/free and overflow/root table alloc/free under NM_DICTIONARY. |
| ddprof-lib/src/main/cpp/counters.h | Adds total native-mem counters (live/avg/max) to the counter table. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Benchmark Results (commit 8a9567f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125902089 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
rkennke
force-pushed
the
feat/native-mem-accounting
branch
from
July 21, 2026 11:52
8a9567f to
3c82245
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/flightRecorder.cpp:1815
- writeNativeMem() serializes values with Buffer::putVar64(u64). If any native-mem gauge goes negative (possible until all free sites are instrumented), the implicit signed->unsigned conversion will emit a huge varint. Clamp to 0 before encoding to keep the JFR/counter stream valid.
auto emit = [&](const char *label, long long value) {
int start = buf->skip(1);
buf->putVar64(T_DATADOG_COUNTER);
buf->putVar64(_start_ticks);
buf->putUtf8(label);
buf->putVar64(value);
writeEventSizePrefix(buf, start);
flushIfNeeded(buf);
Contributor
Benchmark Results (commit 3c82245)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125929510 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/flightRecorder.cpp:1801
- NativeMem::sample() explicitly clamps negative per-category live values to 0 when computing totals/averages, but the exported total live counter uses NativeMem::liveTotal(), which can go negative if any category is temporarily negative (the scenario sample() is already designed to tolerate). This can produce a nonsensical negative "native_mem_live_bytes" in the counter stream and makes the totals inconsistent with the sampled/clamped window.
// per-category values are emitted by writeNativeMem().
Counters::set(NATIVE_MEM_LIVE_BYTES, NativeMem::liveTotal());
Counters::set(NATIVE_MEM_AVG_BYTES, NativeMem::avgTotal());
Counters::set(NATIVE_MEM_MAX_BYTES, NativeMem::maxTotal());
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19db9d3172
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Contributor
Benchmark Results (commit 19db9d3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125953166 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
kaahos
reviewed
Jul 22, 2026
kaahos
left a comment
kaahos
left a comment
Contributor
There was a problem hiding this comment.
Thank you for the work, I've left some comments! let me know what you think.
- Add two tests proving NM_THREAD_LOCAL balances across the ProfiledThread lifecycle: the decrement lives in freeValue(), reached both via release() -> ThreadLocal::clear() and via the pthread-key destructor on thread exit. Both paths return the gauge to baseline. - dictionary_ut.cpp: update copyright to 2025, 2026. - stringDictionary_ut.cpp: add the missing Datadog copyright header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two follow-up review fixes: - ProfiledThread::deleteForTest() (UNIT_TEST helper) deleted the object directly, bypassing freeValue()'s NM_THREAD_LOCAL decrement and leaking live bytes across tests that use it. It stands in for freeValue()'s delete, so mirror the decrement. - In ~ThreadFilter(), record the decrements after the memory is actually freed: delete the chunk before its decrement, and reset the _free_list unique_ptr explicitly before recording (rather than letting it free the array after the destructor body runs). Keeps the gauge from leading the free during teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rkennke
force-pushed
the
feat/native-mem-accounting
branch
from
July 23, 2026 10:59
5740eff to
53fc7e2
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/profiler.cpp:1747
- In updateNativeLibMemStats(), CODECACHE_RUNTIME_STUBS_SIZE_BYTES is set to the native library CodeCacheArray's memoryUsage(). This makes the runtime-stubs counter mirror CODECACHE_NATIVE_SIZE_BYTES and ignores the actual runtime stub cache (e.g., HotSpot JitCodeCache::_runtime_stubs), so the metric name/value pair is inconsistent.
const CodeCacheArray& native_libs = _libs->native_libs();
long long usage = (long long)native_libs.memoryUsage();
Counters::set(CODECACHE_NATIVE_COUNT, native_libs.count());
Counters::set(CODECACHE_NATIVE_SIZE_BYTES, usage);
Counters::set(CODECACHE_RUNTIME_STUBS_SIZE_BYTES, usage);
NativeMem::setLive(NM_NATIVE_SYMBOLS, usage);
Consistency: the two-phase resize recorded the NM_CALLTRACE decrement before free(prev); the other decrement sites (FlightRecorder::stop, ~ThreadFilter) account the free after it happens. Reorder to match. Functionally equivalent (free doesn't read the counter), purely for uniformity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Reordered the calltrace-buffer decrement to after |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/profiler.cpp:1749
updateNativeLibMemStats()publishesCODECACHE_RUNTIME_STUBS_SIZE_BYTESusingnative_libs.memoryUsage(). That makes the runtime-stubs metric identical to the native-libraries metric and does not match the counter name (there is a separate runtime-stubs code cache, e.g.JitCodeCache::_runtime_stubs). This is likely misleading for downstream dashboards; it should either be wired to the actual runtime-stubs cache or be removed/zeroed with a clear comment.
const CodeCacheArray& native_libs = _libs->native_libs();
long long usage = (long long)native_libs.memoryUsage();
Counters::set(CODECACHE_NATIVE_COUNT, native_libs.count());
Counters::set(CODECACHE_NATIVE_SIZE_BYTES, usage);
Counters::set(CODECACHE_RUNTIME_STUBS_SIZE_BYTES, usage);
NativeMem::setLive(NM_NATIVE_SYMBOLS, usage);
Contributor
Benchmark Results (commit 53fc7e2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126483661 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10321 ms (21 iters) | ✅ 10256 ms (21 iters) | ≈ -0.6% (±10.8%) | — / — |
| akka-uct | 25 | ✅ 8855 ms (24 iters) | ✅ 8888 ms (24 iters) | ≈ +0.4% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6029 ms (33 iters) | ✅ 5973 ms (33 iters) | ≈ -0.9% (±25.8%) | |
| fj-kmeans | 21 | ✅ 2823 ms (66 iters) | ✅ 2636 ms (72 iters) | 🟢 -6.6% | — / — |
| fj-kmeans | 25 | ✅ 2837 ms (66 iters) | ✅ 2811 ms (66 iters) | ≈ -0.9% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2077 ms (89 iters) | ✅ 2050 ms (90 iters) | ≈ -1.3% (±2.5%) | — / — |
| future-genetic | 25 | ✅ 2013 ms (92 iters) | ✅ 2039 ms (91 iters) | ≈ +1.3% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1275 ms (134 iters) | ✅ 1248 ms (136 iters) | ≈ -2.1% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 971 ms (175 iters) | ✅ 1015 ms (169 iters) | ≈ +4.5% (±32.5%) | — / — |
| reactors | 21 | ✅ 17367 ms (15 iters) | ✅ 16251 ms (15 iters) | ≈ -6.4% (±7.4%) | — / — |
| reactors | 25 | ✅ 18606 ms (15 iters) | ✅ 18632 ms (15 iters) | ≈ +0.1% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 1900 / 1973 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2125 / 2410 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8625 / 8512 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8261 / 8416 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 1278 / 1275 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1275 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2967 / 2991 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2829 / 2913 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 3548 / 3485 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3449 / 3513 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1770 / 1720 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1931 / 1951 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit d4cad78)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126487357 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Consistency (same as the calltrace-buffer and ~ThreadFilter sites): record the NM_PERF decrement for the old _events array after free(_events) rather than before. Capture the old size first, since _max_events is overwritten by the reallocation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Reordered the |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
ddprof-lib/src/main/cpp/profiler.cpp:1749
- CODECACHE_RUNTIME_STUBS_SIZE_BYTES is being set to native_libs.memoryUsage(), which is the native-library symbol-table footprint. This counter name suggests it should reflect JIT/runtime stubs (e.g., HotSpot JitCodeCache::_runtime_stubs), so publishing native_libs usage here makes the metric incorrect/misleading.
Counters::set(CODECACHE_NATIVE_COUNT, native_libs.count());
Counters::set(CODECACHE_NATIVE_SIZE_BYTES, usage);
Counters::set(CODECACHE_RUNTIME_STUBS_SIZE_BYTES, usage);
NativeMem::setLive(NM_NATIVE_SYMBOLS, usage);
ddprof-lib/src/main/cpp/stringDictionary.h:187
- In StringArena::reset(), NativeMem is decremented before free(). If NativeMem::sample() can run concurrently with reset(), this briefly reports memory as freed even though it is still allocated. Recording after free() avoids that skew and matches the approach used in other decrement sites.
while (c) {
Chunk* n = c->next;
NativeMem::record(NM_DICTIONARY, -(long long)sizeof(Chunk));
free(c);
c = n;
ddprof-lib/src/main/cpp/stringDictionary.h:255
- freeOverflowNodes() records the NativeMem decrement before free(). For consistency with the rest of the accounting (and to avoid transient underreporting during concurrent sampling), record the decrement after the free completes.
if (f.t != table) {
NativeMem::record(NM_DICTIONARY, -(long long)sizeof(SBTable));
free(f.t);
freed++;
}
Consistency with the other decrement sites (calltrace resize, perf _events, ~ThreadFilter): record the NM_THREAD_LOCAL decrement after delete pt rather than before. sizeof is a compile-time constant, so no value is lost by deleting first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Reordered the |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/profiler.cpp:1404
- Updating _max_stack_depth before swapping all per-shard CallTraceBuffer instances can make concurrent sampling/JVMTI code use the new (larger) _max_stack_depth with an old (smaller) buffer, which can lead to out-of-bounds writes/reads via calltrace_buffer[lock_index]->{asgct,jvmti}_frames. During the transition, ensure _max_stack_depth never exceeds the capacity of any shard buffer: publish the smaller depth first when shrinking, and only publish the larger depth after all shards have been replaced.
// Phase 2: all allocations succeeded — commit. Swap each shard under its
// per-shard lock (readers acquire it via tryLock before reading
// _calltrace_buffer, so none observes a freed pointer mid-replacement),
// then free the old buffer and reconcile accounting.
_max_stack_depth = args._jstackdepth;
for (int i = 0; i < CONCURRENCY_LEVEL; i++) {
NativeMem::record(NM_CALLTRACE, (long long)(nelem * sizeof(CallTraceBuffer)));
_locks[i].lock();
CallTraceBuffer *prev = _calltrace_buffer[i];
_calltrace_buffer[i] = fresh[i];
_locks[i].unlock();
Contributor
Benchmark Results (commit 9a8e4dd)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126495477 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10278 ms (21 iters) | ✅ 10328 ms (21 iters) | ≈ +0.5% (±11.4%) | — / — |
| akka-uct | 25 | ✅ 8842 ms (24 iters) | ✅ 8864 ms (24 iters) | ≈ +0.2% (±10.1%) | — / — |
| finagle-chirper | 21 | ✅ 5967 ms (33 iters) | ✅ 6024 ms (33 iters) | ≈ +1% (±25%) | |
| finagle-chirper | 25 | ✅ 5565 ms (35 iters) | ✅ 5478 ms (36 iters) | ≈ -1.6% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2746 ms (68 iters) | ✅ 2729 ms (69 iters) | ≈ -0.6% (±2.8%) | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2829 ms (66 iters) | ≈ +0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2063 ms (90 iters) | ✅ 2138 ms (87 iters) | 🔴 +3.6% | — / — |
| future-genetic | 25 | ✅ 2044 ms (91 iters) | ✅ 2094 ms (88 iters) | ≈ +2.4% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1220 ms (139 iters) | ✅ 1281 ms (134 iters) | ≈ +5% (±33.7%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.2% (±31.8%) | — / — |
| reactors | 21 | ✅ 16442 ms (15 iters) | ✅ 16921 ms (15 iters) | ≈ +2.9% (±6.9%) | — / — |
| reactors | 25 | ✅ 17994 ms (15 iters) | ✅ 18564 ms (15 iters) | ≈ +3.2% (±5.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1953 / 1982 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2525 / 2276 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8754 / 8434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 8075 / 8210 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 1283 / 1306 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1271 / 1267 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2982 / 2954 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2855 / 2899 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 3484 / 3531 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 3499 / 3462 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1693 / 1716 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1880 / 1819 | ✅ / ✅ | ✅ / ✅ |
Sweep the remaining "record before free" decrement sites to record after the free, consistent with the calltrace/perf/thread-local/ThreadFilter sites (avoids a transient underreport if sampling races teardown): - StringArena chunk frees (~StringArena, reset()). - SBTable frees (freeOverflowNodes, ~StringDictionaryBuffer). - Dictionary key strings (capture strlen+1 before free, record after) and overflow DictTable frees. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Swept all remaining |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
ddprof-lib/src/main/cpp/profiler.cpp:1749
CODECACHE_RUNTIME_STUBS_SIZE_BYTESis being set tonative_libs.memoryUsage(). That value represents native-library symbol tables, not HotSpot runtime stubs (tracked underhotspot/JitCodeCache::_runtime_stubs). As-is, the counter name and value don’t match, which makes downstream diagnostics misleading. Consider either wiring this counter to the actual runtime-stub CodeCache memory usage, or stop setting it here (leave it 0/unchanged) until a correct source is available.
long long usage = (long long)native_libs.memoryUsage();
Counters::set(CODECACHE_NATIVE_COUNT, native_libs.count());
Counters::set(CODECACHE_NATIVE_SIZE_BYTES, usage);
Counters::set(CODECACHE_RUNTIME_STUBS_SIZE_BYTES, usage);
NativeMem::setLive(NM_NATIVE_SYMBOLS, usage);
Contributor
Benchmark Results (commit 1a4f8ee)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126505620 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
NativeMemfacility to measure the profiler's own native memory usage — currently we only observe whole-process RSS. Per-category live gauge, moving-window average, and a precise per-category peak, wired through the existing counter/JFR path.Why
We measure the agent's native footprint only as an RSS delta (~150–200 MB) with no breakdown. This gives an in-process, attributable, categorized number we can trend, and a foundation for later pinpointing which code is responsible.
How
NativeMem(nativeMem.h/nativeMem.cpp): aNativeMemCategoryenum whose per-category live gauges partition the total — each backing allocation belongs to exactly one category, so no double-counting.COUNTERSbuild flag:record()is a single relaxed atomic add. Most sites are off the signal path (malloc/new is not async-signal-safe anyway); the exception is CALLTRACE, whose arena allocates inside the sampling signal handler (viaOS::safeAlloc), sorecord()is kept async-signal-safe there.record()maintains each category's high-water mark at allocation time via a relaxed CAS (common path is load+compare; CAS only on a new peak; frees skip it). Spikes between sample ticks are still captured.sample()refreshes the moving-window averages and the observed total, ticked once per JFR chunk finish.NATIVE_MEM_{LIVE,AVG,MAX}_BYTES(JFRT_DATADOG_COUNTER+ JNI debug counters); per-categorynative_mem_{live,avg,max}_bytes.<category>values reuse the same event format — no new event type.Total peak: bracketed, no shared hotspot
A precise total peak would need a single global atomic hit by every allocation (a contention hotspot). Instead the total peak is bracketed:
NATIVE_MEM_MAX_BYTES= upper bound = sum of the precise per-category peaks.native_mem_max_observed_total_bytes= lower bound = the largest instantaneous total seen at a sampling tick.Instrumented categories
Tagged via
record()at their backing alloc/free sites (precise, always-on):LinearAllocatorchunks (call-trace arena) + per-shard calltrace buffers.Dictionary(symbols/packages — root/overflowDictTables + key strings, whose size is recovered viastrlenat bulk-free), and the arenaStringDictionary(arena chunks + root/overflowSBTables; keys live inside chunks so they're not counted separately). Counted unconditionally, independent of the diagnostic counters' offset gate. Single category — a per-role split, if ever wanted, belongs in a nested dimension.ProfiledThreadper thread.Recordingobject (embeds the JFR buffer array).mmap(2 × page_size).ChunkStoragechunks (bounded) + free-list array.Gauge-mirrored (
NativeMem::setLive()at a recompute site):CodeCache; not the JVM code cache). Sourced fromCodeCache::memoryUsage(), which is now accurate onmain(fix(codecache): make memoryUsage() accurate and live #677 merged); the gauge is refreshed at dump/stop, the same cadence as the existingCODECACHE_NATIVE_SIZE_BYTEScounter.Not yet covered (reads 0):
new/mallocacross the remaining files → best captured by malloc interception, not hand-tagging.(
CONTEXTwas removed: the OTel context record is embedded inProfiledThread, already counted underTHREAD_LOCAL; a separate category would double-count or read a permanent 0.)So the total is still a subset of the RSS delta until interception lands, but now covers the major consumers.
Avoiding double-counting
Call-trace bytes appear at three layers (
safeAllocmmap,LINEAR_ALLOCATOR_BYTESreserved chunks,CALLTRACE_STORAGE_BYTESused-within-chunks); summing counters would double/triple-count.NativeMemcounts backing memory once per category; the existing reserved/used/waste counters remain an independent nested diagnostic dimension. The dictionary tagging follows the same rule — e.g. arena keys are inside chunks and are not counted separately.Performance
Designed to be off the hot path. Measured in operations, not observed as a regression in this workload:
CallTraceStorage::putnormally just bump-allocates within an existing arena chunk and touches norecord().record()runs only when the call-trace arena grows a new chunk (8 MiB apart), and there it's a lock-free relaxed atomic add + high-water update — async-signal-safe.record()is one relaxedfetch_add; on allocation it also does a relaxed load + compare, with a CAS only when a new per-category peak is set (rare, since the peak is monotonic). This sits next to amalloc/calloc/mmap/newthat dominates the cost, so it's negligible. These sites (dictionary inserts, chunk/table growth, thread/lib/recording creation) are not per-sample.sample()is O(categories × window) ≈ 9 × 64 folds;writeNativeMem()emits ~28 counter events. This runs once per chunk rotation (seconds–minutes), off the sampling path.clear(): the per-key free now also does astrlento recover the freed size, soclear()cost is O(sum of key lengths) rather than O(key count).clear()already walks every key; this is a marginal constant-factor increase, at dictionary rotation only.No new locks, no shared global counter on the hot path (the total peak is bracketed precisely to avoid one), and the always-on atomics are relaxed.
sample()/writeNativeMem()run within the existing recording flush.Roadmap
Testing
nativeMem_ut(7 tests): live tracking + total partition, single-sample avg==live, moving average, max high-water, precise per-category max catching an inter-sample spike (with the bracket),setLivegauge + peak retention, category names. (The non-negative invariant is enforced by an assert inrecord()rather than a dedicated clamp test.)dictionary_ut,stringDictionary_ut): accounting grows on insert, returns exactly to the construction baseline afterclear(), and to zero after destruction — directly proving inc/dec pairing (incl. the strlen-at-free path and arena chunk growth).thread_teardown_safety_ut, Linux):NM_THREAD_LOCALreturns to baseline after bothrelease()and thread-exit teardown, proving theforTid/freeValuepairing.:ddprof-lib:gtestDebugsuite green (358 cross-platform tests, 0 failures locally; Linux-only perf/thread-teardown coverage runs in CI).🤖 Generated with Claude Code