Skip to content

inet-collector: stop recreating the RIPE Atlas fleet on metadata loss - #4142

Open
elitegreg wants to merge 3 commits into
mainfrom
gm/inet-collector-measurement-metadata-recovery
Open

inet-collector: stop recreating the RIPE Atlas fleet on metadata loss#4142
elitegreg wants to merge 3 commits into
mainfrom
gm/inet-collector-measurement-metadata-recovery

Conversation

@elitegreg

Copy link
Copy Markdown
Contributor

Fixes #4131.

Summary

A degraded RIPE Atlas measurement listing could destroy the collector's local state and, through that, the entire measurement fleet. On 2026-07-31 this stopped and recreated all 29 mainnet-beta measurements at once and left the collector in a recreation loop for hours, with metro-pair coverage oscillating between 380 and 427 of 435 while WheresItUp held a steady 435.

The trigger is configureMeasurements: when GetAllMeasurements failed it logged a warning, substituted an empty list, and continued reconciling. With no measurements in view, every metadata entry looks orphaned and gets pruned. The next run then finds 29 measurements it cannot recognize, and unrecognized measurements are stopped and recreated.

Changes, in dependency order:

  • Abort reconciliation on a failed listing. Reconciling against a view we failed to fetch is never safe. Returns an error and retries next cycle.
  • Rebuild metadata instead of recreating measurements. Everything needed is still available: the description carries the target location and probe (DoubleZero [mainnet-beta] to osl probe 7439), and the new GetMeasurementProbes API call plus the current location-to-probe mapping recovers the sources. A measurement is only recreated when it genuinely cannot be identified.
  • Refuse to prune all metadata at once. If every known entry looks orphaned, a degraded listing is far more likely than a vanished fleet.
  • Write the state file atomically. Save used os.Create, truncating in place, so an interrupted write left a torn file. Now writes a temp file and renames.
  • Fix the tracked-measurement gauge. It was Set at startup then Inc/Dec per operation, so the two drifted; the gauge read -2 during the incident. Now set from state at the end of each run.

Note the atomic-write change alone would not have prevented this incident. The state file was never torn, it was validly overwritten with a pruned map. The listing guard is the actual fix; atomicity closes a related latent hole.

Not included

Issue #4131 lists two further items, both changes to probe-selection behavior that sustain the loop once started rather than causing it. They warrant separate review and are left for follow-up:

  • recreation granularity, where one changed probe out of 29 discards a whole measurement
  • the unresponsive-probe race, where the 1-hour no-export deadline races the ~1-hour recreation interval

Testing Verification

  • parseMeasurementDescription covers both description forms, large probe IDs, and rejects legacy source-qualified descriptions (DoubleZero NYC probe 100 to LON probe 200) that cannot be rebuilt from text alone
  • Metadata rebuilding reconstructs target, probe, and sources from a probe list, and the result survives a reload from disk
  • Unrecoverable cases invent nothing: an unparseable description never reaches the probes API, and a probe that cannot be placed in a known metro leaves the measurement unrecognized
  • A failed listing returns an error, stops no measurements, and leaves metadata intact both in memory and on disk
  • A listing that would orphan every known entry leaves metadata intact
  • Atomic save verified under 10 concurrent writers and 25 concurrent readers: every read sees a complete file with all entries, and no temporary files remain. A failed save into a read-only directory leaves no partial file
  • Full internet-latency-collector suite passes; golangci-lint reports 0 issues

One existing assertion was updated: State_FilePermissionError matched on the old error string. The behavior it checks is unchanged, and the test now also asserts no temp file is left behind.

A failed measurement listing was substituted with an empty list, which made
every live measurement look absent. Metadata was then pruned as orphaned, and
the next run stopped and recreated all 29 mainnet measurements at once.

- abort reconciliation when the measurement listing fails instead of
  reconciling against an empty view
- rebuild metadata for unrecognized measurements from the description and the
  measurement's probe list rather than stopping and recreating them
- refuse to prune orphaned metadata when it would remove every entry
- write the state file atomically via temp file and rename
- set the tracked-measurement gauge from state instead of incrementing and
  decrementing it, which had driven the value negative
@elitegreg elitegreg self-assigned this Aug 1, 2026
@elitegreg
elitegreg marked this pull request as ready for review August 2, 2026 02:10
@elitegreg
elitegreg enabled auto-merge (squash) August 2, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

internet-latency-collector: missing measurement metadata triggers full RIPE Atlas fleet teardown and sustained recreation churn

1 participant