inet-collector: stop recreating the RIPE Atlas fleet on metadata loss - #4142
Open
elitegreg wants to merge 3 commits into
Open
inet-collector: stop recreating the RIPE Atlas fleet on metadata loss#4142elitegreg wants to merge 3 commits into
elitegreg wants to merge 3 commits into
Conversation
A failed measurement listing was substituted with an empty list, which made every live measurement look absent. Metadata was then pruned as orphaned, and the next run stopped and recreated all 29 mainnet measurements at once. - abort reconciliation when the measurement listing fails instead of reconciling against an empty view - rebuild metadata for unrecognized measurements from the description and the measurement's probe list rather than stopping and recreating them - refuse to prune orphaned metadata when it would remove every entry - write the state file atomically via temp file and rename - set the tracked-measurement gauge from state instead of incrementing and decrementing it, which had driven the value negative
elitegreg
marked this pull request as ready for review
August 2, 2026 02:10
elitegreg
enabled auto-merge (squash)
August 2, 2026 02:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4131.
Summary
A degraded RIPE Atlas measurement listing could destroy the collector's local state and, through that, the entire measurement fleet. On 2026-07-31 this stopped and recreated all 29 mainnet-beta measurements at once and left the collector in a recreation loop for hours, with metro-pair coverage oscillating between 380 and 427 of 435 while WheresItUp held a steady 435.
The trigger is
configureMeasurements: whenGetAllMeasurementsfailed it logged a warning, substituted an empty list, and continued reconciling. With no measurements in view, every metadata entry looks orphaned and gets pruned. The next run then finds 29 measurements it cannot recognize, and unrecognized measurements are stopped and recreated.Changes, in dependency order:
DoubleZero [mainnet-beta] to osl probe 7439), and the newGetMeasurementProbesAPI call plus the current location-to-probe mapping recovers the sources. A measurement is only recreated when it genuinely cannot be identified.Saveusedos.Create, truncating in place, so an interrupted write left a torn file. Now writes a temp file and renames.Setat startup thenInc/Decper operation, so the two drifted; the gauge read-2during the incident. Now set from state at the end of each run.Note the atomic-write change alone would not have prevented this incident. The state file was never torn, it was validly overwritten with a pruned map. The listing guard is the actual fix; atomicity closes a related latent hole.
Not included
Issue #4131 lists two further items, both changes to probe-selection behavior that sustain the loop once started rather than causing it. They warrant separate review and are left for follow-up:
Testing Verification
parseMeasurementDescriptioncovers both description forms, large probe IDs, and rejects legacy source-qualified descriptions (DoubleZero NYC probe 100 to LON probe 200) that cannot be rebuilt from text aloneinternet-latency-collectorsuite passes;golangci-lintreports 0 issuesOne existing assertion was updated:
State_FilePermissionErrormatched on the old error string. The behavior it checks is unchanged, and the test now also asserts no temp file is left behind.