fix(reimporter): drain the last partial batch however the loop ended - #15762
Open
devGregA wants to merge 1 commit into
Open
fix(reimporter): drain the last partial batch however the loop ended#15762devGregA wants to merge 1 commit into
devGregA wants to merge 1 commit into
Conversation
process_findings() accumulates per-batch work and flushes it inside the per-finding loop, gated on `len(batch_finding_ids) >= dedupe_batch_max_size or is_final`. The matched branch ends in `continue`. That skip is what makes the tail fragile: the flush sits after it in the loop body, so a report whose LAST finding took the matched branch never reached the is_final flush, and everything appended since the previous size-triggered flush was dropped. Silently, because nothing raises. Those findings got no deduplication, no rules, no issue updater and no JIRA dispatch, and none of their parser or inherited tags; pending location status updates, vulnerability ids and burp request/response pairs were discarded with them. A reimport whose final finding matches an existing one is the ordinary case, not an edge case, so this fires on unchanged re-syncs rather than on unusual reports. Extracts the flush into _flush_post_processing_batch() and calls it once after the loop instead of relying on a flag inside it. The final call runs however the last iteration ended, and is deliberately unconditional: matched findings can accumulate location status updates without appending anything to dispatch, and every step is already a no-op on empty state (close_old_findings calls persist() the same way). The in-loop call keeps the size trigger only. is_final and is_final_batch are now unused and removed, which also leaves the loop's enumerate() index unused, so the loop iterates the batch directly. Tests: unittests/test_reimport_final_drain.py reimports a report whose matches all take the force_continue path, so the final iteration is guaranteed to be the skipping one, and asserts the batch was still flushed. It fails on the code before this change with exactly the dropped-work assertion and passes after. test_reimport_prefetch and test_import_reimport were compared before and after and are unchanged (both carry pre-existing failures on this branch that this change neither causes nor fixes). ruff 0.16.0 clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This pull request contains a critical finding where a sensitive codepath in
🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in
|
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/importers/default_reimporter.py' matches configured sensitive codepath pattern 'dojo/importers/*.py' and was modified by '' (commit d3b0fb6) who is not in the allowed authors list. |
We've notified @mtesauro.
Comment to provide feedback on these findings.
Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]
Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing
All finding details can be found in the DryRun Security Dashboard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
process_findings() accumulates per-batch work and flushes it inside the per-finding loop, gated on
len(batch_finding_ids) >= dedupe_batch_max_size or is_final.The matched branch ends in
continue. That skip is what makes the tail fragile: the flush sits after it in the loop body, so a report whose LAST finding took the matched branch never reached the is_final flush, and everything appended since the previous size-triggered flush was dropped. Silently, because nothing raises.What is lost for those findings: deduplication, rules, the issue updater and JIRA dispatch, plus their parser and inherited tags. Pending location status updates, vulnerability ids and burp request/response pairs are discarded with them.
A reimport whose final finding matches an existing one is the ordinary case rather than an edge case, so this fires on unchanged re-syncs.
The fix extracts the flush into _flush_post_processing_batch() and calls it once after the loop instead of relying on a flag inside it. The final call runs however the last iteration ended, and is deliberately unconditional: matched findings can accumulate location status updates without appending anything to dispatch, and every step is already a no-op on empty state (close_old_findings calls persist() the same way). The in-loop call keeps the size trigger only.
is_final and is_final_batch become unused and are removed. That also leaves the loop's enumerate() index unused, so the loop now iterates the batch directly.
Testing
unittests/test_reimport_final_drain.py reimports a report whose matches all take the force_continue path, so the final iteration is guaranteed to be the skipping one, and asserts the batch was still flushed. force_continue is forced rather than arranged through the corpus so the test does not depend on which finding happens to sort last.
Verified as a real gate, not just a passing test: it fails on the code before this change with exactly the dropped-work assertion, and passes after.
test_reimport_prefetch and test_import_reimport were run before and after and are byte-identical in outcome. Both carry pre-existing failures on this branch which this change neither causes nor fixes.
ruff 0.16.0 (the pinned version in requirements-lint.txt) reports no findings.