Skip to content

fix(reimporter): drain the last partial batch however the loop ended - #15762

Open
devGregA wants to merge 1 commit into
DefectDojo:bugfixfrom
devGregA:fix/reimport-final-drain
Open

fix(reimporter): drain the last partial batch however the loop ended#15762
devGregA wants to merge 1 commit into
DefectDojo:bugfixfrom
devGregA:fix/reimport-final-drain

Conversation

@devGregA

Copy link
Copy Markdown
Contributor

process_findings() accumulates per-batch work and flushes it inside the per-finding loop, gated on len(batch_finding_ids) >= dedupe_batch_max_size or is_final.

The matched branch ends in continue. That skip is what makes the tail fragile: the flush sits after it in the loop body, so a report whose LAST finding took the matched branch never reached the is_final flush, and everything appended since the previous size-triggered flush was dropped. Silently, because nothing raises.

What is lost for those findings: deduplication, rules, the issue updater and JIRA dispatch, plus their parser and inherited tags. Pending location status updates, vulnerability ids and burp request/response pairs are discarded with them.

A reimport whose final finding matches an existing one is the ordinary case rather than an edge case, so this fires on unchanged re-syncs.

The fix extracts the flush into _flush_post_processing_batch() and calls it once after the loop instead of relying on a flag inside it. The final call runs however the last iteration ended, and is deliberately unconditional: matched findings can accumulate location status updates without appending anything to dispatch, and every step is already a no-op on empty state (close_old_findings calls persist() the same way). The in-loop call keeps the size trigger only.

is_final and is_final_batch become unused and are removed. That also leaves the loop's enumerate() index unused, so the loop now iterates the batch directly.

Testing

unittests/test_reimport_final_drain.py reimports a report whose matches all take the force_continue path, so the final iteration is guaranteed to be the skipping one, and asserts the batch was still flushed. force_continue is forced rather than arranged through the corpus so the test does not depend on which finding happens to sort last.

Verified as a real gate, not just a passing test: it fails on the code before this change with exactly the dropped-work assertion, and passes after.

test_reimport_prefetch and test_import_reimport were run before and after and are byte-identical in outcome. Both carry pre-existing failures on this branch which this change neither causes nor fixes.

ruff 0.16.0 (the pinned version in requirements-lint.txt) reports no findings.

process_findings() accumulates per-batch work and flushes it inside the per-finding
loop, gated on `len(batch_finding_ids) >= dedupe_batch_max_size or is_final`.

The matched branch ends in `continue`. That skip is what makes the tail fragile: the
flush sits after it in the loop body, so a report whose LAST finding took the matched
branch never reached the is_final flush, and everything appended since the previous
size-triggered flush was dropped. Silently, because nothing raises. Those findings got
no deduplication, no rules, no issue updater and no JIRA dispatch, and none of their
parser or inherited tags; pending location status updates, vulnerability ids and burp
request/response pairs were discarded with them.

A reimport whose final finding matches an existing one is the ordinary case, not an
edge case, so this fires on unchanged re-syncs rather than on unusual reports.

Extracts the flush into _flush_post_processing_batch() and calls it once after the
loop instead of relying on a flag inside it. The final call runs however the last
iteration ended, and is deliberately unconditional: matched findings can accumulate
location status updates without appending anything to dispatch, and every step is
already a no-op on empty state (close_old_findings calls persist() the same way). The
in-loop call keeps the size trigger only.

is_final and is_final_batch are now unused and removed, which also leaves the loop's
enumerate() index unused, so the loop iterates the batch directly.

Tests: unittests/test_reimport_final_drain.py reimports a report whose matches all
take the force_continue path, so the final iteration is guaranteed to be the skipping
one, and asserts the batch was still flushed. It fails on the code before this change
with exactly the dropped-work assertion and passes after. test_reimport_prefetch and
test_import_reimport were compared before and after and are unchanged (both carry
pre-existing failures on this branch that this change neither causes nor fixes).
ruff 0.16.0 clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dryrunsecurity

Copy link
Copy Markdown

DryRun Security

This pull request contains a critical finding where a sensitive codepath in dojo/importers/default_reimporter.py was modified by an author not on the allowed list. Although the issue is marked as failing, it is not currently blocking the merge.

🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/importers/default_reimporter.py (drs_f6dd02a4)
Vulnerability Configured Sensitive Codepath Modified by Non-Allowed Author
Description File 'dojo/importers/default_reimporter.py' matches configured sensitive codepath pattern 'dojo/importers/*.py' and was modified by '' (commit d3b0fb6) who is not in the allowed authors list.

We've notified @mtesauro.


Comment to provide feedback on these findings.

Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]

Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing

All finding details can be found in the DryRun Security Dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant