Skip to content

fix(email_scan): tolerate attributes on screener's errorlist markup - #617

Merged
kaifcodec merged 1 commit into
kaifcodec:mainfrom
ARAVIND281:fix/screener-errorlist-attributes
Aug 31, 2026
Merged

fix(email_scan): tolerate attributes on screener's errorlist markup#617
kaifcodec merged 1 commit into
kaifcodec:mainfrom
ARAVIND281:fix/screener-errorlist-attributes

Conversation

@ARAVIND281

Copy link
Copy Markdown
Contributor

Fixes #611

email_scan/other/screener.py returned Unexpected response body structure for every address, because the availability check matched the Django error list by exact tag text:

if '<ul class="errorlist"><li>This field is required.</li>' in res_text:

screener.in now renders that list with an id:

<ul class="errorlist" id="id_password_error"><li>This field is required.</li></ul>

<ul class="errorlist" is no longer immediately followed by >, so the substring test fails and every scan falls through to the error branch.

Fix

Match the tag by its class and the message it wraps, rather than verbatim:

_REQUIRED_FIELD_ERROR = re.compile(
    r'<ul[^>]*\sclass="[^"]*\berrorlist\b[^"]*"[^>]*>\s*<li>\s*This field is required\.\s*</li>',
    re.IGNORECASE,
)

This is slightly broader than the regex suggested in the issue: it also tolerates attribute reordering (<ul id="..." class="errorlist">), extra classes (class="errorlist nonfield"), and pretty-printed whitespace between the <ul> and its <li> — so the next markup change doesn't reopen this.

It stays narrow in the ways that matter: the message text is still required, and the errorlist class is matched with \b boundaries, so a different error (Enter a valid email address.) or a different list (class="helptext") is not read as available.

The taken branch matches plain text (User account with this Email already exists) and is untouched.

Verification

Live against screener.in, with an unregistered address:

before:  [!] Screener: Error (Unexpected response body structure, report it via GitHub issues)
after:   [✘] Screener: Not Registered

Local gates from AGENTS.md, all clean:

ruff check .        All checks passed!
mypy user_scanner   Success: no issues found in 529 source files
pytest              380 passed, 3 skipped

Note on the taken path: I verified the available path live, but didn't probe a real registered account to exercise taken, since that means testing against someone's actual email. That branch isn't touched by this diff — happy to confirm it another way if you'd prefer.

I did initially write a mocked regression test for the marker, then dropped it after re-reading AGENTS.md ("Do not add unit tests for individual scan modules"). Let me know if a test would be welcome here as an exception, since this is a detection-regression rather than a new module — I have it ready.

screener.py reported "Unexpected response body structure" for every
address. The availability check matched the error list by exact tag text:

    '<ul class="errorlist"><li>This field is required.</li>'

screener.in now renders that list with an id, as Django emits on field
errors:

    <ul class="errorlist" id="id_password_error"><li>This field is required.</li></ul>

`<ul class="errorlist"` is no longer immediately followed by `>`, so the
substring test failed and every scan fell through to the error branch.

Match the tag by its class and the message it wraps instead, so the check
also survives further attribute additions or reordering. The "already
exists" (taken) branch matches plain text and is unchanged.

Verified live against screener.in with an unregistered address:
before, "Error (Unexpected response body structure)"; after, "Not
Registered". ruff, mypy and pytest all clean.

Fixes kaifcodec#611
@kaifcodec

Copy link
Copy Markdown
Owner

@ARAVIND281 Can you test it again?
I am getting Unexpected response body... error.

@ARAVIND281

Copy link
Copy Markdown
Contributor Author

Re-tested just now, and it passes here — so let's find where our setups differ rather than me just saying "works for me".

A/B on the same machine, same network, same address, minutes apart:

on main (no fix):
  [!] Screener (zq7x4m2p9v.notreal8815@gmail.com): Error (Unexpected response body structure, report it via GitHub issues)

on fix/screener-errorlist-attributes:
  [✘] Screener (zq7x4m2p9v.notreal8815@gmail.com): Not Registered

What the site returns right now, replaying the module's own two-step flow:

GET  200  csrf=True  token=True
POST 200  len=20210
  'already exists' marker : False
  OLD exact-string match  : False      <- the bug
  NEW regex match         : True       <- the fix
  errorlist seen: <ul class="errorlist" id="id_password_error"><li>This field is required.</li></ul>

So the markup is still the id="id_password_error" variant the issue describes, and the old exact-string test still misses it.

Most likely explanation: the error message you're seeing is exactly what main produces, so I suspect the run picked up the released/installed copy rather than this branch. Worth checking:

gh pr checkout 617
python -c "import user_scanner, os; print(os.path.dirname(user_scanner.__file__))"   # should be your worktree, not site-packages
python -m user_scanner -e <some-unregistered-address> -m screener

If it still fails after that, then we differ somewhere I can't see and I'd like the detail rather than guessing. The useful thing would be the raw response — whether 'already exists' is present, whether the errorlist block appears at all, and what its opening tag looks like on your side. If screener.in serves different markup by region, or a WAF page to your IP, the errorlist line would come back NONE and that tells us immediately.

One thing I did not test: the taken branch. That path matches plain text (User account with this Email already exists) and is untouched by this diff, and I didn't want to probe a stranger's registered address to exercise it. If you have an account you're happy to test with, that would close the last gap.

@kaifcodec kaifcodec left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ARAVIND281 Most likely the issue was on my end. I likely had forgot to switch the branch.

LGTM. Merging it now.

Thank you for the PR and feel free to contribute further to the project.

@kaifcodec kaifcodec added the bug fix Fixed a bug in existing files label Aug 31, 2026
@kaifcodec
kaifcodec merged commit 52fc106 into kaifcodec:main Aug 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix Fixed a bug in existing files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

screener: "Unexpected response body structure" for every address — the errorlist markup gained an id attribute

2 participants