Skip to content

Improve user feedback on upload error - #3

Open
joselfrias wants to merge 3 commits into
mainfrom
imp/improve_user_feedback_on_upload_error
Open

Improve user feedback on upload error#3
joselfrias wants to merge 3 commits into
mainfrom
imp/improve_user_feedback_on_upload_error

Conversation

@joselfrias

Copy link
Copy Markdown
Collaborator

Motivation

Database exceptions reached end users as stack traces or as unrelated messages. upload_data_to_tmp_table wrapped everything in except Exception: raise InvalidCSVFile("Error processing the file"), so a numeric overflow was reported as a malformed CSV and the original exception was discarded. Messages never said whether data had been saved or what to do next.

Summary

  • postgres_errors.py (new) — classifies Postgres errors by SQLSTATE into who can fix them (USER / TRANSIENT), plus is_connection_lost(), which detects a dropped connection by the absence of a SQLSTATE on an OperationalError.
  • errors.py (new) — UploadError base carrying title, what_happened, data_saved, next_steps and an incident_id. Copy is declared once per class instead of at each raise site. translate() sends the full traceback to the log and the curated object to the user. The eight exception classes previously in checks.py now live here.
  • checks.pyupload_data_to_tmp_tablevalidate_data_in_existing_mat_views, split into load / probe / cleanup. Cleanup moved into finally.
  • updates.py — archive and insert wrapped separately, so a failure names which one it was.
  • tasks.py — closes dead connections before writing status; previously the handler itself raised InterfaceError, leaving uploads stuck in STARTED with nothing shown in the UI. Switched to .filter().update(), since delete() clears the instance PK and a later rollback made save() insert a duplicate.
  • models.pyfailure_message() renders the structured payload to HTML, falling back to the plain string for older TaskResult rows.
  • settings.pypool_pre_ping, pool_recycle and TCP keepalives on the achilles engine, addressing the SSL SYSCALL error: EOF detected failures that started after Postgres moved to a remote host.

Test plan

Existing Test Suite

  • uploader/tests.py passes (imports moved to file_handler.errors, FileChecksExceptionUploadError).

File validation - each should show a title, cause and numbered next steps in the popover

  • CSV with 8 columns → "The file layout wasn't recognised"
  • Non-CSV file (e.g. .xlsx renamed) → "This file couldn't be read as a CSV"
  • Text in count_value → "Some values in the file aren't in the expected format"
  • File with analysis 0 removed → "The file is missing required metadata"
  • Two exports concatenated → "The file contains conflicting metadata rows"
  • Same file uploaded twice → "This file has already been uploaded", data_saved: true

Postgres error classification

  • Value overflowing a chart's SUM()FileDataCorrupted, not InvalidCSVFile
  • Confirm the SQLSTATE and incident id appear in the worker log for each of the above

Connection loss

  • Mid-upload, SELECT pg_terminate_backend(<pid>) on the worker's connection
    • Classified as TemporaryFailure, not InvalidCSVFile
    • PendingUpload reaches STATE_FAILED (not stuck in STARTED)
    • TaskResult row is written and the message renders in the UI
  • Repeat with the kill during update_achilles_results_data, then check
    SELECT count(*) FROM achilles_results WHERE data_source_id = <id>
    documents the partial-commit behaviour the retry work will need to fix
  • pool_pre_ping: run a query, kill the pooled connection from psql, run
    another — should succeed silently

Regression

  • A successful upload still completes, creates UploadHistory, and deletes the PendingUpload
  • achilles_results_tmp and its sequence are gone after both a successful and a failed run
  • An old TaskResult row (plain-string message) still renders readably

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant