Skip to content

Harden JSON unmarshaling in job rescuer + job executor#1324

Open
brandur wants to merge 1 commit into
masterfrom
brandur-harden-json-unmarshaling
Open

Harden JSON unmarshaling in job rescuer + job executor#1324
brandur wants to merge 1 commit into
masterfrom
brandur-harden-json-unmarshaling

Conversation

@brandur

@brandur brandur commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Here, do a little more to harden unmarshaling in the job rescuer and job
executor. Previously, if a job's JSON didn't unmarshal successfully,
we'd log an error and move on, which could result in unexpected trouble.

Realistically, this would be a very rare circumstance though luckily --
the Postgres/SQLite jsonb fields guarantee the formatting of the JSON
data, so the only way for an unmarshaling problem to occur is if a job's
data was stored initially, and then the JSON Go struct later change to
be incompatible (i.e. string changed to an int or something of that
nature).

Here, during an unmarshal error, use the job's standard retry schedule
and back off, discarding the job if it's at the end of its allowed
retries. This approach is best because it gives the user a chance to
notice the failure and correct a potential unmarshaling problem by
either manipulating the job row's data or fixing their Go struct.

Fixes #1323, but also addresses a similar problem in the job executor
that Codex found while working.

@brandur
brandur force-pushed the brandur-harden-json-unmarshaling branch from d3f0011 to 51b5b61 Compare July 22, 2026 18:18
@brandur brandur changed the title Track unhealthy heartbeat + pause job fetching when unhealthy Harden JSON unmarshaling in job rescuer + job executor Jul 22, 2026
@brandur

brandur commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Tool failure on this one that caused the original title to be wrong. Correct PR title/description now.

Here, do a little more to harden unmarshaling in the job rescuer and job
executor. Previously, if a job's JSON didn't unmarshal successfully,
we'd log an error and move on, which could result in unexpected trouble.

Realistically, this would be a very rare circumstance though luckily --
the Postgres/SQLite `jsonb` fields guarantee the formatting of the JSON
data, so the only way for an unmarshaling problem to occur is if a job's
data was stored initially, and then the JSON Go struct later change to
be incompatible (i.e. string changed to an int or something of that
nature).

Here, during an unmarshal error, use the job's standard retry schedule
and back off, discarding the job if it's at the end of its allowed
retries. This approach is best because it gives the user a chance to
notice the failure and correct a potential unmarshaling problem by
either manipulating the job row's data or fixing their Go struct.

Fixes #1323, but also addresses a similar problem in the job executor
that Codex found while working.
@brandur
brandur force-pushed the brandur-harden-json-unmarshaling branch from 51b5b61 to ece9af6 Compare July 22, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

job_rescuer: UnmarshalJob error doesn't short-circuit makeRetryDecision, causing incorrect rescue behavior

2 participants