fix: avoid 500 errors when job parameters cannot be stored - #1007
Open
aldbr wants to merge 1 commit into
Open
Conversation
Python's JSON parser accepts NaN and (-)Infinity so such values used to survive request parsing and be forwarded to OpenSearch, which rejects them with 'x_content_parse_exception ... failed to parse field [doc]', resulting in an unhandled internal server error. - Reject non-finite numbers in JobMetaData extra fields and HeartbeatData floats at the API boundary (HTTP 422) - Sanitize non-finite numbers echoed back in validation error details so the 422 response itself can be serialized to JSON - Wrap RequestError from BaseOSDB.upsert into a new DocumentUpsertError (HTTP 400) and log the offending document - Re-raise DiracErrors from TaskGroups directly rather than wrapped in an ExceptionGroup so the FastAPI exception handlers can match them Fixes DIRACGrid#582 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fstagni
reviewed
Aug 7, 2026
| return v | ||
|
|
||
| @model_validator(mode="after") | ||
| def validate_extra_fields_are_json_safe(self): |
Contributor
There was a problem hiding this comment.
Suggested change
| def validate_extra_fields_are_json_safe(self): | |
| def validate_extra_fields_are_json_safe(self) -> Self: |
with from typing_extensions import Self
Comment on lines
+100
to
+102
| for name, value in (self.model_extra or {}).items(): | ||
| _ensure_finite_numbers(value, name) | ||
| return self |
Contributor
There was a problem hiding this comment.
Suggested change
| for name, value in (self.model_extra or {}).items(): | |
| _ensure_finite_numbers(value, name) | |
| return self | |
| if self.model_extra: | |
| for name, value in self.model_extra.items(): | |
| _ensure_finite_numbers(value, name) | |
| return self |
| None, alias="BenchMark", description="Pilot benchmark value." | ||
| None, | ||
| alias="BenchMark", | ||
| allow_inf_nan=False, |
Contributor
There was a problem hiding this comment.
As I commented in the related DIRAC PR, I do not see how this can ever happen
| RequestError, and the offending document must be logged. | ||
| """ | ||
| with pytest.raises(DocumentUpsertError, match="Failed to upsert document"): | ||
| await dummy_opensearch_db.upsert("dummyvo", 2, {"IntField": float("nan")}) |
Contributor
There was a problem hiding this comment.
Test also against math.nan?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #582
PATCH /api/jobs/metadataintermittently returns an unhandled 500:Root cause: Python's JSON parser accepts the non-standard literals
NaNand(-)Infinity, so such values survive request parsing.JobMetaDataallows arbitrary extra fields with no value validation, so a non-finite float is forwarded verbatim to OpenSearch, whose strict JSON parser rejects the whole document and nothing logs the offending body.This is caused by DIRAC#6938 where the Watchdog reports
math.nanforLastUpdateCPU(s),DiskSpace(MB),MemoryUsed(MB)andLoadAveragefor any job that ends before the first watchdog cycle (20–30 min), and these reach this endpoint through the legacyJobStateUpdateClientfuture client. A companion DIRAC PR stops sending them (link to be added).This PR provides pydantic models with additional checks to reject non-finite numbers. Client gets a 422 status error instead of 500, naming the offending field.