Skip to content

feat: add pilot database and router - #570

Merged
aldbr merged 34 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management
Aug 5, 2026
Merged

feat: add pilot database and router#570
aldbr merged 34 commits into
DIRACGrid:mainfrom
Robin-Van-de-Merghel:robin-pilot-management

Conversation

@Robin-Van-de-Merghel

@Robin-Van-de-Merghel Robin-Van-de-Merghel commented Jun 13, 2025

Copy link
Copy Markdown
Contributor

Split of #421 , first part : pilot management

closes #960

@Robin-Van-de-Merghel
Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from 8c655b0 to 2cfe44a Compare June 13, 2025 11:19
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
@Robin-Van-de-Merghel
Robin-Van-de-Merghel marked this pull request as draft June 13, 2025 11:36
Comment thread diracx-db/src/diracx/db/sql/utils/functions.py Outdated
Comment thread diracx-logic/src/diracx/logic/auth/token.py
Comment thread diracx-core/src/diracx/core/settings.py
Comment thread diracx-routers/src/diracx/routers/auth/token.py
Comment thread diracx-routers/src/diracx/routers/utils/users.py
Comment thread diracx-routers/tests/auth/test_standard.py
Comment thread diracx-testing/src/diracx/testing/utils.py
@Robin-Van-de-Merghel
Robin-Van-de-Merghel marked this pull request as ready for review June 15, 2025 06:13
Comment thread diracx-logic/src/diracx/logic/pilots/query.py Outdated

@fstagni fstagni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review.

Comment thread diracx-db/src/diracx/db/sql/utils/base.py Outdated
Comment thread diracx-core/src/diracx/core/exceptions.py Outdated
Comment thread diracx-db/src/diracx/db/sql/job/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-core/src/diracx/core/models.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-db/tests/pilots/test_query.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-routers/tests/pilots/test_pilot_creation.py Outdated
@Robin-Van-de-Merghel

Robin-Van-de-Merghel commented Jun 24, 2025

Copy link
Copy Markdown
Contributor Author

Where an issue?

Whether we have a base router with require_auth or not, we won't be able to override it in its children (cf #417 ).
So for pilots we must have require_auth=False at the base router because of secret-exchange. But now every pilot endpoint is opened.

Possibilities

Splitting

Let's start with the routers themselves. We can separate pilots and users endpoints : /api/pilots (only for pilots) and ~/api/pilot_management (only for users).

That brings us:

  1. Cleaner: each their own territory, no overlapping, easier to read (you know that on /pilots its only pilot resources)
  2. More secure: we can add a dependency to the pilot router (same as verify_dirac_pilot_token but for pilots)
  3. We can have require_auth for /pilot_management, and enforce tokens

Using another approach

We could have a bare base router without dependency injection, with sub routers with verify_dirac_access_token:

# Authenticated parent router
protected_router = APIRouter(dependencies=[Depends(verify_dirac_access_token)])

@protected_router.get("/secure")
def secure_route():
    return {"msg": "secure"}

# Public sub-router (no auth)
public_router = APIRouter()

@public_router.get("/public")
def public_route():
    return {"msg": "public"}

Its goal would be to replacer DiracxRouter by fixing the issue, and keeping an explicit depedency injection.
Or have @router.authentificated.get(...) instead of @router.get, to explicitely say if we want auth or not

Comment thread diracx-routers/src/diracx/routers/pilots/query.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-db/src/diracx/db/sql/pilots/db.py Outdated
Comment thread diracx-routers/src/diracx/routers/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
Comment thread diracx-logic/src/diracx/logic/pilots/management.py Outdated
@Robin-Van-de-Merghel
Robin-Van-de-Merghel force-pushed the robin-pilot-management branch 2 times, most recently from a9b353d to 4ba2c9b Compare July 4, 2025 11:48
@fstagni
fstagni force-pushed the robin-pilot-management branch from 070de3e to afe2c12 Compare July 10, 2026 16:02
aldbr and others added 10 commits August 5, 2026 14:15
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JobToPilotMapping has no foreign key in the legacy schema, so the
IntegrityError branch mapping FK violations to PilotNotFoundError could
never fire. Existence checking stays in the logic layer; the branch can
be reintroduced once a FK lands on the DIRAC side (DIRAC#8097).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the PilotStamp index (absent from legacy DIRAC; adding it must go
through a DIRAC-side migration with an update-guide entry) and add
idx_dest_queue_status, present in legacy since DIRAC v9.1.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /api/jobs/jdl and POST /api/pilots/ now return 201 instead of 200,
per RFC 9110. Clients regenerated (diracx + gubbins). Breaking for
previously released clients, whose generated operations only accept 200
from submit_jdl_jobs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HTTP-level tests stub the access policy, so read scoping cannot be
exercised through the routes; seed a second VO directly in the DB and
assert the logic layer's vo_constraint filters search and summary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No route passes job_db/job_ids in this PR; the branch belongs with the
future job-to-pilot assignment route (pilot token work) and was
untested here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirror the PilotAgents column sizes as max_length on the pilot models
so oversized values are rejected with a 422 instead of a
backend-dependent database error (strict MySQL raises DataError, SQLite
silently accepts). Clients regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aldbr
aldbr enabled auto-merge (squash) August 5, 2026 14:53
@aldbr
aldbr merged commit 503d3d2 into DIRACGrid:main Aug 5, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pilot Manager service and DB in DiracX Open access and require auth not working inside a router

4 participants