Skip to content

fix: Avoid Python import race condition for Kubernetes executor stacklets - #844

Merged
sbernauer merged 8 commits into
mainfrom
fix/python-import-race-condition
Aug 13, 2026
Merged

fix: Avoid Python import race condition for Kubernetes executor stacklets#844
sbernauer merged 8 commits into
mainfrom
fix/python-import-race-condition

Conversation

@sbernauer

@sbernauer sbernauer commented Aug 13, 2026

Copy link
Copy Markdown
Member

Description

Part of #732

I noticed this while looking into SUP-396. We already have code in place to prevent Python import race-conditions by pre-cloning the git repo contents before starting the Python processes.
However, that was only done for celery-based stacklets:

    // If the DAG is modularized we may encounter a timing issue whereby the celery worker
    // has started *before* all modules referenced by the DAG have been fetched by gitsync
    // and registered. This will result in ModuleNotFoundError errors. This can be avoided
    // by running a one-off git-sync process in an init-container so that all DAG
    // dependencies are fully loaded. The sidecar git-sync is then used for regular updates.
    let use_git_sync_init_containers = matches!(executor, AirflowExecutor::CeleryExecutors { .. });

While being still on SDP 25.3 (so before our platform fix), the customer in SUP-396 correctly reported that they are using Kubernetes executors, which have the same problem [for anything except executors].

This PR fixes the mechanism to also run on Kubernetes executor-based stacklets

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

Release notes

Fixed

Avoid Python import race conditions by pre-cloning the git repo not only for Celery-based stacklets, but also for Kubernetes executor-based stacklets

@sbernauer sbernauer self-assigned this Aug 13, 2026
@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering Aug 13, 2026
@sbernauer
sbernauer requested review from a team and adwk67 August 13, 2026 06:56
@adwk67 adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Aug 13, 2026
@adwk67
adwk67 removed the request for review from a team August 13, 2026 07:12
@sbernauer sbernauer added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 13, 2026

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to change the existing gitsync test to catch this? If I see correctly, right now we just look for the config map - I am not sure whether that is enough to catch e.g. slow network. Does it make sense to add a check there to the statefulset or something like that, that we actually load the dags?

Comment thread rust/operator-binary/src/controller/build/resource/pod.rs Outdated
Comment thread rust/operator-binary/src/controller/build/resource/pod.rs Outdated
Comment thread rust/operator-binary/src/controller/build/resource/executor.rs
@sbernauer
sbernauer requested review from adwk67 and sweb August 13, 2026 10:16

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - I'd still say the real approval has to be done by Andrew

@adwk67 adwk67 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - thanks!

@sbernauer
sbernauer added this pull request to the merge queue Aug 13, 2026
@sbernauer sbernauer moved this from Development: In Review to Development: Done in Stackable Engineering Aug 13, 2026
Merged via the queue into main with commit 9a989bc Aug 13, 2026
18 checks passed
@sbernauer
sbernauer deleted the fix/python-import-race-condition branch August 13, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/26.11.0 release-note Denotes a PR that will be considered when it comes time to generate release notes. type/bug

Projects

Status: Development: Done

Development

Successfully merging this pull request may close these issues.

3 participants