[FLINK-40114][runtime] Don't erase leader information while a confirmation is pending

[prashantbh]

What is the purpose of the change

Fixes a regression from FLINK-36451 that can permanently erase a component's leader information from the HA backend after a leadership loss and re-acquisition.

Mechanism: leadership is granted once per process (a single issuedLeaderSessionID), but each component (restserver, dispatcher, jobmanager, resourcemanager) confirms its own leader information asynchronously on leadershipOperationExecutor, while leader-information change events are handled synchronously in notifyLeaderInformationChangeInternal. After re-acquiring leadership a component is granted but not yet confirmed, so its confirmedLeaderInformation is empty. A change event in that window takes the confirmedLeaderInformation.isEmpty() branch and publishes empty, which on Kubernetes removes the component's key. The self-correcting rewrite only runs once a confirmation exists, so the erased entry is not restored until a subsequent confirmation succeeds, and remains erased if none does.

The change event is typically self-inflicted: all components share one ConfigMap, so when one component confirms (e.g. the dispatcher), that write fires a watch event evaluated against every other component, including one whose confirmation hasn't landed yet and whose stale prior-session entry is then reset to empty.

Impact: the JobManager keeps running as a healthy leader while its external entry is gone. The REST endpoint cannot be resolved (restserver), or job submission and TaskManager registration fail (dispatcher/jobmanager), with no self-recovery.

Brief change log

• DefaultLeaderElectionService#notifyLeaderInformationChangeInternal: the confirmedLeaderInformation.isEmpty() branch now returns instead of publishing empty. That branch reset the external value on the assumption that confirmed == empty means "not leader", which only held under synchronous confirmation; with async confirmation it also means "leader, but not yet confirmed", so deleting acts on a transient state and destroys still-valid information. Returning is safe: if we hold the lock a pending (session-fenced) confirmation overwrites any stale value, and if we do not, the write would no-op anyway. Intentional cleanup is unchanged (remove -> deleteLeaderInformation).

Verifying this change

Added tests that use ManuallyTriggeredScheduledExecutorService to hold a confirmation while a change event is injected; they fail without the fix and pass with it:

• testLeaderInformationChangeDoesNotEraseComponentWithInflightConfirmation: a change event during an outstanding confirmation must not erase the entry; it self-heals once the confirmation runs.
• testLeaderInformationChangeDoesNotEraseComponentAfterReacquisition: loss and re-acquisition with a new session; the prior session's entry must not be erased and self-heals to the new session.
• Repurposed testGrantDoesNotBlockNotifyLeaderInformationChange and ...AllKnownLeaderInformation to assert non-erasure for both onLeaderInformationChange overloads.
• Also verified against the leaderelection / leaderretrieval / highavailability suites (including ZooKeeper and LeaderChangeClusterComponentsTest) and the flink-kubernetes HA unit tests.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes, fixes HA leader-election behavior affecting JobManager recovery on Kubernetes and ZooKeeper.
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes (Claude Code / Claude Opus 4.8)

Generated-by: Claude Code (Claude Opus 4.8)

[flinkbot]

CI report:

• 93d9e1e Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build