Support ragged_sort_use_single_sparsecore flag to restrict token sorting to 1 SparseCore#4513
Open
Shuwen-Fang wants to merge 1 commit into
Open
Support ragged_sort_use_single_sparsecore flag to restrict token sorting to 1 SparseCore#4513Shuwen-Fang wants to merge 1 commit into
Shuwen-Fang wants to merge 1 commit into
Conversation
Shuwen-Fang
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
michelle-yooh,
richjames0,
shralex,
suexu1025,
vipannalla and
xibinliu
as code owners
July 16, 2026 23:44
Shuwen-Fang
force-pushed
the
sort_one_sparsecore
branch
5 times, most recently
from
July 17, 2026 00:11
ec51e92 to
16ebffc
Compare
…ing to 1 SparseCore Motivation: If token sorting is on both sparsecores, sometimes it interferes with FSDP all gather scheduling and prevent it from overlapping (e.g. custom Qwen model). We can update the sorting kernels to schedule token sorting on one sparsecore only. This can make sorting slightly longer, but will enable FSDP all gathers to overlap better. XProf Traces: - Baseline (2 SparseCores): https://xprof.corp.google.com/?session_id=shuwenf-12635063694751230364 - Single SparseCore (ragged_sort_use_single_sparsecore=True): https://xprof.corp.google.com/?session_id=shuwenf-7290787811420351875
Shuwen-Fang
force-pushed
the
sort_one_sparsecore
branch
from
July 17, 2026 00:20
16ebffc to
87453fe
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
gobbleturk
approved these changes
Jul 17, 2026
gobbleturk
left a comment
gobbleturk
left a comment
Collaborator
There was a problem hiding this comment.
Either in this PR or as a followup we may want more flexibility - choosing number of SC individually of fwd combine, fwd dispatch, bwd combine, bwd dispatch
NuojCheng
reviewed
Jul 17, 2026
| ) | ||
|
|
||
| @pytest.mark.tpu_only | ||
| def test_ragged_sort_single_sparsecore(self): |
Collaborator
There was a problem hiding this comment.
why not use the self._run_ragged_sort_loss_and_grad function, which already implemented the correctness checks
NuojCheng
reviewed
Jul 17, 2026
NuojCheng
left a comment
NuojCheng
left a comment
Collaborator
There was a problem hiding this comment.
You might need to implement the same thing for ragged gather reduce v2 kernel, https://github.com/AI-Hypercomputer/maxtext/blob/main/src/maxtext/kernels/ragged/ragged_gather_reduce_v2.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
If token sorting is on both sparsecores, sometimes it interferes with FSDP all gather scheduling and prevent it from overlapping (e.g. custom Qwen model).
We can update the sorting kernels to schedule token sorting on one sparsecore only. This can make sorting slightly longer, but will enable FSDP all gathers to overlap better.
XProf Traces
ragged_sort_use_single_sparsecore=True): https://xprof.corp.google.com/?session_id=shuwenf-7290787811420351875bug: b/525027581
Checklist