Skip to content

DSV4 NNX support#4497

Open
snehalv2002 wants to merge 2 commits into
mainfrom
dsv4-nnx-support
Open

DSV4 NNX support#4497
snehalv2002 wants to merge 2 commits into
mainfrom
dsv4-nnx-support

Conversation

@snehalv2002

@snehalv2002 snehalv2002 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR adds DeepSeek V4 (DEEPSEEK4) decoder block support and HyperHead routing integration to NNXDecoder.

Context & Implementation Details

  • Instantiates DeepSeek4HyperHead on NNXDecoder when mhc_expansion_rate > 1 and decoder_block == DecoderBlockType.DEEPSEEK4.
  • Routes the post-transformer hidden state reduction in NNXDecoder through self.hc_head(y) for DEEPSEEK4 instead of the default mhc_reduce(y).
  • Added DecoderBlockType.DEEPSEEK4 to the list of supported decoder block types in NNXDecoder.

Tests

  • Added unit test test_nnx_decoder_hyperhead_integration in tests/unit/deepseek_v4_vs_reference_test.py to verify NNXDecoder initializes hc_head as DeepSeek4HyperHead.
  • Ran pytest tests/unit/deepseek_v4_vs_reference_test.py.

Checklist

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@snehalv2002
snehalv2002 changed the base branch from main to dsv4-bug-fixes July 15, 2026 23:51
@snehalv2002
snehalv2002 force-pushed the dsv4-bug-fixes branch 6 times, most recently from 0d18a3e to 7f4b99f Compare July 16, 2026 23:15
…e.py, and match maxtext sinkhorn implementation to HF reference.
@snehalv2002
snehalv2002 marked this pull request as ready for review July 17, 2026 19:56
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 20.00000% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/nnx_decoders.py 20.00% 2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@snehalv2002 snehalv2002 changed the title Dsv4 nnx support DSV4 NNX support Jul 17, 2026
Base automatically changed from dsv4-bug-fixes to main July 18, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant