Bump the pip group across 2 directories with 4 updates by dependabot[bot] · Pull Request #4527 · AI-Hypercomputer/maxtext

dependabot · 2026-07-17T20:05:15Z

Bumps the pip group with 1 update in the /src/dependencies/requirements/base_requirements directory: setuptools.
Bumps the pip group with 3 updates in the /src/maxtext/inference/mlperf directory: sentencepiece, transformers and nltk.

Updates setuptools from 78.1.0 to 78.1.1

Changelog

Sourced from setuptools's changelog.

v78.1.1

Bugfixes

More fully sanitized the filename in PackageIndex._download. (#4946)

Commits

8e4868a Bump version: 78.1.0 → 78.1.1
100e9a6 Merge pull request #4951
8faf1d7 Add news fragment.
2ca4a9f Rely on re.sub to perform the decision in one expression.
e409e80 Extract _sanitize method for sanitizing the filename.
250a6d1 Add a check to ensure the name resolves relative to the tmpdir.
d8390fe Extract _resolve_download_filename with test.
4e1e893 Merge https://github.com/jaraco/skeleton
3a3144f Fix typo: pyproject.license -> project.license (#4931)
d751068 Fix typo: pyproject.license -> project.license
Additional commits viewable in compare view

Updates sentencepiece from 0.1.99 to 0.2.1

Release notes

Sourced from sentencepiece's releases.

v0.2.1

Major changes

[Python] Supported wheels and builds for Python 3.13 and 3.14(rc1) #1134, #1127, #1121, #1111, #1110, #1104, #1103, #1099, #1091

[Python] Added an experimental support for free-threading. #1134, #1127, #1110 https://github.com/google/sentencepiece/tree/master/python#free-threading-support

[Python] Updated the supported Python version to 3.9 or later.

New features

[ALL]: Added new build mode to prevent the precompiled normalization rules being embedded in *.so and *.a. (-DSPM_DISABLE_EMBEDDED_DATA=ON). This reduces the runtime size by approximately 1-2 MB. This mode is enabled to build python wheels. The rules are loaded as the data package.

Bug fixes & minor changes

[ALL]: Security fix to address a heap overflow issue that could occur when using a model containing an invalid precompiled normalization model.

[Python]: Deprecates the wheel package for Linux i686.

[Python]: Supported wheel for Windows Arm64. #1114

[Python]: Fixed the crash issue on batch decoding #1051

[ALL]: Updated the Unicode normalization rule with the latest ICU/Unicode rules.

[ALL]: Unused code and build mode cleanup.

v0.2.0

Major changes

N/A

New features

[ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample

[ALL] Added SentencePieceProcessor::Normalize method in C++/Python Python Sample C++ Sample

[ALL] Added functionality to override the normalization spec before the processing. Python Sample

Bug fixes & minor changes

Introduce better support of using external abseil and protobuf google/sentencepiece#869

Build universal binary in OSX release package google/sentencepiece#892

Add the set_min_log_level function to python to change the loglevel from the python wrapper. google/sentencepiece#893

Uses the logsumexp techniques in marginal probabilities of n-best tokenization to avoid underflow.

Support Python 3.12 google/sentencepiece#932

Improves the thread utilization in batch encoding/decoding.

Fix nasty bug in BPE position encoding.

Fix bugs in the handling of duplicated bigrams

Commits

31646a4 Merge pull request #1136 from crusaderky/pytest-run-parallel
bcd44b9 free-threading tests
135747f install twine before checking wheel
69fe0b2 install setuptools before making sdist
ee1422b install setuptools before making sdist
5ac2fd2 use windows-11-arm runner to test ARM64 wheel on native env.
36b9745 use windows-11-arm runner to test ARM64 wheel on native env.
4f043ae use auto-mode to make wheel with the native binary.
623196e uses arm docker image to build and test wheel
559fd65 re-enable QEMU to enable arm execution
Additional commits viewable in compare view

Updates transformers from 4.31.0 to 5.5.0

Release notes

Sourced from transformers's releases.

Release v5.5.0

New Model additions

Gemma4

Gemma 4 is a multimodal model with pretrained and instruction-tuned variants, available in 1B, 13B, and 27B parameters. The architecture is mostly the same as the previous Gemma versions. The key differences are a vision processor that can output images of fixed token budget and a spatial 2D RoPE to encode vision-specific information across height and width axis.

You can find all the original Gemma 4 checkpoints under the Gemma 4 release.

The key difference from previous Gemma releases is the new design to process images of different sizes using a fixed-budget number of tokens. Unlike many models that squash every image into a fixed square (like 224×224), Gemma 4 keeps the image's natural aspect ratio while making it the right size. There a a couple constraints to follow:

The total number of pixels must fit within a patch budget

Both height and width must be divisible by 48 (= patch size 16 × pooling kernel 3)

[!IMPORTANT] Gemma 4 does not apply the standard ImageNet mean/std normalization that many other vision models use. The model's own patch embedding layer handles the final scaling internally (shifting values to the [-1, 1] range).

The number of "soft tokens" (aka vision tokens) an image processor can produce is configurable. The supported options are outlined below and the default is 280 soft tokens per image.

Soft Tokens Patches (before pooling) Approx. Image Area

70 630 ~161K pixels

140 1,260 ~323K pixels

280 2,520 ~645K pixels

560 5,040 ~1.3M pixels

1,120 10,080 ~2.6M pixels

To encode positional information for each patch in the image, Gemma 4 uses a learned 2D position embedding table. The position table stores up to 10,240 positions per axis, which allows the model to handle very large images. Each position is a learned vector of the same dimensions as the patch embedding. The 2D RoPE which Gemma 4 uses independently rotate half the attention head dimensions for the x-axis and the other half for the y-axis. This allows the model to understand spatial relationships like "above," "below," "left of," and "right of."

NomicBERT

NomicBERT is a BERT-inspired encoder model that applies Rotary Position Embeddings (RoPE) to create reproducible long context text embeddings. It is the first fully reproducible, open-source text embedding model with 8192 context length that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short-context MTEB and long context LoCo benchmarks. The model generates dense vector embeddings for various tasks including search, clustering, and classification using specific instruction prefixes.

Links: Documentation | Paper

Internalise the NomicBERT model (#43067) by @ed22699 in #43067

MusicFlamingo

Music Flamingo is a fully open large audio–language model designed for robust understanding and reasoning over music. It builds upon the Audio Flamingo 3 architecture by including Rotary Time Embeddings (RoTE), which injects temporal position information to enable the model to handle audio sequences up to 20 minutes. The model features a unified audio encoder across speech, sound, and music with special sound boundary tokens for improved audio sequence modeling.

Links: Documentation | Paper

Add Music Flamingo (#43538) by @lashahub in #43538

... (truncated)

Commits

c1c3424 update
20bff68 update release workflow
8956441 v5.5.0
5135e5e casually dropping the most capable open weights on the planet (#45192)
a594e09 Internalise the NomicBERT model (#43067)
4932e97 Fix resized LM head weights being overwritten by post_init (#45079)
57e8413 [Qwen3.5 MoE] Add _tp_plan to ForConditionalGeneration (#45124)
b10552e Fix TypeError: 'NoneType' object is not iterable in GenerationMixin.generate ...
423f2a3 fix(models): Fix dtype mismatch in SwitchTransformers and TimmWrapperModel (#...
ade7a05 Generalize gemma vision mask to videos (#45185)
Additional commits viewable in compare view

Updates nltk from 3.8.1 to 3.10.0

Release notes

Sourced from nltk's releases.

v3.10.0-rc1

What's Changed

CHILDES corpus reader: lazy loading (fixes #1340) by @nschneid in nltk/nltk#1341

Stanseg by @casperlehmann in nltk/nltk#1350

Add optional CCG semantics computation by @tanin47 in nltk/nltk#1321

address of free software foundation updated by @Lightyagami1 in nltk/nltk#1347

Added a verification for empty tokens by @gnardari in nltk/nltk#1353

use data.load for vader lexicon by @afgiel in nltk/nltk#1303

Ensure that proper exceptions are caught correctly by @pombredanne in nltk/nltk#1343

Reset CCGVar everytime we invoke lexicon.fromstring() and skip the unicode test because it fails on python2.7 by @tanin47 in nltk/nltk#1354

Add tanin as an author :) by @tanin47 in nltk/nltk#1355

framenet.py: remove unnecessary str() call that causes problems for names with Unicode characters by @nschneid in nltk/nltk#1360

remove u because it doesn't work in Python 3.0 to 3.2 by @tanin47 in nltk/nltk#1362

moved the import for dispersion_plot out of the else block by @dennisobrien in nltk/nltk#1364

Raise warning for infinite recursion by @alvations in nltk/nltk#1365

Patch None TypeError in align_blocks() by @alvations in nltk/nltk#1366

Safer try-except for catching infinite recursion by @alvations in nltk/nltk#1368

Collections refactor by @nschneid in nltk/nltk#1382

Handles fringe cases in BLEU by @alvations in nltk/nltk#1383

Allow specifying substitution cost in edit distance by @dnc1994 in nltk/nltk#1386

Pl196x fixes by @pkasprzyk in nltk/nltk#1359

Docs/ngrams by @ColCarroll in nltk/nltk#1403

Fix conll.py by @germanferrero in nltk/nltk#1402

Aline by @geoffbacon in nltk/nltk#1407

Handle empty files in a corpus by @alexisdimi in nltk/nltk#1411

fixed bug: StackDecoder would throw exception if it's unable to finish… by @hikui in nltk/nltk#1405

add hikui to authors by @hikui in nltk/nltk#1412

removes download-cache pip flag from tox config by @fievelk in nltk/nltk#1415

Added three tokenizers, one detokenizer and two tokenizer-related word/char list corpora by @alvations in nltk/nltk#1282

Much faster Vader sentiment tagging by @georgeberry in nltk/nltk#1409

Fixes doctest in is_cjk() and REPP tokenizer wrapper by @alvations in nltk/nltk#1420

Fixed doctest for UnicharsCorpusReader and NonbreakingPrefixesCorpusReader by @alvations in nltk/nltk#1422

Updated senna version in docstring by @alvations in nltk/nltk#1431

Removes Python 2.6 and Python 3.1-3.3 code, Improves Python 3.5 syntax by @adamn in nltk/nltk#1429

Cleanup by @dimazest in nltk/nltk#1432

Fix zero division & log(0) in PunktTrainer._col_log_likelihood by @mrecachinas in nltk/nltk#1247

Update stanford_segmenter.py by @janissl in nltk/nltk#1444

explained data format for constructor by @drevicko in nltk/nltk#1450

Use a more efficient idiom for flattening nested lists by @rmalouf in nltk/nltk#1447

improved documentation of load_array() by @drevicko in nltk/nltk#1452

changed "list or iterator" to "sequence" by @drevicko in nltk/nltk#1453

Deal gracefully with degenerate cases. by @drevicko in nltk/nltk#1454

deal with (hopefully) all degenerate cases by @drevicko in nltk/nltk#1455

Enable unordered comparison for gaac clustering by @alvations in nltk/nltk#1456

Rewrite porter.py by @ExplodingCabbage in nltk/nltk#1261

Update naivebayes.py by @andyreagan in nltk/nltk#1461

Fixes bug where download status not returned. by @nsfabina in nltk/nltk#1468

Use lcon and rcon variables in conllned() when trace is False - Issue #1474 by @timleslie in nltk/nltk#1475

Use shutil.copyfileobj for unzipping by @ikorolev93 in nltk/nltk#1478

Fixing Chen&Cherry method 6 smoothing in BLEU score by @alvations in nltk/nltk#1477

... (truncated)

Changelog

Sourced from nltk's changelog.

Version 3.10.0 2026-06-11

Enforce the stricter nltk.pathsec security policy by default

Document the new security model and migration guidance

Harden resource loading against path traversal and SSRF/DNS-rebinding

Harden downloader path handling and block XML entity expansion

Close remaining corpus-reader security edge cases

Replace unsafe exec() usage in the utility CLI

Warn on unpickling user-provided pickles

Add HuggingFace datasets integration (nltk.huggingface)

Align TnT with Brants (2000) specifications

Fix PorterStemmer irregular-form lowercasing in NLTK mode

Fix TransitionParser sparse index dtype for scikit-learn 1.9

Fix TextCat tie handling

Fix WordNet object comparisons for incompatible types

Cache WordNet max depth lazily for lch_similarity()

Fix CCG variable direction, substitution, and type-raising bugs

Fix Jaro similarity for single-character and empty-string cases

Improve CI and release-maintenance workflows

Thanks to the following contributors to 3.10.0: 13rac1, alvations, bowiechen, devesh-2002, ekaf, elias-ba, haosenwang1018, HyperPS, ihitamandal, jancallewaert, jhnwnstd, JuanIMartinezB, Lemm1, LinZiyuu, Mr-Neutr0n, PastelStorm, scruge1, Syzygy2048, ylwango613, yzhaoinuw

Version 3.9.4 2026-03-24

Support Python 3.14

Fix bug in Levenshtein distance when substitution_cost > 2

Fix bug in Treebank detokeniser re quote ordering

Fix bug in Jaro similarity for empty strings

Several security enhancements

Fix GHSA-rf74-v2fm-23pw: unbounded recursion in JSONTaggedDecoder

Implement TextTiling vocabulary introduction method (Hearst 1997)

Fix ALINE feature matrix errors and add comprehensive tests

Support multiple VerbNet versions, fix longid/shortid regex for VerbNet ids

Let downloader fallback to md5 when sha256 is unavailable

Several other minor bugfixes and code cleanups

Thanks to the following contributors to 3.9.4: Min-Yen Kan, Eric Kafe, Emily Voss, bowiechen, Hrudhai01, jancallewaert, Mr-Neutr0n, pollak.peter89, ylwango613,

Version 3.9.3 2026-02-21

Fix CVE-2025-14009: secure ZIP extraction in nltk.downloader (#3468)

Block path traversal/arbitrary reads in nltk.data for protocol-less refs (#3467)

Block path traversal/abs paths in corpus readers and FS pointers (#3479, #3480)

Validate external StanfordSegmenter JARs using SHA256 (#3477)

... (truncated)

Commits

bd49f90 allow escaped brackets in Tree.fromstring (#3694)
27b8ad6 don't crash chomsky_normal_form on terminals with siblings (#3693)
52227d2 Use os.name for Windows path handling (#3605)
06c0e2c Avoid RIBES zero division on empty inputs (#3604)
a167389 Treat missing unzip output as stale (#3607)
c94c967 Fix EOF empty document bug in IEER corpus reader (#3648)
94a259c Enforce restrictive primitive type checking in pathsec wrappers (#3692)
5ac475d fix(security): isolate Stanford Java options and clean temp files (#3683)
986f26e ci(deps): bump the github-actions group with 3 updates (#3691)
f26b375 fix(security): prevent pickle RCE in TransitionParser model loading (CWE-502)...
Additional commits viewable in compare view

Updates transformers from 4.31.0 to 5.5.0

Release notes

Sourced from transformers's releases.

Release v5.5.0

New Model additions

Gemma4

Gemma 4 is a multimodal model with pretrained and instruction-tuned variants, available in 1B, 13B, and 27B parameters. The architecture is mostly the same as the previous Gemma versions. The key differences are a vision processor that can output images of fixed token budget and a spatial 2D RoPE to encode vision-specific information across height and width axis.

You can find all the original Gemma 4 checkpoints under the Gemma 4 release.

The key difference from previous Gemma releases is the new design to process images of different sizes using a fixed-budget number of tokens. Unlike many models that squash every image into a fixed square (like 224×224), Gemma 4 keeps the image's natural aspect ratio while making it the right size. There a a couple constraints to follow:

The total number of pixels must fit within a patch budget

Both height and width must be divisible by 48 (= patch size 16 × pooling kernel 3)

[!IMPORTANT] Gemma 4 does not apply the standard ImageNet mean/std normalization that many other vision models use. The model's own patch embedding layer handles the final scaling internally (shifting values to the [-1, 1] range).

The number of "soft tokens" (aka vision tokens) an image processor can produce is configurable. The supported options are outlined below and the default is 280 soft tokens per image.

Soft Tokens Patches (before pooling) Approx. Image Area

70 630 ~161K pixels

140 1,260 ~323K pixels

280 2,520 ~645K pixels

560 5,040 ~1.3M pixels

1,120 10,080 ~2.6M pixels

To encode positional information for each patch in the image, Gemma 4 uses a learned 2D position embedding table. The position table stores up to 10,240 positions per axis, which allows the model to handle very large images. Each position is a learned vector of the same dimensions as the patch embedding. The 2D RoPE which Gemma 4 uses independently rotate half the attention head dimensions for the x-axis and the other half for the y-axis. This allows the model to understand spatial relationships like "above," "below," "left of," and "right of."

NomicBERT

NomicBERT is a BERT-inspired encoder model that applies Rotary Position Embeddings (RoPE) to create reproducible long context text embeddings. It is the first fully reproducible, open-source text embedding model with 8192 context length that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on short-context MTEB and long context LoCo benchmarks. The model generates dense vector embeddings for various tasks including search, clustering, and classification using specific instruction prefixes.

Links: Documentation | Paper

Internalise the NomicBERT model (#43067) by @ed22699 in #43067

MusicFlamingo

Music Flamingo is a fully open large audio–language model designed for robust understanding and reasoning over music. It builds upon the Audio Flamingo 3 architecture by including Rotary Time Embeddings (RoTE), which injects temporal position information to enable the model to handle audio sequences up to 20 minutes. The model features a unified audio encoder across speech, sound, and music with special sound boundary tokens for improved audio sequence modeling.

Links: Documentation | Paper

Add Music Flamingo (#43538) by @lashahub in #43538

... (truncated)

Commits

c1c3424 update
20bff68 update release workflow
8956441 v5.5.0
5135e5e casually dropping the most capable open weights on the planet (#45192)
a594e09 Internalise the NomicBERT model (#43067)
4932e97 Fix resized LM head weights being overwritten by post_init (#45079)
57e8413 [Qwen3.5 MoE] Add _tp_plan to ForConditionalGeneration (#45124)
b10552e Fix TypeError: 'NoneType' object is not iterable in GenerationMixin.generate ...
423f2a3 fix(models): Fix dtype mismatch in SwitchTransformers and TimmWrapperModel (#...
ade7a05 Generalize gemma vision mask to videos (#45185)
Additional commits viewable in compare view

Updates nltk from 3.8.1 to 3.10.0

Release notes

Sourced from nltk's releases.

v3.10.0-rc1

What's Changed

CHILDES corpus reader: lazy loading (fixes #1340) by @nschneid in nltk/nltk#1341

Stanseg by @casperlehmann in nltk/nltk#1350

Add optional CCG semantics computation by @tanin47 in nltk/nltk#1321

address of free software foundation updated by @Lightyagami1 in nltk/nltk#1347

Added a verification for empty tokens by @gnardari in nltk/nltk#1353

use data.load for vader lexicon by @afgiel in nltk/nltk#1303

Ensure that proper exceptions are caught correctly by @pombredanne in nltk/nltk#1343

Reset CCGVar everytime we invoke lexicon.fromstring() and skip the unicode test because it fails on python2.7 by @tanin47 in nltk/nltk#1354

Add tanin as an author :) by @tanin47 in nltk/nltk#1355

framenet.py: remove unnecessary str() call that causes problems for names with Unicode characters by @nschneid in nltk/nltk#1360

remove u because it doesn't work in Python 3.0 to 3.2 by @tanin47 in nltk/nltk#1362

moved the import for dispersion_plot out of the else block by @dennisobrien in nltk/nltk#1364

Raise warning for infinite recursion by @alvations in nltk/nltk#1365

Patch None TypeError in align_blocks() by @alvations in nltk/nltk#1366

Safer try-except for catching infinite recursion by @alvations in nltk/nltk#1368

Collections refactor by @nschneid in nltk/nltk#1382

Handles fringe cases in BLEU by @alvations in nltk/nltk#1383

Allow specifying substitution cost in edit distance by @dnc1994 in nltk/nltk#1386

Pl196x fixes by @pkasprzyk in nltk/nltk#1359

Docs/ngrams by @ColCarroll in nltk/nltk#1403

Fix conll.py by @germanferrero in nltk/nltk#1402

Aline by @geoffbacon in nltk/nltk#1407

Handle empty files in a corpus by @alexisdimi in nltk/nltk#1411

fixed bug: StackDecoder would throw exception if it's unable to finish… by @hikui in nltk/nltk#1405

add hikui to authors by @hikui in nltk/nltk#1412

removes download-cache pip flag from tox config by @fievelk in nltk/nltk#1415

Added three tokenizers, one detokenizer and two tokenizer-related word/char list corpora by @alvations in nltk/nltk#1282

Much faster Vader sentiment tagging by @georgeberry in nltk/nltk#1409

Fixes doctest in is_cjk() and REPP tokenizer wrapper by @alvations in nltk/nltk#1420

Fixed doctest for UnicharsCorpusReader and NonbreakingPrefixesCorpusReader by @alvations in nltk/nltk#1422

Updated senna version in docstring by @alvations in nltk/nltk#1431

Removes Python 2.6 and Python 3.1-3.3 code, Improves Python 3.5 syntax by @adamn in nltk/nltk#1429

Cleanup by @dimazest in nltk/nltk#1432

Fix zero division & log(0) in PunktTrainer._col_log_likelihood by @mrecachinas in nltk/nltk#1247

Update stanford_segmenter.py by @janissl in nltk/nltk#1444

explained data format for constructor by @drevicko in nltk/nltk#1450

Use a more efficient idiom for flattening nested lists by @rmalouf in nltk/nltk#1447

improved documentation of load_array() by @drevicko in nltk/nltk#1452

changed "list or iterator" to "sequence" by @drevicko in nltk/nltk#1453

Deal gracefully with degenerate cases. by @drevicko in nltk/nltk#1454

deal with (hopefully) all degenerate cases by @drevicko in nltk/nltk#1455

Enable unordered comparison for gaac clustering by @alvations in nltk/nltk#1456

Rewrite porter.py by @ExplodingCabbage in nltk/nltk#1261

Update naivebayes.py by @andyreagan in nltk/nltk#1461

Fixes bug where download status not returned. by @nsfabina in nltk/nltk#1468

Use lcon and rcon variables in conllned() when trace is False - Issue #1474 by @timleslie in nltk/nltk#1475

Use shutil.copyfileobj for unzipping by @ikorolev93 in nltk/nltk#1478

Fixing Chen&Cherry method 6 smoothing in BLEU score by @alvations in nltk/nltk#1477

... (truncated)

Changelog

Sourced from nltk's changelog.

Version 3.10.0 2026-06-11

Enforce the stricter nltk.pathsec security policy by default

Document the new security model and migration guidance

Harden resource loading against path traversal and SSRF/DNS-rebinding

Harden downloader path handling and block XML entity expansion

Close remaining corpus-reader security edge cases

Replace unsafe exec() usage in the utility CLI

Warn on unpickling user-provided pickles

Add HuggingFace datasets integration (nltk.huggingface)

Align TnT with Brants (2000) specifications

Fix PorterStemmer irregular-form lowercasing in NLTK mode

Fix TransitionParser sparse index dtype for scikit-learn 1.9

Fix TextCat tie handling

Fix WordNet object comparisons for incompatible types

Cache WordNet max depth lazily for lch_similarity()

Fix CCG variable direction, substitution, and type-raising bugs

Fix Jaro similarity for single-character and empty-string cases

Improve CI and release-maintenance workflows

Thanks to the following contributors to 3.10.0: 13rac1, alvations, bowiechen, devesh-2002, ekaf, elias-ba, haosenwang1018, HyperPS, ihitamandal, jancallewaert, jhnwnstd, JuanIMartinezB, Lemm1, LinZiyuu, Mr-Neutr0n, PastelStorm, scruge1, Syzygy2048, ylwango613, yzhaoinuw

Version 3.9.4 2026-03-24

Support Python 3.14

Fix bug in Levenshtein distance when substitution_cost > 2

Fix bug in Treebank detokeniser re quote ordering

Fix bug in Jaro similarity for empty strings

Several security enhancements

Fix GHSA-rf74-v2fm-23pw: unbounded recursion in JSONTaggedDecoder

Implement TextTiling vocabulary introduction method (Hearst 1997)

Fix ALINE feature matrix errors and add comprehensive tests

Support multiple VerbNet versions, fix longid/shortid regex for VerbNet ids

Let downloader fallback to md5 when sha256 is unavailable

Several other minor bugfixes and code cleanups

Thanks to the following contributors to 3.9.4: Min-Yen Kan, Eric Kafe, Emily Voss, bowiechen, Hrudhai01, jancallewaert, Mr-Neutr0n, pollak.peter89, ylwango613,

Version 3.9.3 2026-02-21

Fix CVE-2025-14009: secure ZIP extraction in nltk.downloader (#3468)

Block path traversal/arbitrary reads in nltk.data for protocol-less refs (#3467)

Block path traversal/abs paths in corpus readers and FS pointers (#3479, #3480)

Validate external StanfordSegmenter JARs using SHA256 (#3477)

... (truncated)

Commits

bd49f90 allow escaped brackets in Tree.fromstring (#3694)
27b8ad6 don't crash chomsky_normal_form on terminals with siblings (#3693)
52227d2 Use os.name for Windows path handling (#3605)
06c0e2c Avoid RIBES zero division on empty inputs (#3604)
a167389 Treat missing unzip output as stale (#3607)
c94c967 Fix EOF empty document bug in IEER corpus reader (#3648)
94a259c Enforce restrictive primitive type checking in pathsec wrappers (#3692)
5ac475d fix(security): isolate Stanford Java options and clean temp files (#3683)
986f26e ci(deps): bump the github-actions group with 3 updates (#3691)
f26b375 fix(security): prevent pickle RCE in TransitionParser model loading (CWE-502)...
Additional commits viewable in compare view

Updates sentencepiece from 0.1.99 to 0.2.1

Release notes

Sourced from sentencepiece's releases.

v0.2.1

Major changes

[Python] Supported wheels and builds for Python 3.13 and 3.14(rc1) #1134, #1127, #1121, #1111, #1110, #1104, #1103, #1099, #1091

[Python] Added an experimental support for free-threading. #1134, #1127, #1110 https://github.com/google/sentencepiece/tree/master/python#free-threading-support

[Python] Updated the supported Python version to 3.9 or later.

New features

[ALL]: Added new build mode to prevent the precompiled normalization rules being embedded in *.so and *.a. (-DSPM_DISABLE_EMBEDDED_DATA=ON). This reduces the runtime size by approximately 1-2 MB. This mode is enabled to build python wheels. The rules are loaded as the data package.

Bug fixes & minor changes

[ALL]: Security fix to address a heap overflow issue that could occur when using a model containing an invalid precompiled normalization model.

[Python]: Deprecates the wheel package for Linux i686.

[Python]: Supported wheel for Windows Arm64. #1114

[Python]: Fixed the crash issue on batch decoding #1051

[ALL]: Updated the Unicode normalization rule with the latest ICU/Unicode rules.

[ALL]: Unused code and build mode cleanup.

v0.2.0

Major changes

N/A

New features

[ALL] Added SentencePieceNormalizer class in C++/Python. It supports almost the equivalent feature of spm_normalize. Python Sample C++ Sample

[ALL] Added SentencePieceProcessor::Normalize method in C++/Python Python Sample C++ Sample

[ALL] Added functionality to override the normalization spec before the processing. Python Sample

Bug fixes & minor changes

Introduce better support of using external abseil and protobuf google/sentencepiece#869

Build universal binary in OSX release package google/sentencepiece#892

Add the set_min_log_level function to python to change the loglevel from the python wrapper. google/sentencepiece#893

Uses the logsumexp techniques in marginal probabilities of n-best tokenization to avoid underflow.

Support Python 3.12 google/sentencepiece#932

Improves the thread utilization in batch encoding/decoding.

Fix nasty bug in BPE position encoding.

Fix bugs in the handling of duplicated bigrams

Commits

31646a4 Merge pull request #1136 from crusaderky/pytest-run-parallel
bcd44b9 free-threading tests
135747f install twine before checking wheel
69fe0b2 install setuptools before making sdist
ee1422b install setuptools before making sdist
5ac2fd2 use windows-11-arm runner to test ARM64 wheel on native env.
36b9745 use windows-11-arm runner to test ARM64 wheel on native env.
4f043ae use auto-mode to make wheel with the native binary.
623196e uses arm docker image to build and test wheel
559fd65 re-enable QEMU to enable arm execution
Additional commits viewable in compare view

codecov · 2026-07-17T20:17:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Bumps the pip group with 1 update in the /src/dependencies/requirements/base_requirements directory: [setuptools](https://github.com/pypa/setuptools). Bumps the pip group with 3 updates in the /src/maxtext/inference/mlperf directory: [sentencepiece](https://github.com/google/sentencepiece), [transformers](https://github.com/huggingface/transformers) and [nltk](https://github.com/nltk/nltk). Updates `setuptools` from 78.1.0 to 78.1.1 - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v78.1.0...v78.1.1) Updates `sentencepiece` from 0.1.99 to 0.2.1 - [Release notes](https://github.com/google/sentencepiece/releases) - [Commits](google/sentencepiece@v0.1.99...v0.2.1) Updates `transformers` from 4.31.0 to 5.5.0 - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.31.0...v5.5.0) Updates `nltk` from 3.8.1 to 3.10.0 - [Release notes](https://github.com/nltk/nltk/releases) - [Changelog](https://github.com/nltk/nltk/blob/develop/ChangeLog) - [Commits](nltk/nltk@3.8.1...v3.10.0) Updates `transformers` from 4.31.0 to 5.5.0 - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.31.0...v5.5.0) Updates `nltk` from 3.8.1 to 3.10.0 - [Release notes](https://github.com/nltk/nltk/releases) - [Changelog](https://github.com/nltk/nltk/blob/develop/ChangeLog) - [Commits](nltk/nltk@3.8.1...v3.10.0) Updates `sentencepiece` from 0.1.99 to 0.2.1 - [Release notes](https://github.com/google/sentencepiece/releases) - [Commits](google/sentencepiece@v0.1.99...v0.2.1) --- updated-dependencies: - dependency-name: nltk dependency-version: 3.10.0 dependency-type: direct:production - dependency-name: sentencepiece dependency-version: 0.2.1 dependency-type: direct:production - dependency-name: setuptools dependency-version: 78.1.1 dependency-type: direct:production - dependency-name: transformers dependency-version: 5.5.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

dependabot Bot added dependencies Pull requests that update a dependency file python Pull requests that update python code labels Jul 17, 2026

dependabot Bot requested a review from bvandermoon as a code owner July 17, 2026 20:05

dependabot Bot added the dependencies Pull requests that update a dependency file label Jul 17, 2026

dependabot Bot requested a review from SurbhiJainUSC as a code owner July 17, 2026 20:05

dependabot Bot added the python Pull requests that update python code label Jul 17, 2026

dependabot Bot requested review from Lumosis, darisoy, gpolovets1, igorts-git, jrplatin, mailvijayasingh, mitalisi, parambole, patemotter, richjames0, shralex and vipannalla as code owners July 17, 2026 20:05

dependabot Bot force-pushed the dependabot/pip/src/dependencies/requirements/base_requirements/pip-154efd262d branch from 1508b2b to 91fa3c7 Compare July 17, 2026 20:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bump the pip group across 2 directories with 4 updates#4527

Bump the pip group across 2 directories with 4 updates#4527
dependabot[bot] wants to merge 1 commit into
mainfrom
dependabot/pip/src/dependencies/requirements/base_requirements/pip-154efd262d

dependabot Bot commented on behalf of github Jul 17, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

Soft Tokens	Patches (before pooling)	Approx. Image Area
70	630	~161K pixels
140	1,260	~323K pixels
280	2,520	~645K pixels
560	5,040	~1.3M pixels
1,120	10,080	~2.6M pixels

Uh oh!

Conversation

dependabot Bot commented on behalf of github Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

v78.1.1

Bugfixes

v0.2.1

Major changes

New features

Bug fixes & minor changes

v0.2.0

Major changes

New features

Bug fixes & minor changes

Release v5.5.0

New Model additions

Gemma4

NomicBERT

MusicFlamingo

v3.10.0-rc1

What's Changed

Release v5.5.0

New Model additions

Gemma4

NomicBERT

MusicFlamingo

v3.10.0-rc1

What's Changed

v0.2.1

Major changes

New features

Bug fixes & minor changes

v0.2.0

Major changes

New features

Bug fixes & minor changes

Uh oh!

codecov Bot commented Jul 17, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

dependabot Bot commented on behalf of github Jul 17, 2026 •

edited

Loading