Skip to content

Add support for huggingface datasets to vortex-python#8733

Open
robert3005 wants to merge 2 commits into
developfrom
rk/hfd
Open

Add support for huggingface datasets to vortex-python#8733
robert3005 wants to merge 2 commits into
developfrom
rk/hfd

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Add IterableDataset implementation backed by vortex files. Majority of the logic in this pr deal with adapting hugging face options into compatible vortex options

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the changelog/feature A new feature label Jul 13, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 13, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 11.87%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 5 improved benchmarks
✅ 1701 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation bitwise_not_vortex_buffer_mut[128] 215.3 ns 186.1 ns +15.67%
Simulation bitwise_not_vortex_buffer_mut[1024] 275.6 ns 246.4 ns +11.84%
Simulation encode_varbin[(1000, 8)] 160.6 µs 144.7 µs +10.98%
Simulation encode_varbin[(1000, 4)] 159.2 µs 144.1 µs +10.48%
Simulation encode_varbin[(1000, 2)] 158.5 µs 143.5 µs +10.47%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/hfd (30b65d7) with develop (0c7e0ff)

Open in CodSpeed

Signed-off-by: Robert Kruszewski <github@robertk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant