Add bounded-concurrency async league ingestion - #48
Merged
Conversation
Upgrade python-mlb-statsapi to 1.1.0 to use its released AsyncMlb client, and add ingest_league_season_async alongside the existing sequential ingest_league_season. Fetches for each team (schedule, hitting/pitching logs) run concurrently up to a configurable limit while persistence stays serialized through a shared write lock, reusing the same validation, normalization, and persistence logic as the sync path. Exposes --async/--concurrency on scripts/import_league_season.py.
…lure asyncio.gather propagates the first exception without cancelling still-running siblings, letting another team keep hitting MLB or reach persist_team_season after the caller has already been told the run failed. Team coroutines are now owned as explicit asyncio.Tasks so an unexpected exception (including one from on_team_complete) cancels and drains every other task before it escapes ingest_league_season_async. Also strengthens the sync/async parity test to compare full persisted batting and pitching domain objects between the two paths, rather than a handful of batting columns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gjqhyq4iPHf51Mr3PyHtMb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
python-mlb-statsapi1.1.0 is now released with the publicAsyncMlbclient. League-season ingestion is network-bound, and prior live testing showed the async path is roughly 3x faster on average while preserving the existing SQLite persistence model.This PR adopts the released async client without changing the browser or database architecture.
What
python-mlb-statsapito the released 1.1.0 package with the async extra.ingest_league_season_asyncalongside the existing sequential ingestion path.AsyncMlbclient and connection pool.--asyncand--concurrencyoptions toscripts/import_league_season.py; sequential ingestion remains available as the default/reference path.Tests
Adds offline deterministic coverage for:
Also validated with the existing full pytest and Ruff checks during development.
Risk and impact
Normal
The change introduces concurrent network orchestration and therefore adds lifecycle/error-handling complexity. The database side remains deliberately conservative: synchronous SQLAlchemy, serialized writes, and the same persistence functions and transaction boundaries as the sequential path.
If the async path fails unexpectedly, the league run remains
RUNNINGrather than claiming trustworthy coverage, and outstanding sibling tasks are cancelled and drained before the error escapes. Successfully committed team-seasons remain committed.The sequential ingestion path remains available as a reference and fallback.
Closes #31