Skip to content

Fix pytest CI matrix compatibility and prevent hangs#26027

Merged
eleanorjboyd merged 12 commits into
mainfrom
copilot/fix-ci-pre-release-pytest
Jul 21, 2026
Merged

Fix pytest CI matrix compatibility and prevent hangs#26027
eleanorjboyd merged 12 commits into
mainfrom
copilot/fix-ci-pre-release-pytest

Conversation

Copilot AI commented Jul 2, 2026

Copy link
Copy Markdown

Summary

Fix the Python test matrix so every job runs its requested pytest version, cover supported Python versions against recent and upcoming pytest releases, remove the obsolete global pytest pin, and prevent subprocess failures from hanging CI indefinitely.

Root cause

pytest-black==0.6.0 uses the deprecated pytest_collect_file(path, ...) hook removed by pytest 8.1. The resulting pytest<8.1 workaround lived in shared test requirements, which were installed after the matrix-selected pytest version and silently downgraded stable and pre-release jobs to pytest 8.0.2.

The downgrade also made the manual pytest-cov test reject --disable-plugin-autoload. Its subprocess exited, but the named-pipe listener waited indefinitely, causing the multi-hour CI hangs.

Test matrix

On Ubuntu and Windows:

  • Python 3.10, 3.11, 3.12, 3.13, and 3.14
  • Latest pytest 8.4 patch (pytest==8.4.*) for approximately one year of compatibility coverage
  • Pytest pre-release channel to catch upcoming breaking changes

This produces 20 jobs. The 8.4 wildcard currently resolves to pytest 8.4.2.

Changes

  • Install shared test requirements before matrix-selected pytest.
  • Remove pytest<8.1 and pytest-black from shared requirements.
  • Skip the obsolete pytest-black-specific discovery case; pytest-ruff continues to cover plugin-generated test items.
  • Explicitly install stable pytest in the build workflow.
  • Use PYTEST_DISABLE_PLUGIN_AUTOLOAD for cross-version manual plugin loading.
  • Bound subprocess and pipe waits and propagate failures promptly.
  • Add regression tests for subprocess timeout, failure propagation, and pipe timeout.

Validation

  • Python 3.10 / pytest 8.4.2: 219 passed, 1 skipped
  • Python 3.13 / pytest 9.1.1: 219 passed, 1 skipped
  • Ruff checks and git diff --check pass.

Copilot AI changed the title [WIP] Fix CI configuration for pre-release pytest Install matrix-specific pytest after test-requirements so pre-release job actually tests pre-release Jul 2, 2026
Copilot AI requested a review from eleanorjboyd July 2, 2026 22:01
@eleanorjboyd eleanorjboyd added bug Issue identified by VS Code Team member as probable bug skip tests Updates to tests unnecessary skip-issue-check labels Jul 2, 2026
eleanorjboyd and others added 2 commits July 21, 2026 10:36
Use pytest 7.0.1 as the oldest supported lane on Python 3.10 and install matrix selections after shared requirements. Keep non-matrix builds on stable pytest, make manual plugin loading compatible across pytest versions, and fail bounded pipe harness waits instead of hanging CI indefinitely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cover Python 3.10 through 3.14 with the latest pytest 8.4 patch and the pytest pre-release channel. This keeps approximately one year of pytest compatibility coverage while detecting upcoming breaking changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eleanorjboyd
eleanorjboyd force-pushed the copilot/fix-ci-pre-release-pytest branch from 279bf86 to b9fe1a4 Compare July 21, 2026 17:37
@eleanorjboyd eleanorjboyd changed the title Install matrix-specific pytest after test-requirements so pre-release job actually tests pre-release Fix pytest CI matrix compatibility and prevent hangs Jul 21, 2026
Keep the subprocess cwd annotation compatible with pathlib callers and simplify the local Black fixture for the supported pytest 8.4+ range so the CI Pyright version can analyze it cleanly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pytest-black does not support pytest 8.1 or newer, and pytest-ruff already covers discovery of plugin-generated test items. Skip the obsolete Black-specific parameter instead of maintaining a local replacement for the third-party plugin.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Python test/CI setup so the GitHub Actions matrix actually runs the intended pytest versions (stable wildcard + pre-release), removes an obsolete pytest pin caused by pytest-black, and hardens the pytestadapter test harness to avoid indefinite hangs when subprocess/pipe communication fails.

Changes:

  • Reorders/install-adjusts pytest in CI workflows and removes the global pytest<8.1 / pytest-black constraints that were downgrading matrix-selected pytest.
  • Adds bounded waits/timeouts and clearer failure propagation in python_files/tests/pytestadapter/helpers.py to prevent CI hangs.
  • Adds regression tests for subprocess timeout, pipe timeout, and failure surfacing; introduces a local replacement conftest for the black-related test data.
Show a summary per file
File Description
python_files/tests/pytestadapter/test_helpers.py Adds regression tests covering subprocess timeout and pipe/failure handling.
python_files/tests/pytestadapter/test_coverage.py Switches to PYTEST_DISABLE_PLUGIN_AUTOLOAD env var for cross-version compatibility.
python_files/tests/pytestadapter/helpers.py Adds subprocess timeout and bounded pipe waits; refactors runner to avoid hang scenarios.
python_files/tests/pytestadapter/.data/2496-black-formatter/conftest.py Provides a local, pytest-8.4+ compatible black-check collector for test data.
build/test-requirements.txt Removes pytest<8.1 and pytest-black from shared test requirements.
.github/workflows/pr-check.yml Expands Python versions and installs matrix-selected pytest after test requirements.
.github/workflows/build.yml Ensures pytest is explicitly installed in the build workflow.

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment thread .github/workflows/pr-check.yml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eleanorjboyd
eleanorjboyd marked this pull request as ready for review July 21, 2026 20:21
The supported pytest 8.4+ range provides --disable-plugin-autoload, so use the option exercised by the regression test instead of the compatibility environment variable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eleanorjboyd
eleanorjboyd merged commit fa5639b into main Jul 21, 2026
91 checks passed
@eleanorjboyd
eleanorjboyd deleted the copilot/fix-ci-pre-release-pytest branch July 21, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Issue identified by VS Code Team member as probable bug skip tests Updates to tests unnecessary skip-issue-check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI is no longer running against pre-release pytest

4 participants