diff --git a/.github/docs-versions.yml b/.github/docs-versions.yml index 264024d9a3..81c1dc6f3f 100644 --- a/.github/docs-versions.yml +++ b/.github/docs-versions.yml @@ -5,8 +5,9 @@ # * the version picker injected into every doc page # * doc/contributing/10_release_process.md - tells the releaser to edit this file # -# To add a new release version: append an entry under `versions:`, and update -# `default:` / `stable:` if the new release should become the landing page. +# To add a new release version: insert an entry at the top of `versions:`, immediately +# after `latest`, and update `default:` / `stable:` if the new release should become the +# landing page. The picker renders this list in file order, so the newest release goes first. default: "1.0.1" # served at the site root (microsoft.github.io/PyRIT/) stable: "1.0.1" # served at /stable/ and shown as "(stable)" in the picker diff --git a/build_scripts/prepare_package.py b/build_scripts/prepare_package.py index 83f81ee83b..065b5e4227 100644 --- a/build_scripts/prepare_package.py +++ b/build_scripts/prepare_package.py @@ -171,7 +171,8 @@ def main() -> int: print("=" * 60) print("\nNext steps:") print(" 1. Build the Python package: python -m build") - print(" 2. Upload to PyPI: python -m twine upload dist/*") + print(" 2. Upload to PyPI by explicit filename, e.g.:") + print(" python -m twine upload dist/pyrit-x.y.z-py3-none-any.whl dist/pyrit-x.y.z.tar.gz") return 0 diff --git a/doc/contributing/10_release_process.md b/doc/contributing/10_release_process.md index a92d90d118..af08cc06d7 100644 --- a/doc/contributing/10_release_process.md +++ b/doc/contributing/10_release_process.md @@ -64,70 +64,113 @@ We follow semantic versioning for Python projects; see https://semver.org/ for more details. Below, we refer to the version as `x.y.z`. `x` is the major version, `y` the minor version, and `z` the patch version. -Every Python project starts at `0.1.0`. -Backwards compatible bug fixes increase the patch version. -Importantly, they are backward compatible, so upgrading from `0.1.0` to -`0.1.1` (or higher ones like `0.1.38`) should not break your code. -More significant changes, such as major features, require at least a new -minor version. -They should still be backwards compatible, so if you're upgrading from -`1.1.0` to `1.2.0` your code shouldn't break. -The major version `1.0.0` is the first "stable" release. -Anything before (i.e., leading with major version `0`) indicates that it is -not stable and anything may change at any time. -For that reason, the minor version may indicate breaking changes, too, -at least until we hit major version `1`. - -We will always have a .dev version in main, but there are circumstances when -we might want to release versions that aren't final; consult + +PyRIT is past `1.0.0`, its first "stable" release, so in practice: + +- **Regular releases increment the minor version** (e.g., `1.0.1` to `1.1.0`). + This is the default for a normal release, whatever mix of features and fixes it contains. + Minor releases are broadly backward compatible: the main exception is that functionality + which was deprecated in an earlier release may be removed once its announced removal version + arrives, so check step 3 and document any such removal as a breaking change per step 1. +- **Patch releases are reserved for targeted fixes** shipped outside the normal cadence, such + as a security patch or a critical bug fix cherry-picked onto the previous release + (e.g., `1.0.0` to `1.0.1`). See the patch release appendix at the end of this document. +- **Major releases** are for deliberate, wide-reaching breaking changes, and are planned with + the maintainers rather than chosen at release time. Some functionality is already scheduled + for removal in a future major version, so treat step 3 as applying to whichever version + number you are incrementing. + +`main` always carries a `.dev0` version for the next planned release (e.g., `1.1.0.dev0` while +`1.0.1` is the latest published release). There are circumstances when we might want to release +versions that aren't final; consult https://packaging.python.org/en/latest/discussions/versioning/ to determine whether there should be any postfixes on the release version. -With that in mind, the reason for the release and the set of changes -that happened since the last release will influence the new version number. +Confirm the choice with the other maintainers if this release is anything other than the next +minor version. ## 3. Remove deprecated functionality -If you are incrementing the minor version, search the entire codebase for the new minor -version, e.g., "0.11.0" (no leading "v"), +If you are incrementing the minor version, search the entire codebase for the new version, +e.g., "1.2.0" (no leading "v"), to find occurrences where we deprecated functionality and announced that it will be removed in the new version. Typically, functionality is deprecated and then stays for two minor versions before getting removed. +Deprecations are usually declared with a `removed_in=` argument to the helpers in +`pyrit/common/deprecation.py`, so searching for `removed_in=` is a quick way to find +everything that is scheduled for removal: + +```bash +grep -rn "removed_in=" --include=*.py pyrit/ +``` + If you find functionality to remove make sure to merge the PR to `main` before proceeding. +If nothing is scheduled for removal in this version, record that in the release work item so +it is not re-investigated later. -In some cases, we know that the next version will not be a patch upgrade (e.g., -0.10.0 to 0.10.1) but rather a minor upgrade (0.10.3 to 0.11.0) and we can remove -functionality that is to be deprecated beforehand. -This is preferable but it may not be clear if a release is patch or minor until the full -scope of the release is known (see step 2). +Because regular releases are always a minor bump (step 2), functionality scheduled for removal +in the next minor version can be removed as soon as that version is the next planned release, +rather than waiting for the release branch to be cut. That is preferable, because it keeps the +removal and its announcement in separate pull requests. ## 4. Update the version -### __init__.py, pyproject.toml, and package.json +### Version files -Set the version in `pyrit/__init__.py`, `pyproject.toml`, and `package.json` to the version -established in step 2. +Set the version established in step 2 in every one of these files. The runtime version lives in +`pyrit/_version.py`, not `pyrit/__init__.py`, which only re-exports it lazily. -### Update README File +| File | Format | Notes | +|---|---|---| +| `pyrit/_version.py` | `__version__ = "x.y.z"` | Canonical runtime version | +| `pyproject.toml` | `version = "x.y.z"` | Packaging metadata | +| `frontend/package.json` | `"version": "x.y.z"` | npm form: `x.y.z-dev.0`, not `x.y.z.dev0` | +| `frontend/package-lock.json` | `"version": "x.y.z"` | Two occurrences near the top of the file | +| `uv.lock` | `version = "x.y.z"` | Under the `[[package]]` entry named `pyrit` | + +Do not edit `.github/docs-versions.yml` here; that is updated on `main` after the release +in step 10. -The README file is published to PyPI and also needs to be updated so the -links work properly. Note: There may not be any links to update, but it is -good practice to check in case our README changes. +Bump `pyrit/_version.py` first. It is the source of truth that the frontend development server +syncs `frontend/package.json` from, so if the two disagree the next dev-server run silently +rewrites `package.json` back, and it never updates `frontend/package-lock.json` to match. -Replace all "main" links like -`https://github.com/microsoft/PyRIT/blob/main/doc/index.md` with "raw" links that have -the correct version number, i.e., -`https://raw.githubusercontent.com/microsoft/PyRIT/releases/vx.y.z/doc/index.md`. +Take care not to modify dependency versions that happen to resemble the PyRIT version. Examples +that exist today are `starlette` and `python-docx` in `pyproject.toml`, `rfc3987-syntax` in +`uv.lock`, and several packages in `frontend/package-lock.json`. In the lock files, only the +root `pyrit` / `pyrit-frontend` entries are the project version; every other `[[package]]` +stanza or `node_modules/**` entry belongs to a dependency. + +Confirm that no development suffix survives in the files you just changed: + +```bash +grep -rn 'x\.y\.z\.dev0\|x\.y\.z-dev\.0' pyrit/ pyproject.toml frontend/package.json \ + frontend/package-lock.json uv.lock +``` + +Then confirm the runtime version resolves correctly: + +```bash +python -c "import pyrit; assert pyrit.__version__ == 'x.y.z', pyrit.__version__" +``` + +### Update README File -For images, update using the "raw" link, e.g., -`https://raw.githubusercontent.com/microsoft/PyRIT/releases/vx.y.z/assets/pyrit_architecture.png`. +The README file is published to PyPI, where relative links do not resolve, so any +repository-relative link has to be rewritten to point at the release branch. Check the +current README, since the set of links changes over time. -For directories, update using the "tree" link, e.g., -`https://github.com/microsoft/PyRIT/tree/releases/vx.y.z/doc/code` +Rewrite relative links, such as `./doc/roakey.png` or `./CITATION.cff`, to absolute +links pinned to this release. For files and images use the "raw" form, e.g. +`https://raw.githubusercontent.com/microsoft/PyRIT/releases/vx.y.z/doc/roakey.png`. For +directories use the "tree" form, e.g. +`https://github.com/microsoft/PyRIT/tree/releases/vx.y.z/doc/code`. Links that already +point at a stable external URL, such as the documentation site or the security policy, +stay as they are. -This is required for the release branch because PyPI does not pick up -other files besides the README, which results in local links breaking. +Make this change only on the release branch. `main` keeps the relative links so that +they resolve while browsing the repository. ## 5. Publish to GitHub @@ -136,14 +179,25 @@ Commit your changes and push them to the repository on a branch called ```bash git checkout -b "releases/vx.y.z" +git add pyrit/_version.py pyproject.toml frontend/package.json \ + frontend/package-lock.json uv.lock README.md git commit -m "release vx.y.z" git push origin releases/vx.y.z git tag -a vx.y.z -m "vx.y.z release" -git push --tags +git push origin vx.y.z ``` After pushing the branch to remote, check the release branch to make sure it looks as intended (e.g. check the links in the README work properly). +Confirm the GitHub Actions checks ran and passed on the release branch itself, and re-run them +after any cherry-pick, rather than relying only on the checks that ran on `main` before the +branch was cut. + +A patch release branch is cut from an older tag (see the appendix), so it carries that +release's workflow files. If the tag predates these release-branch triggers, nothing will run +on push; cherry-pick the workflow change onto the branch, or start each workflow manually from +the Actions tab. + ## 6. Build Package You'll need the build package to build the project. If it’s not already installed, install it `pip install build`. @@ -165,6 +219,13 @@ This will: ### Build the Python Package +`dist/` is not cleaned automatically and may still hold wheels from an earlier release. Remove +them so the `python -m build` output is unambiguous and no stale artifact can reach PyPI: + +```bash +rm -rf dist/ +``` + To build the package wheel and archive for PyPI run ```bash @@ -183,6 +244,22 @@ Create a new environment with the equivalent of `uv venv --python 3.11`. You do Once the package is successfully installed in the new environment, run `uv pip show pyrit`. Ensure that the version matches the release `vx.y.z` and that the package is found under the site-packages directory of the environment, like `..\venv\Lib\site-packages`. +Also confirm the *runtime* version, which is what users actually see. Two things can shadow the +installed package: a `PYTHONPATH` that points at the repository, and the current directory, which +Python puts on `sys.path` for `python -c`. Running the check from inside the checkout therefore +tests the working tree rather than the wheel, and it passes even when the wheel is wrong. Run it +from a directory outside the repository, with `PYTHONPATH` unset, and assert both the location +and the version: + +```bash +cd .. +env -u PYTHONPATH python -c "import pyrit; assert 'site-packages' in pyrit.__file__, pyrit.__file__; assert pyrit.__version__ == 'x.y.z', pyrit.__version__" +``` + +In PowerShell, change to a directory outside the repository, clear the variable with +`Remove-Item Env:PYTHONPATH -ErrorAction SilentlyContinue`, and then run the same `python -c` +command. + Make sure to set up the Jupyter kernel as described in our [Jupyter setup](../getting_started/troubleshooting/jupyter_setup.md) guide. To test the demos outside the PyRIT repository, copy the `doc`, `assets`, and `.env` files to a new folder created outside the PyRIT directory. For better organization, you could create a main folder called `releases` and a subfolder named `releasevx.y.z`, and then place the copied folders within this structure. @@ -191,12 +268,18 @@ Before running the demos, execute `az login` or `az login --use-device-code`, as Additionally, verify that your environment file includes all the test secrets needed to run the demos. If not, update your .env file using the secrets from the key vault. -In the new location, run all notebooks that are currently skipped by integration tests (there are less than 10) in VS Code. These are listed in `skipped_files` in each `tests/integration//test_notebooks_*.py` file and are located in the doc folder that you copied into your new `releases\releasevx.y.z` folder. Note that some of these notebooks have known issues and it may make sense to skip testing them until those are fixed. Check with the last person to deploy or look for the relevant release work item for more information. In running the notebooks, you may also see exceptions. If this happens, make sure to look for existing bugs open on the ADO board or create a new one if it does not exist! If it is easy to fix, we prefer to fix the issue before the release continues. +In the new location, run all notebooks that are currently skipped by integration tests in VS Code. To find them, search for `skipped_files` under `tests/integration/`, and also check the separate `_azure_key_auth_notebooks` set described below; the lists live in several differently-named test modules, so search rather than relying on a fixed filename pattern or a remembered count. The notebooks themselves are in the doc folder that you copied into your new `releases\releasevx.y.z` folder. Note that some of these notebooks have known issues and it may make sense to skip testing them until those are fixed. Check with the last person to deploy or look for the relevant release work item for more information. In running the notebooks, you may also see exceptions. If this happens, make sure to look for existing bugs open on the ADO board or create a new one if it does not exist! If it is easy to fix, we prefer to fix the issue before the release continues. Some notebooks that use Azure targets with API-key (local) auth are skipped by the integration tests for a separate reason: key-based auth is disabled in our Azure subscription (see the note under *Release Readiness* above). These are tracked in the `_azure_key_auth_notebooks` set in `tests/integration/targets/test_notebooks_targets.py` (distinct from the long-standing `skipped_files` list). Validate them with Entra auth (`az login`) rather than API keys, or rely on the equivalent Entra-auth integration tests; running them with API keys fails with HTTP 403 `AuthenticationTypeDisabled`. A reminder that you should ensure that the integration tests pass in the version you are releasing in addition to the skipped files. +The Azure DevOps test pipelines do not run themselves on a release branch. `integration-tests`, +`end-to-end-tests` and `partner-integration-tests` all declare `trigger: none` and schedule only +`main`, so cutting the release branch produces no automatic run and the absence of a result is +easy to mistake for a passing one. Queue each pipeline manually against `releases/vx.y.z` and +wait for the results before continuing; the previous release was validated this way. + Note: copying the `doc` folder elsewhere is essential since we store data files in the repository that should be shipped with the package. If we run inside the repository, we may not face errors that users encounter @@ -206,17 +289,20 @@ If at any point you need to make changes to fix bugs discovered while testing, o ```bash git checkout main -git fetch main -git log main # to identify the commit hash of the change you want to cherry-pick +git fetch origin +git log origin/main # to identify the commit hash of the change you want to cherry-pick git checkout releases/vx.y.z git cherry-pick git push origin releases/vx.y.z git tag -a vx.y.z -m "vx.y.z release" --force # to update the tag to the correct commit +git push origin vx.y.z --force ``` Note: You may need to build the package again if those changes modify any dependencies, and consider retesting the notebooks if the changes affect them. If you reuse the same environment, it is best to `uv pip uninstall pyrit` to force the reinstall. -Lastly, **Verify pyrit-internal is up to date.** Follow the instructions at [aka.ms/internal-release](https://aka.ms/internal-release) to ensure the internal package is current. +Only move the tag while you are still validating, before step 9. Once the package is on PyPI +that version is permanent, so the tag must keep pointing at the commit that produced it; ship +any later fix as a new version instead. ## 8. Migrate Production Database Schema @@ -285,54 +371,62 @@ Note: Before publishing to PyPI, have your API token for scope 'Project: pyrit' ```bash uv pip install twine -twine upload dist/* # this should be the same as dist/pyrit-x.y.z-py3-none-any.whl +twine upload dist/pyrit-x.y.z-py3-none-any.whl dist/pyrit-x.y.z.tar.gz ``` +Upload the two files by name rather than `dist/*` so that a stale artifact from an earlier +release cannot be published by accident. + If successful, it will print > View at: > https://pypi.org/project/pyrit/x.y.z/ +PyPI permanently reserves a filename once it is uploaded. If one artifact uploads and the other +fails, do not rebuild, because a rebuilt file will not match the one already published. Retry +the missing file as-is, or contact PyPI support. + ## 10. Update main After the release is on PyPI, make sure to create a PR for the `main` branch where the changes are: -- the version increase in `__init__.py` (while keeping suffix `.dev0`). +- the version increase in the version files listed in step 4, this time keeping the development + suffix: `1.2.0.dev0` for the Python files and `1.2.0-dev.0` for the frontend files. - Search for the previous release version in the codebase and replace any occurrences with the new version (without `.dev0`). For example, some installation pages refer to the latest release. - Update the documentation site versions in `.github/docs-versions.yml` (see below). -The PR should be made from your fork and should be a different branch than the releases branch you created earlier. -This should be something like `x.y.z+1.dev0`. +The PR should be made from your fork and should be a different branch than the releases branch you created earlier, +named after the next development version, for example `1.2.0.dev0`. ### Update the documentation site versions Add the new release as a version on the [documentation site](https://microsoft.github.io/PyRIT/) by editing `.github/docs-versions.yml`: -- Append the new release under `versions:` with `slug: "x.y.z"`, `name: "x.y.z"`, and `ref: releases/vx.y.z`. +- Insert the new release at the top of `versions:`, immediately after the `latest` entry, with `slug: "x.y.z"`, `name: "x.y.z"`, and `ref: releases/vx.y.z`. The picker renders the list in file order, which is newest first. - Update `stable:` and `default:` at the top of the file to point at the new version (so the root URL `microsoft.github.io/PyRIT/` and the `/stable/` alias redirect to it). Once merged, the docs workflow rebuilds the site and the new version appears in the version picker on every page of every version. -For example, releasing `0.14.0` would change: +For example, releasing `1.1.0` would change: ```yaml -default: "0.13.0" -stable: "0.13.0" +default: "1.0.1" +stable: "1.0.1" ``` to: ```yaml -default: "0.14.0" -stable: "0.14.0" +default: "1.1.0" +stable: "1.1.0" ``` -and add an entry under `versions:` for `0.14.0` pointing at `releases/v0.14.0`. +and add an entry under `versions:` for `1.1.0` pointing at `releases/v1.1.0`. ## 11. Create GitHub Release @@ -349,9 +443,31 @@ If you are unsure about whether to include certain changes please consult with y maintainers. When you're done, hit "Publish release" and mark it as the latest release. +## 12. Update internal deployments + +The release is not complete until Microsoft-internal consumers pick up the new release. These +steps apply to maintainers with access to the internal environments; external contributors can +skip this section. The details live at +[aka.ms/internal-release](https://aka.ms/internal-release). + +1. **Verify pyrit-internal is up to date** so the internal package tracks the new release. +2. **Coordinate the CoPyRIT production deployment before it runs.** A production deployment + restarts the application and clears its in-memory state, so targets onboarded into a running + instance are lost when it reloads, and an unannounced run can interrupt work in progress. Agree + with the CoPyRIT maintainers on who queues it and when, rather than assuming that falls to the + release owner; they have taken it themselves in the past. Whoever runs it should do so only + after the package is on PyPI, so production runs a version users can install, and after the + production database migration in step 8 has completed, so the application and the shared + database schema agree. The pipeline definition, today `gui-deploy.yml` in this repository, + restricts which branch may deploy to production and fails the run if it is queued from anywhere + else; read it rather than assuming the rule. It also builds the image from a specific commit + rather than from the published package, so the deployed application tracks that commit and not + the release tag. Record the deployed commit in the release work item once the run finishes; + step 12 is complete when that commit is recorded. + ## Appendix: Patch Releases (Cherry-Pick Process) -A patch release (e.g., `0.12.0` → `0.12.1`) ships a targeted fix — typically a security +A patch release (e.g., `1.0.0` → `1.0.1`) ships a targeted fix — typically a security patch or critical bug fix — without including other in-flight changes from `main`. The process follows the same steps as a regular release with a few key differences. @@ -368,13 +484,13 @@ The process follows the same steps as a regular release with a few key differenc ```bash git fetch origin -git checkout -b release/vx.y.z vx.y.(z-1) +git checkout -b releases/vx.y.z vprevious.x.y.z ``` -For example, to create `0.12.1` from `0.12.0`: +For example, to create `1.0.1` from `1.0.0`: ```bash -git checkout -b release/v0.12.1 v0.12.0 +git checkout -b releases/v1.0.1 v1.0.0 ``` **2. Cherry-pick the fix from `main`:** @@ -390,23 +506,25 @@ the fix should apply cleanly in most cases. **3. Bump the version:** -Update the version in all three files (`pyproject.toml`, `pyrit/__init__.py`, `frontend/package.json`) -to the new patch version (e.g., `0.12.1`). Also update any version-pinned links in `README.md` -(e.g., image URLs pointing to `releases/v0.12.0` → `releases/v0.12.1`). +Update the version in every file listed in step 4 (`pyrit/_version.py`, `pyproject.toml`, +`frontend/package.json`, `frontend/package-lock.json`, and `uv.lock`) +to the new patch version (e.g., `1.0.1`). Also update any version-pinned links in `README.md` +(e.g., image URLs pointing to `releases/v1.0.0` → `releases/v1.0.1`). Commit the version bump: ```bash -git add pyproject.toml pyrit/__init__.py frontend/package.json README.md -git commit -m "Bump version to x.y.z" +git add pyrit/_version.py pyproject.toml frontend/package.json \ + frontend/package-lock.json uv.lock README.md +git commit -m "release vx.y.z" ``` **4. Push and tag:** -Push the release branch with the `releases/` prefix and create the tag: +Push the release branch and create the tag: ```bash -git push origin release/vx.y.z:releases/vx.y.z +git push origin releases/vx.y.z git tag -a vx.y.z -m "vx.y.z release" git push origin vx.y.z ``` @@ -416,6 +534,8 @@ git push origin vx.y.z - Build the package (step 6) - Test the built package in a clean environment (step 7) - Run integration tests (step 7) +- Migrate the production database (step 8) — required if the cherry-picked change carries an + Alembic revision, and harmless to run as confirmation if it does not - Publish to PyPI (step 9) - Update `main` with the next dev version (step 10) — for a patch release after `x.y.z`, the next version on `main` may be either `x.y.(z+1).dev0` or `x.(y+1).0.dev0` @@ -428,12 +548,13 @@ git push origin vx.y.z changed?" summary and the full changelog will be much shorter than a regular release. Make sure to call out the specific issue or vulnerability that prompted the patch so users can quickly assess whether they need to upgrade. +- Update internal deployments (step 12) ### Key differences from a regular release | Aspect | Regular release | Patch release | |---|---|---| -| Branch base | `main` | Previous release tag (e.g., `v0.12.0`) | +| Branch base | `main` | Previous release tag (e.g., `v1.0.0`) | | Changes included | Everything on `main` | Only cherry-picked fix(es) | | Deprecated code removal | Yes (if minor bump) | No | | Integration test scope | Full | Focused on affected areas |