Skip to content

ci: fix documentation artefacts automatically after merge#8024

Open
emyller wants to merge 5 commits into
mainfrom
fix/auto-docs-artefacts
Open

ci: fix documentation artefacts automatically after merge#8024
emyller wants to merge 5 commits into
mainfrom
fix/auto-docs-artefacts

Conversation

@emyller

@emyller emyller commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Human text.

Closes #8021

The team has been made responsible to include auto-generated documentation changes in their PRs, leading to some time wasted in back-and-forth make targets and git-fu — not to mention the need to authenticate to AWS for CodeArtefact (private packages). This breaks us free!

Changes

  • Pull requests no longer fail when documentation artefacts are outdated.
  • Documentation artefacts are regenerated and committed automatically after each merge to the default branch.

Review effort: 2/5

FAQ

  1. Why we didn't use pre-commit's CI?
    Because it runs in an isolated environment lacking the runtime to auto-generate docs, and it doesn't have access to private packages.
  2. Why not push changes to the PR?
    Because 1. it uses less resources per work; and 2. it allows community PR not to fail CI.

@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Jul 16, 2026 11:51pm
flagsmith-frontend-preview Ignored Ignored Preview Jul 16, 2026 11:51pm
flagsmith-frontend-staging Ignored Ignored Preview Jul 16, 2026 11:51pm

Request Review

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a GitHub Actions workflow that regenerates public and private API documentation artefacts on changes to api/** on main, then commits and pushes generated changes. Existing pull-request and private-package workflows no longer perform documentation freshness checks. The API also gains more contextual experiment error logging and wording updates to an endpoint description, docstring, and Prometheus metric.

Estimated code review effort: 3 (Moderate) | ~20 minutes


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added api Issue related to the REST API ci-cd Build, test and deployment related labels Jul 15, 2026
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.64%. Comparing base (b51e6b7) to head (e61a1a6).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #8024    +/-   ##
========================================
  Coverage   98.64%   98.64%            
========================================
  Files        1501     1506     +5     
  Lines       59397    59572   +175     
========================================
+ Hits        58591    58766   +175     
  Misses        806      806            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Jul 16, 2026
@github-actions github-actions Bot added the docs Documentation updates label Jul 16, 2026
@emyller emyller self-assigned this Jul 16, 2026
@emyller
emyller force-pushed the fix/auto-docs-artefacts branch from 1235b2e to 380c28c Compare July 16, 2026 14:30
@github-actions github-actions Bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related docs Documentation updates labels Jul 16, 2026
@github-actions github-actions Bot added the docs Documentation updates label Jul 16, 2026
emyller and others added 3 commits July 16, 2026 11:39
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HWahXvKyob4DTBgmGzV1gm
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@emyller
emyller force-pushed the fix/auto-docs-artefacts branch from d104773 to f38778a Compare July 16, 2026 14:40
@github-actions github-actions Bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related docs Documentation updates labels Jul 16, 2026
@github-actions github-actions Bot added the docs Documentation updates label Jul 16, 2026
@emyller
emyller force-pushed the fix/auto-docs-artefacts branch from 2804a78 to e61a1a6 Compare July 16, 2026 16:13
@github-actions github-actions Bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related docs Documentation updates labels Jul 16, 2026
@emyller
emyller marked this pull request as ready for review July 16, 2026 16:15
@emyller
emyller requested review from a team as code owners July 16, 2026 16:15
@emyller
emyller requested review from khvn26 and removed request for a team July 16, 2026 16:15
@github-actions github-actions Bot added ci-cd Build, test and deployment related and removed ci-cd Build, test and deployment related labels Jul 16, 2026
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-8024 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api-test:pr-8024 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-8024 Finished ✅ Results
ghcr.io/flagsmith/flagsmith-api:pr-8024 Finished ✅ Results
ghcr.io/flagsmith/flagsmith:pr-8024 Finished ✅ Results
ghcr.io/flagsmith/flagsmith-private-cloud:pr-8024 Finished ✅ Results

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e148b71d-cbcf-442b-b625-2909e46046a1

📥 Commits

Reviewing files that changed from the base of the PR and between 4df9bfb and e61a1a6.

📒 Files selected for processing (7)
  • .github/workflows/api-fix-docs-artefacts.yml
  • .github/workflows/api-pull-request.yml
  • .github/workflows/api-tests-with-private-packages.yml
  • api/experimentation/tasks.py
  • api/experimentation/views.py
  • api/features/views.py
  • api/segment_membership/metrics.py
💤 Files with no reviewable changes (2)
  • .github/workflows/api-pull-request.yml
  • .github/workflows/api-tests-with-private-packages.yml

Comment on lines +3 to +5
permissions:
contents: read # For actions/checkout
id-token: write # For CodeArtifact OIDC

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

Restrict id-token: write permission to the job level.

Applying id-token: write at the workflow level grants broad access to all jobs in the workflow, which violates the principle of least privilege. While there is only one job here, it is best practice to move the permissions block to the specific job that requires it.

🛡️ Proposed refactor

Remove the global permissions block:

-permissions:
-  contents: read # For actions/checkout
-  id-token: write # For CodeArtifact OIDC

And add it to the fix-docs-artefacts job:

 jobs:
   fix-docs-artefacts:
     runs-on: depot-ubuntu-latest
     name: Fix Documentation Artefacts
+    permissions:
+      contents: read
+      id-token: write
🧰 Tools
🪛 zizmor (1.26.1)

[error] 5-5: overly broad permissions (excessive-permissions): id-token: write is overly broad at the workflow level

(excessive-permissions)

Source: Linters/SAST tools

Comment on lines +68 to +78
- name: Commit and push fixes
run: |
git add --all
if git diff --cached --quiet; then
echo "Documentation artefacts are up to date."
exit 0
fi
git config user.name '${{ steps.app-token.outputs.app-slug }}[bot]'
git config user.email '${{ steps.app-token.outputs.app-slug }}[bot]@users.noreply.github.com'
git commit --message 'chore: Update documentation artefacts'
git push || { git pull --rebase && git push; }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Improve commit step reliability and prevent redundant CI runs.

  • Prevent Redundant Triggers: Because this workflow uses a GitHub App token to push changes, the push will re-trigger this and any other on: push workflows. Appending [skip ci] to the commit message prevents these redundant executions, saving CI resources in direct alignment with the PR objectives.
  • Working Directory: The default working directory for the job is api/. While git add --all operates on the entire working tree regardless of the current directory (in Git 2.0+), it is clearer and safer to explicitly set the working directory to the repository root for repository-wide Git commands to avoid any pathspec ambiguity with artefacts generated in ../docs.
♻️ Proposed refactor
       - name: Commit and push fixes
+        working-directory: ${{ github.workspace }}
         run: |
           git add --all
           if git diff --cached --quiet; then
             echo "Documentation artefacts are up to date."
             exit 0
           fi
           git config user.name '${{ steps.app-token.outputs.app-slug }}[bot]'
           git config user.email '${{ steps.app-token.outputs.app-slug }}[bot]`@users.noreply.github.com`'
-          git commit --message 'chore: Update documentation artefacts'
+          git commit --message 'chore: Update documentation artefacts [skip ci]'
           git push || { git pull --rebase && git push; }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Commit and push fixes
run: |
git add --all
if git diff --cached --quiet; then
echo "Documentation artefacts are up to date."
exit 0
fi
git config user.name '${{ steps.app-token.outputs.app-slug }}[bot]'
git config user.email '${{ steps.app-token.outputs.app-slug }}[bot]@users.noreply.github.com'
git commit --message 'chore: Update documentation artefacts'
git push || { git pull --rebase && git push; }
- name: Commit and push fixes
working-directory: ${{ github.workspace }}
run: |
git add --all
if git diff --cached --quiet; then
echo "Documentation artefacts are up to date."
exit 0
fi
git config user.name '${{ steps.app-token.outputs.app-slug }}[bot]'
git config user.email '${{ steps.app-token.outputs.app-slug }}[bot]`@users.noreply.github.com`'
git commit --message 'chore: Update documentation artefacts [skip ci]'
git push || { git pull --rebase && git push; }
🧰 Tools
🪛 zizmor (1.26.1)

[info] 75-75: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)


[info] 76-76: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  6.6 seconds
commit  e61a1a6
info  🔄 Run: #18375 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  5.8 seconds
commit  e61a1a6
info  🔄 Run: #18375 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  4 passed

Details

stats  4 tests across 4 suites
duration  41.1 seconds
commit  e61a1a6
info  🔄 Run: #18375 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  5 passed

Details

stats  5 tests across 5 suites
duration  37.5 seconds
commit  e61a1a6
info  🔄 Run: #18375 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  37.7 seconds
commit  102acf0
info  🔄 Run: #18414 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

failed  1 failed

Details

stats  1 test across 1 suite
duration  12.2 seconds
commit  102acf0
info  📦 Artifacts: View test results and HTML report
🔄 Run: #18414 (attempt 1)

Failed tests

firefox › tests/onboarding-tests.pw.ts › Onboarding › New user connects via the single-page onboarding flow @oss

### Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  40 seconds
commit  102acf0
info  🔄 Run: #18414 (attempt 2)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  38.6 seconds
commit  102acf0
info  🔄 Run: #18414 (attempt 2)

@github-actions

Copy link
Copy Markdown
Contributor

Visual Regression

19 screenshots compared. See report for details.
View full report

@khvn26 khvn26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Responding to the original reasoning:

Why we didn't use pre-commit's CI?
Because it runs in an isolated environment lacking the runtime to auto-generate docs, and it doesn't have access to private packages.

Is there a way to run pre-commit hooks in a runtime we need in CI? This would allow us to rely on hooks more rather than less, and lean on standardised CI vs introducing another custom workflow.

Why not push changes to the PR?
Because 1. it uses less resources per work; and 2. it allows community PR not to fail CI.

This approach decouples autogenerated changes from the actual changes that cause them; i.e. I do want to see specific changes to the API schema, should the PR introduce them, and review them together with the code.

The concern number 2 should be addressed if we solve the running hooks part — our GitHub app's credentials will be used to run them on any PR.

If we decide to keep with the current workflow, we'll have to avoid pushing directly to main.

Comment on lines +77 to +78
git commit --message 'chore: Update documentation artefacts'
git push || { git pull --rebase && git push; }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this violates SOC2.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean having app pushing to main?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean having anything pushing to main bypassing the PR approval mechanism.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Caution

Review failed

An error occurred during the review process. Please try again later.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API ci-cd Build, test and deployment related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI fails too often on checking documentation artefacts

2 participants