Skip to content

[LTS 8.6] hugetlb: CVE-2024-57883, CVE-2025-38084, CVE-2025-38085, CVE-2026-23100 - #1491

Open
pvts-mat wants to merge 18 commits into
ctrliq:ciqlts8_6from
pvts-mat:CVE-batch-42_ciqlts8_6
Open

[LTS 8.6] hugetlb: CVE-2024-57883, CVE-2025-38084, CVE-2025-38085, CVE-2026-23100#1491
pvts-mat wants to merge 18 commits into
ctrliq:ciqlts8_6from
pvts-mat:CVE-batch-42_ciqlts8_6

Conversation

@pvts-mat

Copy link
Copy Markdown
Contributor

[LTS 8.6]

CVE-2024-57883 VULN-46924
CVE-2025-38084 VULN-71572
CVE-2025-38085 VULN-71581
CVE-2026-23100 VULN-193890

PR structure

This PR consists of two sets of patches and their supporting bugfixes/prerequisites.

The first patch set, provided by Jann Horn, is part of a larger set of hotfixes merged in d3c82f6 and addresses CVE-2025-38084 and CVE-2025-38085 in two commits:

git log -n 14 --oneline --graph d3c82f618a9c2b764b7651afe16594ffeb50ade9
*   d3c82f618 Merge tag 'mm-hotfixes-stable-2025-06-06-16-02' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|\  
| * 2da20fd90 kernel/rcu/tree_stall: add /sys/kernel/rcu_stall_count
| * 7266f590c MAINTAINERS: add mm swap section
| * ffa74d44f kmsan: test: add module description
| * 1b11a2830 MAINTAINERS: add tlb trace events to MMU GATHER AND TLB INVALIDATION
| * 1013af4f5 mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race                        <- CVE-2025-38085
| * 081056dc0 mm/hugetlb: unshare page tables during VMA split, not before               <- CVE-2025-38084
| * a2946fb27 MAINTAINERS: add Alistair as reviewer of mm memory policy
| * 334d7c4fb iov_iter: use iov_offset for length calculation in iov_iter_aligned_bvec
| * 41ffaa0ea mm/mempolicy: fix incorrect freeing of wi_kobj
| * 044d2aee6 alloc_tag: handle module codetag load errors as module load failures
| * 9c49e5d09 mm/madvise: handle madvise_lock() failure during race unwinding
| * 684088173 mm: fix vmstat after removing NR_BOUNCE
| * 15ac613f1 KVM: s390: rename PROT_NONE to PROT_TYPE_DUMMY

The second patch set, provided by David Hildenbrand, is similarly embedded in a larger set of hotfixes, consists of 4 commits (all backported in this PR), and was motivated by the performance regression introduced by the CVE-2025-38085 fix - the "excessive IPI broadcasts" issue. Additionally it addresses CVE-2026-23100.

git log -n 18 --oneline --graph c25f2fb1f469deaed2df8db524d91f3321a0f816
*   c25f2fb1f Merge tag 'mm-hotfixes-stable-2026-01-20-13-09' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
|\  
| * 16aca2c98 mm: restore per-memcg proactive reclaim with !CONFIG_NUMA
| * 9bc9ccbf4 mm/kfence: fix potential deadlock in reboot notifier
| * cb7d761bf Docs/mm/allocation-profiling: describe sysctrl limitations in debug mode
| * 35e247032 mm: do not copy page tables unnecessarily for VM_UFFD_WP
| * 8ce720d5b mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather  <- performance patch
| * a8682d500 mm/rmap: fix two comments related to huge_pmd_unshare()                              <-
| * 3937027ca mm/hugetlb: fix two comments related to huge_pmd_unshare()                           <-
| * ca1a47cd3 mm/hugetlb: fix hugetlb_pmd_shared()                                                 <- CVE-2026-23100
| * 90888b4ae mm: remove unnecessary and incorrect mmap lock assert
| * b505f1944 x86/kfence: avoid writing L1TF-vulnerable PTEs
| * 605f6586e mm/vma: do not leak memory when .mmap_prepare swaps the file
| * b7880cb16 migrate: correct lock ordering for hugetlb file folios
| * 90f3c1232 panic: only warn about deprecated panic_print on write access
| * f9a49aa30 fs/writeback: skip AS_NO_DATA_INTEGRITY mappings in wait_sb_inodes()
| * be31340a4 mm: take into account mm_cid size for mm_struct static definitions
| * 6ac433f8b mm: rename cpu_bitmap field to flexible_array
| * 12a6ddfc7 mm: add missing static initializer for init_mm::mm_cid.lock

The CVE-2025-38085 problem was about unsharing a shared page table on one processor, thus potentially changing its meaning (turning it into a normal page table) while another processor could have been in the middle of walking that table (resolving a virtual address to a physical one). The solution was to send IPI (Inter-Processor Interrupt) to other processors as a synchronization mechanism. This resulted in a noticable performance drop on some setups of RH clients (https://lore.kernel.org/all/4d3878531c76479d9f8ca9789dc6485d@amazon.de/). A quick mitigation of that problem in the form of "fix disable" was merged into CentOS 9 and CentOS 10. The 8ce720d mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather commit implements a proper solution and was merged into the upstream, then backported, at least to centos9 (797ec6e) and rocky8_10 (ecba853).

One important prerequisite for this fix was the stand-alone commit 59d9094 mm: hugetlb: independent PMD page table shared count, which also solved CVE-2024-57883 (and introduced CVE-2026-23100 - solved, however, in the same patch set as the performance fix).

CVE-2025-38085 fix

Commit 1013af4 mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race is identified as the fix for CVE-2025-38085. However, it depends on a function tlb_remove_table_sync_one which is not defined in LTS 8.6. It was introduced in 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE. No solution to CVE-2025-38085 was found which didn't have this commit in place:

  • centos9 12a6db3,
  • centos10 41f7eb5,
  • rocky9_6 2872192,
  • rocky8_10 e2c95cd,
  • rocky10_0 68b3612,
  • linux-5.10.y: 952596b08c74e8fe9e2883d1dc8a8f54a37384ec,
  • linux-5.15.y: a3d864c901a300c295692d129159fc3001a56185,
  • linux-6.1.y: b7754d3aa7bf9f62218d096c0c8f6c13698fac8b,
  • linux-6.6.y: fe684290418ef9ef76630072086ee530b92f02b8,
  • linux-6.12.y: 034a52b,
  • linux-6.15.y: a6bfeb97941a9187833b526bc6cc4ff5706d0ce9,

although also none of those solutions had 0d6e24d backported as part of it.

The commit 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE introduces a new option, but it's not user-selectable; rather, it's used in the configuration as part of architecture's features makeup, eg. as in (implied by):

select MMU_GATHER_RCU_TABLE_FREE

The commit therefore doesn't introduce a new feature but addresses the hardare setup not accounted for sufficiently before:

This however leaves architectures that don't have page based directories
but don't need RCU in a bind. For those, provide MMU_GATHER_TABLE_FREE,
which provides the independent batching for directories without the
additional RCU freeing.

Nevertheless backporting it may seem risky and simply ad-hoc defining the one line long tlb_remove_table_sync_one may be preferred. However, the definition of tlb_remove_table_sync_one function used in CVE-2025-38085 depends on an option. For CONFIG_MMU_GATHER_RCU_TABLE_FREE:

static void tlb_remove_table_sync_one(void)
{
/*
* This isn't an RCU grace period and hence the page-tables cannot be
* assumed to be actually RCU-freed.
*
* It is however sufficient for software page-table walkers that rely on
* IRQ disabling.
*/
smp_call_function(tlb_remove_table_smp_sync, NULL, 1);
}

For !CONFIG_MMU_GATHER_RCU_TABLE_FREE:

static void tlb_remove_table_sync_one(void) { }

In the upstream the tlb_remove_table_one function is expressed by tlb_remove_table_sync_one:

static void tlb_remove_table_one(void *table)
{
tlb_remove_table_sync_one();
__tlb_remove_table(table);
}

If it was defined like that in LTS 8.6 as well then its definition would start depending on CONFIG_MMU_GATHER_RCU_TABLE_FREE too, which would change code semantics in an uncontrolled manner. Putting it under control by following 0d6e24d's options dependency would boil down to simply backporting that commit. Not expressing tlb_remove_table_one by tlb_remove_table_sync_one, on the other hand, would lead to code duplication and enigmatic deviation from the upstream. It was therefore decided to backport 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE despite some complications it implied.

That commit depends on options naming scheme not yet backported to LTS 8.6:

Old New Renaming ocmmit
`HAVE_RCU_TABLE_FREE` `MMU_GATHER_RCU_TABLE_FREE` ff2e6d7
`HAVE_MMU_GATHER_PAGE_SIZE` `MMU_GATHER_PAGE_SIZE` 3af4bd0
`HAVE_MMU_GATHER_NO_GATHER` `MMU_GATHER_NO_GATHER` 580a586
- `MMU_GATHER_TABLE_FREE` 0d6e24d

The renaming was done in a bulk, right before 0d6e24d:

git log --oneline -n 4 0d6e24d430ef23280d8dea0ba1faeefc66c26a57
0d6e24d430ef23280d8dea0ba1faeefc66c26a57 asm-generic/tlb: provide MMU_GATHER_TABLE_FREE
580a586c409ab3040b7284a19cd9e281692c40c7 asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER
3af4bd033759c4dab4f0ff594f0aa1e8d182b9d7 asm-generic/tlb: rename HAVE_MMU_GATHER_PAGE_SIZE
ff2e6d7259f82ccc9a5aaa7f41194161d9262392 asm-generic/tlb: rename HAVE_RCU_TABLE_FREE

Between sticking to an old scheme and using the new one the latter was chosen, because it made tha backporting much simpler, less prone to mistakes and it avoided devising a completely unique option name HAVE_MMU_GATHER_TABLE_FREE. Only the MMU_GATHER_NO_GATHER option was renamed though, to simplify the bug fix. This introduced some option naming inconsistency, but it's not a big problem.

All other commits in the CVE-2025-38085 fix are preparing the code for either the option renaming done in 580a586 asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER, or for prviding the MMU_GATHER_TABLE_FREE option with the required tlb_remove_table_sync_one function in 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE.

Finally, the CVE-2025-38085 fix 1013af4 mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race includes the exposition of tlb_remove_table_sync_one, which in the upstream was done in a similar fix 2ba99c5 mm/khugepaged: fix GUP-fast interaction by sending IPI, not backported to LTS 8.6.

Linux stable 5.10 as the main source of changes

The PR strongly relies on the Linux stable 5.10 backports, which provided clean cherry picks for the most part. Although the mm/hugetlb codebase in ciqlts8_6 branched off the upstream roughly two years earlier than linux-5.10.y (2018 vs 2020), it was heavily backported with features until the beginning of 2021, such that it closely approximates Linux stable 5.10.

CVE-2026-63875 issue

The performance-fixing commit 8ce720d mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather introduces arm64-specific vulnerability CVE-2026-63875, fixed by c2ff476. The modified code fragments don't even exist in LTS 8.6, so it's highly probable that the problem doesn't apply to this version. However, given the very non-local nature of page tables management a deeper dive may be required. Because the volume of this PR is already big the the solution of CVE-2026-63875 (or marking LTS 8.6 as not affected by it) was deferred to a different PR. This aligns with RH's https://access.redhat.com/security/cve/cve-2026-63875.

The CVE-2025-38084 bugfix b4aea43 issue

The CVE-2025-38084 fix 081056d mm/hugetlb: unshare page tables during VMA split, not before introduces a bug fixed by b4aea43 mm/hugetlb: avoid false positive lockdep assertion. However, it doesn't apply to LTS 8.6. From the commit's message:

Commit 081056d ("mm/hugetlb: unshare page tables during VMA split,
not before") changed the locking model around hugetlbfs PMD unsharing on
VMA split, but did not update the function which asserts the locks,
hugetlb_vma_assert_locked().

The hugetlb_vma_assert_locked() function is not present in LTS 8.6 and neither it has any differently named equivalents. It was introduced in 8d9bfb2 hugetlb: add vma based lock for pmd sharing, not backported to LTS 8.6. It also wasn't backported to linux-5.10.y and linux-5.15.y and these versions don't have the bugfix b4aea43 backported either (unlike versions linux-6.1.y, linux-6.6.y, linux-6.12.y which do have 8d9bfb2 in history and do backport the bugfix backported, as 3e1b69f02e71f73ad47e1c2dcaca02234a512c93, 428a33573dcb921971d1965f618cc4431c00b126 and b987d94)

Commits

The order of the commits in this PR reflects the natural order in the upstream. Because the prerequisites and bugfixes were not linear, the relationships between them can best be depicted with the following graph:

relations-graph
mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather

jira VULN-71581
cve-bf CVE-2025-38085
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit 8ce720d5bd91e9dc16db3604aa4b1bf76770a9a1
upstream-diff Used linux-5.10.y backport
  6495204f8219a57859669d618b25d06176d5a872 for an almost clean pick.
  Resolved conflicts in the `hugetlb_unshare_pmds' function: unlike
  ciqlts8_6 the linux-5.10.y version doesn't have
  4ddb4d91b82f4b64458fe35bc8e395c7c082ea2b ("hugetlb: do not update
  address in huge_pmd_unshare") backported. That commit changed the
  signature of `huge_pmd_unshare()', converting the third argument
  `unsigned long *addr' to `unsigned long addr'. This allowed for code
  reduction in `hugetlb_unshare_all_pmds()' by dropping the use of
  temporary `address' placeholder protecting that variable from being
  changed by the `huge_pmd_unshare()' call. From the backported commit
  perspective this is irrelevant though, because it's only concerned
  with the change of the first argument of `huge_pmd_unshare()'. Applied
  the first argument change while preserving `&address' used as the
  third.
mm/rmap: fix two comments related to huge_pmd_unshare()

jira VULN-46924
cve-bf CVE-2024-57883
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit a8682d500f691b6dfaa16ae1502d990aeb86e8be
upstream-diff Used linux-5.10.y backport
  2723af9e1e283cee312946c3d6cb20d2c731c139 for the clean pick
mm/hugetlb: fix two comments related to huge_pmd_unshare()

jira VULN-46924
cve-bf CVE-2024-57883
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit 3937027caecb4f8251e82dd857ba1d749bb5a428
upstream-diff Used linux-5.10.y backport
  44b18bac2b3063784ebb871a922803b11d9c339d for the clean pick
mm/hugetlb: fix hugetlb_pmd_shared()

jira VULN-193890
cve-bf CVE-2026-23100
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit ca1a47cd3f5f4c46ca188b1c9a27af87d1ab2216
upstream-diff Used linux-5.10.y backport
  8ae48255bcb17b32436be97553dca848730d365f for the clean pick
mm/hugetlb: fix copy_hugetlb_page_range() to use ->pt_share_count

jira VULN-71572
cve-bf CVE-2025-38084
commit-author Jane Chu <jane.chu@oracle.com>
commit 59d9094df3d79 ("mm: hugetlb: independent PMD page table shared
commit 14967a9c7d247841b0312c48dcf8cd29e55a4cc8
upstream-diff Used linux-5.10.y backport
  ee67c4ac1aed403688dd359918099a745b60fb69 for the clean pick
mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race

jira VULN-71581
cve CVE-2025-38085
commit-author Jann Horn <jannh@google.com>
commit 1013af4f585fccc4d3e5c5824d174de2257f7d6d
upstream-diff Used linux-5.10.y backport
  952596b08c74e8fe9e2883d1dc8a8f54a37384ec for the clean pick. Exposed
  the `tlb_remove_table_sync_one' function for use in the solution
  (missing 2ba99c5e08812494bc57f319fb562f527d9bacd8 ("mm/khugepaged: fix
  GUP-fast interaction by sending IPI")).

The CVE-2025-38085 fixing commit. Compared to the upstream the tlb_remove_table_sync_one function was made public as part of the fix.

mm/hugetlb: unshare page tables during VMA split, not before

jira VULN-71572
cve CVE-2025-38084
commit-author Jann Horn <jannh@google.com>
commit 081056dc00a27bccb55ccc3c6f230a3d5fd3f7e0
upstream-diff Used linux-5.10.y backport
  e8847d18cd9fff1edbb45e963d9141273c3b539c for the clean pick
mm: hugetlb: independent PMD page table shared count

jira VULN-46924
cve CVE-2024-57883
commit-author Liu Shixin <liushixin2@huawei.com>
commit 59d9094df3d79443937add8700b2ef1a866b1081
upstream-diff Stable linux-5.15.y backport
  8410996eb6fea116fe1483ed977aacf580eee7b4 was used for the actual (clean)
  cherry-pick. The newly inserted field `atomic_t pt_share_count'
  preserves the intent of the `RH_KABI_REPLACE' macro call it's embedded
  in: the field has the same type as another field in the union `atomic_t
  pt_frag_refcount', thus preserving its size and offset. Additionally,
  this union serves as a scratch space for the subsystems using the struct
  page. Upon releasing the ownership to buddy allocator the union contents
  no longer matter. When the page is allocated again the scratch space
  will be used by the new owner in its own way.

The kABI issue described was discussed in the ciqlts9_2 backport of CVE-2024-57883: #731. While the situation here is slightly different (RH_KABI_REPLACE macro used) the buddy allocator argumentation is the same.

mm: hugetlb: proc: check for hugetlb shared PMD in /proc/PID/smaps

jira VULN-193890
cve-pre CVE-2026-23100
commit-author Mike Kravetz <mike.kravetz@oracle.com>
commit 3489dbb696d25602aea8c3e669a6d43b76bd5358
upstream-diff Used linux-5.10.y backport
  556959327b190a4efcfccc4ed311a08f6ea520ed for an almost clean pick.
  Resolved conflict at the end due to missing
  cf11e85fc08cc6a4fe3ac2ba2e610c962bf20bc3 ("mm: hugetlb: optionally
  allocate gigantic hugepages using cma")
hugetlb: unshare some PMDs when splitting VMAs

jira VULN-71572
cve-pre CVE-2025-38084
commit-author James Houghton <jthoughton@google.com>
commit b30c14cd61025eeea2f2e8569606cd167ba9ad2d
upstream-diff Used linux-5.10.y backport
  f1082f5f3d02f2f9b4ca36642274529cf5ddaf34 for an almost clean pick.
  Context conflicts resolved due to the missing feature
  cf11e85fc08cc6a4fe3ac2ba2e610c962bf20bc3 ("mm: hugetlb: optionally
  allocate gigantic hugepages using cma")
mm/hugetlb: make detecting shared pte more reliable

jira VULN-71572
cve-pre CVE-2025-38084
commit-author Miaohe Lin <linmiaohe@huawei.com>
commit 3aa4ed8040e1535d95c03cef8b52cf11bf0d8546
upstream-diff Fixed small conflict in copy_hugetlb_page_range due to
  missing e95a9851787bbb3cd4deb40fe8bab03f731852d1 ("hugetlb: skip to
  end of PT page mapping when pte not present")
mm: proc: Invalidate TLB after clearing soft-dirty page state

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Will Deacon <will@kernel.org>
commit 912efa17e5121693dfbadae29768f4144a3f9e62
upstream-diff Fixed context conflicts due to the lack of the following
  commits:
  - 4d45e75a9955ade5c2f49bd96fc4173b2cec9a72 ("mm: remove the
    now-unnecessary mmget_still_valid() hack")
  - 29a951dfb3c3263c3a0f3bd9f7f2c2cfde4baedb ("mm: fix clear_refs_write
    locking")
  Backporting them, although syntactically clean, would require including
  additional patches to provide the logic required for them to be correct.
asm-generic/tlb: provide MMU_GATHER_TABLE_FREE

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit 0d6e24d430ef23280d8dea0ba1faeefc66c26a57
upstream-diff |
  include/asm-generic/tlb.h
        Small conflict in the comments
  mm/mmu_gather.c
        - Context conflict in tlb_flush_mmu_free because
          580a586c409ab3040b7284a19cd9e281692c40c7 ("asm-generic/tlb:
          rename HAVE_MMU_GATHER_NO_GATHER") wasn't backported and
          CONFIG_MMU_GATHER_NO_GATHER still has its old name
          CONFIG_HAVE_MMU_GATHER_NO_GATHER.
        - Context conflict in tlb_gather_mmu because
          3af4bd033759c4dab4f0ff594f0aa1e8d182b9d7 ("asm-generic/tlb:
          rename HAVE_MMU_GATHER_PAGE_SIZE") wasn't backported and
          CONFIG_MMU_GATHER_PAGE_SIZE still has its old name
          CONFIG_HAVE_MMU_GATHER_PAGE_SIZE.

This commit provides the necessary options for the tlb_remove_table_sync_one function to be defined properly. The new option MMU_GATHER_TABLE_FREE is not user-selectable and it doesn't introduce a new feature. Rather it can be thought of as fine-tuning the existing solution.

asm-generic/tlb: rename HAVE_RCU_TABLE_FREE

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit ff2e6d7259f82ccc9a5aaa7f41194161d9262392
upstream-diff |
  arch/arm64/Kconfig
        Context conflicts due to options HAVE_FUTEX_CMPXCHG,
        HAVE_FUNCTION_ARG_ACCESS_API not yet introduced into LTS 8.6.
  arch/x86/Kconfig
        Context conflicts due to the option HAVE_RCU_TABLE_INVALIDATE
        still being in LTS 8.6, but no longer in the upstream

Code refactoring commit, functionally neutral. Backporting it significantly reduces conflict in backporting the main CVE-2025-38085 prerequisite 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE.

asm-generic/tlb: avoid potential double flush

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit 0758cd8304942292e95a0f750c374533db378b32
arm/tlb: Convert to generic mmu_gather

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit b78180b97dcf667350aac716cd3f32356eaf4984

This commit massively simplifies the arch/arm/include/asm/tlb.h, easing the backport of 0d6e24d asm-generic/tlb: provide MMU_GATHER_TABLE_FREE. Important note from the commit's message:

No change in behavior intended.

ARM: Kconfig: remove useless parenthesis

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Russell King <rmk+kernel@armlinux.org.uk>
commit f00790aaf2e240f57d19dab39e07f0ab87c6d95c

Similarly to the commit above this one prepares the codebase for the cherry pick of ff2e6d7 asm-generic/tlb: rename HAVE_RCU_TABLE_FREE. Functionally neutral.

arm/asm/tlb.h: Fix build error implicit func declaration

jira VULN-71581
cve-pre CVE-2025-38085
commit-author Anders Roxell <anders.roxell@linaro.org>
commit 063daa8129f94d651c4d30a8333b577c5bbf8090

This commit is just a small and cheap prerequisite to make clean the cherry pick of b78180b arm/tlb: Convert to generic mmu_gather. Functionally neutral.

kABI check: passed

+ dist_git_version=el-8.6
+ local_version=ciqlts8_6-CVE-batch-39
+ arch=x86_64
+ user=pvts
+ buildmachine=x86_64--build--ciqlts8_6
+ virsh_timeout=600
+ ssh_daemon_wait=20
+ src_dir=/mnt/code/kernel-dist-git-el-8.6
+ build_dir=/mnt/build_files/kernel-src-tree-ciqlts8_6-CVE-batch-39
+ sudo chmod +x /data/src/ctrliq-github-haskell/kernel-dist-git-el-8.6/SOURCES/check-kabi
+ ninja-back/virssh.xsh --max 8 --shutdown-on-success --shutdown-on-failure --timeout 600 --ssh-daemon-wait 20 pvts x86_64--build--ciqlts8_6 ''\''/mnt/code/kernel-dist-git-el-8.6/SOURCES/check-kabi'\'' -k '\''/mnt/code/kernel-dist-git-el-8.6/SOURCES/Module.kabi_x86_64'\'' -s '\''/mnt/build_files/kernel-src-tree-ciqlts8_6-CVE-batch-39/Module.symvers'\'''
kABI check passed
+ touch state/kernels/ciqlts8_6-CVE-batch-39/x86_64/kabi_checked

Boot test: passed

boot-test.log

Kselftests: passed relative

Reference

kselftests–ciqlts8_6–run1.log
kselftests–ciqlts8_6–run2.log
kselftests–ciqlts8_6–run3.log

Patch

kselftests–ciqlts8_6-CVE-batch-42–run1.log
kselftests–ciqlts8_6-CVE-batch-42–run2.log
kselftests–ciqlts8_6-CVE-batch-42–run3.log

Comparison

The tests results for the reference and the patch are the same.

$ ktests.xsh diff -d kselftests*.log

Column    File
--------  --------------------------------------------
Status0   kselftests--ciqlts8_6--run1.log
Status1   kselftests--ciqlts8_6--run2.log
Status2   kselftests--ciqlts8_6--run3.log
Status3   kselftests--ciqlts8_6-CVE-batch-42--run1.log
Status4   kselftests--ciqlts8_6-CVE-batch-42--run2.log
Status5   kselftests--ciqlts8_6-CVE-batch-42--run3.log

full-test-results-comparison.log

pvts-mat added 18 commits July 29, 2026 22:46
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Anders Roxell <anders.roxell@linaro.org>
commit 063daa8

Building on arm 32 with LPAE enabled we don't include asm-generic/tlb.h,
where we have tlb_flush_remove_tables_local and tlb_flush_remove_tables
defined.

The build fails with:

  mm/memory.c: In function ‘tlb_remove_table_smp_sync’:
  mm/memory.c:339:2: error: implicit declaration of function ‘tlb_flush_remove_tables_local’; did you mean ‘tlb_remove_table’? [-Werror=implicit-function-declaration]
  ...

This bug got introduced in:

  2ff6ddf ("x86/mm/tlb: Leave lazy TLB mode at page table free time")

To fix this issue we define them in arm 32's specific asm/tlb.h file as well.

	Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
	Cc: Linus Torvalds <torvalds@linux-foundation.org>
	Cc: Peter Zijlstra <peterz@infradead.org>
	Cc: Thomas Gleixner <tglx@linutronix.de>
	Cc: dave.hansen@intel.com
	Cc: linux-arm-kernel@lists.infradead.org
	Cc: linux@armlinux.org.uk
	Cc: riel@surriel.com
	Cc: songliubraving@fb.com
Fixes: 2ff6ddf ("x86/mm/tlb: Leave lazy TLB mode at page table free time")
Link: http://lkml.kernel.org/r/20180725095557.19668-1-anders.roxell@linaro.org
	Signed-off-by: Ingo Molnar <mingo@kernel.org>
(cherry picked from commit 063daa8)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Russell King <rmk+kernel@armlinux.org.uk>
commit f00790a

Remove useless parenthesis from the Kconfig - Kconfig is not C, and
can cope without parenthesising e.g. single symbols to 'if'.

	Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
(cherry picked from commit f00790a)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit b78180b

Generic mmu_gather provides everything that ARM needs:

 - range tracking
 - RCU table free
 - VM_EXEC tracking
 - VIPT cache flushing

The one notable curiosity is the 'funny' range tracking for classical
ARM in __pte_free_tlb().

No change in behavior intended.

	Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Acked-by: Will Deacon <will.deacon@arm.com>
	Cc: Andrew Morton <akpm@linux-foundation.org>
	Cc: Andy Lutomirski <luto@kernel.org>
	Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
	Cc: Borislav Petkov <bp@alien8.de>
	Cc: Dave Hansen <dave.hansen@linux.intel.com>
	Cc: H. Peter Anvin <hpa@zytor.com>
	Cc: Linus Torvalds <torvalds@linux-foundation.org>
	Cc: Nick Piggin <npiggin@gmail.com>
	Cc: Peter Zijlstra <peterz@infradead.org>
	Cc: Rik van Riel <riel@surriel.com>
	Cc: Russell King <linux@armlinux.org.uk>
	Cc: Thomas Gleixner <tglx@linutronix.de>
	Signed-off-by: Ingo Molnar <mingo@kernel.org>
(cherry picked from commit b78180b)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit 0758cd8

Aneesh reported that:

	tlb_flush_mmu()
	  tlb_flush_mmu_tlbonly()
	    tlb_flush()			<-- ctrliq#1
	  tlb_flush_mmu_free()
	    tlb_table_flush()
	      tlb_table_invalidate()
		tlb_flush_mmu_tlbonly()
		  tlb_flush()		<-- ctrliq#2

does two TLBIs when tlb->fullmm, because __tlb_reset_range() will not
clear tlb->end in that case.

Observe that any caller to __tlb_adjust_range() also sets at least one of
the tlb->freed_tables || tlb->cleared_p* bits, and those are
unconditionally cleared by __tlb_reset_range().

Change the condition for actually issuing TLBI to having one of those bits
set, as opposed to having tlb->end != 0.

Link: http://lkml.kernel.org/r/20200116064531.483522-4-aneesh.kumar@linux.ibm.com
	Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
	Reported-by: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
	Cc: Michael Ellerman <mpe@ellerman.id.au>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
	Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 0758cd8)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit ff2e6d7
upstream-diff |
  arch/arm64/Kconfig
        Context conflicts due to options HAVE_FUTEX_CMPXCHG,
        HAVE_FUNCTION_ARG_ACCESS_API not yet introduced into LTS 8.6.
  arch/x86/Kconfig
        Context conflicts due to the option HAVE_RCU_TABLE_INVALIDATE
        still being in LTS 8.6, but no longer in the upstream

Towards a more consistent naming scheme.

[akpm@linux-foundation.org: fix sparc64 Kconfig]
Link: http://lkml.kernel.org/r/20200116064531.483522-7-aneesh.kumar@linux.ibm.com
	Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
	Cc: Michael Ellerman <mpe@ellerman.id.au>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
	Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit 6adfe77d47bd5db38fd2e5b5786c77b86e462e13)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Peter Zijlstra <peterz@infradead.org>
commit 0d6e24d
upstream-diff |
  include/asm-generic/tlb.h
        Small conflict in the comments
  mm/mmu_gather.c
        - Context conflict in tlb_flush_mmu_free because
          580a586 ("asm-generic/tlb:
          rename HAVE_MMU_GATHER_NO_GATHER") wasn't backported and
          CONFIG_MMU_GATHER_NO_GATHER still has its old name
          CONFIG_HAVE_MMU_GATHER_NO_GATHER.
        - Context conflict in tlb_gather_mmu because
          3af4bd0 ("asm-generic/tlb:
          rename HAVE_MMU_GATHER_PAGE_SIZE") wasn't backported and
          CONFIG_MMU_GATHER_PAGE_SIZE still has its old name
          CONFIG_HAVE_MMU_GATHER_PAGE_SIZE.

As described in the comment, the correct order for freeing pages is:

 1) unhook page
 2) TLB invalidate page
 3) free page

This order equally applies to page directories.

Currently there are two correct options:

 - use tlb_remove_page(), when all page directores are full pages and
   there are no futher contraints placed by things like software
   walkers (HAVE_FAST_GUP).

 - use MMU_GATHER_RCU_TABLE_FREE and tlb_remove_table() when the
   architecture does not do IPI based TLB invalidate and has
   HAVE_FAST_GUP (or software TLB fill).

This however leaves architectures that don't have page based directories
but don't need RCU in a bind.  For those, provide MMU_GATHER_TABLE_FREE,
which provides the independent batching for directories without the
additional RCU freeing.

Link: http://lkml.kernel.org/r/20200116064531.483522-10-aneesh.kumar@linux.ibm.com
	Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
	Cc: Michael Ellerman <mpe@ellerman.id.au>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
	Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
(cherry picked from commit d420cc00ee07b95443e2d9a0c6b4f4c1ee00bc36)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve-pre CVE-2025-38085
commit-author Will Deacon <will@kernel.org>
commit 912efa1
upstream-diff Fixed context conflicts due to the lack of the following
  commits:
  - 4d45e75 ("mm: remove the
    now-unnecessary mmget_still_valid() hack")
  - 29a951d ("mm: fix clear_refs_write
    locking")
  Backporting them, although syntactically clean, would require including
  additional patches to provide the logic required for them to be correct.

Since commit 0758cd8 ("asm-generic/tlb: avoid potential double
flush"), TLB invalidation is elided in tlb_finish_mmu() if no entries
were batched via the tlb_remove_*() functions. Consequently, the
page-table modifications performed by clear_refs_write() in response to
a write to /proc/<pid>/clear_refs do not perform TLB invalidation.
Although this is fine when simply aging the ptes, in the case of
clearing the "soft-dirty" state we can end up with entries where
pte_write() is false, yet a writable mapping remains in the TLB.

Fix this by avoiding the mmu_gather API altogether: managing both the
'tlb_flush_pending' flag on the 'mm_struct' and explicit TLB
invalidation for the sort-dirty path, much like mprotect() does already.

Fixes: 0758cd8 ("asm-generic/tlb: avoid potential double flush”)
	Signed-off-by: Will Deacon <will@kernel.org>
	Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Reviewed-by: Yu Zhao <yuzhao@google.com>
	Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
	Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Link: https://lkml.kernel.org/r/20210127235347.1402-2-will@kernel.org
(cherry picked from commit 912efa1)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572
cve-pre CVE-2025-38084
commit-author Miaohe Lin <linmiaohe@huawei.com>
commit 3aa4ed8
upstream-diff Fixed small conflict in copy_hugetlb_page_range due to
  missing e95a985 ("hugetlb: skip to
  end of PT page mapping when pte not present")

If the pagetables are shared, we shouldn't copy or take references.  Since
src could have unshared and dst shares with another vma, huge_pte_none()
is thus used to determine whether dst_pte is shared.  But this check isn't
reliable.  A shared pte could have pte none in pagetable in fact.  The
page count of ptep page should be checked here in order to reliably
determine whether pte is shared.

[lukas.bulwahn@gmail.com: remove unused local variable dst_entry in copy_hugetlb_page_range()]
  Link: https://lkml.kernel.org/r/20220822082525.26071-1-lukas.bulwahn@gmail.com
Link: https://lkml.kernel.org/r/20220816130553.31406-7-linmiaohe@huawei.com
	Signed-off-by: Miaohe Lin <linmiaohe@huawei.com>
	Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
	Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
	Cc: Muchun Song <songmuchun@bytedance.com>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 3aa4ed8)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572
cve-pre CVE-2025-38084
commit-author James Houghton <jthoughton@google.com>
commit b30c14c
upstream-diff Used linux-5.10.y backport
  f1082f5f3d02f2f9b4ca36642274529cf5ddaf34 for an almost clean pick.
  Context conflicts resolved due to the missing feature
  cf11e85 ("mm: hugetlb: optionally
  allocate gigantic hugepages using cma")

PMD sharing can only be done in PUD_SIZE-aligned pieces of VMAs; however,
it is possible that HugeTLB VMAs are split without unsharing the PMDs
first.

Without this fix, it is possible to hit the uffd-wp-related WARN_ON_ONCE
in hugetlb_change_protection [1].  The key there is that
hugetlb_unshare_all_pmds will not attempt to unshare PMDs in
non-PUD_SIZE-aligned sections of the VMA.

It might seem ideal to unshare in hugetlb_vm_op_open, but we need to
unshare in both the new and old VMAs, so unsharing in hugetlb_vm_op_split
seems natural.

[1]: https://lore.kernel.org/linux-mm/CADrL8HVeOkj0QH5VZZbRzybNE8CG-tEGFshnA+bG9nMgcWtBSg@mail.gmail.com/

Link: https://lkml.kernel.org/r/20230104231910.1464197-1-jthoughton@google.com
Fixes: 6dfeaff ("hugetlb/userfaultfd: unshare all pmds for hugetlbfs when register wp")
	Signed-off-by: James Houghton <jthoughton@google.com>
	Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
	Acked-by: Peter Xu <peterx@redhat.com>
	Cc: Axel Rasmussen <axelrasmussen@google.com>
	Cc: Muchun Song <songmuchun@bytedance.com>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit f1082f5f3d02f2f9b4ca36642274529cf5ddaf34)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-193890
cve-pre CVE-2026-23100
commit-author Mike Kravetz <mike.kravetz@oracle.com>
commit 3489dbb
upstream-diff Used linux-5.10.y backport
  5569593 for an almost clean pick.
  Resolved conflict at the end due to missing
  cf11e85 ("mm: hugetlb: optionally
  allocate gigantic hugepages using cma")

Patch series "Fixes for hugetlb mapcount at most 1 for shared PMDs".

This issue of mapcount in hugetlb pages referenced by shared PMDs was
discussed in [1].  The following two patches address user visible behavior
caused by this issue.

[1] https://lore.kernel.org/linux-mm/Y9BF+OCdWnCSilEu@monkey/

This patch (of 2):

A hugetlb page will have a mapcount of 1 if mapped by multiple processes
via a shared PMD.  This is because only the first process increases the
map count, and subsequent processes just add the shared PMD page to their
page table.

page_mapcount is being used to decide if a hugetlb page is shared or
private in /proc/PID/smaps.  Pages referenced via a shared PMD were
incorrectly being counted as private.

To fix, check for a shared PMD if mapcount is 1.  If a shared PMD is found
count the hugetlb page as shared.  A new helper to check for a shared PMD
is added.

[akpm@linux-foundation.org: simplification, per David]
[akpm@linux-foundation.org: hugetlb.h: include page_ref.h for page_count()]
Link: https://lkml.kernel.org/r/20230126222721.222195-2-mike.kravetz@oracle.com
Fixes: 25ee01a ("mm: hugetlb: proc: add hugetlb-related fields to /proc/PID/smaps")
	Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
	Acked-by: Peter Xu <peterx@redhat.com>
	Cc: David Hildenbrand <david@redhat.com>
	Cc: James Houghton <jthoughton@google.com>
	Cc: Matthew Wilcox <willy@infradead.org>
	Cc: Michal Hocko <mhocko@suse.com>
	Cc: Muchun Song <songmuchun@bytedance.com>
	Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev>
	Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com>
	Cc: Yang Shi <shy828301@gmail.com>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 3489dbb)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924
cve CVE-2024-57883
commit-author Liu Shixin <liushixin2@huawei.com>
commit 59d9094
upstream-diff Stable linux-5.15.y backport
  8410996eb6fea116fe1483ed977aacf580eee7b4 was used for the actual (clean)
  cherry-pick. The newly inserted field `atomic_t pt_share_count'
  preserves the intent of the `RH_KABI_REPLACE' macro call it's embedded
  in: the field has the same type as another field in the union `atomic_t
  pt_frag_refcount', thus preserving its size and offset. Additionally,
  this union serves as a scratch space for the subsystems using the struct
  page. Upon releasing the ownership to buddy allocator the union contents
  no longer matter. When the page is allocated again the scratch space
  will be used by the new owner in its own way.

The folio refcount may be increased unexpectly through try_get_folio() by
caller such as split_huge_pages.  In huge_pmd_unshare(), we use refcount
to check whether a pmd page table is shared.  The check is incorrect if
the refcount is increased by the above caller, and this can cause the page
table leaked:

 BUG: Bad page state in process sh  pfn:109324
 page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x66 pfn:0x109324
 flags: 0x17ffff800000000(node=0|zone=2|lastcpupid=0xfffff)
 page_type: f2(table)
 raw: 017ffff800000000 0000000000000000 0000000000000000 0000000000000000
 raw: 0000000000000066 0000000000000000 00000000f2000000 0000000000000000
 page dumped because: nonzero mapcount
 ...
 CPU: 31 UID: 0 PID: 7515 Comm: sh Kdump: loaded Tainted: G    B              6.13.0-rc2master+ ctrliq#7
 Tainted: [B]=BAD_PAGE
 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
 Call trace:
  show_stack+0x20/0x38 (C)
  dump_stack_lvl+0x80/0xf8
  dump_stack+0x18/0x28
  bad_page+0x8c/0x130
  free_page_is_bad_report+0xa4/0xb0
  free_unref_page+0x3cc/0x620
  __folio_put+0xf4/0x158
  split_huge_pages_all+0x1e0/0x3e8
  split_huge_pages_write+0x25c/0x2d8
  full_proxy_write+0x64/0xd8
  vfs_write+0xcc/0x280
  ksys_write+0x70/0x110
  __arm64_sys_write+0x24/0x38
  invoke_syscall+0x50/0x120
  el0_svc_common.constprop.0+0xc8/0xf0
  do_el0_svc+0x24/0x38
  el0_svc+0x34/0x128
  el0t_64_sync_handler+0xc8/0xd0
  el0t_64_sync+0x190/0x198

The issue may be triggered by damon, offline_page, page_idle, etc, which
will increase the refcount of page table.

1. The page table itself will be discarded after reporting the
   "nonzero mapcount".

2. The HugeTLB page mapped by the page table miss freeing since we
   treat the page table as shared and a shared page table will not be
   unmapped.

Fix it by introducing independent PMD page table shared count.  As
described by comment, pt_index/pt_mm/pt_frag_refcount are used for s390
gmap, x86 pgds and powerpc, pt_share_count is used for x86/arm64/riscv
pmds, so we can reuse the field as pt_share_count.

Link: https://lkml.kernel.org/r/20241216071147.3984217-1-liushixin2@huawei.com
Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page")
	Signed-off-by: Liu Shixin <liushixin2@huawei.com>
	Cc: Kefeng Wang <wangkefeng.wang@huawei.com>
	Cc: Ken Chen <kenneth.w.chen@intel.com>
	Cc: Muchun Song <muchun.song@linux.dev>
	Cc: Nanyong Sun <sunnanyong@huawei.com>
	Cc: Jane Chu <jane.chu@oracle.com>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 8410996eb6fea116fe1483ed977aacf580eee7b4)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572
cve CVE-2025-38084
commit-author Jann Horn <jannh@google.com>
commit 081056d
upstream-diff Used linux-5.10.y backport
  e8847d18cd9fff1edbb45e963d9141273c3b539c for the clean pick

Currently, __split_vma() triggers hugetlb page table unsharing through
vm_ops->may_split().  This happens before the VMA lock and rmap locks are
taken - which is too early, it allows racing VMA-locked page faults in our
process and racing rmap walks from other processes to cause page tables to
be shared again before we actually perform the split.

Fix it by explicitly calling into the hugetlb unshare logic from
__split_vma() in the same place where THP splitting also happens.  At that
point, both the VMA and the rmap(s) are write-locked.

An annoying detail is that we can now call into the helper
hugetlb_unshare_pmds() from two different locking contexts:

1. from hugetlb_split(), holding:
    - mmap lock (exclusively)
    - VMA lock
    - file rmap lock (exclusively)
2. hugetlb_unshare_all_pmds(), which I think is designed to be able to
   call us with only the mmap lock held (in shared mode), but currently
   only runs while holding mmap lock (exclusively) and VMA lock

Backporting note:
This commit fixes a racy protection that was introduced in commit
b30c14c ("hugetlb: unshare some PMDs when splitting VMAs"); that
commit claimed to fix an issue introduced in 5.13, but it should actually
also go all the way back.

[jannh@google.com: v2]
  Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-1-1329349bad1a@google.com
Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-0-1329349bad1a@google.com
Link: https://lkml.kernel.org/r/20250527-hugetlb-fixes-splitrace-v1-1-f4136f5ec58a@google.com
Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page")
	Signed-off-by: Jann Horn <jannh@google.com>
	Cc: Liam Howlett <liam.howlett@oracle.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Reviewed-by: Oscar Salvador <osalvador@suse.de>
	Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Cc: Vlastimil Babka <vbabka@suse.cz>
	Cc: <stable@vger.kernel.org>	[b30c14c: hugetlb: unshare some PMDs when splitting VMAs]
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit e8847d18cd9fff1edbb45e963d9141273c3b539c)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581
cve CVE-2025-38085
commit-author Jann Horn <jannh@google.com>
commit 1013af4
upstream-diff Used linux-5.10.y backport
  952596b08c74e8fe9e2883d1dc8a8f54a37384ec for the clean pick. Exposed
  the `tlb_remove_table_sync_one' function for use in the solution
  (missing 2ba99c5 ("mm/khugepaged: fix
  GUP-fast interaction by sending IPI")).

huge_pmd_unshare() drops a reference on a page table that may have
previously been shared across processes, potentially turning it into a
normal page table used in another process in which unrelated VMAs can
afterwards be installed.

If this happens in the middle of a concurrent gup_fast(), gup_fast() could
end up walking the page tables of another process.  While I don't see any
way in which that immediately leads to kernel memory corruption, it is
really weird and unexpected.

Fix it with an explicit broadcast IPI through tlb_remove_table_sync_one(),
just like we do in khugepaged when removing page tables for a THP
collapse.

Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-2-1329349bad1a@google.com
Link: https://lkml.kernel.org/r/20250527-hugetlb-fixes-splitrace-v1-2-f4136f5ec58a@google.com
Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page")
	Signed-off-by: Jann Horn <jannh@google.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Cc: Liam Howlett <liam.howlett@oracle.com>
	Cc: Muchun Song <muchun.song@linux.dev>
	Cc: Oscar Salvador <osalvador@suse.de>
	Cc: Vlastimil Babka <vbabka@suse.cz>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 952596b08c74e8fe9e2883d1dc8a8f54a37384ec)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572
cve-bf CVE-2025-38084
commit-author Jane Chu <jane.chu@oracle.com>
commit 59d9094 ("mm: hugetlb: independent PMD page table shared
commit 14967a9
upstream-diff Used linux-5.10.y backport
  ee67c4ac1aed403688dd359918099a745b60fb69 for the clean pick

count") introduced ->pt_share_count dedicated to hugetlb PMD share count
tracking, but omitted fixing copy_hugetlb_page_range(), leaving the
function relying on page_count() for tracking that no longer works.

When lazy page table copy for hugetlb is disabled, that is, revert commit
bcd51a3 ("hugetlb: lazy page table copies in fork()") fork()'ing with
hugetlb PMD sharing quickly lockup -

[  239.446559] watchdog: BUG: soft lockup - CPU#75 stuck for 27s!
[  239.446611] RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x2e0
[  239.446631] Call Trace:
[  239.446633]  <TASK>
[  239.446636]  _raw_spin_lock+0x3f/0x60
[  239.446639]  copy_hugetlb_page_range+0x258/0xb50
[  239.446645]  copy_page_range+0x22b/0x2c0
[  239.446651]  dup_mmap+0x3e2/0x770
[  239.446654]  dup_mm.constprop.0+0x5e/0x230
[  239.446657]  copy_process+0xd17/0x1760
[  239.446660]  kernel_clone+0xc0/0x3e0
[  239.446661]  __do_sys_clone+0x65/0xa0
[  239.446664]  do_syscall_64+0x82/0x930
[  239.446668]  ? count_memcg_events+0xd2/0x190
[  239.446671]  ? syscall_trace_enter+0x14e/0x1f0
[  239.446676]  ? syscall_exit_work+0x118/0x150
[  239.446677]  ? arch_exit_to_user_mode_prepare.constprop.0+0x9/0xb0
[  239.446681]  ? clear_bhb_loop+0x30/0x80
[  239.446684]  ? clear_bhb_loop+0x30/0x80
[  239.446686]  entry_SYSCALL_64_after_hwframe+0x76/0x7e

There are two options to resolve the potential latent issue:
  1. warn against PMD sharing in copy_hugetlb_page_range(),
  2. fix it.
This patch opts for the second option.
While at it, simplify the comment, the details are not actually relevant
anymore.

Link: https://lkml.kernel.org/r/20250916004520.1604530-1-jane.chu@oracle.com
Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count")
	Signed-off-by: Jane Chu <jane.chu@oracle.com>
	Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
	Acked-by: Oscar Salvador <osalvador@suse.de>
	Acked-by: David Hildenbrand <david@redhat.com>
	Cc: Jann Horn <jannh@google.com>
	Cc: Liu Shixin <liushixin2@huawei.com>
	Cc: Muchun Song <muchun.song@linux.dev>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit ee67c4ac1aed403688dd359918099a745b60fb69)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-193890
cve-bf CVE-2026-23100
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit ca1a47c
upstream-diff Used linux-5.10.y backport
  8ae48255bcb17b32436be97553dca848730d365f for the clean pick

Patch series "mm/hugetlb: fixes for PMD table sharing (incl.  using
mmu_gather)", v3.

One functional fix, one performance regression fix, and two related
comment fixes.

I cleaned up my prototype I recently shared [1] for the performance fix,
deferring most of the cleanups I had in the prototype to a later point.
While doing that I identified the other things.

The goal of this patch set is to be backported to stable trees "fairly"
easily. At least patch ctrliq#1 and ctrliq#4.

Patch ctrliq#1 fixes hugetlb_pmd_shared() not detecting any sharing
Patch ctrliq#2 + ctrliq#3 are simple comment fixes that patch ctrliq#4 interacts with.
Patch ctrliq#4 is a fix for the reported performance regression due to excessive
IPI broadcasts during fork()+exit().

The last patch is all about TLB flushes, IPIs and mmu_gather.
Read: complicated

There are plenty of cleanups in the future to be had + one reasonable
optimization on x86. But that's all out of scope for this series.

Runtime tested, with a focus on fixing the performance regression using
the original reproducer [2] on x86.

This patch (of 4):

We switched from (wrongly) using the page count to an independent shared
count.  Now, shared page tables have a refcount of 1 (excluding
speculative references) and instead use ptdesc->pt_share_count to identify
sharing.

We didn't convert hugetlb_pmd_shared(), so right now, we would never
detect a shared PMD table as such, because sharing/unsharing no longer
touches the refcount of a PMD table.

Page migration, like mbind() or migrate_pages() would allow for migrating
folios mapped into such shared PMD tables, even though the folios are not
exclusive.  In smaps we would account them as "private" although they are
"shared", and we would be wrongly setting the PM_MMAP_EXCLUSIVE in the
pagemap interface.

Fix it by properly using ptdesc_pmd_is_shared() in hugetlb_pmd_shared().

Link: https://lkml.kernel.org/r/20251223214037.580860-1-david@kernel.org
Link: https://lkml.kernel.org/r/20251223214037.580860-2-david@kernel.org
Link: https://lore.kernel.org/all/8cab934d-4a56-44aa-b641-bfd7e23bd673@kernel.org/ [1]
Link: https://lore.kernel.org/all/8cab934d-4a56-44aa-b641-bfd7e23bd673@kernel.org/ [2]
Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count")
	Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org>
	Reviewed-by: Rik van Riel <riel@surriel.com>
	Reviewed-by: Lance Yang <lance.yang@linux.dev>
	Tested-by: Lance Yang <lance.yang@linux.dev>
	Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
	Tested-by: Laurence Oberman <loberman@redhat.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Acked-by: Oscar Salvador <osalvador@suse.de>
	Cc: Liu Shixin <liushixin2@huawei.com>
	Cc: Uschakow, Stanislav" <suschako@amazon.de>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 8ae48255bcb17b32436be97553dca848730d365f)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924
cve-bf CVE-2024-57883
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit 3937027
upstream-diff Used linux-5.10.y backport
  44b18bac2b3063784ebb871a922803b11d9c339d for the clean pick

Ever since we stopped using the page count to detect shared PMD page
tables, these comments are outdated.

The only reason we have to flush the TLB early is because once we drop the
i_mmap_rwsem, the previously shared page table could get freed (to then
get reallocated and used for other purpose).  So we really have to flush
the TLB before that could happen.

So let's simplify the comments a bit.

The "If we unshared PMDs, the TLB flush was not recorded in mmu_gather."
part introduced as in commit a4a118f ("hugetlbfs: flush TLBs
correctly after huge_pmd_unshare") was confusing: sure it is recorded in
the mmu_gather, otherwise tlb_flush_mmu_tlbonly() wouldn't do anything.
So let's drop that comment while at it as well.

We'll centralize these comments in a single helper as we rework the code
next.

Link: https://lkml.kernel.org/r/20251223214037.580860-3-david@kernel.org
Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count")
	Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org>
	Reviewed-by: Rik van Riel <riel@surriel.com>
	Tested-by: Laurence Oberman <loberman@redhat.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Acked-by: Oscar Salvador <osalvador@suse.de>
	Reviewed-by: Harry Yoo <harry.yoo@oracle.com>
	Cc: Liu Shixin <liushixin2@huawei.com>
	Cc: Lance Yang <lance.yang@linux.dev>
	Cc: "Uschakow, Stanislav" <suschako@amazon.de>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 44b18bac2b3063784ebb871a922803b11d9c339d)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924
cve-bf CVE-2024-57883
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit a8682d5
upstream-diff Used linux-5.10.y backport
  2723af9e1e283cee312946c3d6cb20d2c731c139 for the clean pick

PMD page table unsharing no longer touches the refcount of a PMD page
table.  Also, it is not about dropping the refcount of a "PMD page" but
the "PMD page table".

Let's just simplify by saying that the PMD page table was unmapped,
consequently also unmapping the folio that was mapped into this page.

This code should be deduplicated in the future.

Link: https://lkml.kernel.org/r/20251223214037.580860-4-david@kernel.org
Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count")
	Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org>
	Reviewed-by: Rik van Riel <riel@surriel.com>
	Tested-by: Laurence Oberman <loberman@redhat.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Acked-by: Oscar Salvador <osalvador@suse.de>
	Cc: Liu Shixin <liushixin2@huawei.com>
	Cc: Harry Yoo <harry.yoo@oracle.com>
	Cc: Lance Yang <lance.yang@linux.dev>
	Cc: "Uschakow, Stanislav" <suschako@amazon.de>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 2723af9e1e283cee312946c3d6cb20d2c731c139)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
…ing mmu_gather

jira VULN-71581
cve-bf CVE-2025-38085
commit-author David Hildenbrand (Red Hat) <david@kernel.org>
commit 8ce720d
upstream-diff Used linux-5.10.y backport
  6495204f8219a57859669d618b25d06176d5a872 for an almost clean pick.
  Resolved conflicts in the `hugetlb_unshare_pmds' function: unlike
  ciqlts8_6 the linux-5.10.y version doesn't have
  4ddb4d9 ("hugetlb: do not update
  address in huge_pmd_unshare") backported. That commit changed the
  signature of `huge_pmd_unshare()', converting the third argument
  `unsigned long *addr' to `unsigned long addr'. This allowed for code
  reduction in `hugetlb_unshare_all_pmds()' by dropping the use of
  temporary `address' placeholder protecting that variable from being
  changed by the `huge_pmd_unshare()' call. From the backported commit
  perspective this is irrelevant though, because it's only concerned
  with the change of the first argument of `huge_pmd_unshare()'. Applied
  the first argument change while preserving `&address' used as the
  third.

As reported, ever since commit 1013af4 ("mm/hugetlb: fix
huge_pmd_unshare() vs GUP-fast race") we can end up in some situations
where we perform so many IPI broadcasts when unsharing hugetlb PMD page
tables that it severely regresses some workloads.

In particular, when we fork()+exit(), or when we munmap() a large
area backed by many shared PMD tables, we perform one IPI broadcast per
unshared PMD table.

There are two optimizations to be had:

(1) When we process (unshare) multiple such PMD tables, such as during
    exit(), it is sufficient to send a single IPI broadcast (as long as
    we respect locking rules) instead of one per PMD table.

    Locking prevents that any of these PMD tables could get reused before
    we drop the lock.

(2) When we are not the last sharer (> 2 users including us), there is
    no need to send the IPI broadcast. The shared PMD tables cannot
    become exclusive (fully unshared) before an IPI will be broadcasted
    by the last sharer.

    Concurrent GUP-fast could walk into a PMD table just before we
    unshared it. It could then succeed in grabbing a page from the
    shared page table even after munmap() etc succeeded (and supressed
    an IPI). But there is not difference compared to GUP-fast just
    sleeping for a while after grabbing the page and re-enabling IRQs.

    Most importantly, GUP-fast will never walk into page tables that are
    no-longer shared, because the last sharer will issue an IPI
    broadcast.

    (if ever required, checking whether the PUD changed in GUP-fast
     after grabbing the page like we do in the PTE case could handle
     this)

So let's rework PMD sharing TLB flushing + IPI sync to use the mmu_gather
infrastructure so we can implement these optimizations and demystify the
code at least a bit. Extend the mmu_gather infrastructure to be able to
deal with our special hugetlb PMD table sharing implementation.

To make initialization of the mmu_gather easier when working on a single
VMA (in particular, when dealing with hugetlb), provide
tlb_gather_mmu_vma().

We'll consolidate the handling for (full) unsharing of PMD tables in
tlb_unshare_pmd_ptdesc() and tlb_flush_unshared_tables(), and track
in "struct mmu_gather" whether we had (full) unsharing of PMD tables.

Because locking is very special (concurrent unsharing+reuse must be
prevented), we disallow deferring flushing to tlb_finish_mmu() and instead
require an explicit earlier call to tlb_flush_unshared_tables().

From hugetlb code, we call huge_pmd_unshare_flush() where we make sure
that the expected lock protecting us from concurrent unsharing+reuse is
still held.

Check with a VM_WARN_ON_ONCE() in tlb_finish_mmu() that
tlb_flush_unshared_tables() was properly called earlier.

Document it all properly.

Notes about tlb_remove_table_sync_one() interaction with unsharing:

There are two fairly tricky things:

(1) tlb_remove_table_sync_one() is a NOP on architectures without
    CONFIG_MMU_GATHER_RCU_TABLE_FREE.

    Here, the assumption is that the previous TLB flush would send an
    IPI to all relevant CPUs. Careful: some architectures like x86 only
    send IPIs to all relevant CPUs when tlb->freed_tables is set.

    The relevant architectures should be selecting
    MMU_GATHER_RCU_TABLE_FREE, but x86 might not do that in stable
    kernels and it might have been problematic before this patch.

    Also, the arch flushing behavior (independent of IPIs) is different
    when tlb->freed_tables is set. Do we have to enlighten them to also
    take care of tlb->unshared_tables? So far we didn't care, so
    hopefully we are fine. Of course, we could be setting
    tlb->freed_tables as well, but that might then unnecessarily flush
    too much, because the semantics of tlb->freed_tables are a bit
    fuzzy.

    This patch changes nothing in this regard.

(2) tlb_remove_table_sync_one() is not a NOP on architectures with
    CONFIG_MMU_GATHER_RCU_TABLE_FREE that actually don't need a sync.

    Take x86 as an example: in the common case (!pv, !X86_FEATURE_INVLPGB)
    we still issue IPIs during TLB flushes and don't actually need the
    second tlb_remove_table_sync_one().

    This optimized can be implemented on top of this, by checking e.g., in
    tlb_remove_table_sync_one() whether we really need IPIs. But as
    described in (1), it really must honor tlb->freed_tables then to
    send IPIs to all relevant CPUs.

Notes on TLB flushing changes:

(1) Flushing for non-shared PMD tables

    We're converting from flush_hugetlb_tlb_range() to
    tlb_remove_huge_tlb_entry(). Given that we properly initialize the
    MMU gather in tlb_gather_mmu_vma() to be hugetlb aware, similar to
    __unmap_hugepage_range(), that should be fine.

(2) Flushing for shared PMD tables

    We're converting from various things (flush_hugetlb_tlb_range(),
    tlb_flush_pmd_range(), flush_tlb_range()) to tlb_flush_pmd_range().

    tlb_flush_pmd_range() achieves the same that
    tlb_remove_huge_tlb_entry() would achieve in these scenarios.
    Note that tlb_remove_huge_tlb_entry() also calls
    __tlb_remove_tlb_entry(), however that is only implemented on
    powerpc, which does not support PMD table sharing.

    Similar to (1), tlb_gather_mmu_vma() should make sure that TLB
    flushing keeps on working as expected.

Further, note that the ptdesc_pmd_pts_dec() in huge_pmd_share() is not a
concern, as we are holding the i_mmap_lock the whole time, preventing
concurrent unsharing. That ptdesc_pmd_pts_dec() usage will be removed
separately as a cleanup later.

There are plenty more cleanups to be had, but they have to wait until
this is fixed.

[david@kernel.org: fix kerneldoc]
  Link: https://lkml.kernel.org/r/f223dd74-331c-412d-93fc-69e360a5006c@kernel.org
Link: https://lkml.kernel.org/r/20251223214037.580860-5-david@kernel.org
Fixes: 1013af4 ("mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race")
	Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org>
	Reported-by: Uschakow, Stanislav" <suschako@amazon.de>
Closes: https://lore.kernel.org/all/4d3878531c76479d9f8ca9789dc6485d@amazon.de/
	Tested-by: Laurence Oberman <loberman@redhat.com>
	Acked-by: Harry Yoo <harry.yoo@oracle.com>
	Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
	Cc: Lance Yang <lance.yang@linux.dev>
	Cc: Liu Shixin <liushixin2@huawei.com>
	Cc: Oscar Salvador <osalvador@suse.de>
	Cc: Rik van Riel <riel@surriel.com>
	Cc: <stable@vger.kernel.org>
	Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 6495204f8219a57859669d618b25d06176d5a872)
	Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
@github-actions

Copy link
Copy Markdown

🤖 Validation Checks In Progress Workflow run: https://github.com/ctrliq/kernel-src-tree/actions/runs/30552170473

@github-actions

Copy link
Copy Markdown

🔍 Upstream Linux Kernel Commit Check

  • ⚠️ PR commit f9ab2b5c060b (mm/hugetlb: unshare page tables during VMA split, not before) references upstream commit
    081056dc00a2 which has been referenced by a Fixes: tag in the upstream
    Linux kernel:
    b4aea43cd37a mm/hugetlb: avoid false positive lockdep assertion (Lorenzo Stoakes)
  • ⚠️ PR commit 9cb602e171c1 (mm/hugetlb: fix hugetlb_pmd_shared()) does not reference a CVE but
    upstream commit ca1a47cd3f5f is associated with CVE-2026-23100

  • ⚠️ PR commit 0e0c9bcf167a (mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather) references upstream commit
    8ce720d5bd91 which has been referenced by a Fixes: tag in the upstream
    Linux kernel:

    c2ff4764e03e arm64: tlb: Flush walk cache when unsharing PMD tables (Zeng Heng) (CVE-2026-63875)

This is an automated message from the kernel commit checker workflow.

@github-actions

Copy link
Copy Markdown

🔍 Interdiff Analysis

  • ⚠️ PR commit 76cb54d2095b (asm-generic/tlb: rename HAVE_RCU_TABLE_FREE) → upstream ff2e6d7259f8
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -203,8 +203,8 @@
 	select HAVE_HARDLOCKUP_DETECTOR_PERF	if PERF_EVENTS && HAVE_PERF_EVENTS_NMI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
-	select MMU_GATHER_RCU_TABLE_FREE		if PARAVIRT
-	select HAVE_RCU_TABLE_INVALIDATE	if MMU_GATHER_RCU_TABLE_FREE
+	select HAVE_RCU_TABLE_FREE		if PARAVIRT
+	select HAVE_RCU_TABLE_INVALIDATE	if HAVE_RCU_TABLE_FREE
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE		if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
 	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -165,7 +165,7 @@
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_FUTEX_CMPXCHG if FUTEX
-	select HAVE_RCU_TABLE_FREE
+	select MMU_GATHER_RCU_TABLE_FREE
 	select HAVE_RSEQ
 	select HAVE_STACKPROTECTOR
 	select HAVE_SYSCALL_TRACEPOINTS
--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -203,7 +203,7 @@
 	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
-	select HAVE_RCU_TABLE_FREE		if PARAVIRT
+	select MMU_GATHER_RCU_TABLE_FREE		if PARAVIRT
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE		if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
 	select HAVE_FUNCTION_ARG_ACCESS_API

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -149,4 +149,6 @@
 	select HAVE_REGS_AND_STACK_ACCESS_API
+	select HAVE_FUNCTION_ARG_ACCESS_API
+	select HAVE_FUTEX_CMPXCHG if FUTEX
 	select HAVE_RCU_TABLE_FREE
 	select HAVE_RSEQ
 	select HAVE_STACKPROTECTOR
--- b/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -159,5 +159,7 @@
 	select HAVE_OPROFILE
+	select HAVE_PCI
 	select HAVE_PERF_EVENTS
 	select HAVE_RCU_TABLE_FREE
 	select HAVE_REGS_AND_STACK_ACCESS_API
+	select HAVE_RELIABLE_STACKTRACE
 	select HAVE_RSEQ
--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -200,8 +200,7 @@
-	select HAVE_HARDLOCKUP_DETECTOR_PERF	if PERF_EVENTS && HAVE_PERF_EVENTS_NMI
+	select HAVE_PCI
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
 	select HAVE_RCU_TABLE_FREE		if PARAVIRT
-	select HAVE_RCU_TABLE_INVALIDATE	if HAVE_RCU_TABLE_FREE
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE		if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
-	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
+	select HAVE_FUNCTION_ARG_ACCESS_API
  • ⚠️ PR commit 98c3edd9f59e (asm-generic/tlb: provide MMU_GATHER_TABLE_FREE) → upstream 0d6e24d430ef
    Differences found:
################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -238,7 +297,6 @@
 {
 #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
 	tlb_table_flush(tlb);
-#endif
 #ifndef CONFIG_MMU_GATHER_NO_GATHER
 	tlb_batch_pages_flush(tlb);
 #endif
@@ -279,9 +337,7 @@
 	tlb->batch_count = 0;
 #endif
 
-#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
-	tlb->batch = NULL;
-#endif
+	tlb_table_init(tlb);
 #ifdef CONFIG_MMU_GATHER_PAGE_SIZE
 	tlb->page_size = 0;
 #endif

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/arch/Kconfig
+++ b/arch/Kconfig
@@ -375,3 +375,3 @@
 
-config HAVE_MMU_GATHER_PAGE_SIZE
+config MMU_GATHER_PAGE_SIZE
 	bool
@@ -378,6 +378,6 @@
 
-config HAVE_MMU_GATHER_NO_GATHER
+config MMU_GATHER_NO_GATHER
 	bool
 
-config ARCH_WANT_IRQS_OFF_ACTIVATE_MM
+config ARCH_HAVE_NMI_SAFE_CMPXCHG
 	bool
--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -127,5 +123,5 @@
- *  This ensures we call tlb_flush() every time tlb_change_page_size() actually
- *  changes the size and provides mmu_gather::page_size to tlb_flush().
+ *  This might be useful if your architecture has size specific TLB
+ *  invalidation instructions.
  *
  *  MMU_GATHER_RCU_TABLE_FREE
  *
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -89,5 +89,5 @@
 
-#endif /* HAVE_MMU_GATHER_NO_GATHER */
+#endif /* MMU_GATHER_NO_GATHER */
 
 #ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
 
@@ -181,5 +181,5 @@
 	tlb_table_flush(tlb);
 #endif
-#ifndef CONFIG_HAVE_MMU_GATHER_NO_GATHER
+#ifndef CONFIG_MMU_GATHER_NO_GATHER
 	tlb_batch_pages_flush(tlb);
 #endif
@@ -224,5 +224,5 @@
 	tlb->batch = NULL;
 #endif
-#ifdef CONFIG_HAVE_MMU_GATHER_PAGE_SIZE
+#ifdef CONFIG_MMU_GATHER_PAGE_SIZE
 	tlb->page_size = 0;
 #endif
  • ⚠️ PR commit 8f7d45b6f15e (mm: proc: Invalidate TLB after clearing soft-dirty page state) → upstream 912efa17e512
    Differences found:
################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1248,7 +1247,6 @@
 			goto out_unlock;
 		}
 
-		tlb_gather_mmu(&tlb, mm, 0, -1);
 		if (type == CLEAR_REFS_SOFT_DIRTY) {
 			for (vma = mm->mmap; vma; vma = vma->vm_next) {
 				if (!(vma->vm_flags & VM_SOFTDIRTY))
@@ -1263,7 +1262,9 @@
 				&cp);
 		if (type == CLEAR_REFS_SOFT_DIRTY)
 			mmu_notifier_invalidate_range_end(&range);
-		tlb_finish_mmu(&tlb, 0, -1);
+			flush_tlb_mm(mm);
+			dec_tlb_flush_pending(mm);
+		}
 out_unlock:
 		mmap_write_unlock(mm);
 out_mm:

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1198,3 +1269,4 @@
 		tlb_finish_mmu(&tlb, 0, -1);
-		mmap_read_unlock(mm);
+out_unlock:
+		mmap_write_unlock(mm);
 out_mm:
  • ⚠️ PR commit c48c7b99c598 (mm/hugetlb: make detecting shared pte more reliable) → upstream 3aa4ed8040e1
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3834,7 +3834,8 @@
 		 * another vma. So page_count of ptep page is checked instead
 		 * to reliably determine whether pte is shared.
 		 */
-		if (page_count(virt_to_page(dst_pte)) > 1)
+		dst_entry = huge_ptep_get(dst_pte);
+		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
 			continue;
 
 		dst_ptl = huge_pte_lock(h, dst, dst_pte);

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4801,8 +4800,7 @@
 		 * Check here before taking page table lock, and once again
 		 * after taking the lock below.
 		 */
-		dst_entry = huge_ptep_get(dst_pte);
-		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) {
+		if (page_count(virt_to_page(dst_pte)) > 1) {
 			addr |= last_addr_mask;
 			continue;
 		}

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3784,5 +3786,5 @@
 {
 	pte_t *src_pte, *dst_pte, entry, dst_entry;
 	struct page *ptepage;
 	unsigned long addr;
-	int cow;
+	bool cow = is_cow_mapping(src_vma->vm_flags);
@@ -3836,10 +4803,10 @@
 		 * after taking the lock below.
 		 */
 		dst_entry = huge_ptep_get(dst_pte);
-		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
+		if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) {
+			addr |= last_addr_mask;
 			continue;
-
-		dst_ptl = huge_pte_lock(h, dst, dst_pte);
+		}
 		src_ptl = huge_pte_lockptr(h, src, src_pte);
 		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
 		entry = huge_ptep_get(src_pte);
@@ -3852,4 +4819,4 @@
 			 * sharing with another vma.
 			 */
 			;
-		} else if (unlikely(is_hugetlb_entry_migration(entry) ||
+		} else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) {
@@ -3902,7 +4894,7 @@
-				entry = huge_ptep_get(src_pte);
-				if (!pte_same(src_pte_old, entry)) {
+					restore_reserve_on_error(h, dst_vma, addr,
+								new);
 					put_page(new);
 					/* dst_entry won't change as in child */
 					goto again;
 				}
-				hugetlb_install_page(vma, dst_pte, addr, new);
+				hugetlb_install_page(dst_vma, dst_pte, addr, new);
  • ⚠️ PR commit 305f3092087c (hugetlb: unshare some PMDs when splitting VMAs) → upstream b30c14cd6102
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5754,47 +5752,3 @@
 		spin_unlock(&hugetlb_lock);
 	}
 }
-
-static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
-				   unsigned long start,
-				   unsigned long end)
-{
-	struct hstate *h = hstate_vma(vma);
-	unsigned long sz = huge_page_size(h);
-	struct mm_struct *mm = vma->vm_mm;
-	struct mmu_notifier_range range;
-	unsigned long address;
-	spinlock_t *ptl;
-	pte_t *ptep;
-
-	if (!(vma->vm_flags & VM_MAYSHARE))
-		return;
-
-	if (start >= end)
-		return;
-
-	flush_cache_range(vma, start, end);
-	/*
-	 * No need to call adjust_range_if_pmd_sharing_possible(), because
-	 * we have already done the PUD_SIZE alignment.
-	 */
-	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
-				start, end);
-	mmu_notifier_invalidate_range_start(&range);
-	i_mmap_lock_write(vma->vm_file->f_mapping);
-	for (address = start; address < end; address += PUD_SIZE) {
-		ptep = huge_pte_offset(mm, address, sz);
-		if (!ptep)
-			continue;
-		ptl = huge_pte_lock(h, mm, ptep);
-		huge_pmd_unshare(mm, vma, &address, ptep);
-		spin_unlock(ptl);
-	}
-	flush_hugetlb_tlb_range(vma, start, end);
-	i_mmap_unlock_write(vma->vm_file->f_mapping);
-	/*
-	 * No need to call mmu_notifier_invalidate_range(), see
-	 * Documentation/mm/mmu_notifier.rst.
-	 */
-	mmu_notifier_invalidate_range_end(&range);
-}

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -94,6 +94,8 @@
 static void hugetlb_vma_lock_free(struct vm_area_struct *vma);
 static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
 static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma);
+static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
+		unsigned long start, unsigned long end);
 
 static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
@@ -7341,11 +7362,9 @@
 	}
 }
 
-/*
- * This function will unconditionally remove all the shared pmd pgtable entries
- * within the specific vma for a hugetlbfs memory range.
- */
-void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
+static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
+				   unsigned long start,
+				   unsigned long end)
 {
 	struct hstate *h = hstate_vma(vma);
 	unsigned long sz = huge_page_size(h);
@@ -7349,7 +7368,7 @@
 	unsigned long sz = huge_page_size(h);
 	struct mm_struct *mm = vma->vm_mm;
 	struct mmu_notifier_range range;
-	unsigned long address, start, end;
+	unsigned long address;
 	spinlock_t *ptl;
 	pte_t *ptep;
 
@@ -7353,9 +7372,6 @@
 	if (!(vma->vm_flags & VM_MAYSHARE))
 		return;
 
-	start = ALIGN(vma->vm_start, PUD_SIZE);
-	end = ALIGN_DOWN(vma->vm_end, PUD_SIZE);
-
 	if (start >= end)
 		return;
 
@@ -7392,6 +7408,16 @@
 	mmu_notifier_invalidate_range_end(&range);
 }
 
+/*
+ * This function will unconditionally remove all the shared pmd pgtable entries
+ * within the specific vma for a hugetlbfs memory range.
+ */
+void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
+{
+	hugetlb_unshare_pmds(vma, ALIGN(vma->vm_start, PUD_SIZE),
+			ALIGN_DOWN(vma->vm_end, PUD_SIZE));
+}
+
 #ifdef CONFIG_CMA
 static bool cma_reserve_called __initdata;
 

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -70,6 +73,3 @@
 
-/* Forward declaration */
-static int hugetlb_acct_memory(struct hstate *h, long delta);
-
-static inline void unlock_or_release_subpool(struct hugepage_subpool *spool)
+static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
  • ⚠️ PR commit 900585336f8a (mm: hugetlb: proc: check for hugetlb shared PMD in /proc/PID/smaps) → upstream 3489dbb696d2
    Differences found:
################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -1188,6 +1189,18 @@
 }
 #endif
 
+#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
+static inline bool hugetlb_pmd_shared(pte_t *pte)
+{
+	return page_count(virt_to_page(pte)) > 1;
+}
+#else
+static inline bool hugetlb_pmd_shared(pte_t *pte)
+{
+	return false;
+}
+#endif
+
 bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
 
 #ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -9,4 +9,4 @@
 #include <linux/cgroup.h>
 #include <linux/list.h>
 #include <linux/kref.h>
-#include <asm/pgtable.h>
+#include <linux/pgtable.h>
@@ -1004,3 +1184,6 @@
 }
+#endif
+
+bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
 
-#endif /* _LINUX_HUGETLB_H */
+#ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
  • ⚠️ PR commit 7fa9f009bd64 (mm: hugetlb: independent PMD page table shared count) → upstream 59d9094df3d7
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2157,9 +2157,6 @@
 	if (!pmd_ptlock_init(page))
 		return false;
 	__SetPageTable(page);
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
-	atomic_set(&page->pt_share_count, 0);
-#endif
 	inc_lruvec_page_state(page, NR_PAGETABLE);
 	return true;
 }
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -161,9 +161,6 @@
 					union {
 						struct mm_struct *pt_mm; /* x86 pgds only */
 						atomic_t pt_frag_refcount; /* powerpc */
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
-						atomic_t pt_share_count;
-#endif
 			})
 #if ALLOC_SPLIT_PTLOCKS
 			spinlock_t *ptl;
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5457,7 +5457,7 @@
 			spte = huge_pte_offset(svma->vm_mm, saddr,
 					       vma_mmu_pagesize(svma));
 			if (spte) {
-				atomic_inc(&virt_to_page(spte)->pt_share_count);
+				ptdesc_pmd_pts_inc(virt_to_ptdesc(spte));
 				break;
 			}
 		}
@@ -5472,7 +5472,7 @@
 				(pmd_t *)((unsigned long)spte & PAGE_MASK));
 		mm_inc_nr_pmds(mm);
 	} else {
-		atomic_dec(&virt_to_page(spte)->pt_share_count);
+		ptdesc_pmd_pts_dec(virt_to_ptdesc(spte));
 	}
 	spin_unlock(ptl);
 out:
@@ -5483,7 +5483,11 @@
 /*
  * unmap huge page backed by shared pte.
  *
- * Called with page table lock held.
+ * Hugetlb pte page is ref counted at the time of mapping.  If pte is shared
+ * indicated by page_count > 1, unmap is achieved by clearing pud and
+ * decrementing the ref count. If count == 1, the pte page is not shared.
+ *
+ * Called with page table lock held and i_mmap_rwsem held in write mode.
  *
  * returns: 1 successfully unmapped a shared pte page
  *	    0 the underlying pte page is not shared, or it is the last user
@@ -5491,19 +5495,17 @@
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 					unsigned long *addr, pte_t *ptep)
 {
-	unsigned long sz = huge_page_size(hstate_vma(vma));
 	pgd_t *pgd = pgd_offset(mm, *addr);
 	p4d_t *p4d = p4d_offset(pgd, *addr);
 	pud_t *pud = pud_offset(p4d, *addr);
 
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
-	if (sz != PMD_SIZE)
-		return 0;
-	if (!atomic_read(&virt_to_page(ptep)->pt_share_count))
+	BUG_ON(page_count(virt_to_page(ptep)) == 0);
+	if (page_count(virt_to_page(ptep)) == 1)
 		return 0;
 
 	pud_clear(pud);
-	atomic_dec(&virt_to_page(ptep)->pt_share_count);
+	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
 	mm_dec_nr_pmds(mm);
 	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3125,6 +3125,7 @@
 	if (!pmd_ptlock_init(ptdesc))
 		return false;
 	__folio_set_pgtable(folio);
+	ptdesc_pmd_pts_init(ptdesc);
 	lruvec_stat_add_folio(folio, NR_PAGETABLE);
 	return true;
 }
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -445,6 +445,7 @@
  * @pt_index:         Used for s390 gmap.
  * @pt_mm:            Used for x86 pgds.
  * @pt_frag_refcount: For fragmented page table tracking. Powerpc only.
+ * @pt_share_count:   Used for HugeTLB PMD page table share count.
  * @_pt_pad_2:        Padding to ensure proper alignment.
  * @ptl:              Lock for the page table.
  * @__page_type:      Same as page->page_type. Unused for page tables.
@@ -471,6 +472,9 @@
 		pgoff_t pt_index;
 		struct mm_struct *pt_mm;
 		atomic_t pt_frag_refcount;
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+		atomic_t pt_share_count;
+#endif
 	};
 
 	union {
@@ -516,6 +520,32 @@
 	const struct page *:		(const struct ptdesc *)(p),	\
 	struct page *:			(struct ptdesc *)(p)))
 
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+	atomic_set(&ptdesc->pt_share_count, 0);
+}
+
+static inline void ptdesc_pmd_pts_inc(struct ptdesc *ptdesc)
+{
+	atomic_inc(&ptdesc->pt_share_count);
+}
+
+static inline void ptdesc_pmd_pts_dec(struct ptdesc *ptdesc)
+{
+	atomic_dec(&ptdesc->pt_share_count);
+}
+
+static inline int ptdesc_pmd_pts_count(struct ptdesc *ptdesc)
+{
+	return atomic_read(&ptdesc->pt_share_count);
+}
+#else
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+}
+#endif
+
 /*
  * Used for sizing the vmemmap region on some architectures
  */
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7238,10 +7238,6 @@
 /*
  * unmap huge page backed by shared pte.
  *
- * Hugetlb pte page is ref counted at the time of mapping.  If pte is shared
- * indicated by page_count > 1, unmap is achieved by clearing pud and
- * decrementing the ref count. If count == 1, the pte page is not shared.
- *
  * Called with page table lock held.
  *
  * returns: 1 successfully unmapped a shared pte page
@@ -7250,6 +7246,7 @@
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 					unsigned long addr, pte_t *ptep)
 {
+	unsigned long sz = huge_page_size(hstate_vma(vma));
 	pgd_t *pgd = pgd_offset(mm, addr);
 	p4d_t *p4d = p4d_offset(pgd, addr);
 	pud_t *pud = pud_offset(p4d, addr);
@@ -7253,8 +7250,9 @@
 
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
 	hugetlb_vma_assert_locked(vma);
-	BUG_ON(page_count(virt_to_page(ptep)) == 0);
-	if (page_count(virt_to_page(ptep)) == 1)
+	if (sz != PMD_SIZE)
+		return 0;
+	if (!ptdesc_pmd_pts_count(virt_to_ptdesc(ptep)))
 		return 0;
 
 	pud_clear(pud);

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2154,5 +2155,5 @@
 		return false;
-	__SetPageTable(page);
-	inc_lruvec_page_state(page, NR_PAGETABLE);
+	__folio_set_pgtable(folio);
+	lruvec_stat_add_folio(folio, NR_PAGETABLE);
 	return true;
 }
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5476,3 +7230,3 @@
 	}
-	spin_unlock(ptl);
+	spin_unlock(&mm->page_table_lock);
 out:
@@ -5488,5 +7238,5 @@
  * decrementing the ref count. If count == 1, the pte page is not shared.
  *
- * Called with page table lock held and i_mmap_rwsem held in write mode.
+ * Called with page table lock held.
  *
  * returns: 1 successfully unmapped a shared pte page
@@ -5494,11 +7243,12 @@
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-					unsigned long *addr, pte_t *ptep)
+					unsigned long addr, pte_t *ptep)
 {
-	pgd_t *pgd = pgd_offset(mm, *addr);
-	p4d_t *p4d = p4d_offset(pgd, *addr);
-	pud_t *pud = pud_offset(p4d, *addr);
+	pgd_t *pgd = pgd_offset(mm, addr);
+	p4d_t *p4d = p4d_offset(pgd, addr);
+	pud_t *pud = pud_offset(p4d, addr);
 
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+	hugetlb_vma_assert_locked(vma);
 	BUG_ON(page_count(virt_to_page(ptep)) == 0);
 	if (page_count(virt_to_page(ptep)) == 1)
 		return 0;
@@ -5509,3 +7258,2 @@
 	mm_dec_nr_pmds(mm);
-	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;
  • ⚠️ PR commit f9ab2b5c060b (mm/hugetlb: unshare page tables during VMA split, not before) → upstream 081056dc00a2
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -189,8 +189,6 @@
 		unsigned long address, unsigned long end, pgprot_t newprot);
 
 bool is_hugetlb_entry_migration(pte_t pte);
-void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
-void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
 
 #else /* !CONFIG_HUGETLB_PAGE */
 
@@ -372,8 +370,6 @@
 	return 0;
 }
 
-static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }
-
 static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {}
 
 #endif /* !CONFIG_HUGETLB_PAGE */
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3695,7 +3695,7 @@
 	 * MM, VMA and rmap all write-locked to prevent concurrent page table
 	 * walks (except hardware and gup_fast()).
 	 */
-	mmap_assert_write_locked(vma->vm_mm);
+	vma_assert_write_locked(vma);
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
 
 	if (addr & ~PUD_MASK) {
@@ -5800,11 +5800,7 @@
 	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
 				start, end);
 	mmu_notifier_invalidate_range_start(&range);
-	if (take_locks) {
-		i_mmap_lock_write(vma->vm_file->f_mapping);
-	} else {
-		i_mmap_assert_write_locked(vma->vm_file->f_mapping);
-	}
+	i_mmap_lock_write(vma->vm_file->f_mapping);
 	for (address = start; address < end; address += PUD_SIZE) {
 		ptep = huge_pte_offset(mm, address, sz);
 		if (!ptep)
@@ -5814,9 +5810,7 @@
 		spin_unlock(ptl);
 	}
 	flush_hugetlb_tlb_range(vma, start, end);
-	if (take_locks) {
-		i_mmap_unlock_write(vma->vm_file->f_mapping);
-	}
+	i_mmap_unlock_write(vma->vm_file->f_mapping);
 	/*
 	 * No need to call mmu_notifier_invalidate_range(), see
 	 * Documentation/mm/mmu_notifier.rst.

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -279,6 +279,7 @@
 bool is_hugetlb_entry_hwpoisoned(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
 void fixup_hugetlb_reservations(struct vm_area_struct *vma);
+void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
 
 #else /* !CONFIG_HUGETLB_PAGE */
 
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -121,7 +121,7 @@
 static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
 static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma);
 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
-		unsigned long start, unsigned long end);
+		unsigned long start, unsigned long end, bool take_locks);
 static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
 
 static void hugetlb_free_folio(struct folio *folio)
@@ -7932,8 +7953,12 @@
 	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
 				start, end);
 	mmu_notifier_invalidate_range_start(&range);
-	hugetlb_vma_lock_write(vma);
-	i_mmap_lock_write(vma->vm_file->f_mapping);
+	if (take_locks) {
+		hugetlb_vma_lock_write(vma);
+		i_mmap_lock_write(vma->vm_file->f_mapping);
+	} else {
+		i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+	}
 	for (address = start; address < end; address += PUD_SIZE) {
 		ptep = hugetlb_walk(vma, address, sz);
 		if (!ptep)
@@ -7943,8 +7968,10 @@
 		spin_unlock(ptl);
 	}
 	flush_hugetlb_tlb_range(vma, start, end);
-	i_mmap_unlock_write(vma->vm_file->f_mapping);
-	hugetlb_vma_unlock_write(vma);
+	if (take_locks) {
+		i_mmap_unlock_write(vma->vm_file->f_mapping);
+		hugetlb_vma_unlock_write(vma);
+	}
 	/*
 	 * No need to call mmu_notifier_arch_invalidate_secondary_tlbs(), see
 	 * Documentation/mm/mmu_notifier.rst.
@@ -7959,7 +7986,8 @@
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
 {
 	hugetlb_unshare_pmds(vma, ALIGN(vma->vm_start, PUD_SIZE),
-			ALIGN_DOWN(vma->vm_end, PUD_SIZE));
+			ALIGN_DOWN(vma->vm_end, PUD_SIZE),
+			/* take_locks = */ true);
 }
 
 /*

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -367,5 +458,5 @@
 }
 
 #endif /* !CONFIG_HUGETLB_PAGE */
-/*
- * hugepages at page global directory. If arch support
+
+#ifndef pgd_write
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -71,5 +73,5 @@
 static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
 		unsigned long start, unsigned long end);
+static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
 
-static inline void unlock_or_release_subpool(struct hugepage_subpool *spool)
-{
+static void hugetlb_free_folio(struct folio *folio)
@@ -5776,7 +7908,8 @@
-	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
 				start, end);
 	mmu_notifier_invalidate_range_start(&range);
+	hugetlb_vma_lock_write(vma);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
 	for (address = start; address < end; address += PUD_SIZE) {
-		ptep = huge_pte_offset(mm, address, sz);
+		ptep = hugetlb_walk(vma, address, sz);
 		if (!ptep)
@@ -5786,4 +7919,5 @@
 	i_mmap_unlock_write(vma->vm_file->f_mapping);
+	hugetlb_vma_unlock_write(vma);
 	/*
-	 * No need to call mmu_notifier_invalidate_range(), see
+	 * No need to call mmu_notifier_arch_invalidate_secondary_tlbs(), see
 	 * Documentation/mm/mmu_notifier.rst.

================================================================================
*    ONLY IN PATCH1 - files not modified by patch2                             *
================================================================================

--- b/mm/mmap.c
+++ a/mm/mmap.c
@@ -812,15 +812,7 @@
 		}
 	}
 again:
-	/*
-	 * Get rid of huge pages and shared page tables straddling the split
-	 * boundary.
-	 */
 	vma_adjust_trans_huge(orig_vma, start, end, adjust_next);
-	if (is_vm_hugetlb_page(orig_vma)) {
-		hugetlb_split(orig_vma, start);
-		hugetlb_split(orig_vma, end);
-	}
 
 	if (file) {
 		mapping = file->f_mapping;

================================================================================
*    ONLY IN PATCH2 - files not modified by patch1                             *
================================================================================

--- a/mm/vma.c
+++ b/mm/vma.c
@@ -539,7 +539,14 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
 	init_vma_prep(&vp, vma);
 	vp.insert = new;
 	vma_prepare(&vp);
+
+	/*
+	 * Get rid of huge pages and shared page tables straddling the split
+	 * boundary.
+	 */
 	vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL);
+	if (is_vm_hugetlb_page(vma))
+		hugetlb_split(vma, addr);
 
 	if (new_below) {
 		vma->vm_start = addr;
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -932,6 +932,8 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
 	(void)next;
 }
 
+static inline void hugetlb_split(struct vm_area_struct *, unsigned long) {}
+
 static inline void vma_iter_free(struct vma_iterator *vmi)
 {
 	mas_destroy(&vmi->mas);
  • ⚠️ PR commit 116b86e2d9a8 (mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race) → upstream 1013af4f585f
    Differences found:
################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7629,6 +7629,13 @@
 		return 0;
 
 	pud_clear(pud);
+	/*
+	 * Once our caller drops the rmap lock, some other process might be
+	 * using this page table as a normal, non-hugetlb page table.
+	 * Wait for pending gup_fast() in other threads to finish before letting
+	 * that happen.
+	 */
+	tlb_remove_table_sync_one();
 	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
 	mm_dec_nr_pmds(mm);
 	return 1;

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5514,6 +5514,6 @@
 		return 0;
 
 	pud_clear(pud);
-	atomic_dec(&virt_to_page(ptep)->pt_share_count);
+	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
 	mm_dec_nr_pmds(mm);
-	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
+	return 1;

================================================================================
*    ONLY IN PATCH1 - files not modified by patch2                             *
================================================================================

--- b/include/asm-generic/tlb.h
+++ a/include/asm-generic/tlb.h
@@ -202,16 +202,12 @@
 #define tlb_needs_table_invalidate() (true)
 #endif
 
-void tlb_remove_table_sync_one(void);
-
 #else
 
 #ifdef tlb_needs_table_invalidate
 #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE
 #endif
 
-static inline void tlb_remove_table_sync_one(void) { }
-
 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
 
--- b/mm/mmu_gather.c
+++ a/mm/mmu_gather.c
@@ -139,7 +139,7 @@
 	/* Simply deliver the interrupt */
 }
 
+static void tlb_remove_table_sync_one(void)
-void tlb_remove_table_sync_one(void)
 {
 	/*
 	 * This isn't an RCU grace period and hence the page-tables cannot be
@@ -163,6 +163,8 @@
 
 #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
 
+static void tlb_remove_table_sync_one(void) { }
+
 static void tlb_remove_table_free(struct mmu_table_batch *batch)
 {
 	__tlb_remove_table_free(batch);
  • ⚠️ PR commit 27fe17e8751e (mm/hugetlb: fix copy_hugetlb_page_range() to use ->pt_share_count) → upstream 59d9094df3d7
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3861,11 +3861,16 @@
 			break;
 		}
 
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
-		/* If the pagetables are shared, there is nothing to do */
-		if (atomic_read(&virt_to_page(dst_pte)->pt_share_count))
+		/*
+		 * If the pagetables are shared don't copy or take references.
+		 *
+		 * dst_pte == src_pte is the common case of src/dest sharing.
+		 * However, src could have 'unshared' and dst shares with
+		 * another vma. So page_count of ptep page is checked instead
+		 * to reliably determine whether pte is shared.
+		 */
+		if (page_count(virt_to_page(dst_pte)) > 1)
 			continue;
-#endif
 
 		dst_ptl = huge_pte_lock(h, dst, dst_pte);
 		src_ptl = huge_pte_lockptr(h, src, src_pte);

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7211,7 +7211,7 @@
 			spte = hugetlb_walk(svma, saddr,
 					    vma_mmu_pagesize(svma));
 			if (spte) {
-				get_page(virt_to_page(spte));
+				ptdesc_pmd_pts_inc(virt_to_ptdesc(spte));
 				break;
 			}
 		}
@@ -7226,7 +7226,7 @@
 				(pmd_t *)((unsigned long)spte & PAGE_MASK));
 		mm_inc_nr_pmds(mm);
 	} else {
-		put_page(virt_to_page(spte));
+		ptdesc_pmd_pts_dec(virt_to_ptdesc(spte));
 	}
 	spin_unlock(&mm->page_table_lock);
 out:
@@ -7238,10 +7238,6 @@
 /*
  * unmap huge page backed by shared pte.
  *
- * Hugetlb pte page is ref counted at the time of mapping.  If pte is shared
- * indicated by page_count > 1, unmap is achieved by clearing pud and
- * decrementing the ref count. If count == 1, the pte page is not shared.
- *
  * Called with page table lock held.
  *
  * returns: 1 successfully unmapped a shared pte page
@@ -7250,6 +7246,7 @@
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 					unsigned long addr, pte_t *ptep)
 {
+	unsigned long sz = huge_page_size(hstate_vma(vma));
 	pgd_t *pgd = pgd_offset(mm, addr);
 	p4d_t *p4d = p4d_offset(pgd, addr);
 	pud_t *pud = pud_offset(p4d, addr);
@@ -7253,8 +7250,9 @@
 
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
 	hugetlb_vma_assert_locked(vma);
-	BUG_ON(page_count(virt_to_page(ptep)) == 0);
-	if (page_count(virt_to_page(ptep)) == 1)
+	if (sz != PMD_SIZE)
+		return 0;
+	if (!ptdesc_pmd_pts_count(virt_to_ptdesc(ptep)))
 		return 0;
 
 	pud_clear(pud);
@@ -7258,7 +7256,7 @@
 		return 0;
 
 	pud_clear(pud);
-	put_page(virt_to_page(ptep));
+	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
 	mm_dec_nr_pmds(mm);
 	return 1;
 }

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3859,14 +3888,12 @@
 
-		/*
-		 * If the pagetables are shared don't copy or take references.
-		 *
-		 * dst_pte == src_pte is the common case of src/dest sharing.
-		 * However, src could have 'unshared' and dst shares with
-		 * another vma. So page_count of ptep page is checked instead
-		 * to reliably determine whether pte is shared.
-		 */
-		if (page_count(virt_to_page(dst_pte)) > 1)
-			continue;
+	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+	hugetlb_vma_assert_locked(vma);
+	BUG_ON(page_count(virt_to_page(ptep)) == 0);
+	if (page_count(virt_to_page(ptep)) == 1)
+		return 0;
 
-		dst_ptl = huge_pte_lock(h, dst, dst_pte);
-		src_ptl = huge_pte_lockptr(h, src, src_pte);
+	pud_clear(pud);
+	put_page(virt_to_page(ptep));
+	mm_dec_nr_pmds(mm);
+	return 1;
+}

================================================================================
*    ONLY IN PATCH2 - files not modified by patch1                             *
================================================================================

--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3125,6 +3125,7 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc)
 	if (!pmd_ptlock_init(ptdesc))
 		return false;
 	__folio_set_pgtable(folio);
+	ptdesc_pmd_pts_init(ptdesc);
 	lruvec_stat_add_folio(folio, NR_PAGETABLE);
 	return true;
 }
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -445,6 +445,7 @@ FOLIO_MATCH(compound_head, _head_2a);
  * @pt_index:         Used for s390 gmap.
  * @pt_mm:            Used for x86 pgds.
  * @pt_frag_refcount: For fragmented page table tracking. Powerpc only.
+ * @pt_share_count:   Used for HugeTLB PMD page table share count.
  * @_pt_pad_2:        Padding to ensure proper alignment.
  * @ptl:              Lock for the page table.
  * @__page_type:      Same as page->page_type. Unused for page tables.
@@ -471,6 +472,9 @@ struct ptdesc {
 		pgoff_t pt_index;
 		struct mm_struct *pt_mm;
 		atomic_t pt_frag_refcount;
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+		atomic_t pt_share_count;
+#endif
 	};
 
 	union {
@@ -516,6 +520,32 @@ static_assert(sizeof(struct ptdesc) <= sizeof(struct page));
 	const struct page *:		(const struct ptdesc *)(p),	\
 	struct page *:			(struct ptdesc *)(p)))
 
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+	atomic_set(&ptdesc->pt_share_count, 0);
+}
+
+static inline void ptdesc_pmd_pts_inc(struct ptdesc *ptdesc)
+{
+	atomic_inc(&ptdesc->pt_share_count);
+}
+
+static inline void ptdesc_pmd_pts_dec(struct ptdesc *ptdesc)
+{
+	atomic_dec(&ptdesc->pt_share_count);
+}
+
+static inline int ptdesc_pmd_pts_count(struct ptdesc *ptdesc)
+{
+	return atomic_read(&ptdesc->pt_share_count);
+}
+#else
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+}
+#endif
+
 /*
  * Used for sizing the vmemmap region on some architectures
  */
  • ⚠️ PR commit 9cb602e171c1 (mm/hugetlb: fix hugetlb_pmd_shared()) → upstream ca1a47cd3f5f
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -1016,7 +1016,7 @@
 #ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
 static inline bool hugetlb_pmd_shared(pte_t *pte)
 {
-	return atomic_read(&virt_to_page(pte)->pt_share_count);
+	return ptdesc_pmd_is_shared(virt_to_ptdesc(pte));
 }
 #else
 static inline bool hugetlb_pmd_shared(pte_t *pte)
  • ⚠️ PR commit 499defa97c33 (mm/hugetlb: fix two comments related to huge_pmd_unshare()) → upstream 3937027caecb
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4069,10 +4069,17 @@
 	tlb_end_vma(tlb, vma);
 
 	/*
-	 * There is nothing protecting a previously-shared page table that we
-	 * unshared through huge_pmd_unshare() from getting freed after we
-	 * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
-	 * succeeded, flush the range corresponding to the pud.
+	 * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
+	 * could defer the flush until now, since by holding i_mmap_rwsem we
+	 * guaranteed that the last refernece would not be dropped. But we must
+	 * do the flushing before we return, as otherwise i_mmap_rwsem will be
+	 * dropped and the last reference to the shared PMDs page might be
+	 * dropped as well.
+	 *
+	 * In theory we could defer the freeing of the PMD pages as well, but
+	 * huge_pmd_unshare() relies on the exact page_count for the PMD page to
+	 * detect sharing, so we cannot defer the release of the page either.
+	 * Instead, do flush now.
 	 */
 	if (force_flush)
 		tlb_flush_mmu_tlbonly(tlb);

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5320,17 +5320,10 @@
 	tlb_end_vma(tlb, vma);
 
 	/*
-	 * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
-	 * could defer the flush until now, since by holding i_mmap_rwsem we
-	 * guaranteed that the last reference would not be dropped. But we must
-	 * do the flushing before we return, as otherwise i_mmap_rwsem will be
-	 * dropped and the last reference to the shared PMDs page might be
-	 * dropped as well.
-	 *
-	 * In theory we could defer the freeing of the PMD pages as well, but
-	 * huge_pmd_unshare() relies on the exact page_count for the PMD page to
-	 * detect sharing, so we cannot defer the release of the page either.
-	 * Instead, do flush now.
+	 * There is nothing protecting a previously-shared page table that we
+	 * unshared through huge_pmd_unshare() from getting freed after we
+	 * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
+	 * succeeded, flush the range corresponding to the pud.
 	 */
 	if (force_flush)
 		tlb_flush_mmu_tlbonly(tlb);

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4068,7 +4068,7 @@
 	/*
 	 * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
 	 * could defer the flush until now, since by holding i_mmap_rwsem we
-	 * guaranteed that the last refernece would not be dropped. But we must
+	 * guaranteed that the last reference would not be dropped. But we must
 	 * do the flushing before we return, as otherwise i_mmap_rwsem will be
 	 * dropped and the last reference to the shared PMDs page might be
 	 * dropped as well.
  • ⚠️ PR commit 91b5a0b3962b (mm/rmap: fix two comments related to huge_pmd_unshare()) → upstream a8682d500f69
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1508,8 +1508,13 @@
 							      range.end);
 
 				/*
-				 * The PMD table was unmapped,
-				 * consequently unmapping the folio.
+				 * The ref count of the PMD page was dropped
+				 * which is part of the way map counting
+				 * is done for shared PMDs.  Return 'true'
+				 * here.  When there is no other sharing,
+				 * huge_pmd_unshare returns false and we will
+				 * unmap the actual page and drop map count
+				 * to zero.
 				 */
 				page_vma_mapped_walk_done(&pvmw);
 				break;

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -2016,14 +2016,8 @@
 					flush_tlb_range(vma,
 						range.start, range.end);
 					/*
-					 * The ref count of the PMD page was
-					 * dropped which is part of the way map
-					 * counting is done for shared PMDs.
-					 * Return 'true' here.  When there is
-					 * no other sharing, huge_pmd_unshare
-					 * returns false and we will unmap the
-					 * actual page and drop map count
-					 * to zero.
+					 * The PMD table was unmapped,
+					 * consequently unmapping the folio.
 					 */
 					goto walk_done;
 				}
@@ -2416,14 +2410,8 @@
 						range.start, range.end);
 
 					/*
-					 * The ref count of the PMD page was
-					 * dropped which is part of the way map
-					 * counting is done for shared PMDs.
-					 * Return 'true' here.  When there is
-					 * no other sharing, huge_pmd_unshare
-					 * returns false and we will unmap the
-					 * actual page and drop map count
-					 * to zero.
+					 * The PMD table was unmapped,
+					 * consequently unmapping the folio.
 					 */
 					page_vma_mapped_walk_done(&pvmw);
 					break;

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1505,12 +1506,13 @@
 
-				/*
-				 * The ref count of the PMD page was dropped
-				 * which is part of the way map counting
-				 * is done for shared PMDs.  Return 'true'
-				 * here.  When there is no other sharing,
-				 * huge_pmd_unshare returns false and we will
-				 * unmap the actual page and drop map count
-				 * to zero.
-				 */
-				page_vma_mapped_walk_done(&pvmw);
-				break;
+					/*
+					 * The ref count of the PMD page was
+					 * dropped which is part of the way map
+					 * counting is done for shared PMDs.
+					 * Return 'true' here.  When there is
+					 * no other sharing, huge_pmd_unshare
+					 * returns false and we will unmap the
+					 * actual page and drop map count
+					 * to zero.
+					 */
+					page_vma_mapped_walk_done(&pvmw);
+					break;
  • ⚠️ PR commit 0e0c9bcf167a (mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather) → upstream 8ce720d5bd91
    Differences found:
================================================================================
*    DELTA DIFFERENCES - code changes that differ between the patches          *
================================================================================

--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -50,8 +50,7 @@
  *
  * The mmu_gather API consists of:
  *
- *  - tlb_gather_mmu() / tlb_gather_mmu_vma() / tlb_finish_mmu(); start and
- *    finish a mmu_gather
+ *  - tlb_gather_mmu() / tlb_finish_mmu(); start and finish a mmu_gather
  *
  *    Finish in particular will issue a (final) TLB invalidate and free
  *    all (remaining) queued pages.
@@ -675,16 +674,16 @@
 #endif
 #endif
 
-#if defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && defined(CONFIG_HUGETLB_PAGE)
-static inline void tlb_unshare_pmd_ptdesc(struct mmu_gather *tlb, struct page *pt,
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void tlb_unshare_pmd_ptdesc(struct mmu_gather *tlb, struct ptdesc *pt,
 					  unsigned long addr)
 {
 	/*
 	 * The caller must make sure that concurrent unsharing + exclusive
 	 * reuse is impossible until tlb_flush_unshared_tables() was called.
 	 */
-	VM_WARN_ON_ONCE(!atomic_read(&pt->pt_share_count));
-	atomic_dec(&pt->pt_share_count);
+	VM_WARN_ON_ONCE(!ptdesc_pmd_is_shared(pt));
+	ptdesc_pmd_pts_dec(pt);
 
 	/* Clearing a PUD pointing at a PMD table with PMD leaves. */
 	tlb_flush_pmd_range(tlb, addr & PUD_MASK, PUD_SIZE);
@@ -693,7 +692,7 @@
 	 * If the page table is now exclusively owned, we fully unshared
 	 * a page table.
 	 */
-	if (!atomic_read(&pt->pt_share_count))
+	if (!ptdesc_pmd_is_shared(pt))
 		tlb->fully_unshared_tables = true;
 	tlb->unshared_tables = true;
 }
@@ -730,7 +729,7 @@
 		tlb->fully_unshared_tables = false;
 	}
 }
-#endif
+#endif /* CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING */
 
 #endif /* CONFIG_MMU */
 
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -167,9 +167,8 @@
 			unsigned long addr, unsigned long sz);
 pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
-		unsigned long *addr, pte_t *ptep);
-void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma);
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+				unsigned long *addr, pte_t *ptep);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
 				unsigned long *start, unsigned long *end);
 struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -210,8 +209,9 @@
 	return NULL;
 }
 
-static inline int huge_pmd_unshare(struct mmu_gather *tlb,
-		struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep)
+static inline int huge_pmd_unshare(struct mm_struct *mm,
+					struct vm_area_struct *vma,
+					unsigned long *addr, pte_t *ptep)
 {
 	return 0;
 }
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -635,8 +635,6 @@
 struct mmu_gather;
 extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
 				unsigned long start, unsigned long end);
-void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma,
-		unsigned long start, unsigned long end);
 extern void tlb_finish_mmu(struct mmu_gather *tlb,
 				unsigned long start, unsigned long end);
 
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4006,8 +4007,10 @@
 			continue;
 
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(tlb, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
 			spin_unlock(ptl);
+			tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
+			force_flush = true;
 			continue;
 		}
 
@@ -5088,7 +5098,7 @@
 
 	BUG_ON(address >= end);
 	flush_cache_range(vma, range.start, range.end);
-	tlb_gather_mmu_vma(&tlb, vma, range.start, range.end);
+	tlb_gather_mmu_vma(&tlb, vma);
 
 	mmu_notifier_invalidate_range_start(&range);
 	i_mmap_lock_write(vma->vm_file->f_mapping);
@@ -5098,9 +5108,10 @@
 		if (!ptep)
 			continue;
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(&tlb, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
 			pages++;
 			spin_unlock(ptl);
+			shared_pmd = true;
 			continue;
 		}
 		pte = huge_ptep_get(ptep);
@@ -5146,7 +5157,7 @@
 	 */
 	i_mmap_unlock_write(vma->vm_file->f_mapping);
 	mmu_notifier_invalidate_range_end(&range);
-	tlb_finish_mmu(&tlb, range.start, range.end);
+	tlb_finish_mmu(&tlb);
 
 	return pages << h->order;
 }
@@ -5470,7 +5481,7 @@
  * huge_pmd_unshare - Unmap a pmd table if it is shared by multiple users
  * @tlb: the current mmu_gather.
  * @vma: the vma covering the pmd table.
- * @addr: pointer to the address we are trying to unshare.
+ * @addr: the address we are trying to unshare.
  * @ptep: pointer into the (pmd) page table.
  *
  * Called with the page table lock held, the i_mmap_rwsem held in write mode
@@ -5482,11 +5493,10 @@
  * Returns: 1 if it was a shared PMD table and it got unmapped, or 0 if it
  *	    was not a shared PMD table.
  */
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
-		unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+					unsigned long *addr, pte_t *ptep)
 {
 	unsigned long sz = huge_page_size(hstate_vma(vma));
-	struct mm_struct *mm = vma->vm_mm;
 	pgd_t *pgd = pgd_offset(mm, *addr);
 	p4d_t *p4d = p4d_offset(pgd, *addr);
 	pud_t *pud = pud_offset(p4d, *addr);
@@ -5498,36 +5508,18 @@
 		return 0;
 
 	pud_clear(pud);
-	tlb_unshare_pmd_ptdesc(tlb, virt_to_page(ptep), *addr);
-
+	/*
+	 * Once our caller drops the rmap lock, some other process might be
+	 * using this page table as a normal, non-hugetlb page table.
+	 * Wait for pending gup_fast() in other threads to finish before letting
+	 * that happen.
+	 */
+	tlb_remove_table_sync_one();
+	atomic_dec(&virt_to_page(ptep)->pt_share_count);
 	mm_dec_nr_pmds(mm);
 	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;
 }
-
-/*
- * huge_pmd_unshare_flush - Complete a sequence of huge_pmd_unshare() calls
- * @tlb: the current mmu_gather.
- * @vma: the vma covering the pmd table.
- *
- * Perform necessary TLB flushes or IPI broadcasts to synchronize PMD table
- * unsharing with concurrent page table walkers.
- *
- * This function must be called after a sequence of huge_pmd_unshare()
- * calls while still holding the i_mmap_rwsem.
- */
-void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma)
-{
-	/*
-	 * We must synchronize page table unsharing such that nobody will
-	 * try reusing a previously-shared page table while it might still
-	 * be in use by previous sharers (TLB, GUP_fast).
-	 */
-	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
-
-	tlb_flush_unshared_tables(tlb);
-}
-
 #define want_pmd_share()	(1)
 #else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
 pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
@@ -5535,8 +5527,8 @@
 	return NULL;
 }
 
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
-		unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+				unsigned long *addr, pte_t *ptep)
 {
 	return 0;
 }
@@ -5804,7 +5796,7 @@
 		return;
 
 	flush_cache_range(vma, start, end);
-	tlb_gather_mmu_vma(&tlb, vma, start, end);
+	tlb_gather_mmu_vma(&tlb, vma);
 
 	/*
 	 * No need to call adjust_range_if_pmd_sharing_possible(), because
@@ -5823,7 +5815,7 @@
 		if (!ptep)
 			continue;
 		ptl = huge_pte_lock(h, mm, ptep);
-		huge_pmd_unshare(&tlb, vma, &address, ptep);
+		huge_pmd_unshare(mm, vma, &address, ptep);
 		spin_unlock(ptl);
 	}
 	huge_pmd_unshare_flush(&tlb, vma);
@@ -5837,3 +5829,3 @@
 	mmu_notifier_invalidate_range_end(&range);
-	tlb_finish_mmu(&tlb, start, end);
+	tlb_finish_mmu(&tlb);
 }
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -292,8 +291,6 @@
  *			single VMA
  * @tlb: the mmu_gather structure to initialize
  * @vma: the vm_area_struct
- * @start: start of the region that will be removed from the page-table
- * @end: end of the region that will be removed from the page-table
  *
  * Called to initialize an (on-stack) mmu_gather structure for operating on
  * a single VMA. In contrast to tlb_gather_mmu(), calling this function will
@@ -305,10 +302,9 @@
  * tlb_change_page_size().
  *
  */
-void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma,
-		unsigned long start, unsigned long end)
+void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma)
 {
-	tlb_gather_mmu(tlb, vma->vm_mm, start, end);
+	tlb_gather_mmu(tlb, vma->vm_mm);
 	tlb_update_vma_flags(tlb, vma);
 	if (is_vm_hugetlb_page(vma))
 		/* All entries have the same size. */
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1488,16 +1488,13 @@
 		address = pvmw.address;
 
 		if (PageHuge(page) && !PageAnon(page)) {
-			struct mmu_gather tlb;
-
 			/*
 			 * To call huge_pmd_unshare, i_mmap_rwsem must be
 			 * held in write mode.  Caller needs to explicitly
 			 * do this outside rmap routines.
 			 */
 			VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
-			tlb_gather_mmu_vma(&tlb, vma, range.start, range.end);
-			if (huge_pmd_unshare(&tlb, vma, &address, pvmw.pte)) {
+			if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
 				/*
 				 * huge_pmd_unshare unmapped an entire PMD
 				 * page.  There is no way of knowing exactly
@@ -1506,10 +1503,9 @@
 				 * already adjusted above to cover this range.
 				 */
 				flush_cache_range(vma, range.start, range.end);
-				huge_pmd_unshare_flush(&tlb, vma);
+				flush_tlb_range(vma, range.start, range.end);
 				mmu_notifier_invalidate_range(mm, range.start,
 							      range.end);
-				tlb_finish_mmu(&tlb, range.start, range.end);
 
 				/*
 				 * The PMD table was unmapped,
@@ -1518,7 +1514,6 @@
 				page_vma_mapped_walk_done(&pvmw);
 				break;
 			}
-			tlb_finish_mmu(&tlb, range.start, range.end);
 		}
 
 		if (IS_ENABLED(CONFIG_MIGRATION) &&

################################################################################
!    REJECTED PATCH2 HUNKS - could not be compared; manual review needed       !
################################################################################

--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -46,7 +46,8 @@
  *
  * The mmu_gather API consists of:
  *
- *  - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_finish_mmu()
+ *  - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_gather_mmu_vma() /
+ *    tlb_finish_mmu()
  *
  *    start and finish a mmu_gather
  *
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -240,8 +240,9 @@
 pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
 unsigned long hugetlb_mask_last_page(struct hstate *h);
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long addr, pte_t *ptep);
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep);
+void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
 				unsigned long *start, unsigned long *end);
 
@@ -300,9 +301,8 @@
 	return NULL;
 }
 
-static inline int huge_pmd_unshare(struct mm_struct *mm,
-					struct vm_area_struct *vma,
-					unsigned long addr, pte_t *ptep)
+static inline int huge_pmd_unshare(struct mmu_gather *tlb,
+		struct vm_area_struct *vma, unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1530,6 +1530,7 @@
 struct mmu_gather;
 extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
 extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
+void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma);
 extern void tlb_finish_mmu(struct mmu_gather *tlb);
 
 struct vm_fault;
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5112,7 +5112,7 @@
 	unsigned long last_addr_mask;
 	pte_t *src_pte, *dst_pte;
 	struct mmu_notifier_range range;
-	bool shared_pmd = false;
+	struct mmu_gather tlb;
 
 	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, old_addr,
 				old_end);
@@ -5139,8 +5140,7 @@
 		if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte)))
 			continue;
 
-		if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) {
-			shared_pmd = true;
+		if (huge_pmd_unshare(&tlb, vma, old_addr, src_pte)) {
 			old_addr |= last_addr_mask;
 			new_addr |= last_addr_mask;
 			continue;
@@ -5151,6 +5151,7 @@
 			break;
 
 		move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte, sz);
+		tlb_remove_huge_tlb_entry(h, &tlb, src_pte, old_addr);
 	}
 
 	if (shared_pmd)
@@ -5153,10 +5154,9 @@
 		move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte, sz);
 	}
 
-	if (shared_pmd)
-		flush_hugetlb_tlb_range(vma, range.start, range.end);
-	else
-		flush_hugetlb_tlb_range(vma, old_end - len, old_end);
+	tlb_flush_mmu_tlbonly(&tlb);
+	huge_pmd_unshare_flush(&tlb, vma);
+
 	mmu_notifier_invalidate_range_end(&range);
 	i_mmap_unlock_write(mapping);
 	hugetlb_vma_unlock_write(vma);
@@ -5160,6 +5160,7 @@
 	mmu_notifier_invalidate_range_end(&range);
 	i_mmap_unlock_write(mapping);
 	hugetlb_vma_unlock_write(vma);
+	tlb_finish_mmu(&tlb);
 
 	return len + old_addr - old_end;
 }
@@ -5178,7 +5179,6 @@
 	unsigned long sz = huge_page_size(h);
 	bool adjust_reservation;
 	unsigned long last_addr_mask;
-	bool force_flush = false;
 
 	WARN_ON(!is_vm_hugetlb_page(vma));
 	BUG_ON(start & ~huge_page_mask(h));
@@ -5201,7 +5201,7 @@
 		}
 
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, address, ptep)) {
+		if (huge_pmd_unshare(tlb, vma, address, ptep)) {
 			spin_unlock(ptl);
 			tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
 			force_flush = true;
@@ -5203,8 +5203,6 @@
 		ptl = huge_pte_lock(h, mm, ptep);
 		if (huge_pmd_unshare(mm, vma, address, ptep)) {
 			spin_unlock(ptl);
-			tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
-			force_flush = true;
 			address |= last_addr_mask;
 			continue;
 		}
@@ -5320,14 +5318,7 @@
 	}
 	tlb_end_vma(tlb, vma);
 
-	/*
-	 * There is nothing protecting a previously-shared page table that we
-	 * unshared through huge_pmd_unshare() from getting freed after we
-	 * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
-	 * succeeded, flush the range corresponding to the pud.
-	 */
-	if (force_flush)
-		tlb_flush_mmu_tlbonly(tlb);
+	huge_pmd_unshare_flush(tlb, vma);
 }
 
 void __hugetlb_zap_begin(struct vm_area_struct *vma,
@@ -6426,7 +6417,6 @@
 	pte_t pte;
 	struct hstate *h = hstate_vma(vma);
 	long pages = 0, psize = huge_page_size(h);
-	bool shared_pmd = false;
 	struct mmu_notifier_range range;
 	unsigned long last_addr_mask;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
@@ -6430,6 +6420,7 @@
 	unsigned long last_addr_mask;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+	struct mmu_gather tlb;
 
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
@@ -6443,6 +6434,7 @@
 
 	BUG_ON(address >= end);
 	flush_cache_range(vma, range.start, range.end);
+	tlb_gather_mmu_vma(&tlb, vma);
 
 	mmu_notifier_invalidate_range_start(&range);
 	hugetlb_vma_lock_write(vma);
@@ -6469,7 +6461,7 @@
 			}
 		}
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, address, ptep)) {
+		if (huge_pmd_unshare(&tlb, vma, address, ptep)) {
 			/*
 			 * When uffd-wp is enabled on the vma, unshare
 			 * shouldn't happen at all.  Warn about it if it
@@ -6478,7 +6470,6 @@
 			WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
 			pages++;
 			spin_unlock(ptl);
-			shared_pmd = true;
 			address |= last_addr_mask;
 			continue;
 		}
@@ -6932,8 +6926,8 @@
  * returns: 1 successfully unmapped a shared pte page
  *	    0 the underlying pte page is not shared, or it is the last user
  */
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-					unsigned long addr, pte_t *ptep)
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep)
 {
 	unsigned long sz = huge_page_size(hstate_vma(vma));
 	pgd_t *pgd = pgd_offset(mm, addr);
@@ -6936,6 +6930,7 @@
 					unsigned long addr, pte_t *ptep)
 {
 	unsigned long sz = huge_page_size(hstate_vma(vma));
+	struct mm_struct *mm = vma->vm_mm;
 	pgd_t *pgd = pgd_offset(mm, addr);
 	p4d_t *p4d = p4d_offset(pgd, addr);
 	pud_t *pud = pud_offset(p4d, addr);
@@ -6947,14 +6942,9 @@
 	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
 	hugetlb_vma_assert_locked(vma);
 	pud_clear(pud);
-	/*
-	 * Once our caller drops the rmap lock, some other process might be
-	 * using this page table as a normal, non-hugetlb page table.
-	 * Wait for pending gup_fast() in other threads to finish before letting
-	 * that happen.
-	 */
-	tlb_remove_table_sync_one();
-	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
+
+	tlb_unshare_pmd_ptdesc(tlb, virt_to_ptdesc(ptep), addr);
+
 	mm_dec_nr_pmds(mm);
 	return 1;
 }
@@ -6958,6 +6948,29 @@
 	return 1;
 }
 
+/*
+ * huge_pmd_unshare_flush - Complete a sequence of huge_pmd_unshare() calls
+ * @tlb: the current mmu_gather.
+ * @vma: the vma covering the pmd table.
+ *
+ * Perform necessary TLB flushes or IPI broadcasts to synchronize PMD table
+ * unsharing with concurrent page table walkers.
+ *
+ * This function must be called after a sequence of huge_pmd_unshare()
+ * calls while still holding the i_mmap_rwsem.
+ */
+void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma)
+{
+	/*
+	 * We must synchronize page table unsharing such that nobody will
+	 * try reusing a previously-shared page table while it might still
+	 * be in use by previous sharers (TLB, GUP_fast).
+	 */
+	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+
+	tlb_flush_unshared_tables(tlb);
+}
+
 #else /* !CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING */
 
 pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
@@ -6967,8 +6980,8 @@
 	return NULL;
 }
 
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long addr, pte_t *ptep)
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
@@ -7275,7 +7295,7 @@
 		if (!ptep)
 			continue;
 		ptl = huge_pte_lock(h, mm, ptep);
-		huge_pmd_unshare(mm, vma, address, ptep);
+		huge_pmd_unshare(&tlb, vma, address, ptep);
 		spin_unlock(ptl);
 	}
 	flush_hugetlb_tlb_range(vma, start, end);
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -10,6 +10,7 @@
 #include <linux/swap.h>
 #include <linux/rmap.h>
 #include <linux/pgalloc.h>
+#include <linux/hugetlb.h>
 
 #include <asm/tlb.h>
 
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -2008,6 +2008,8 @@
 			 * if unsuccessful.
 			 */
 			if (!anon) {
+				struct mmu_gather tlb;
+
 				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
 				if (!hugetlb_vma_trylock_write(vma))
 					goto walk_abort;
@@ -2011,7 +2013,9 @@
 				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
 				if (!hugetlb_vma_trylock_write(vma))
 					goto walk_abort;
-				if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
+
+				tlb_gather_mmu_vma(&tlb, vma);
+				if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
 					hugetlb_vma_unlock_write(vma);
 					flush_tlb_range(vma,
 						range.start, range.end);
@@ -2013,8 +2017,8 @@
 					goto walk_abort;
 				if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
 					hugetlb_vma_unlock_write(vma);
-					flush_tlb_range(vma,
-						range.start, range.end);
+					huge_pmd_unshare_flush(&tlb, vma);
+					tlb_finish_mmu(&tlb);
 					/*
 					 * The PMD table was unmapped,
 					 * consequently unmapping the folio.
@@ -2022,6 +2026,7 @@
 					goto walk_done;
 				}
 				hugetlb_vma_unlock_write(vma);
+				tlb_finish_mmu(&tlb);
 			}
 			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
 			if (pte_dirty(pteval))
@@ -2398,6 +2403,8 @@
 			 * fail if unsuccessful.
 			 */
 			if (!anon) {
+				struct mmu_gather tlb;
+
 				VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
 				if (!hugetlb_vma_trylock_write(vma)) {
 					page_vma_mapped_walk_done(&pvmw);
@@ -2401,10 +2408,6 @@
 					ret = false;
 					break;
 				}
-				if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
-					hugetlb_vma_unlock_write(vma);
-					flush_tlb_range(vma,
-						range.start, range.end);
 
 					/*
 					 * The PMD table was unmapped,
@@ -2406,6 +2409,11 @@
 					flush_tlb_range(vma,
 						range.start, range.end);
 
+				tlb_gather_mmu_vma(&tlb, vma);
+				if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
+					hugetlb_vma_unlock_write(vma);
+					huge_pmd_unshare_flush(&tlb, vma);
+					tlb_finish_mmu(&tlb);
 					/*
 					 * The PMD table was unmapped,
 					 * consequently unmapping the folio.
@@ -2417,6 +2425,7 @@
 					break;
 				}
 				hugetlb_vma_unlock_write(vma);
+				tlb_finish_mmu(&tlb);
 			}
 			/* Nuke the hugetlb page table entry */
 			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);

================================================================================
*    CONTEXT DIFFERENCES - surrounding code differences between the patches    *
================================================================================

--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -45,3 +45,5 @@
  *
- *  - tlb_gather_mmu() / tlb_finish_mmu(); start and finish a mmu_gather
+ *  - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_finish_mmu()
+ *
+ *    start and finish a mmu_gather
  *
@@ -291,4 +366,4 @@
 
 	unsigned int		batch_count;
 
-#ifndef CONFIG_HAVE_MMU_GATHER_NO_GATHER
+#ifndef CONFIG_MMU_GATHER_NO_GATHER
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -164,7 +164,8 @@
 pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
+unsigned long hugetlb_mask_last_page(struct hstate *h);
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long *addr, pte_t *ptep);
+				unsigned long addr, pte_t *ptep);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
 				unsigned long *start, unsigned long *end);
-struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
+
@@ -211,7 +302,7 @@
 
 static inline int huge_pmd_unshare(struct mm_struct *mm,
 					struct vm_area_struct *vma,
-					unsigned long *addr, pte_t *ptep)
+					unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -632,6 +632,5 @@
 struct mmu_gather;
-extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
-				unsigned long start, unsigned long end);
-extern void tlb_finish_mmu(struct mmu_gather *tlb,
-				unsigned long start, unsigned long end);
+extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
+extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
+extern void tlb_finish_mmu(struct mmu_gather *tlb);
 
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3977,5 +3979,11 @@
 	struct mmu_notifier_range range;
-	bool force_flush = false;
+	bool shared_pmd = false;
 
-	WARN_ON(!is_vm_hugetlb_page(vma));
-	BUG_ON(start & ~huge_page_mask(h));
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, old_addr,
+				old_end);
+	 * range.
+	 */
+	flush_cache_range(vma, range.start, range.end);
+
+	mmu_notifier_invalidate_range_start(&range);
+	last_addr_mask = hugetlb_mask_last_page(h);
@@ -4004,8 +5166,9 @@
 
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, address, ptep)) {
 			spin_unlock(ptl);
 			tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
 			force_flush = true;
+			address |= last_addr_mask;
 			continue;
 		}
@@ -4078,4 +5329,4 @@
 		tlb_flush_mmu_tlbonly(tlb);
 }
 
-void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+void __hugetlb_zap_begin(struct vm_area_struct *vma,
@@ -5083,8 +6424,11 @@
 	pte_t pte;
 	struct hstate *h = hstate_vma(vma);
-	unsigned long pages = 0;
+	long pages = 0, psize = huge_page_size(h);
 	bool shared_pmd = false;
 	struct mmu_notifier_range range;
+	unsigned long last_addr_mask;
+	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
@@ -5091,7 +6443,12 @@
 		ptl = huge_pte_lock(h, mm, ptep);
-		if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+		if (huge_pmd_unshare(mm, vma, address, ptep)) {
+			/*
+			 * When uffd-wp is enabled on the vma, unshare
+			 * shouldn't happen at all.  Warn about it if it
+			WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
 			pages++;
 			spin_unlock(ptl);
 			shared_pmd = true;
+			address |= last_addr_mask;
 			continue;
 		}
@@ -5138,7 +6504,10 @@
 			huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);
 			pages++;
 		}
+
+next:
 		spin_unlock(ptl);
+		cond_resched();
 	}
 	/*
 	 * There is nothing protecting a previously-shared page table that we
@@ -5160,6 +6558,8 @@
-	 */
+	 * No need to call mmu_notifier_arch_invalidate_secondary_tlbs() we are
+	 * downgrading page table protection not changing it to point to a new
 	i_mmap_unlock_write(vma->vm_file->f_mapping);
+	hugetlb_vma_unlock_write(vma);
 	mmu_notifier_invalidate_range_end(&range);
 
-	return pages << h->order;
+	return pages > 0 ? (pages << h->order) : pages;
 }
@@ -5491,5 +6930,5 @@
  */
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-					unsigned long *addr, pte_t *ptep)
+					unsigned long addr, pte_t *ptep)
 {
 	unsigned long sz = huge_page_size(hstate_vma(vma));
@@ -5499,5 +6935,8 @@
-		return 0;
-
+	pgd_t *pgd = pgd_offset(mm, addr);
+	p4d_t *p4d = p4d_offset(pgd, addr);
+	pud_t *pud = pud_offset(p4d, addr);
+	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+	hugetlb_vma_assert_locked(vma);
 	pud_clear(pud);
 	/*
 	 * Once our caller drops the rmap lock, some other process might be
@@ -5512,7 +6946,6 @@
 	 */
 	tlb_remove_table_sync_one();
-	atomic_dec(&virt_to_page(ptep)->pt_share_count);
+	ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
 	mm_dec_nr_pmds(mm);
-	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;
 }
@@ -5523,7 +6957,7 @@
 }
 
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
-				unsigned long *addr, pte_t *ptep)
+				unsigned long addr, pte_t *ptep)
 {
 	return 0;
 }
@@ -5801,9 +7255,9 @@
 		if (!ptep)
 			continue;
 		ptl = huge_pte_lock(h, mm, ptep);
-		huge_pmd_unshare(mm, vma, &address, ptep);
+		huge_pmd_unshare(mm, vma, address, ptep);
 		spin_unlock(ptl);
 	}
 	flush_hugetlb_tlb_range(vma, start, end);
 	if (take_locks) {
 		i_mmap_unlock_write(vma->vm_file->f_mapping);
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -6,4 +4,5 @@
 #include <linux/swap.h>
+#include <linux/rmap.h>
+#include <linux/pgalloc.h>
 
-#include <asm/pgalloc.h>
 #include <asm/tlb.h>
@@ -279,5 +423,6 @@
 #endif
+	tlb->vma_pfn = 0;
 
 	__tlb_reset_range(tlb);
 	inc_tlb_flush_pending(tlb->mm);
 }
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -69,7 +69,7 @@
-#include <linux/memremap.h>
-#include <linux/userfaultfd_k.h>
+#include <linux/mm_inline.h>
+#include <linux/oom.h>
 
 #include <asm/tlbflush.h>
 
-#include <trace/events/tlb.h>
-
+#define CREATE_TRACE_POINTS
+#include <trace/events/migrate.h>
@@ -1485,2 +1944,3 @@
 			}
-		}
+			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
+			if (pte_dirty(pteval))
@@ -1487,2 +1960,10 @@
 
-		if (IS_ENABLED(CONFIG_MIGRATION) &&
+					/*
+					 * The PMD table was unmapped,
+					 * consequently unmapping the folio.
+					break;
+				}
+				hugetlb_vma_unlock_write(vma);
+			}
+			/* Nuke the hugetlb page table entry */
+			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);

This is an automated interdiff check for backported commits.

@github-actions

Copy link
Copy Markdown

JIRA PR Check Results

18 commit(s) with issues found:

Commit 0e0c9bcf167a

Summary: mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit 91b5a0b3962b

Summary: mm/rmap: fix two comments related to huge_pmd_unshare()

❌ Errors:

  • VULN-46924: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-46924: No time logged - please log time manually

Commit 499defa97c33

Summary: mm/hugetlb: fix two comments related to huge_pmd_unshare()

❌ Errors:

  • VULN-46924: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-46924: No time logged - please log time manually

Commit 9cb602e171c1

Summary: mm/hugetlb: fix hugetlb_pmd_shared()

❌ Errors:

  • VULN-193890: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-193890: No time logged - please log time manually

Commit 27fe17e8751e

Summary: mm/hugetlb: fix copy_hugetlb_page_range() to use ->pt_share_count

❌ Errors:

  • VULN-71572: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71572: No time logged - please log time manually

Commit 116b86e2d9a8

Summary: mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit f9ab2b5c060b

Summary: mm/hugetlb: unshare page tables during VMA split, not before

❌ Errors:

  • VULN-71572: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71572: No time logged - please log time manually

Commit 7fa9f009bd64

Summary: mm: hugetlb: independent PMD page table shared count

❌ Errors:

  • VULN-46924: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-46924: No time logged - please log time manually

Commit 900585336f8a

Summary: mm: hugetlb: proc: check for hugetlb shared PMD in /proc/PID/smaps

❌ Errors:

  • VULN-193890: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-193890: No time logged - please log time manually

Commit 305f3092087c

Summary: hugetlb: unshare some PMDs when splitting VMAs

❌ Errors:

  • VULN-71572: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71572: No time logged - please log time manually

Commit c48c7b99c598

Summary: mm/hugetlb: make detecting shared pte more reliable

❌ Errors:

  • VULN-71572: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71572: No time logged - please log time manually

Commit 8f7d45b6f15e

Summary: mm: proc: Invalidate TLB after clearing soft-dirty page state

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit 98c3edd9f59e

Summary: asm-generic/tlb: provide MMU_GATHER_TABLE_FREE

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit 76cb54d2095b

Summary: asm-generic/tlb: rename HAVE_RCU_TABLE_FREE

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit 82b0e4d9469b

Summary: asm-generic/tlb: avoid potential double flush

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit c3be31f36874

Summary: arm/tlb: Convert to generic mmu_gather

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit 97a5c0fdf5d3

Summary: ARM: Kconfig: remove useless parenthesis

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Commit eef386801c6f

Summary: arm/asm/tlb.h: Fix build error implicit func declaration

❌ Errors:

  • VULN-71581: Status is 'To Do', expected 'In Progress'

⚠️ Warnings:

  • VULN-71581: No time logged - please log time manually

Summary: Checked 18 commit(s) total.

@github-actions

Copy link
Copy Markdown

Validation checks completed with issues View full results: https://github.com/ctrliq/kernel-src-tree/actions/runs/30552170473

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant