[LTS 8.6] hugetlb: CVE-2024-57883, CVE-2025-38084, CVE-2025-38085, CVE-2026-23100 - #1491
[LTS 8.6] hugetlb: CVE-2024-57883, CVE-2025-38084, CVE-2025-38085, CVE-2026-23100#1491pvts-mat wants to merge 18 commits into
Conversation
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Anders Roxell <anders.roxell@linaro.org> commit 063daa8 Building on arm 32 with LPAE enabled we don't include asm-generic/tlb.h, where we have tlb_flush_remove_tables_local and tlb_flush_remove_tables defined. The build fails with: mm/memory.c: In function ‘tlb_remove_table_smp_sync’: mm/memory.c:339:2: error: implicit declaration of function ‘tlb_flush_remove_tables_local’; did you mean ‘tlb_remove_table’? [-Werror=implicit-function-declaration] ... This bug got introduced in: 2ff6ddf ("x86/mm/tlb: Leave lazy TLB mode at page table free time") To fix this issue we define them in arm 32's specific asm/tlb.h file as well. Signed-off-by: Anders Roxell <anders.roxell@linaro.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: dave.hansen@intel.com Cc: linux-arm-kernel@lists.infradead.org Cc: linux@armlinux.org.uk Cc: riel@surriel.com Cc: songliubraving@fb.com Fixes: 2ff6ddf ("x86/mm/tlb: Leave lazy TLB mode at page table free time") Link: http://lkml.kernel.org/r/20180725095557.19668-1-anders.roxell@linaro.org Signed-off-by: Ingo Molnar <mingo@kernel.org> (cherry picked from commit 063daa8) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Russell King <rmk+kernel@armlinux.org.uk> commit f00790a Remove useless parenthesis from the Kconfig - Kconfig is not C, and can cope without parenthesising e.g. single symbols to 'if'. Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk> (cherry picked from commit f00790a) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Peter Zijlstra <peterz@infradead.org> commit b78180b Generic mmu_gather provides everything that ARM needs: - range tracking - RCU table free - VM_EXEC tracking - VIPT cache flushing The one notable curiosity is the 'funny' range tracking for classical ARM in __pte_free_tlb(). No change in behavior intended. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Lutomirski <luto@kernel.org> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com> Cc: Borislav Petkov <bp@alien8.de> Cc: Dave Hansen <dave.hansen@linux.intel.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Nick Piggin <npiggin@gmail.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rik van Riel <riel@surriel.com> Cc: Russell King <linux@armlinux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Ingo Molnar <mingo@kernel.org> (cherry picked from commit b78180b) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Peter Zijlstra <peterz@infradead.org> commit 0758cd8 Aneesh reported that: tlb_flush_mmu() tlb_flush_mmu_tlbonly() tlb_flush() <-- ctrliq#1 tlb_flush_mmu_free() tlb_table_flush() tlb_table_invalidate() tlb_flush_mmu_tlbonly() tlb_flush() <-- ctrliq#2 does two TLBIs when tlb->fullmm, because __tlb_reset_range() will not clear tlb->end in that case. Observe that any caller to __tlb_adjust_range() also sets at least one of the tlb->freed_tables || tlb->cleared_p* bits, and those are unconditionally cleared by __tlb_reset_range(). Change the condition for actually issuing TLBI to having one of those bits set, as opposed to having tlb->end != 0. Link: http://lkml.kernel.org/r/20200116064531.483522-4-aneesh.kumar@linux.ibm.com Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Reported-by: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 0758cd8) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Peter Zijlstra <peterz@infradead.org> commit ff2e6d7 upstream-diff | arch/arm64/Kconfig Context conflicts due to options HAVE_FUTEX_CMPXCHG, HAVE_FUNCTION_ARG_ACCESS_API not yet introduced into LTS 8.6. arch/x86/Kconfig Context conflicts due to the option HAVE_RCU_TABLE_INVALIDATE still being in LTS 8.6, but no longer in the upstream Towards a more consistent naming scheme. [akpm@linux-foundation.org: fix sparc64 Kconfig] Link: http://lkml.kernel.org/r/20200116064531.483522-7-aneesh.kumar@linux.ibm.com Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit 6adfe77d47bd5db38fd2e5b5786c77b86e462e13) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Peter Zijlstra <peterz@infradead.org> commit 0d6e24d upstream-diff | include/asm-generic/tlb.h Small conflict in the comments mm/mmu_gather.c - Context conflict in tlb_flush_mmu_free because 580a586 ("asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER") wasn't backported and CONFIG_MMU_GATHER_NO_GATHER still has its old name CONFIG_HAVE_MMU_GATHER_NO_GATHER. - Context conflict in tlb_gather_mmu because 3af4bd0 ("asm-generic/tlb: rename HAVE_MMU_GATHER_PAGE_SIZE") wasn't backported and CONFIG_MMU_GATHER_PAGE_SIZE still has its old name CONFIG_HAVE_MMU_GATHER_PAGE_SIZE. As described in the comment, the correct order for freeing pages is: 1) unhook page 2) TLB invalidate page 3) free page This order equally applies to page directories. Currently there are two correct options: - use tlb_remove_page(), when all page directores are full pages and there are no futher contraints placed by things like software walkers (HAVE_FAST_GUP). - use MMU_GATHER_RCU_TABLE_FREE and tlb_remove_table() when the architecture does not do IPI based TLB invalidate and has HAVE_FAST_GUP (or software TLB fill). This however leaves architectures that don't have page based directories but don't need RCU in a bind. For those, provide MMU_GATHER_TABLE_FREE, which provides the independent batching for directories without the additional RCU freeing. Link: http://lkml.kernel.org/r/20200116064531.483522-10-aneesh.kumar@linux.ibm.com Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> (cherry picked from commit d420cc00ee07b95443e2d9a0c6b4f4c1ee00bc36) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve-pre CVE-2025-38085 commit-author Will Deacon <will@kernel.org> commit 912efa1 upstream-diff Fixed context conflicts due to the lack of the following commits: - 4d45e75 ("mm: remove the now-unnecessary mmget_still_valid() hack") - 29a951d ("mm: fix clear_refs_write locking") Backporting them, although syntactically clean, would require including additional patches to provide the logic required for them to be correct. Since commit 0758cd8 ("asm-generic/tlb: avoid potential double flush"), TLB invalidation is elided in tlb_finish_mmu() if no entries were batched via the tlb_remove_*() functions. Consequently, the page-table modifications performed by clear_refs_write() in response to a write to /proc/<pid>/clear_refs do not perform TLB invalidation. Although this is fine when simply aging the ptes, in the case of clearing the "soft-dirty" state we can end up with entries where pte_write() is false, yet a writable mapping remains in the TLB. Fix this by avoiding the mmu_gather API altogether: managing both the 'tlb_flush_pending' flag on the 'mm_struct' and explicit TLB invalidation for the sort-dirty path, much like mprotect() does already. Fixes: 0758cd8 ("asm-generic/tlb: avoid potential double flush”) Signed-off-by: Will Deacon <will@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Reviewed-by: Yu Zhao <yuzhao@google.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Link: https://lkml.kernel.org/r/20210127235347.1402-2-will@kernel.org (cherry picked from commit 912efa1) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572 cve-pre CVE-2025-38084 commit-author Miaohe Lin <linmiaohe@huawei.com> commit 3aa4ed8 upstream-diff Fixed small conflict in copy_hugetlb_page_range due to missing e95a985 ("hugetlb: skip to end of PT page mapping when pte not present") If the pagetables are shared, we shouldn't copy or take references. Since src could have unshared and dst shares with another vma, huge_pte_none() is thus used to determine whether dst_pte is shared. But this check isn't reliable. A shared pte could have pte none in pagetable in fact. The page count of ptep page should be checked here in order to reliably determine whether pte is shared. [lukas.bulwahn@gmail.com: remove unused local variable dst_entry in copy_hugetlb_page_range()] Link: https://lkml.kernel.org/r/20220822082525.26071-1-lukas.bulwahn@gmail.com Link: https://lkml.kernel.org/r/20220816130553.31406-7-linmiaohe@huawei.com Signed-off-by: Miaohe Lin <linmiaohe@huawei.com> Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com> Cc: Muchun Song <songmuchun@bytedance.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 3aa4ed8) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572 cve-pre CVE-2025-38084 commit-author James Houghton <jthoughton@google.com> commit b30c14c upstream-diff Used linux-5.10.y backport f1082f5f3d02f2f9b4ca36642274529cf5ddaf34 for an almost clean pick. Context conflicts resolved due to the missing feature cf11e85 ("mm: hugetlb: optionally allocate gigantic hugepages using cma") PMD sharing can only be done in PUD_SIZE-aligned pieces of VMAs; however, it is possible that HugeTLB VMAs are split without unsharing the PMDs first. Without this fix, it is possible to hit the uffd-wp-related WARN_ON_ONCE in hugetlb_change_protection [1]. The key there is that hugetlb_unshare_all_pmds will not attempt to unshare PMDs in non-PUD_SIZE-aligned sections of the VMA. It might seem ideal to unshare in hugetlb_vm_op_open, but we need to unshare in both the new and old VMAs, so unsharing in hugetlb_vm_op_split seems natural. [1]: https://lore.kernel.org/linux-mm/CADrL8HVeOkj0QH5VZZbRzybNE8CG-tEGFshnA+bG9nMgcWtBSg@mail.gmail.com/ Link: https://lkml.kernel.org/r/20230104231910.1464197-1-jthoughton@google.com Fixes: 6dfeaff ("hugetlb/userfaultfd: unshare all pmds for hugetlbfs when register wp") Signed-off-by: James Houghton <jthoughton@google.com> Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com> Acked-by: Peter Xu <peterx@redhat.com> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Muchun Song <songmuchun@bytedance.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit f1082f5f3d02f2f9b4ca36642274529cf5ddaf34) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-193890 cve-pre CVE-2026-23100 commit-author Mike Kravetz <mike.kravetz@oracle.com> commit 3489dbb upstream-diff Used linux-5.10.y backport 5569593 for an almost clean pick. Resolved conflict at the end due to missing cf11e85 ("mm: hugetlb: optionally allocate gigantic hugepages using cma") Patch series "Fixes for hugetlb mapcount at most 1 for shared PMDs". This issue of mapcount in hugetlb pages referenced by shared PMDs was discussed in [1]. The following two patches address user visible behavior caused by this issue. [1] https://lore.kernel.org/linux-mm/Y9BF+OCdWnCSilEu@monkey/ This patch (of 2): A hugetlb page will have a mapcount of 1 if mapped by multiple processes via a shared PMD. This is because only the first process increases the map count, and subsequent processes just add the shared PMD page to their page table. page_mapcount is being used to decide if a hugetlb page is shared or private in /proc/PID/smaps. Pages referenced via a shared PMD were incorrectly being counted as private. To fix, check for a shared PMD if mapcount is 1. If a shared PMD is found count the hugetlb page as shared. A new helper to check for a shared PMD is added. [akpm@linux-foundation.org: simplification, per David] [akpm@linux-foundation.org: hugetlb.h: include page_ref.h for page_count()] Link: https://lkml.kernel.org/r/20230126222721.222195-2-mike.kravetz@oracle.com Fixes: 25ee01a ("mm: hugetlb: proc: add hugetlb-related fields to /proc/PID/smaps") Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> Acked-by: Peter Xu <peterx@redhat.com> Cc: David Hildenbrand <david@redhat.com> Cc: James Houghton <jthoughton@google.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Muchun Song <songmuchun@bytedance.com> Cc: Naoya Horiguchi <naoya.horiguchi@linux.dev> Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com> Cc: Yang Shi <shy828301@gmail.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 3489dbb) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924 cve CVE-2024-57883 commit-author Liu Shixin <liushixin2@huawei.com> commit 59d9094 upstream-diff Stable linux-5.15.y backport 8410996eb6fea116fe1483ed977aacf580eee7b4 was used for the actual (clean) cherry-pick. The newly inserted field `atomic_t pt_share_count' preserves the intent of the `RH_KABI_REPLACE' macro call it's embedded in: the field has the same type as another field in the union `atomic_t pt_frag_refcount', thus preserving its size and offset. Additionally, this union serves as a scratch space for the subsystems using the struct page. Upon releasing the ownership to buddy allocator the union contents no longer matter. When the page is allocated again the scratch space will be used by the new owner in its own way. The folio refcount may be increased unexpectly through try_get_folio() by caller such as split_huge_pages. In huge_pmd_unshare(), we use refcount to check whether a pmd page table is shared. The check is incorrect if the refcount is increased by the above caller, and this can cause the page table leaked: BUG: Bad page state in process sh pfn:109324 page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x66 pfn:0x109324 flags: 0x17ffff800000000(node=0|zone=2|lastcpupid=0xfffff) page_type: f2(table) raw: 017ffff800000000 0000000000000000 0000000000000000 0000000000000000 raw: 0000000000000066 0000000000000000 00000000f2000000 0000000000000000 page dumped because: nonzero mapcount ... CPU: 31 UID: 0 PID: 7515 Comm: sh Kdump: loaded Tainted: G B 6.13.0-rc2master+ ctrliq#7 Tainted: [B]=BAD_PAGE Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: show_stack+0x20/0x38 (C) dump_stack_lvl+0x80/0xf8 dump_stack+0x18/0x28 bad_page+0x8c/0x130 free_page_is_bad_report+0xa4/0xb0 free_unref_page+0x3cc/0x620 __folio_put+0xf4/0x158 split_huge_pages_all+0x1e0/0x3e8 split_huge_pages_write+0x25c/0x2d8 full_proxy_write+0x64/0xd8 vfs_write+0xcc/0x280 ksys_write+0x70/0x110 __arm64_sys_write+0x24/0x38 invoke_syscall+0x50/0x120 el0_svc_common.constprop.0+0xc8/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x34/0x128 el0t_64_sync_handler+0xc8/0xd0 el0t_64_sync+0x190/0x198 The issue may be triggered by damon, offline_page, page_idle, etc, which will increase the refcount of page table. 1. The page table itself will be discarded after reporting the "nonzero mapcount". 2. The HugeTLB page mapped by the page table miss freeing since we treat the page table as shared and a shared page table will not be unmapped. Fix it by introducing independent PMD page table shared count. As described by comment, pt_index/pt_mm/pt_frag_refcount are used for s390 gmap, x86 pgds and powerpc, pt_share_count is used for x86/arm64/riscv pmds, so we can reuse the field as pt_share_count. Link: https://lkml.kernel.org/r/20241216071147.3984217-1-liushixin2@huawei.com Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page") Signed-off-by: Liu Shixin <liushixin2@huawei.com> Cc: Kefeng Wang <wangkefeng.wang@huawei.com> Cc: Ken Chen <kenneth.w.chen@intel.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nanyong Sun <sunnanyong@huawei.com> Cc: Jane Chu <jane.chu@oracle.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 8410996eb6fea116fe1483ed977aacf580eee7b4) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572 cve CVE-2025-38084 commit-author Jann Horn <jannh@google.com> commit 081056d upstream-diff Used linux-5.10.y backport e8847d18cd9fff1edbb45e963d9141273c3b539c for the clean pick Currently, __split_vma() triggers hugetlb page table unsharing through vm_ops->may_split(). This happens before the VMA lock and rmap locks are taken - which is too early, it allows racing VMA-locked page faults in our process and racing rmap walks from other processes to cause page tables to be shared again before we actually perform the split. Fix it by explicitly calling into the hugetlb unshare logic from __split_vma() in the same place where THP splitting also happens. At that point, both the VMA and the rmap(s) are write-locked. An annoying detail is that we can now call into the helper hugetlb_unshare_pmds() from two different locking contexts: 1. from hugetlb_split(), holding: - mmap lock (exclusively) - VMA lock - file rmap lock (exclusively) 2. hugetlb_unshare_all_pmds(), which I think is designed to be able to call us with only the mmap lock held (in shared mode), but currently only runs while holding mmap lock (exclusively) and VMA lock Backporting note: This commit fixes a racy protection that was introduced in commit b30c14c ("hugetlb: unshare some PMDs when splitting VMAs"); that commit claimed to fix an issue introduced in 5.13, but it should actually also go all the way back. [jannh@google.com: v2] Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-1-1329349bad1a@google.com Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-0-1329349bad1a@google.com Link: https://lkml.kernel.org/r/20250527-hugetlb-fixes-splitrace-v1-1-f4136f5ec58a@google.com Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page") Signed-off-by: Jann Horn <jannh@google.com> Cc: Liam Howlett <liam.howlett@oracle.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Reviewed-by: Oscar Salvador <osalvador@suse.de> Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: <stable@vger.kernel.org> [b30c14c: hugetlb: unshare some PMDs when splitting VMAs] Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit e8847d18cd9fff1edbb45e963d9141273c3b539c) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71581 cve CVE-2025-38085 commit-author Jann Horn <jannh@google.com> commit 1013af4 upstream-diff Used linux-5.10.y backport 952596b08c74e8fe9e2883d1dc8a8f54a37384ec for the clean pick. Exposed the `tlb_remove_table_sync_one' function for use in the solution (missing 2ba99c5 ("mm/khugepaged: fix GUP-fast interaction by sending IPI")). huge_pmd_unshare() drops a reference on a page table that may have previously been shared across processes, potentially turning it into a normal page table used in another process in which unrelated VMAs can afterwards be installed. If this happens in the middle of a concurrent gup_fast(), gup_fast() could end up walking the page tables of another process. While I don't see any way in which that immediately leads to kernel memory corruption, it is really weird and unexpected. Fix it with an explicit broadcast IPI through tlb_remove_table_sync_one(), just like we do in khugepaged when removing page tables for a THP collapse. Link: https://lkml.kernel.org/r/20250528-hugetlb-fixes-splitrace-v2-2-1329349bad1a@google.com Link: https://lkml.kernel.org/r/20250527-hugetlb-fixes-splitrace-v1-2-f4136f5ec58a@google.com Fixes: 39dde65 ("[PATCH] shared page table for hugetlb page") Signed-off-by: Jann Horn <jannh@google.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Cc: Liam Howlett <liam.howlett@oracle.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Oscar Salvador <osalvador@suse.de> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 952596b08c74e8fe9e2883d1dc8a8f54a37384ec) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-71572 cve-bf CVE-2025-38084 commit-author Jane Chu <jane.chu@oracle.com> commit 59d9094 ("mm: hugetlb: independent PMD page table shared commit 14967a9 upstream-diff Used linux-5.10.y backport ee67c4ac1aed403688dd359918099a745b60fb69 for the clean pick count") introduced ->pt_share_count dedicated to hugetlb PMD share count tracking, but omitted fixing copy_hugetlb_page_range(), leaving the function relying on page_count() for tracking that no longer works. When lazy page table copy for hugetlb is disabled, that is, revert commit bcd51a3 ("hugetlb: lazy page table copies in fork()") fork()'ing with hugetlb PMD sharing quickly lockup - [ 239.446559] watchdog: BUG: soft lockup - CPU#75 stuck for 27s! [ 239.446611] RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x2e0 [ 239.446631] Call Trace: [ 239.446633] <TASK> [ 239.446636] _raw_spin_lock+0x3f/0x60 [ 239.446639] copy_hugetlb_page_range+0x258/0xb50 [ 239.446645] copy_page_range+0x22b/0x2c0 [ 239.446651] dup_mmap+0x3e2/0x770 [ 239.446654] dup_mm.constprop.0+0x5e/0x230 [ 239.446657] copy_process+0xd17/0x1760 [ 239.446660] kernel_clone+0xc0/0x3e0 [ 239.446661] __do_sys_clone+0x65/0xa0 [ 239.446664] do_syscall_64+0x82/0x930 [ 239.446668] ? count_memcg_events+0xd2/0x190 [ 239.446671] ? syscall_trace_enter+0x14e/0x1f0 [ 239.446676] ? syscall_exit_work+0x118/0x150 [ 239.446677] ? arch_exit_to_user_mode_prepare.constprop.0+0x9/0xb0 [ 239.446681] ? clear_bhb_loop+0x30/0x80 [ 239.446684] ? clear_bhb_loop+0x30/0x80 [ 239.446686] entry_SYSCALL_64_after_hwframe+0x76/0x7e There are two options to resolve the potential latent issue: 1. warn against PMD sharing in copy_hugetlb_page_range(), 2. fix it. This patch opts for the second option. While at it, simplify the comment, the details are not actually relevant anymore. Link: https://lkml.kernel.org/r/20250916004520.1604530-1-jane.chu@oracle.com Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count") Signed-off-by: Jane Chu <jane.chu@oracle.com> Reviewed-by: Harry Yoo <harry.yoo@oracle.com> Acked-by: Oscar Salvador <osalvador@suse.de> Acked-by: David Hildenbrand <david@redhat.com> Cc: Jann Horn <jannh@google.com> Cc: Liu Shixin <liushixin2@huawei.com> Cc: Muchun Song <muchun.song@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit ee67c4ac1aed403688dd359918099a745b60fb69) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-193890 cve-bf CVE-2026-23100 commit-author David Hildenbrand (Red Hat) <david@kernel.org> commit ca1a47c upstream-diff Used linux-5.10.y backport 8ae48255bcb17b32436be97553dca848730d365f for the clean pick Patch series "mm/hugetlb: fixes for PMD table sharing (incl. using mmu_gather)", v3. One functional fix, one performance regression fix, and two related comment fixes. I cleaned up my prototype I recently shared [1] for the performance fix, deferring most of the cleanups I had in the prototype to a later point. While doing that I identified the other things. The goal of this patch set is to be backported to stable trees "fairly" easily. At least patch ctrliq#1 and ctrliq#4. Patch ctrliq#1 fixes hugetlb_pmd_shared() not detecting any sharing Patch ctrliq#2 + ctrliq#3 are simple comment fixes that patch ctrliq#4 interacts with. Patch ctrliq#4 is a fix for the reported performance regression due to excessive IPI broadcasts during fork()+exit(). The last patch is all about TLB flushes, IPIs and mmu_gather. Read: complicated There are plenty of cleanups in the future to be had + one reasonable optimization on x86. But that's all out of scope for this series. Runtime tested, with a focus on fixing the performance regression using the original reproducer [2] on x86. This patch (of 4): We switched from (wrongly) using the page count to an independent shared count. Now, shared page tables have a refcount of 1 (excluding speculative references) and instead use ptdesc->pt_share_count to identify sharing. We didn't convert hugetlb_pmd_shared(), so right now, we would never detect a shared PMD table as such, because sharing/unsharing no longer touches the refcount of a PMD table. Page migration, like mbind() or migrate_pages() would allow for migrating folios mapped into such shared PMD tables, even though the folios are not exclusive. In smaps we would account them as "private" although they are "shared", and we would be wrongly setting the PM_MMAP_EXCLUSIVE in the pagemap interface. Fix it by properly using ptdesc_pmd_is_shared() in hugetlb_pmd_shared(). Link: https://lkml.kernel.org/r/20251223214037.580860-1-david@kernel.org Link: https://lkml.kernel.org/r/20251223214037.580860-2-david@kernel.org Link: https://lore.kernel.org/all/8cab934d-4a56-44aa-b641-bfd7e23bd673@kernel.org/ [1] Link: https://lore.kernel.org/all/8cab934d-4a56-44aa-b641-bfd7e23bd673@kernel.org/ [2] Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count") Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org> Reviewed-by: Rik van Riel <riel@surriel.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Tested-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Harry Yoo <harry.yoo@oracle.com> Tested-by: Laurence Oberman <loberman@redhat.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Acked-by: Oscar Salvador <osalvador@suse.de> Cc: Liu Shixin <liushixin2@huawei.com> Cc: Uschakow, Stanislav" <suschako@amazon.de> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 8ae48255bcb17b32436be97553dca848730d365f) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924 cve-bf CVE-2024-57883 commit-author David Hildenbrand (Red Hat) <david@kernel.org> commit 3937027 upstream-diff Used linux-5.10.y backport 44b18bac2b3063784ebb871a922803b11d9c339d for the clean pick Ever since we stopped using the page count to detect shared PMD page tables, these comments are outdated. The only reason we have to flush the TLB early is because once we drop the i_mmap_rwsem, the previously shared page table could get freed (to then get reallocated and used for other purpose). So we really have to flush the TLB before that could happen. So let's simplify the comments a bit. The "If we unshared PMDs, the TLB flush was not recorded in mmu_gather." part introduced as in commit a4a118f ("hugetlbfs: flush TLBs correctly after huge_pmd_unshare") was confusing: sure it is recorded in the mmu_gather, otherwise tlb_flush_mmu_tlbonly() wouldn't do anything. So let's drop that comment while at it as well. We'll centralize these comments in a single helper as we rework the code next. Link: https://lkml.kernel.org/r/20251223214037.580860-3-david@kernel.org Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count") Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org> Reviewed-by: Rik van Riel <riel@surriel.com> Tested-by: Laurence Oberman <loberman@redhat.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Acked-by: Oscar Salvador <osalvador@suse.de> Reviewed-by: Harry Yoo <harry.yoo@oracle.com> Cc: Liu Shixin <liushixin2@huawei.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: "Uschakow, Stanislav" <suschako@amazon.de> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 44b18bac2b3063784ebb871a922803b11d9c339d) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
jira VULN-46924 cve-bf CVE-2024-57883 commit-author David Hildenbrand (Red Hat) <david@kernel.org> commit a8682d5 upstream-diff Used linux-5.10.y backport 2723af9e1e283cee312946c3d6cb20d2c731c139 for the clean pick PMD page table unsharing no longer touches the refcount of a PMD page table. Also, it is not about dropping the refcount of a "PMD page" but the "PMD page table". Let's just simplify by saying that the PMD page table was unmapped, consequently also unmapping the folio that was mapped into this page. This code should be deduplicated in the future. Link: https://lkml.kernel.org/r/20251223214037.580860-4-david@kernel.org Fixes: 59d9094 ("mm: hugetlb: independent PMD page table shared count") Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org> Reviewed-by: Rik van Riel <riel@surriel.com> Tested-by: Laurence Oberman <loberman@redhat.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Acked-by: Oscar Salvador <osalvador@suse.de> Cc: Liu Shixin <liushixin2@huawei.com> Cc: Harry Yoo <harry.yoo@oracle.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: "Uschakow, Stanislav" <suschako@amazon.de> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 2723af9e1e283cee312946c3d6cb20d2c731c139) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
…ing mmu_gather jira VULN-71581 cve-bf CVE-2025-38085 commit-author David Hildenbrand (Red Hat) <david@kernel.org> commit 8ce720d upstream-diff Used linux-5.10.y backport 6495204f8219a57859669d618b25d06176d5a872 for an almost clean pick. Resolved conflicts in the `hugetlb_unshare_pmds' function: unlike ciqlts8_6 the linux-5.10.y version doesn't have 4ddb4d9 ("hugetlb: do not update address in huge_pmd_unshare") backported. That commit changed the signature of `huge_pmd_unshare()', converting the third argument `unsigned long *addr' to `unsigned long addr'. This allowed for code reduction in `hugetlb_unshare_all_pmds()' by dropping the use of temporary `address' placeholder protecting that variable from being changed by the `huge_pmd_unshare()' call. From the backported commit perspective this is irrelevant though, because it's only concerned with the change of the first argument of `huge_pmd_unshare()'. Applied the first argument change while preserving `&address' used as the third. As reported, ever since commit 1013af4 ("mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race") we can end up in some situations where we perform so many IPI broadcasts when unsharing hugetlb PMD page tables that it severely regresses some workloads. In particular, when we fork()+exit(), or when we munmap() a large area backed by many shared PMD tables, we perform one IPI broadcast per unshared PMD table. There are two optimizations to be had: (1) When we process (unshare) multiple such PMD tables, such as during exit(), it is sufficient to send a single IPI broadcast (as long as we respect locking rules) instead of one per PMD table. Locking prevents that any of these PMD tables could get reused before we drop the lock. (2) When we are not the last sharer (> 2 users including us), there is no need to send the IPI broadcast. The shared PMD tables cannot become exclusive (fully unshared) before an IPI will be broadcasted by the last sharer. Concurrent GUP-fast could walk into a PMD table just before we unshared it. It could then succeed in grabbing a page from the shared page table even after munmap() etc succeeded (and supressed an IPI). But there is not difference compared to GUP-fast just sleeping for a while after grabbing the page and re-enabling IRQs. Most importantly, GUP-fast will never walk into page tables that are no-longer shared, because the last sharer will issue an IPI broadcast. (if ever required, checking whether the PUD changed in GUP-fast after grabbing the page like we do in the PTE case could handle this) So let's rework PMD sharing TLB flushing + IPI sync to use the mmu_gather infrastructure so we can implement these optimizations and demystify the code at least a bit. Extend the mmu_gather infrastructure to be able to deal with our special hugetlb PMD table sharing implementation. To make initialization of the mmu_gather easier when working on a single VMA (in particular, when dealing with hugetlb), provide tlb_gather_mmu_vma(). We'll consolidate the handling for (full) unsharing of PMD tables in tlb_unshare_pmd_ptdesc() and tlb_flush_unshared_tables(), and track in "struct mmu_gather" whether we had (full) unsharing of PMD tables. Because locking is very special (concurrent unsharing+reuse must be prevented), we disallow deferring flushing to tlb_finish_mmu() and instead require an explicit earlier call to tlb_flush_unshared_tables(). From hugetlb code, we call huge_pmd_unshare_flush() where we make sure that the expected lock protecting us from concurrent unsharing+reuse is still held. Check with a VM_WARN_ON_ONCE() in tlb_finish_mmu() that tlb_flush_unshared_tables() was properly called earlier. Document it all properly. Notes about tlb_remove_table_sync_one() interaction with unsharing: There are two fairly tricky things: (1) tlb_remove_table_sync_one() is a NOP on architectures without CONFIG_MMU_GATHER_RCU_TABLE_FREE. Here, the assumption is that the previous TLB flush would send an IPI to all relevant CPUs. Careful: some architectures like x86 only send IPIs to all relevant CPUs when tlb->freed_tables is set. The relevant architectures should be selecting MMU_GATHER_RCU_TABLE_FREE, but x86 might not do that in stable kernels and it might have been problematic before this patch. Also, the arch flushing behavior (independent of IPIs) is different when tlb->freed_tables is set. Do we have to enlighten them to also take care of tlb->unshared_tables? So far we didn't care, so hopefully we are fine. Of course, we could be setting tlb->freed_tables as well, but that might then unnecessarily flush too much, because the semantics of tlb->freed_tables are a bit fuzzy. This patch changes nothing in this regard. (2) tlb_remove_table_sync_one() is not a NOP on architectures with CONFIG_MMU_GATHER_RCU_TABLE_FREE that actually don't need a sync. Take x86 as an example: in the common case (!pv, !X86_FEATURE_INVLPGB) we still issue IPIs during TLB flushes and don't actually need the second tlb_remove_table_sync_one(). This optimized can be implemented on top of this, by checking e.g., in tlb_remove_table_sync_one() whether we really need IPIs. But as described in (1), it really must honor tlb->freed_tables then to send IPIs to all relevant CPUs. Notes on TLB flushing changes: (1) Flushing for non-shared PMD tables We're converting from flush_hugetlb_tlb_range() to tlb_remove_huge_tlb_entry(). Given that we properly initialize the MMU gather in tlb_gather_mmu_vma() to be hugetlb aware, similar to __unmap_hugepage_range(), that should be fine. (2) Flushing for shared PMD tables We're converting from various things (flush_hugetlb_tlb_range(), tlb_flush_pmd_range(), flush_tlb_range()) to tlb_flush_pmd_range(). tlb_flush_pmd_range() achieves the same that tlb_remove_huge_tlb_entry() would achieve in these scenarios. Note that tlb_remove_huge_tlb_entry() also calls __tlb_remove_tlb_entry(), however that is only implemented on powerpc, which does not support PMD table sharing. Similar to (1), tlb_gather_mmu_vma() should make sure that TLB flushing keeps on working as expected. Further, note that the ptdesc_pmd_pts_dec() in huge_pmd_share() is not a concern, as we are holding the i_mmap_lock the whole time, preventing concurrent unsharing. That ptdesc_pmd_pts_dec() usage will be removed separately as a cleanup later. There are plenty more cleanups to be had, but they have to wait until this is fixed. [david@kernel.org: fix kerneldoc] Link: https://lkml.kernel.org/r/f223dd74-331c-412d-93fc-69e360a5006c@kernel.org Link: https://lkml.kernel.org/r/20251223214037.580860-5-david@kernel.org Fixes: 1013af4 ("mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race") Signed-off-by: David Hildenbrand (Red Hat) <david@kernel.org> Reported-by: Uschakow, Stanislav" <suschako@amazon.de> Closes: https://lore.kernel.org/all/4d3878531c76479d9f8ca9789dc6485d@amazon.de/ Tested-by: Laurence Oberman <loberman@redhat.com> Acked-by: Harry Yoo <harry.yoo@oracle.com> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liu Shixin <liushixin2@huawei.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Rik van Riel <riel@surriel.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> (cherry picked from commit 6495204f8219a57859669d618b25d06176d5a872) Signed-off-by: Marcin Wcisło <marcin.wcislo@conclusive.pl>
|
🤖 Validation Checks In Progress Workflow run: https://github.com/ctrliq/kernel-src-tree/actions/runs/30552170473 |
🔍 Upstream Linux Kernel Commit Check
This is an automated message from the kernel commit checker workflow. |
🔍 Interdiff Analysis
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -203,8 +203,8 @@
select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
- select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT
- select HAVE_RCU_TABLE_INVALIDATE if MMU_GATHER_RCU_TABLE_FREE
+ select HAVE_RCU_TABLE_FREE if PARAVIRT
+ select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -165,7 +165,7 @@
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_FUNCTION_ARG_ACCESS_API
select HAVE_FUTEX_CMPXCHG if FUTEX
- select HAVE_RCU_TABLE_FREE
+ select MMU_GATHER_RCU_TABLE_FREE
select HAVE_RSEQ
select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -203,7 +203,7 @@
select HAVE_PCI
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
- select HAVE_RCU_TABLE_FREE if PARAVIRT
+ select MMU_GATHER_RCU_TABLE_FREE if PARAVIRT
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
select HAVE_FUNCTION_ARG_ACCESS_API
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -149,4 +149,6 @@
select HAVE_REGS_AND_STACK_ACCESS_API
+ select HAVE_FUNCTION_ARG_ACCESS_API
+ select HAVE_FUTEX_CMPXCHG if FUTEX
select HAVE_RCU_TABLE_FREE
select HAVE_RSEQ
select HAVE_STACKPROTECTOR
--- b/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -159,5 +159,7 @@
select HAVE_OPROFILE
+ select HAVE_PCI
select HAVE_PERF_EVENTS
select HAVE_RCU_TABLE_FREE
select HAVE_REGS_AND_STACK_ACCESS_API
+ select HAVE_RELIABLE_STACKTRACE
select HAVE_RSEQ
--- b/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -200,8 +200,7 @@
- select HAVE_HARDLOCKUP_DETECTOR_PERF if PERF_EVENTS && HAVE_PERF_EVENTS_NMI
+ select HAVE_PCI
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_RCU_TABLE_FREE if PARAVIRT
- select HAVE_RCU_TABLE_INVALIDATE if HAVE_RCU_TABLE_FREE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
- select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR
+ select HAVE_FUNCTION_ARG_ACCESS_API
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -238,7 +297,6 @@
{
#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
tlb_table_flush(tlb);
-#endif
#ifndef CONFIG_MMU_GATHER_NO_GATHER
tlb_batch_pages_flush(tlb);
#endif
@@ -279,9 +337,7 @@
tlb->batch_count = 0;
#endif
-#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
- tlb->batch = NULL;
-#endif
+ tlb_table_init(tlb);
#ifdef CONFIG_MMU_GATHER_PAGE_SIZE
tlb->page_size = 0;
#endif
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/arch/Kconfig
+++ b/arch/Kconfig
@@ -375,3 +375,3 @@
-config HAVE_MMU_GATHER_PAGE_SIZE
+config MMU_GATHER_PAGE_SIZE
bool
@@ -378,6 +378,6 @@
-config HAVE_MMU_GATHER_NO_GATHER
+config MMU_GATHER_NO_GATHER
bool
-config ARCH_WANT_IRQS_OFF_ACTIVATE_MM
+config ARCH_HAVE_NMI_SAFE_CMPXCHG
bool
--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -127,5 +123,5 @@
- * This ensures we call tlb_flush() every time tlb_change_page_size() actually
- * changes the size and provides mmu_gather::page_size to tlb_flush().
+ * This might be useful if your architecture has size specific TLB
+ * invalidation instructions.
*
* MMU_GATHER_RCU_TABLE_FREE
*
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -89,5 +89,5 @@
-#endif /* HAVE_MMU_GATHER_NO_GATHER */
+#endif /* MMU_GATHER_NO_GATHER */
#ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE
@@ -181,5 +181,5 @@
tlb_table_flush(tlb);
#endif
-#ifndef CONFIG_HAVE_MMU_GATHER_NO_GATHER
+#ifndef CONFIG_MMU_GATHER_NO_GATHER
tlb_batch_pages_flush(tlb);
#endif
@@ -224,5 +224,5 @@
tlb->batch = NULL;
#endif
-#ifdef CONFIG_HAVE_MMU_GATHER_PAGE_SIZE
+#ifdef CONFIG_MMU_GATHER_PAGE_SIZE
tlb->page_size = 0;
#endif
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1248,7 +1247,6 @@
goto out_unlock;
}
- tlb_gather_mmu(&tlb, mm, 0, -1);
if (type == CLEAR_REFS_SOFT_DIRTY) {
for (vma = mm->mmap; vma; vma = vma->vm_next) {
if (!(vma->vm_flags & VM_SOFTDIRTY))
@@ -1263,7 +1262,9 @@
&cp);
if (type == CLEAR_REFS_SOFT_DIRTY)
mmu_notifier_invalidate_range_end(&range);
- tlb_finish_mmu(&tlb, 0, -1);
+ flush_tlb_mm(mm);
+ dec_tlb_flush_pending(mm);
+ }
out_unlock:
mmap_write_unlock(mm);
out_mm:
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1198,3 +1269,4 @@
tlb_finish_mmu(&tlb, 0, -1);
- mmap_read_unlock(mm);
+out_unlock:
+ mmap_write_unlock(mm);
out_mm:
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3834,7 +3834,8 @@
* another vma. So page_count of ptep page is checked instead
* to reliably determine whether pte is shared.
*/
- if (page_count(virt_to_page(dst_pte)) > 1)
+ dst_entry = huge_ptep_get(dst_pte);
+ if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
continue;
dst_ptl = huge_pte_lock(h, dst, dst_pte);
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4801,8 +4800,7 @@
* Check here before taking page table lock, and once again
* after taking the lock below.
*/
- dst_entry = huge_ptep_get(dst_pte);
- if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) {
+ if (page_count(virt_to_page(dst_pte)) > 1) {
addr |= last_addr_mask;
continue;
}
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3784,5 +3786,5 @@
{
pte_t *src_pte, *dst_pte, entry, dst_entry;
struct page *ptepage;
unsigned long addr;
- int cow;
+ bool cow = is_cow_mapping(src_vma->vm_flags);
@@ -3836,10 +4803,10 @@
* after taking the lock below.
*/
dst_entry = huge_ptep_get(dst_pte);
- if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
+ if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) {
+ addr |= last_addr_mask;
continue;
-
- dst_ptl = huge_pte_lock(h, dst, dst_pte);
+ }
src_ptl = huge_pte_lockptr(h, src, src_pte);
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
entry = huge_ptep_get(src_pte);
@@ -3852,4 +4819,4 @@
* sharing with another vma.
*/
;
- } else if (unlikely(is_hugetlb_entry_migration(entry) ||
+ } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) {
@@ -3902,7 +4894,7 @@
- entry = huge_ptep_get(src_pte);
- if (!pte_same(src_pte_old, entry)) {
+ restore_reserve_on_error(h, dst_vma, addr,
+ new);
put_page(new);
/* dst_entry won't change as in child */
goto again;
}
- hugetlb_install_page(vma, dst_pte, addr, new);
+ hugetlb_install_page(dst_vma, dst_pte, addr, new);
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5754,47 +5752,3 @@
spin_unlock(&hugetlb_lock);
}
}
-
-static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
- unsigned long start,
- unsigned long end)
-{
- struct hstate *h = hstate_vma(vma);
- unsigned long sz = huge_page_size(h);
- struct mm_struct *mm = vma->vm_mm;
- struct mmu_notifier_range range;
- unsigned long address;
- spinlock_t *ptl;
- pte_t *ptep;
-
- if (!(vma->vm_flags & VM_MAYSHARE))
- return;
-
- if (start >= end)
- return;
-
- flush_cache_range(vma, start, end);
- /*
- * No need to call adjust_range_if_pmd_sharing_possible(), because
- * we have already done the PUD_SIZE alignment.
- */
- mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
- start, end);
- mmu_notifier_invalidate_range_start(&range);
- i_mmap_lock_write(vma->vm_file->f_mapping);
- for (address = start; address < end; address += PUD_SIZE) {
- ptep = huge_pte_offset(mm, address, sz);
- if (!ptep)
- continue;
- ptl = huge_pte_lock(h, mm, ptep);
- huge_pmd_unshare(mm, vma, &address, ptep);
- spin_unlock(ptl);
- }
- flush_hugetlb_tlb_range(vma, start, end);
- i_mmap_unlock_write(vma->vm_file->f_mapping);
- /*
- * No need to call mmu_notifier_invalidate_range(), see
- * Documentation/mm/mmu_notifier.rst.
- */
- mmu_notifier_invalidate_range_end(&range);
-}
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -94,6 +94,8 @@
static void hugetlb_vma_lock_free(struct vm_area_struct *vma);
static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma);
+static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end);
static inline bool subpool_is_free(struct hugepage_subpool *spool)
{
@@ -7341,11 +7362,9 @@
}
}
-/*
- * This function will unconditionally remove all the shared pmd pgtable entries
- * within the specific vma for a hugetlbfs memory range.
- */
-void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
+static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
+ unsigned long start,
+ unsigned long end)
{
struct hstate *h = hstate_vma(vma);
unsigned long sz = huge_page_size(h);
@@ -7349,7 +7368,7 @@
unsigned long sz = huge_page_size(h);
struct mm_struct *mm = vma->vm_mm;
struct mmu_notifier_range range;
- unsigned long address, start, end;
+ unsigned long address;
spinlock_t *ptl;
pte_t *ptep;
@@ -7353,9 +7372,6 @@
if (!(vma->vm_flags & VM_MAYSHARE))
return;
- start = ALIGN(vma->vm_start, PUD_SIZE);
- end = ALIGN_DOWN(vma->vm_end, PUD_SIZE);
-
if (start >= end)
return;
@@ -7392,6 +7408,16 @@
mmu_notifier_invalidate_range_end(&range);
}
+/*
+ * This function will unconditionally remove all the shared pmd pgtable entries
+ * within the specific vma for a hugetlbfs memory range.
+ */
+void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
+{
+ hugetlb_unshare_pmds(vma, ALIGN(vma->vm_start, PUD_SIZE),
+ ALIGN_DOWN(vma->vm_end, PUD_SIZE));
+}
+
#ifdef CONFIG_CMA
static bool cma_reserve_called __initdata;
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -70,6 +73,3 @@
-/* Forward declaration */
-static int hugetlb_acct_memory(struct hstate *h, long delta);
-
-static inline void unlock_or_release_subpool(struct hugepage_subpool *spool)
+static inline bool subpool_is_free(struct hugepage_subpool *spool)
{
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -1188,6 +1189,18 @@
}
#endif
+#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
+static inline bool hugetlb_pmd_shared(pte_t *pte)
+{
+ return page_count(virt_to_page(pte)) > 1;
+}
+#else
+static inline bool hugetlb_pmd_shared(pte_t *pte)
+{
+ return false;
+}
+#endif
+
bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
#ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -9,4 +9,4 @@
#include <linux/cgroup.h>
#include <linux/list.h>
#include <linux/kref.h>
-#include <asm/pgtable.h>
+#include <linux/pgtable.h>
@@ -1004,3 +1184,6 @@
}
+#endif
+
+bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr);
-#endif /* _LINUX_HUGETLB_H */
+#ifndef __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2157,9 +2157,6 @@
if (!pmd_ptlock_init(page))
return false;
__SetPageTable(page);
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
- atomic_set(&page->pt_share_count, 0);
-#endif
inc_lruvec_page_state(page, NR_PAGETABLE);
return true;
}
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -161,9 +161,6 @@
union {
struct mm_struct *pt_mm; /* x86 pgds only */
atomic_t pt_frag_refcount; /* powerpc */
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
- atomic_t pt_share_count;
-#endif
})
#if ALLOC_SPLIT_PTLOCKS
spinlock_t *ptl;
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5457,7 +5457,7 @@
spte = huge_pte_offset(svma->vm_mm, saddr,
vma_mmu_pagesize(svma));
if (spte) {
- atomic_inc(&virt_to_page(spte)->pt_share_count);
+ ptdesc_pmd_pts_inc(virt_to_ptdesc(spte));
break;
}
}
@@ -5472,7 +5472,7 @@
(pmd_t *)((unsigned long)spte & PAGE_MASK));
mm_inc_nr_pmds(mm);
} else {
- atomic_dec(&virt_to_page(spte)->pt_share_count);
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(spte));
}
spin_unlock(ptl);
out:
@@ -5483,7 +5483,11 @@
/*
* unmap huge page backed by shared pte.
*
- * Called with page table lock held.
+ * Hugetlb pte page is ref counted at the time of mapping. If pte is shared
+ * indicated by page_count > 1, unmap is achieved by clearing pud and
+ * decrementing the ref count. If count == 1, the pte page is not shared.
+ *
+ * Called with page table lock held and i_mmap_rwsem held in write mode.
*
* returns: 1 successfully unmapped a shared pte page
* 0 the underlying pte page is not shared, or it is the last user
@@ -5491,19 +5495,17 @@
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long *addr, pte_t *ptep)
{
- unsigned long sz = huge_page_size(hstate_vma(vma));
pgd_t *pgd = pgd_offset(mm, *addr);
p4d_t *p4d = p4d_offset(pgd, *addr);
pud_t *pud = pud_offset(p4d, *addr);
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
- if (sz != PMD_SIZE)
- return 0;
- if (!atomic_read(&virt_to_page(ptep)->pt_share_count))
+ BUG_ON(page_count(virt_to_page(ptep)) == 0);
+ if (page_count(virt_to_page(ptep)) == 1)
return 0;
pud_clear(pud);
- atomic_dec(&virt_to_page(ptep)->pt_share_count);
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
mm_dec_nr_pmds(mm);
*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
return 1;
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3125,6 +3125,7 @@
if (!pmd_ptlock_init(ptdesc))
return false;
__folio_set_pgtable(folio);
+ ptdesc_pmd_pts_init(ptdesc);
lruvec_stat_add_folio(folio, NR_PAGETABLE);
return true;
}
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -445,6 +445,7 @@
* @pt_index: Used for s390 gmap.
* @pt_mm: Used for x86 pgds.
* @pt_frag_refcount: For fragmented page table tracking. Powerpc only.
+ * @pt_share_count: Used for HugeTLB PMD page table share count.
* @_pt_pad_2: Padding to ensure proper alignment.
* @ptl: Lock for the page table.
* @__page_type: Same as page->page_type. Unused for page tables.
@@ -471,6 +472,9 @@
pgoff_t pt_index;
struct mm_struct *pt_mm;
atomic_t pt_frag_refcount;
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+ atomic_t pt_share_count;
+#endif
};
union {
@@ -516,6 +520,32 @@
const struct page *: (const struct ptdesc *)(p), \
struct page *: (struct ptdesc *)(p)))
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+ atomic_set(&ptdesc->pt_share_count, 0);
+}
+
+static inline void ptdesc_pmd_pts_inc(struct ptdesc *ptdesc)
+{
+ atomic_inc(&ptdesc->pt_share_count);
+}
+
+static inline void ptdesc_pmd_pts_dec(struct ptdesc *ptdesc)
+{
+ atomic_dec(&ptdesc->pt_share_count);
+}
+
+static inline int ptdesc_pmd_pts_count(struct ptdesc *ptdesc)
+{
+ return atomic_read(&ptdesc->pt_share_count);
+}
+#else
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+}
+#endif
+
/*
* Used for sizing the vmemmap region on some architectures
*/
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7238,10 +7238,6 @@
/*
* unmap huge page backed by shared pte.
*
- * Hugetlb pte page is ref counted at the time of mapping. If pte is shared
- * indicated by page_count > 1, unmap is achieved by clearing pud and
- * decrementing the ref count. If count == 1, the pte page is not shared.
- *
* Called with page table lock held.
*
* returns: 1 successfully unmapped a shared pte page
@@ -7250,6 +7246,7 @@
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep)
{
+ unsigned long sz = huge_page_size(hstate_vma(vma));
pgd_t *pgd = pgd_offset(mm, addr);
p4d_t *p4d = p4d_offset(pgd, addr);
pud_t *pud = pud_offset(p4d, addr);
@@ -7253,8 +7250,9 @@
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
hugetlb_vma_assert_locked(vma);
- BUG_ON(page_count(virt_to_page(ptep)) == 0);
- if (page_count(virt_to_page(ptep)) == 1)
+ if (sz != PMD_SIZE)
+ return 0;
+ if (!ptdesc_pmd_pts_count(virt_to_ptdesc(ptep)))
return 0;
pud_clear(pud);
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2154,5 +2155,5 @@
return false;
- __SetPageTable(page);
- inc_lruvec_page_state(page, NR_PAGETABLE);
+ __folio_set_pgtable(folio);
+ lruvec_stat_add_folio(folio, NR_PAGETABLE);
return true;
}
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5476,3 +7230,3 @@
}
- spin_unlock(ptl);
+ spin_unlock(&mm->page_table_lock);
out:
@@ -5488,5 +7238,5 @@
* decrementing the ref count. If count == 1, the pte page is not shared.
*
- * Called with page table lock held and i_mmap_rwsem held in write mode.
+ * Called with page table lock held.
*
* returns: 1 successfully unmapped a shared pte page
@@ -5494,11 +7243,12 @@
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+ unsigned long addr, pte_t *ptep)
{
- pgd_t *pgd = pgd_offset(mm, *addr);
- p4d_t *p4d = p4d_offset(pgd, *addr);
- pud_t *pud = pud_offset(p4d, *addr);
+ pgd_t *pgd = pgd_offset(mm, addr);
+ p4d_t *p4d = p4d_offset(pgd, addr);
+ pud_t *pud = pud_offset(p4d, addr);
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+ hugetlb_vma_assert_locked(vma);
BUG_ON(page_count(virt_to_page(ptep)) == 0);
if (page_count(virt_to_page(ptep)) == 1)
return 0;
@@ -5509,3 +7258,2 @@
mm_dec_nr_pmds(mm);
- *addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
return 1;
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -189,8 +189,6 @@
unsigned long address, unsigned long end, pgprot_t newprot);
bool is_hugetlb_entry_migration(pte_t pte);
-void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
-void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
#else /* !CONFIG_HUGETLB_PAGE */
@@ -372,8 +370,6 @@
return 0;
}
-static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }
-
static inline void hugetlb_split(struct vm_area_struct *vma, unsigned long addr) {}
#endif /* !CONFIG_HUGETLB_PAGE */
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3695,7 +3695,7 @@
* MM, VMA and rmap all write-locked to prevent concurrent page table
* walks (except hardware and gup_fast()).
*/
- mmap_assert_write_locked(vma->vm_mm);
+ vma_assert_write_locked(vma);
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
if (addr & ~PUD_MASK) {
@@ -5800,11 +5800,7 @@
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
start, end);
mmu_notifier_invalidate_range_start(&range);
- if (take_locks) {
- i_mmap_lock_write(vma->vm_file->f_mapping);
- } else {
- i_mmap_assert_write_locked(vma->vm_file->f_mapping);
- }
+ i_mmap_lock_write(vma->vm_file->f_mapping);
for (address = start; address < end; address += PUD_SIZE) {
ptep = huge_pte_offset(mm, address, sz);
if (!ptep)
@@ -5814,9 +5810,7 @@
spin_unlock(ptl);
}
flush_hugetlb_tlb_range(vma, start, end);
- if (take_locks) {
- i_mmap_unlock_write(vma->vm_file->f_mapping);
- }
+ i_mmap_unlock_write(vma->vm_file->f_mapping);
/*
* No need to call mmu_notifier_invalidate_range(), see
* Documentation/mm/mmu_notifier.rst.
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -279,6 +279,7 @@
bool is_hugetlb_entry_hwpoisoned(pte_t pte);
void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
void fixup_hugetlb_reservations(struct vm_area_struct *vma);
+void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
#else /* !CONFIG_HUGETLB_PAGE */
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -121,7 +121,7 @@
static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma);
static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
- unsigned long start, unsigned long end);
+ unsigned long start, unsigned long end, bool take_locks);
static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
static void hugetlb_free_folio(struct folio *folio)
@@ -7932,8 +7953,12 @@
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
start, end);
mmu_notifier_invalidate_range_start(&range);
- hugetlb_vma_lock_write(vma);
- i_mmap_lock_write(vma->vm_file->f_mapping);
+ if (take_locks) {
+ hugetlb_vma_lock_write(vma);
+ i_mmap_lock_write(vma->vm_file->f_mapping);
+ } else {
+ i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+ }
for (address = start; address < end; address += PUD_SIZE) {
ptep = hugetlb_walk(vma, address, sz);
if (!ptep)
@@ -7943,8 +7968,10 @@
spin_unlock(ptl);
}
flush_hugetlb_tlb_range(vma, start, end);
- i_mmap_unlock_write(vma->vm_file->f_mapping);
- hugetlb_vma_unlock_write(vma);
+ if (take_locks) {
+ i_mmap_unlock_write(vma->vm_file->f_mapping);
+ hugetlb_vma_unlock_write(vma);
+ }
/*
* No need to call mmu_notifier_arch_invalidate_secondary_tlbs(), see
* Documentation/mm/mmu_notifier.rst.
@@ -7959,7 +7986,8 @@
void hugetlb_unshare_all_pmds(struct vm_area_struct *vma)
{
hugetlb_unshare_pmds(vma, ALIGN(vma->vm_start, PUD_SIZE),
- ALIGN_DOWN(vma->vm_end, PUD_SIZE));
+ ALIGN_DOWN(vma->vm_end, PUD_SIZE),
+ /* take_locks = */ true);
}
/*
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -367,5 +458,5 @@
}
#endif /* !CONFIG_HUGETLB_PAGE */
-/*
- * hugepages at page global directory. If arch support
+
+#ifndef pgd_write
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -71,5 +73,5 @@
static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
unsigned long start, unsigned long end);
+static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
-static inline void unlock_or_release_subpool(struct hugepage_subpool *spool)
-{
+static void hugetlb_free_folio(struct folio *folio)
@@ -5776,7 +7908,8 @@
- mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm,
+ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm,
start, end);
mmu_notifier_invalidate_range_start(&range);
+ hugetlb_vma_lock_write(vma);
i_mmap_lock_write(vma->vm_file->f_mapping);
for (address = start; address < end; address += PUD_SIZE) {
- ptep = huge_pte_offset(mm, address, sz);
+ ptep = hugetlb_walk(vma, address, sz);
if (!ptep)
@@ -5786,4 +7919,5 @@
i_mmap_unlock_write(vma->vm_file->f_mapping);
+ hugetlb_vma_unlock_write(vma);
/*
- * No need to call mmu_notifier_invalidate_range(), see
+ * No need to call mmu_notifier_arch_invalidate_secondary_tlbs(), see
* Documentation/mm/mmu_notifier.rst.
================================================================================
* ONLY IN PATCH1 - files not modified by patch2 *
================================================================================
--- b/mm/mmap.c
+++ a/mm/mmap.c
@@ -812,15 +812,7 @@
}
}
again:
- /*
- * Get rid of huge pages and shared page tables straddling the split
- * boundary.
- */
vma_adjust_trans_huge(orig_vma, start, end, adjust_next);
- if (is_vm_hugetlb_page(orig_vma)) {
- hugetlb_split(orig_vma, start);
- hugetlb_split(orig_vma, end);
- }
if (file) {
mapping = file->f_mapping;
================================================================================
* ONLY IN PATCH2 - files not modified by patch1 *
================================================================================
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -539,7 +539,14 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
init_vma_prep(&vp, vma);
vp.insert = new;
vma_prepare(&vp);
+
+ /*
+ * Get rid of huge pages and shared page tables straddling the split
+ * boundary.
+ */
vma_adjust_trans_huge(vma, vma->vm_start, addr, NULL);
+ if (is_vm_hugetlb_page(vma))
+ hugetlb_split(vma, addr);
if (new_below) {
vma->vm_start = addr;
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -932,6 +932,8 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
(void)next;
}
+static inline void hugetlb_split(struct vm_area_struct *, unsigned long) {}
+
static inline void vma_iter_free(struct vma_iterator *vmi)
{
mas_destroy(&vmi->mas);
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7629,6 +7629,13 @@
return 0;
pud_clear(pud);
+ /*
+ * Once our caller drops the rmap lock, some other process might be
+ * using this page table as a normal, non-hugetlb page table.
+ * Wait for pending gup_fast() in other threads to finish before letting
+ * that happen.
+ */
+ tlb_remove_table_sync_one();
ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
mm_dec_nr_pmds(mm);
return 1;
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5514,6 +5514,6 @@
return 0;
pud_clear(pud);
- atomic_dec(&virt_to_page(ptep)->pt_share_count);
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
mm_dec_nr_pmds(mm);
- *addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
+ return 1;
================================================================================
* ONLY IN PATCH1 - files not modified by patch2 *
================================================================================
--- b/include/asm-generic/tlb.h
+++ a/include/asm-generic/tlb.h
@@ -202,16 +202,12 @@
#define tlb_needs_table_invalidate() (true)
#endif
-void tlb_remove_table_sync_one(void);
-
#else
#ifdef tlb_needs_table_invalidate
#error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE
#endif
-static inline void tlb_remove_table_sync_one(void) { }
-
#endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */
--- b/mm/mmu_gather.c
+++ a/mm/mmu_gather.c
@@ -139,7 +139,7 @@
/* Simply deliver the interrupt */
}
+static void tlb_remove_table_sync_one(void)
-void tlb_remove_table_sync_one(void)
{
/*
* This isn't an RCU grace period and hence the page-tables cannot be
@@ -163,6 +163,8 @@
#else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */
+static void tlb_remove_table_sync_one(void) { }
+
static void tlb_remove_table_free(struct mmu_table_batch *batch)
{
__tlb_remove_table_free(batch);
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3861,11 +3861,16 @@
break;
}
-#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
- /* If the pagetables are shared, there is nothing to do */
- if (atomic_read(&virt_to_page(dst_pte)->pt_share_count))
+ /*
+ * If the pagetables are shared don't copy or take references.
+ *
+ * dst_pte == src_pte is the common case of src/dest sharing.
+ * However, src could have 'unshared' and dst shares with
+ * another vma. So page_count of ptep page is checked instead
+ * to reliably determine whether pte is shared.
+ */
+ if (page_count(virt_to_page(dst_pte)) > 1)
continue;
-#endif
dst_ptl = huge_pte_lock(h, dst, dst_pte);
src_ptl = huge_pte_lockptr(h, src, src_pte);
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -7211,7 +7211,7 @@
spte = hugetlb_walk(svma, saddr,
vma_mmu_pagesize(svma));
if (spte) {
- get_page(virt_to_page(spte));
+ ptdesc_pmd_pts_inc(virt_to_ptdesc(spte));
break;
}
}
@@ -7226,7 +7226,7 @@
(pmd_t *)((unsigned long)spte & PAGE_MASK));
mm_inc_nr_pmds(mm);
} else {
- put_page(virt_to_page(spte));
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(spte));
}
spin_unlock(&mm->page_table_lock);
out:
@@ -7238,10 +7238,6 @@
/*
* unmap huge page backed by shared pte.
*
- * Hugetlb pte page is ref counted at the time of mapping. If pte is shared
- * indicated by page_count > 1, unmap is achieved by clearing pud and
- * decrementing the ref count. If count == 1, the pte page is not shared.
- *
* Called with page table lock held.
*
* returns: 1 successfully unmapped a shared pte page
@@ -7250,6 +7246,7 @@
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep)
{
+ unsigned long sz = huge_page_size(hstate_vma(vma));
pgd_t *pgd = pgd_offset(mm, addr);
p4d_t *p4d = p4d_offset(pgd, addr);
pud_t *pud = pud_offset(p4d, addr);
@@ -7253,8 +7250,9 @@
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
hugetlb_vma_assert_locked(vma);
- BUG_ON(page_count(virt_to_page(ptep)) == 0);
- if (page_count(virt_to_page(ptep)) == 1)
+ if (sz != PMD_SIZE)
+ return 0;
+ if (!ptdesc_pmd_pts_count(virt_to_ptdesc(ptep)))
return 0;
pud_clear(pud);
@@ -7258,7 +7256,7 @@
return 0;
pud_clear(pud);
- put_page(virt_to_page(ptep));
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
mm_dec_nr_pmds(mm);
return 1;
}
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3859,14 +3888,12 @@
- /*
- * If the pagetables are shared don't copy or take references.
- *
- * dst_pte == src_pte is the common case of src/dest sharing.
- * However, src could have 'unshared' and dst shares with
- * another vma. So page_count of ptep page is checked instead
- * to reliably determine whether pte is shared.
- */
- if (page_count(virt_to_page(dst_pte)) > 1)
- continue;
+ i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+ hugetlb_vma_assert_locked(vma);
+ BUG_ON(page_count(virt_to_page(ptep)) == 0);
+ if (page_count(virt_to_page(ptep)) == 1)
+ return 0;
- dst_ptl = huge_pte_lock(h, dst, dst_pte);
- src_ptl = huge_pte_lockptr(h, src, src_pte);
+ pud_clear(pud);
+ put_page(virt_to_page(ptep));
+ mm_dec_nr_pmds(mm);
+ return 1;
+}
================================================================================
* ONLY IN PATCH2 - files not modified by patch1 *
================================================================================
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3125,6 +3125,7 @@ static inline bool pagetable_pmd_ctor(struct ptdesc *ptdesc)
if (!pmd_ptlock_init(ptdesc))
return false;
__folio_set_pgtable(folio);
+ ptdesc_pmd_pts_init(ptdesc);
lruvec_stat_add_folio(folio, NR_PAGETABLE);
return true;
}
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -445,6 +445,7 @@ FOLIO_MATCH(compound_head, _head_2a);
* @pt_index: Used for s390 gmap.
* @pt_mm: Used for x86 pgds.
* @pt_frag_refcount: For fragmented page table tracking. Powerpc only.
+ * @pt_share_count: Used for HugeTLB PMD page table share count.
* @_pt_pad_2: Padding to ensure proper alignment.
* @ptl: Lock for the page table.
* @__page_type: Same as page->page_type. Unused for page tables.
@@ -471,6 +472,9 @@ struct ptdesc {
pgoff_t pt_index;
struct mm_struct *pt_mm;
atomic_t pt_frag_refcount;
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+ atomic_t pt_share_count;
+#endif
};
union {
@@ -516,6 +520,32 @@ static_assert(sizeof(struct ptdesc) <= sizeof(struct page));
const struct page *: (const struct ptdesc *)(p), \
struct page *: (struct ptdesc *)(p)))
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+ atomic_set(&ptdesc->pt_share_count, 0);
+}
+
+static inline void ptdesc_pmd_pts_inc(struct ptdesc *ptdesc)
+{
+ atomic_inc(&ptdesc->pt_share_count);
+}
+
+static inline void ptdesc_pmd_pts_dec(struct ptdesc *ptdesc)
+{
+ atomic_dec(&ptdesc->pt_share_count);
+}
+
+static inline int ptdesc_pmd_pts_count(struct ptdesc *ptdesc)
+{
+ return atomic_read(&ptdesc->pt_share_count);
+}
+#else
+static inline void ptdesc_pmd_pts_init(struct ptdesc *ptdesc)
+{
+}
+#endif
+
/*
* Used for sizing the vmemmap region on some architectures
*/
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -1016,7 +1016,7 @@
#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
static inline bool hugetlb_pmd_shared(pte_t *pte)
{
- return atomic_read(&virt_to_page(pte)->pt_share_count);
+ return ptdesc_pmd_is_shared(virt_to_ptdesc(pte));
}
#else
static inline bool hugetlb_pmd_shared(pte_t *pte)
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4069,10 +4069,17 @@
tlb_end_vma(tlb, vma);
/*
- * There is nothing protecting a previously-shared page table that we
- * unshared through huge_pmd_unshare() from getting freed after we
- * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
- * succeeded, flush the range corresponding to the pud.
+ * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
+ * could defer the flush until now, since by holding i_mmap_rwsem we
+ * guaranteed that the last refernece would not be dropped. But we must
+ * do the flushing before we return, as otherwise i_mmap_rwsem will be
+ * dropped and the last reference to the shared PMDs page might be
+ * dropped as well.
+ *
+ * In theory we could defer the freeing of the PMD pages as well, but
+ * huge_pmd_unshare() relies on the exact page_count for the PMD page to
+ * detect sharing, so we cannot defer the release of the page either.
+ * Instead, do flush now.
*/
if (force_flush)
tlb_flush_mmu_tlbonly(tlb);
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5320,17 +5320,10 @@
tlb_end_vma(tlb, vma);
/*
- * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
- * could defer the flush until now, since by holding i_mmap_rwsem we
- * guaranteed that the last reference would not be dropped. But we must
- * do the flushing before we return, as otherwise i_mmap_rwsem will be
- * dropped and the last reference to the shared PMDs page might be
- * dropped as well.
- *
- * In theory we could defer the freeing of the PMD pages as well, but
- * huge_pmd_unshare() relies on the exact page_count for the PMD page to
- * detect sharing, so we cannot defer the release of the page either.
- * Instead, do flush now.
+ * There is nothing protecting a previously-shared page table that we
+ * unshared through huge_pmd_unshare() from getting freed after we
+ * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
+ * succeeded, flush the range corresponding to the pud.
*/
if (force_flush)
tlb_flush_mmu_tlbonly(tlb);
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4068,7 +4068,7 @@
/*
* If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
* could defer the flush until now, since by holding i_mmap_rwsem we
- * guaranteed that the last refernece would not be dropped. But we must
+ * guaranteed that the last reference would not be dropped. But we must
* do the flushing before we return, as otherwise i_mmap_rwsem will be
* dropped and the last reference to the shared PMDs page might be
* dropped as well.
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1508,8 +1508,13 @@
range.end);
/*
- * The PMD table was unmapped,
- * consequently unmapping the folio.
+ * The ref count of the PMD page was dropped
+ * which is part of the way map counting
+ * is done for shared PMDs. Return 'true'
+ * here. When there is no other sharing,
+ * huge_pmd_unshare returns false and we will
+ * unmap the actual page and drop map count
+ * to zero.
*/
page_vma_mapped_walk_done(&pvmw);
break;
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -2016,14 +2016,8 @@
flush_tlb_range(vma,
range.start, range.end);
/*
- * The ref count of the PMD page was
- * dropped which is part of the way map
- * counting is done for shared PMDs.
- * Return 'true' here. When there is
- * no other sharing, huge_pmd_unshare
- * returns false and we will unmap the
- * actual page and drop map count
- * to zero.
+ * The PMD table was unmapped,
+ * consequently unmapping the folio.
*/
goto walk_done;
}
@@ -2416,14 +2410,8 @@
range.start, range.end);
/*
- * The ref count of the PMD page was
- * dropped which is part of the way map
- * counting is done for shared PMDs.
- * Return 'true' here. When there is
- * no other sharing, huge_pmd_unshare
- * returns false and we will unmap the
- * actual page and drop map count
- * to zero.
+ * The PMD table was unmapped,
+ * consequently unmapping the folio.
*/
page_vma_mapped_walk_done(&pvmw);
break;
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1505,12 +1506,13 @@
- /*
- * The ref count of the PMD page was dropped
- * which is part of the way map counting
- * is done for shared PMDs. Return 'true'
- * here. When there is no other sharing,
- * huge_pmd_unshare returns false and we will
- * unmap the actual page and drop map count
- * to zero.
- */
- page_vma_mapped_walk_done(&pvmw);
- break;
+ /*
+ * The ref count of the PMD page was
+ * dropped which is part of the way map
+ * counting is done for shared PMDs.
+ * Return 'true' here. When there is
+ * no other sharing, huge_pmd_unshare
+ * returns false and we will unmap the
+ * actual page and drop map count
+ * to zero.
+ */
+ page_vma_mapped_walk_done(&pvmw);
+ break;
================================================================================
* DELTA DIFFERENCES - code changes that differ between the patches *
================================================================================
--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -50,8 +50,7 @@
*
* The mmu_gather API consists of:
*
- * - tlb_gather_mmu() / tlb_gather_mmu_vma() / tlb_finish_mmu(); start and
- * finish a mmu_gather
+ * - tlb_gather_mmu() / tlb_finish_mmu(); start and finish a mmu_gather
*
* Finish in particular will issue a (final) TLB invalidate and free
* all (remaining) queued pages.
@@ -675,16 +674,16 @@
#endif
#endif
-#if defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && defined(CONFIG_HUGETLB_PAGE)
-static inline void tlb_unshare_pmd_ptdesc(struct mmu_gather *tlb, struct page *pt,
+#ifdef CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING
+static inline void tlb_unshare_pmd_ptdesc(struct mmu_gather *tlb, struct ptdesc *pt,
unsigned long addr)
{
/*
* The caller must make sure that concurrent unsharing + exclusive
* reuse is impossible until tlb_flush_unshared_tables() was called.
*/
- VM_WARN_ON_ONCE(!atomic_read(&pt->pt_share_count));
- atomic_dec(&pt->pt_share_count);
+ VM_WARN_ON_ONCE(!ptdesc_pmd_is_shared(pt));
+ ptdesc_pmd_pts_dec(pt);
/* Clearing a PUD pointing at a PMD table with PMD leaves. */
tlb_flush_pmd_range(tlb, addr & PUD_MASK, PUD_SIZE);
@@ -693,7 +692,7 @@
* If the page table is now exclusively owned, we fully unshared
* a page table.
*/
- if (!atomic_read(&pt->pt_share_count))
+ if (!ptdesc_pmd_is_shared(pt))
tlb->fully_unshared_tables = true;
tlb->unshared_tables = true;
}
@@ -730,7 +729,7 @@
tlb->fully_unshared_tables = false;
}
}
-#endif
+#endif /* CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING */
#endif /* CONFIG_MMU */
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -167,9 +167,8 @@
unsigned long addr, unsigned long sz);
pte_t *huge_pte_offset(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep);
-void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma);
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep);
void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
unsigned long *start, unsigned long *end);
struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -210,8 +209,9 @@
return NULL;
}
-static inline int huge_pmd_unshare(struct mmu_gather *tlb,
- struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep)
+static inline int huge_pmd_unshare(struct mm_struct *mm,
+ struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
return 0;
}
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -635,8 +635,6 @@
struct mmu_gather;
extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
unsigned long start, unsigned long end);
-void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long start, unsigned long end);
extern void tlb_finish_mmu(struct mmu_gather *tlb,
unsigned long start, unsigned long end);
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4006,8 +4007,10 @@
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(tlb, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, &address, ptep)) {
spin_unlock(ptl);
+ tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
+ force_flush = true;
continue;
}
@@ -5088,7 +5098,7 @@
BUG_ON(address >= end);
flush_cache_range(vma, range.start, range.end);
- tlb_gather_mmu_vma(&tlb, vma, range.start, range.end);
+ tlb_gather_mmu_vma(&tlb, vma);
mmu_notifier_invalidate_range_start(&range);
i_mmap_lock_write(vma->vm_file->f_mapping);
@@ -5098,9 +5108,10 @@
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(&tlb, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, &address, ptep)) {
pages++;
spin_unlock(ptl);
+ shared_pmd = true;
continue;
}
pte = huge_ptep_get(ptep);
@@ -5146,7 +5157,7 @@
*/
i_mmap_unlock_write(vma->vm_file->f_mapping);
mmu_notifier_invalidate_range_end(&range);
- tlb_finish_mmu(&tlb, range.start, range.end);
+ tlb_finish_mmu(&tlb);
return pages << h->order;
}
@@ -5470,7 +5481,7 @@
* huge_pmd_unshare - Unmap a pmd table if it is shared by multiple users
* @tlb: the current mmu_gather.
* @vma: the vma covering the pmd table.
- * @addr: pointer to the address we are trying to unshare.
+ * @addr: the address we are trying to unshare.
* @ptep: pointer into the (pmd) page table.
*
* Called with the page table lock held, the i_mmap_rwsem held in write mode
@@ -5482,11 +5493,10 @@
* Returns: 1 if it was a shared PMD table and it got unmapped, or 0 if it
* was not a shared PMD table.
*/
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
unsigned long sz = huge_page_size(hstate_vma(vma));
- struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd = pgd_offset(mm, *addr);
p4d_t *p4d = p4d_offset(pgd, *addr);
pud_t *pud = pud_offset(p4d, *addr);
@@ -5498,36 +5508,18 @@
return 0;
pud_clear(pud);
- tlb_unshare_pmd_ptdesc(tlb, virt_to_page(ptep), *addr);
-
+ /*
+ * Once our caller drops the rmap lock, some other process might be
+ * using this page table as a normal, non-hugetlb page table.
+ * Wait for pending gup_fast() in other threads to finish before letting
+ * that happen.
+ */
+ tlb_remove_table_sync_one();
+ atomic_dec(&virt_to_page(ptep)->pt_share_count);
mm_dec_nr_pmds(mm);
*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
return 1;
}
-
-/*
- * huge_pmd_unshare_flush - Complete a sequence of huge_pmd_unshare() calls
- * @tlb: the current mmu_gather.
- * @vma: the vma covering the pmd table.
- *
- * Perform necessary TLB flushes or IPI broadcasts to synchronize PMD table
- * unsharing with concurrent page table walkers.
- *
- * This function must be called after a sequence of huge_pmd_unshare()
- * calls while still holding the i_mmap_rwsem.
- */
-void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma)
-{
- /*
- * We must synchronize page table unsharing such that nobody will
- * try reusing a previously-shared page table while it might still
- * be in use by previous sharers (TLB, GUP_fast).
- */
- i_mmap_assert_write_locked(vma->vm_file->f_mapping);
-
- tlb_flush_unshared_tables(tlb);
-}
-
#define want_pmd_share() (1)
#else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
@@ -5535,8 +5527,8 @@
return NULL;
}
-int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
return 0;
}
@@ -5804,7 +5796,7 @@
return;
flush_cache_range(vma, start, end);
- tlb_gather_mmu_vma(&tlb, vma, start, end);
+ tlb_gather_mmu_vma(&tlb, vma);
/*
* No need to call adjust_range_if_pmd_sharing_possible(), because
@@ -5823,7 +5815,7 @@
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- huge_pmd_unshare(&tlb, vma, &address, ptep);
+ huge_pmd_unshare(mm, vma, &address, ptep);
spin_unlock(ptl);
}
huge_pmd_unshare_flush(&tlb, vma);
@@ -5837,3 +5829,3 @@
mmu_notifier_invalidate_range_end(&range);
- tlb_finish_mmu(&tlb, start, end);
+ tlb_finish_mmu(&tlb);
}
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -292,8 +291,6 @@
* single VMA
* @tlb: the mmu_gather structure to initialize
* @vma: the vm_area_struct
- * @start: start of the region that will be removed from the page-table
- * @end: end of the region that will be removed from the page-table
*
* Called to initialize an (on-stack) mmu_gather structure for operating on
* a single VMA. In contrast to tlb_gather_mmu(), calling this function will
@@ -305,10 +302,9 @@
* tlb_change_page_size().
*
*/
-void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long start, unsigned long end)
+void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma)
{
- tlb_gather_mmu(tlb, vma->vm_mm, start, end);
+ tlb_gather_mmu(tlb, vma->vm_mm);
tlb_update_vma_flags(tlb, vma);
if (is_vm_hugetlb_page(vma))
/* All entries have the same size. */
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -1488,16 +1488,13 @@
address = pvmw.address;
if (PageHuge(page) && !PageAnon(page)) {
- struct mmu_gather tlb;
-
/*
* To call huge_pmd_unshare, i_mmap_rwsem must be
* held in write mode. Caller needs to explicitly
* do this outside rmap routines.
*/
VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
- tlb_gather_mmu_vma(&tlb, vma, range.start, range.end);
- if (huge_pmd_unshare(&tlb, vma, &address, pvmw.pte)) {
+ if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
/*
* huge_pmd_unshare unmapped an entire PMD
* page. There is no way of knowing exactly
@@ -1506,10 +1503,9 @@
* already adjusted above to cover this range.
*/
flush_cache_range(vma, range.start, range.end);
- huge_pmd_unshare_flush(&tlb, vma);
+ flush_tlb_range(vma, range.start, range.end);
mmu_notifier_invalidate_range(mm, range.start,
range.end);
- tlb_finish_mmu(&tlb, range.start, range.end);
/*
* The PMD table was unmapped,
@@ -1518,7 +1514,6 @@
page_vma_mapped_walk_done(&pvmw);
break;
}
- tlb_finish_mmu(&tlb, range.start, range.end);
}
if (IS_ENABLED(CONFIG_MIGRATION) &&
################################################################################
! REJECTED PATCH2 HUNKS - could not be compared; manual review needed !
################################################################################
--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -46,7 +46,8 @@
*
* The mmu_gather API consists of:
*
- * - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_finish_mmu()
+ * - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_gather_mmu_vma() /
+ * tlb_finish_mmu()
*
* start and finish a mmu_gather
*
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -240,8 +240,9 @@
pte_t *huge_pte_offset(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
unsigned long hugetlb_mask_last_page(struct hstate *h);
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep);
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep);
+void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma);
void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
unsigned long *start, unsigned long *end);
@@ -300,9 +301,8 @@
return NULL;
}
-static inline int huge_pmd_unshare(struct mm_struct *mm,
- struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep)
+static inline int huge_pmd_unshare(struct mmu_gather *tlb,
+ struct vm_area_struct *vma, unsigned long addr, pte_t *ptep)
{
return 0;
}
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1530,6 +1530,7 @@
struct mmu_gather;
extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
+void tlb_gather_mmu_vma(struct mmu_gather *tlb, struct vm_area_struct *vma);
extern void tlb_finish_mmu(struct mmu_gather *tlb);
struct vm_fault;
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -5112,7 +5112,7 @@
unsigned long last_addr_mask;
pte_t *src_pte, *dst_pte;
struct mmu_notifier_range range;
- bool shared_pmd = false;
+ struct mmu_gather tlb;
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, old_addr,
old_end);
@@ -5139,8 +5140,7 @@
if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte)))
continue;
- if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) {
- shared_pmd = true;
+ if (huge_pmd_unshare(&tlb, vma, old_addr, src_pte)) {
old_addr |= last_addr_mask;
new_addr |= last_addr_mask;
continue;
@@ -5151,6 +5151,7 @@
break;
move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte, sz);
+ tlb_remove_huge_tlb_entry(h, &tlb, src_pte, old_addr);
}
if (shared_pmd)
@@ -5153,10 +5154,9 @@
move_huge_pte(vma, old_addr, new_addr, src_pte, dst_pte, sz);
}
- if (shared_pmd)
- flush_hugetlb_tlb_range(vma, range.start, range.end);
- else
- flush_hugetlb_tlb_range(vma, old_end - len, old_end);
+ tlb_flush_mmu_tlbonly(&tlb);
+ huge_pmd_unshare_flush(&tlb, vma);
+
mmu_notifier_invalidate_range_end(&range);
i_mmap_unlock_write(mapping);
hugetlb_vma_unlock_write(vma);
@@ -5160,6 +5160,7 @@
mmu_notifier_invalidate_range_end(&range);
i_mmap_unlock_write(mapping);
hugetlb_vma_unlock_write(vma);
+ tlb_finish_mmu(&tlb);
return len + old_addr - old_end;
}
@@ -5178,7 +5179,6 @@
unsigned long sz = huge_page_size(h);
bool adjust_reservation;
unsigned long last_addr_mask;
- bool force_flush = false;
WARN_ON(!is_vm_hugetlb_page(vma));
BUG_ON(start & ~huge_page_mask(h));
@@ -5201,7 +5201,7 @@
}
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, address, ptep)) {
+ if (huge_pmd_unshare(tlb, vma, address, ptep)) {
spin_unlock(ptl);
tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
force_flush = true;
@@ -5203,8 +5203,6 @@
ptl = huge_pte_lock(h, mm, ptep);
if (huge_pmd_unshare(mm, vma, address, ptep)) {
spin_unlock(ptl);
- tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
- force_flush = true;
address |= last_addr_mask;
continue;
}
@@ -5320,14 +5318,7 @@
}
tlb_end_vma(tlb, vma);
- /*
- * There is nothing protecting a previously-shared page table that we
- * unshared through huge_pmd_unshare() from getting freed after we
- * release i_mmap_rwsem, so flush the TLB now. If huge_pmd_unshare()
- * succeeded, flush the range corresponding to the pud.
- */
- if (force_flush)
- tlb_flush_mmu_tlbonly(tlb);
+ huge_pmd_unshare_flush(tlb, vma);
}
void __hugetlb_zap_begin(struct vm_area_struct *vma,
@@ -6426,7 +6417,6 @@
pte_t pte;
struct hstate *h = hstate_vma(vma);
long pages = 0, psize = huge_page_size(h);
- bool shared_pmd = false;
struct mmu_notifier_range range;
unsigned long last_addr_mask;
bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
@@ -6430,6 +6420,7 @@
unsigned long last_addr_mask;
bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+ struct mmu_gather tlb;
/*
* In the case of shared PMDs, the area to flush could be beyond
@@ -6443,6 +6434,7 @@
BUG_ON(address >= end);
flush_cache_range(vma, range.start, range.end);
+ tlb_gather_mmu_vma(&tlb, vma);
mmu_notifier_invalidate_range_start(&range);
hugetlb_vma_lock_write(vma);
@@ -6469,7 +6461,7 @@
}
}
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, address, ptep)) {
+ if (huge_pmd_unshare(&tlb, vma, address, ptep)) {
/*
* When uffd-wp is enabled on the vma, unshare
* shouldn't happen at all. Warn about it if it
@@ -6478,7 +6470,6 @@
WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
pages++;
spin_unlock(ptl);
- shared_pmd = true;
address |= last_addr_mask;
continue;
}
@@ -6932,8 +6926,8 @@
* returns: 1 successfully unmapped a shared pte page
* 0 the underlying pte page is not shared, or it is the last user
*/
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep)
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep)
{
unsigned long sz = huge_page_size(hstate_vma(vma));
pgd_t *pgd = pgd_offset(mm, addr);
@@ -6936,6 +6930,7 @@
unsigned long addr, pte_t *ptep)
{
unsigned long sz = huge_page_size(hstate_vma(vma));
+ struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd = pgd_offset(mm, addr);
p4d_t *p4d = p4d_offset(pgd, addr);
pud_t *pud = pud_offset(p4d, addr);
@@ -6947,14 +6942,9 @@
i_mmap_assert_write_locked(vma->vm_file->f_mapping);
hugetlb_vma_assert_locked(vma);
pud_clear(pud);
- /*
- * Once our caller drops the rmap lock, some other process might be
- * using this page table as a normal, non-hugetlb page table.
- * Wait for pending gup_fast() in other threads to finish before letting
- * that happen.
- */
- tlb_remove_table_sync_one();
- ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
+
+ tlb_unshare_pmd_ptdesc(tlb, virt_to_ptdesc(ptep), addr);
+
mm_dec_nr_pmds(mm);
return 1;
}
@@ -6958,6 +6948,29 @@
return 1;
}
+/*
+ * huge_pmd_unshare_flush - Complete a sequence of huge_pmd_unshare() calls
+ * @tlb: the current mmu_gather.
+ * @vma: the vma covering the pmd table.
+ *
+ * Perform necessary TLB flushes or IPI broadcasts to synchronize PMD table
+ * unsharing with concurrent page table walkers.
+ *
+ * This function must be called after a sequence of huge_pmd_unshare()
+ * calls while still holding the i_mmap_rwsem.
+ */
+void huge_pmd_unshare_flush(struct mmu_gather *tlb, struct vm_area_struct *vma)
+{
+ /*
+ * We must synchronize page table unsharing such that nobody will
+ * try reusing a previously-shared page table while it might still
+ * be in use by previous sharers (TLB, GUP_fast).
+ */
+ i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+
+ tlb_flush_unshared_tables(tlb);
+}
+
#else /* !CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING */
pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
@@ -6967,8 +6980,8 @@
return NULL;
}
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep)
+int huge_pmd_unshare(struct mmu_gather *tlb, struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep)
{
return 0;
}
@@ -7275,7 +7295,7 @@
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- huge_pmd_unshare(mm, vma, address, ptep);
+ huge_pmd_unshare(&tlb, vma, address, ptep);
spin_unlock(ptl);
}
flush_hugetlb_tlb_range(vma, start, end);
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -10,6 +10,7 @@
#include <linux/swap.h>
#include <linux/rmap.h>
#include <linux/pgalloc.h>
+#include <linux/hugetlb.h>
#include <asm/tlb.h>
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -2008,6 +2008,8 @@
* if unsuccessful.
*/
if (!anon) {
+ struct mmu_gather tlb;
+
VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
if (!hugetlb_vma_trylock_write(vma))
goto walk_abort;
@@ -2011,7 +2013,9 @@
VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
if (!hugetlb_vma_trylock_write(vma))
goto walk_abort;
- if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
+
+ tlb_gather_mmu_vma(&tlb, vma);
+ if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
hugetlb_vma_unlock_write(vma);
flush_tlb_range(vma,
range.start, range.end);
@@ -2013,8 +2017,8 @@
goto walk_abort;
if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
hugetlb_vma_unlock_write(vma);
- flush_tlb_range(vma,
- range.start, range.end);
+ huge_pmd_unshare_flush(&tlb, vma);
+ tlb_finish_mmu(&tlb);
/*
* The PMD table was unmapped,
* consequently unmapping the folio.
@@ -2022,6 +2026,7 @@
goto walk_done;
}
hugetlb_vma_unlock_write(vma);
+ tlb_finish_mmu(&tlb);
}
pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
if (pte_dirty(pteval))
@@ -2398,6 +2403,8 @@
* fail if unsuccessful.
*/
if (!anon) {
+ struct mmu_gather tlb;
+
VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
if (!hugetlb_vma_trylock_write(vma)) {
page_vma_mapped_walk_done(&pvmw);
@@ -2401,10 +2408,6 @@
ret = false;
break;
}
- if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) {
- hugetlb_vma_unlock_write(vma);
- flush_tlb_range(vma,
- range.start, range.end);
/*
* The PMD table was unmapped,
@@ -2406,6 +2409,11 @@
flush_tlb_range(vma,
range.start, range.end);
+ tlb_gather_mmu_vma(&tlb, vma);
+ if (huge_pmd_unshare(&tlb, vma, address, pvmw.pte)) {
+ hugetlb_vma_unlock_write(vma);
+ huge_pmd_unshare_flush(&tlb, vma);
+ tlb_finish_mmu(&tlb);
/*
* The PMD table was unmapped,
* consequently unmapping the folio.
@@ -2417,6 +2425,7 @@
break;
}
hugetlb_vma_unlock_write(vma);
+ tlb_finish_mmu(&tlb);
}
/* Nuke the hugetlb page table entry */
pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
================================================================================
* CONTEXT DIFFERENCES - surrounding code differences between the patches *
================================================================================
--- b/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -45,3 +45,5 @@
*
- * - tlb_gather_mmu() / tlb_finish_mmu(); start and finish a mmu_gather
+ * - tlb_gather_mmu() / tlb_gather_mmu_fullmm() / tlb_finish_mmu()
+ *
+ * start and finish a mmu_gather
*
@@ -291,4 +366,4 @@
unsigned int batch_count;
-#ifndef CONFIG_HAVE_MMU_GATHER_NO_GATHER
+#ifndef CONFIG_MMU_GATHER_NO_GATHER
--- b/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -164,7 +164,8 @@
pte_t *huge_pte_offset(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
+unsigned long hugetlb_mask_last_page(struct hstate *h);
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep);
+ unsigned long addr, pte_t *ptep);
void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
unsigned long *start, unsigned long *end);
-struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
+
@@ -211,7 +302,7 @@
static inline int huge_pmd_unshare(struct mm_struct *mm,
struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+ unsigned long addr, pte_t *ptep)
{
return 0;
}
--- b/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -632,6 +632,5 @@
struct mmu_gather;
-extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
- unsigned long start, unsigned long end);
-extern void tlb_finish_mmu(struct mmu_gather *tlb,
- unsigned long start, unsigned long end);
+extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
+extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
+extern void tlb_finish_mmu(struct mmu_gather *tlb);
--- b/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3977,5 +3979,11 @@
struct mmu_notifier_range range;
- bool force_flush = false;
+ bool shared_pmd = false;
- WARN_ON(!is_vm_hugetlb_page(vma));
- BUG_ON(start & ~huge_page_mask(h));
+ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, old_addr,
+ old_end);
+ * range.
+ */
+ flush_cache_range(vma, range.start, range.end);
+
+ mmu_notifier_invalidate_range_start(&range);
+ last_addr_mask = hugetlb_mask_last_page(h);
@@ -4004,8 +5166,9 @@
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, address, ptep)) {
spin_unlock(ptl);
tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
force_flush = true;
+ address |= last_addr_mask;
continue;
}
@@ -4078,4 +5329,4 @@
tlb_flush_mmu_tlbonly(tlb);
}
-void __unmap_hugepage_range_final(struct mmu_gather *tlb,
+void __hugetlb_zap_begin(struct vm_area_struct *vma,
@@ -5083,8 +6424,11 @@
pte_t pte;
struct hstate *h = hstate_vma(vma);
- unsigned long pages = 0;
+ long pages = 0, psize = huge_page_size(h);
bool shared_pmd = false;
struct mmu_notifier_range range;
+ unsigned long last_addr_mask;
+ bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+ bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
/*
* In the case of shared PMDs, the area to flush could be beyond
@@ -5091,7 +6443,12 @@
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, address, ptep)) {
+ /*
+ * When uffd-wp is enabled on the vma, unshare
+ * shouldn't happen at all. Warn about it if it
+ WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
pages++;
spin_unlock(ptl);
shared_pmd = true;
+ address |= last_addr_mask;
continue;
}
@@ -5138,7 +6504,10 @@
huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);
pages++;
}
+
+next:
spin_unlock(ptl);
+ cond_resched();
}
/*
* There is nothing protecting a previously-shared page table that we
@@ -5160,6 +6558,8 @@
- */
+ * No need to call mmu_notifier_arch_invalidate_secondary_tlbs() we are
+ * downgrading page table protection not changing it to point to a new
i_mmap_unlock_write(vma->vm_file->f_mapping);
+ hugetlb_vma_unlock_write(vma);
mmu_notifier_invalidate_range_end(&range);
- return pages << h->order;
+ return pages > 0 ? (pages << h->order) : pages;
}
@@ -5491,5 +6930,5 @@
*/
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+ unsigned long addr, pte_t *ptep)
{
unsigned long sz = huge_page_size(hstate_vma(vma));
@@ -5499,5 +6935,8 @@
- return 0;
-
+ pgd_t *pgd = pgd_offset(mm, addr);
+ p4d_t *p4d = p4d_offset(pgd, addr);
+ pud_t *pud = pud_offset(p4d, addr);
+ i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+ hugetlb_vma_assert_locked(vma);
pud_clear(pud);
/*
* Once our caller drops the rmap lock, some other process might be
@@ -5512,7 +6946,6 @@
*/
tlb_remove_table_sync_one();
- atomic_dec(&virt_to_page(ptep)->pt_share_count);
+ ptdesc_pmd_pts_dec(virt_to_ptdesc(ptep));
mm_dec_nr_pmds(mm);
- *addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
return 1;
}
@@ -5523,7 +6957,7 @@
}
int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+ unsigned long addr, pte_t *ptep)
{
return 0;
}
@@ -5801,9 +7255,9 @@
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- huge_pmd_unshare(mm, vma, &address, ptep);
+ huge_pmd_unshare(mm, vma, address, ptep);
spin_unlock(ptl);
}
flush_hugetlb_tlb_range(vma, start, end);
if (take_locks) {
i_mmap_unlock_write(vma->vm_file->f_mapping);
--- b/mm/mmu_gather.c
+++ b/mm/mmu_gather.c
@@ -6,4 +4,5 @@
#include <linux/swap.h>
+#include <linux/rmap.h>
+#include <linux/pgalloc.h>
-#include <asm/pgalloc.h>
#include <asm/tlb.h>
@@ -279,5 +423,6 @@
#endif
+ tlb->vma_pfn = 0;
__tlb_reset_range(tlb);
inc_tlb_flush_pending(tlb->mm);
}
--- b/mm/rmap.c
+++ b/mm/rmap.c
@@ -69,7 +69,7 @@
-#include <linux/memremap.h>
-#include <linux/userfaultfd_k.h>
+#include <linux/mm_inline.h>
+#include <linux/oom.h>
#include <asm/tlbflush.h>
-#include <trace/events/tlb.h>
-
+#define CREATE_TRACE_POINTS
+#include <trace/events/migrate.h>
@@ -1485,2 +1944,3 @@
}
- }
+ pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
+ if (pte_dirty(pteval))
@@ -1487,2 +1960,10 @@
- if (IS_ENABLED(CONFIG_MIGRATION) &&
+ /*
+ * The PMD table was unmapped,
+ * consequently unmapping the folio.
+ break;
+ }
+ hugetlb_vma_unlock_write(vma);
+ }
+ /* Nuke the hugetlb page table entry */
+ pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);This is an automated interdiff check for backported commits. |
JIRA PR Check Results18 commit(s) with issues found: Commit
|
|
❌ Validation checks completed with issues View full results: https://github.com/ctrliq/kernel-src-tree/actions/runs/30552170473 |
[LTS 8.6]
PR structure
This PR consists of two sets of patches and their supporting bugfixes/prerequisites.
The first patch set, provided by Jann Horn, is part of a larger set of hotfixes merged in d3c82f6 and addresses CVE-2025-38084 and CVE-2025-38085 in two commits:
The second patch set, provided by David Hildenbrand, is similarly embedded in a larger set of hotfixes, consists of 4 commits (all backported in this PR), and was motivated by the performance regression introduced by the CVE-2025-38085 fix - the "excessive IPI broadcasts" issue. Additionally it addresses CVE-2026-23100.
The CVE-2025-38085 problem was about unsharing a shared page table on one processor, thus potentially changing its meaning (turning it into a normal page table) while another processor could have been in the middle of walking that table (resolving a virtual address to a physical one). The solution was to send IPI (Inter-Processor Interrupt) to other processors as a synchronization mechanism. This resulted in a noticable performance drop on some setups of RH clients (https://lore.kernel.org/all/4d3878531c76479d9f8ca9789dc6485d@amazon.de/). A quick mitigation of that problem in the form of "fix disable" was merged into CentOS 9 and CentOS 10. The 8ce720d
mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gathercommit implements a proper solution and was merged into the upstream, then backported, at least tocentos9(797ec6e) androcky8_10(ecba853).One important prerequisite for this fix was the stand-alone commit 59d9094
mm: hugetlb: independent PMD page table shared count, which also solved CVE-2024-57883 (and introduced CVE-2026-23100 - solved, however, in the same patch set as the performance fix).CVE-2025-38085 fix
Commit 1013af4
mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast raceis identified as the fix for CVE-2025-38085. However, it depends on a functiontlb_remove_table_sync_onewhich is not defined in LTS 8.6. It was introduced in 0d6e24dasm-generic/tlb: provide MMU_GATHER_TABLE_FREE. No solution to CVE-2025-38085 was found which didn't have this commit in place:centos912a6db3,centos1041f7eb5,rocky9_62872192,rocky8_10e2c95cd,rocky10_068b3612,linux-5.10.y: 952596b08c74e8fe9e2883d1dc8a8f54a37384ec,linux-5.15.y: a3d864c901a300c295692d129159fc3001a56185,linux-6.1.y: b7754d3aa7bf9f62218d096c0c8f6c13698fac8b,linux-6.6.y: fe684290418ef9ef76630072086ee530b92f02b8,linux-6.12.y: 034a52b,linux-6.15.y: a6bfeb97941a9187833b526bc6cc4ff5706d0ce9,although also none of those solutions had 0d6e24d backported as part of it.
The commit 0d6e24d
asm-generic/tlb: provide MMU_GATHER_TABLE_FREEintroduces a new option, but it's not user-selectable; rather, it's used in the configuration as part of architecture's features makeup, eg. as in (implied by):kernel-src-tree/arch/arm64/Kconfig
Line 247 in 1013af4
The commit therefore doesn't introduce a new feature but addresses the hardare setup not accounted for sufficiently before:
Nevertheless backporting it may seem risky and simply ad-hoc defining the one line long
tlb_remove_table_sync_onemay be preferred. However, the definition oftlb_remove_table_sync_onefunction used in CVE-2025-38085 depends on an option. ForCONFIG_MMU_GATHER_RCU_TABLE_FREE:kernel-src-tree/mm/mmu_gather.c
Lines 142 to 152 in 0d6e24d
For
!CONFIG_MMU_GATHER_RCU_TABLE_FREE:kernel-src-tree/mm/mmu_gather.c
Line 166 in 0d6e24d
In the upstream the
tlb_remove_table_onefunction is expressed bytlb_remove_table_sync_one:kernel-src-tree/mm/mmu_gather.c
Lines 190 to 194 in 0d6e24d
If it was defined like that in LTS 8.6 as well then its definition would start depending on
CONFIG_MMU_GATHER_RCU_TABLE_FREEtoo, which would change code semantics in an uncontrolled manner. Putting it under control by following 0d6e24d's options dependency would boil down to simply backporting that commit. Not expressingtlb_remove_table_onebytlb_remove_table_sync_one, on the other hand, would lead to code duplication and enigmatic deviation from the upstream. It was therefore decided to backport 0d6e24dasm-generic/tlb: provide MMU_GATHER_TABLE_FREEdespite some complications it implied.That commit depends on options naming scheme not yet backported to LTS 8.6:
The renaming was done in a bulk, right before 0d6e24d:
Between sticking to an old scheme and using the new one the latter was chosen, because it made tha backporting much simpler, less prone to mistakes and it avoided devising a completely unique option name
HAVE_MMU_GATHER_TABLE_FREE. Only theMMU_GATHER_NO_GATHERoption was renamed though, to simplify the bug fix. This introduced some option naming inconsistency, but it's not a big problem.All other commits in the CVE-2025-38085 fix are preparing the code for either the option renaming done in 580a586
asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER, or for prviding theMMU_GATHER_TABLE_FREEoption with the requiredtlb_remove_table_sync_onefunction in 0d6e24dasm-generic/tlb: provide MMU_GATHER_TABLE_FREE.Finally, the CVE-2025-38085 fix 1013af4
mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast raceincludes the exposition oftlb_remove_table_sync_one, which in the upstream was done in a similar fix 2ba99c5mm/khugepaged: fix GUP-fast interaction by sending IPI, not backported to LTS 8.6.Linux stable 5.10 as the main source of changes
The PR strongly relies on the Linux stable 5.10 backports, which provided clean cherry picks for the most part. Although the
mm/hugetlbcodebase inciqlts8_6branched off the upstream roughly two years earlier thanlinux-5.10.y(2018 vs 2020), it was heavily backported with features until the beginning of 2021, such that it closely approximates Linux stable 5.10.CVE-2026-63875 issue
The performance-fixing commit 8ce720d
mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gatherintroduces arm64-specific vulnerability CVE-2026-63875, fixed by c2ff476. The modified code fragments don't even exist in LTS 8.6, so it's highly probable that the problem doesn't apply to this version. However, given the very non-local nature of page tables management a deeper dive may be required. Because the volume of this PR is already big the the solution of CVE-2026-63875 (or marking LTS 8.6 as not affected by it) was deferred to a different PR. This aligns with RH's https://access.redhat.com/security/cve/cve-2026-63875.The CVE-2025-38084 bugfix b4aea43 issue
The CVE-2025-38084 fix 081056d
mm/hugetlb: unshare page tables during VMA split, not beforeintroduces a bug fixed by b4aea43mm/hugetlb: avoid false positive lockdep assertion. However, it doesn't apply to LTS 8.6. From the commit's message:The
hugetlb_vma_assert_locked()function is not present in LTS 8.6 and neither it has any differently named equivalents. It was introduced in 8d9bfb2hugetlb: add vma based lock for pmd sharing, not backported to LTS 8.6. It also wasn't backported tolinux-5.10.yandlinux-5.15.yand these versions don't have the bugfix b4aea43 backported either (unlike versionslinux-6.1.y,linux-6.6.y,linux-6.12.ywhich do have 8d9bfb2 in history and do backport the bugfix backported, as 3e1b69f02e71f73ad47e1c2dcaca02234a512c93, 428a33573dcb921971d1965f618cc4431c00b126 and b987d94)Commits
The order of the commits in this PR reflects the natural order in the upstream. Because the prerequisites and bugfixes were not linear, the relationships between them can best be depicted with the following graph:
The CVE-2025-38085 fixing commit. Compared to the upstream the
tlb_remove_table_sync_onefunction was made public as part of the fix.The kABI issue described was discussed in the ciqlts9_2 backport of CVE-2024-57883: #731. While the situation here is slightly different (
RH_KABI_REPLACEmacro used) the buddy allocator argumentation is the same.This commit provides the necessary options for the
tlb_remove_table_sync_onefunction to be defined properly. The new optionMMU_GATHER_TABLE_FREEis not user-selectable and it doesn't introduce a new feature. Rather it can be thought of as fine-tuning the existing solution.Code refactoring commit, functionally neutral. Backporting it significantly reduces conflict in backporting the main CVE-2025-38085 prerequisite 0d6e24d
asm-generic/tlb: provide MMU_GATHER_TABLE_FREE.This commit massively simplifies the
arch/arm/include/asm/tlb.h, easing the backport of 0d6e24dasm-generic/tlb: provide MMU_GATHER_TABLE_FREE. Important note from the commit's message:Similarly to the commit above this one prepares the codebase for the cherry pick of ff2e6d7
asm-generic/tlb: rename HAVE_RCU_TABLE_FREE. Functionally neutral.This commit is just a small and cheap prerequisite to make clean the cherry pick of b78180b
arm/tlb: Convert to generic mmu_gather. Functionally neutral.kABI check: passed
Boot test: passed
boot-test.log
Kselftests: passed relative
Reference
kselftests–ciqlts8_6–run1.log
kselftests–ciqlts8_6–run2.log
kselftests–ciqlts8_6–run3.log
Patch
kselftests–ciqlts8_6-CVE-batch-42–run1.log
kselftests–ciqlts8_6-CVE-batch-42–run2.log
kselftests–ciqlts8_6-CVE-batch-42–run3.log
Comparison
The tests results for the reference and the patch are the same.
full-test-results-comparison.log