Re: [PATCH v4 0/7] mm: Remember a/d bits for migration entries

From: Raghavendra K T
Date: Mon Nov 21 2022 - 00:16:09 EST

On 8/11/2022 9:43 PM, Peter Xu wrote:
v4:
- Added r-bs for Ying
- Some cosmetic changes here and there [Ying]
- Fix smaps to only dump PFN for pfn swap entries for both pte/pmd [Ying]
- Remove max_swapfile_size(), export swapfile_maximum_size variable [Ying]
- In migrate_vma_collect_pmd() only read A/D if pte_present()

rfc: https://lore.kernel.org/all/20220729014041.21292-1-peterx@xxxxxxxxxx
v1: https://lore.kernel.org/all/20220803012159.36551-1-peterx@xxxxxxxxxx
v2: https://lore.kernel.org/all/20220804203952.53665-1-peterx@xxxxxxxxxx
v3: https://lore.kernel.org/all/20220809220100.20033-1-peterx@xxxxxxxxxx

Problem
=======

When migrate a page, right now we always mark the migrated page as old &
clean.

However that could lead to at least two problems:

(1) We lost the real hot/cold information while we could have persisted.
That information shouldn't change even if the backing page is changed
after the migration,

(2) There can be always extra overhead on the immediate next access to
any migrated page, because hardware MMU needs cycles to set the young
bit again for reads, and dirty bits for write, as long as the
hardware MMU supports these bits.

Many of the recent upstream works showed that (2) is not something trivial
and actually very measurable. In my test case, reading 1G chunk of memory
- jumping in page size intervals - could take 99ms just because of the
extra setting on the young bit on a generic x86_64 system, comparing to 4ms
if young set.

This issue is originally reported by Andrea Arcangeli.

Solution
========

To solve this problem, this patchset tries to remember the young/dirty bits
in the migration entries and carry them over when recovering the ptes.

We have the chance to do so because in many systems the swap offset is not
really fully used. Migration entries use swp offset to store PFN only,
while the PFN is normally not as large as swp offset and normally smaller.
It means we do have some free bits in swp offset that we can use to store
things like A/D bits, and that's how this series tried to approach this
problem.

max_swapfile_size() is used here to detect per-arch offset length in swp
entries. We'll automatically remember the A/D bits when we find that we
have enough swp offset field to keep both the PFN and the extra bits.

Since max_swapfile_size() can be slow, the last two patches cache the
results for it and also swap_migration_ad_supported as a whole.

Known Issues / TODOs
====================

We still haven't taught madvise() to recognize the new A/D bits in
migration entries, namely MADV_COLD/MADV_FREE. E.g. when MADV_COLD upon a
migration entry. It's not clear yet on whether we should clear the A bit,
or we should just drop the entry directly.

We didn't teach idle page tracking on the new migration entries, because
it'll need larger rework on the tree on rmap pgtable walk. However it
should make it already better because before this patchset page will be old
page after migration, so the series will fix potential false negative of
idle page tracking when pages were migrated before observing.

The other thing is migration A/D bits will not start to working for private
device swap entries. The code is there for completeness but since private
device swap entries do not yet have fields to store A/D bits, even if we'll
persistent A/D across present pte switching to migration entry, we'll lose
it again when the migration entry converted to private device swap entry.

Tests
=====

After the patchset applied, the immediate read access test [1] of above 1G
chunk after migration can shrink from 99ms to 4ms. The test is done by
moving 1G pages from node 0->1->0 then read it in page size jumps. The
test is with Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz.

Similar effect can also be measured when writting the memory the 1st time
after migration.

After applying the patchset, both initial immediate read/write after page
migrated will perform similarly like before migration happened.

I was able to test on AMD EPYC 64 core 2 numa node (Milan) 3.72 GHz clocked system

am seeing the similar improvement for the test mentioned above (swap-young)

base: (6.0)
--------------
Write (node 0) took 562202 (us)
Read (node 0) took 7790 (us)
Move to node 1 took 474876(us)
Move to node 0 took 642805(us)
Read (node 0) took 81364 (us)
Write (node 0) took 12887 (us)
Read (node 0) took 5202 (us)
Write (node 0) took 4533 (us)
Read (node 0) took 5229 (us)
Write (node 0) took 4558 (us)
Read (node 0) took 5198 (us)
Write (node 0) took 4551 (us)
Read (node 0) took 5218 (us)
Write (node 0) took 4534 (us)

patched
-------------
Write (node 0) took 250232 (us)
Read (node 0) took 3262 (us)
Move to node 1 took 640636(us)
Move to node 0 took 449051(us)
Read (node 0) took 2966 (us)
Write (node 0) took 2720 (us)
Read (node 0) took 2891 (us)
Write (node 0) took 2560 (us)
Read (node 0) took 2899 (us)
Write (node 0) took 2568 (us)
Read (node 0) took 2890 (us)
Write (node 0) took 2568 (us)
Read (node 0) took 2897 (us)
Write (node 0) took 2563 (us)

Please feel free to add FWIW
Tested-by: Raghavendra K T <raghavendra.kt@xxxxxxx>


Patch Layout
============

Patch 1-2: Cleanups from either previous versions or on swapops.h macros.

Patch 3-4: Prepare for the introduction of migration A/D bits

Patch 5: The core patch to remember young/dirty bit in swap offsets.

Patch 6-7: Cache relevant fields to make migration_entry_supports_ad() fast.

Please review, thanks.

[1] https://github.com/xzpeter/clibs/blob/master/misc/swap-young.c

Peter Xu (7):
mm/x86: Use SWP_TYPE_BITS in 3-level swap macros
mm/swap: Comment all the ifdef in swapops.h
mm/swap: Add swp_offset_pfn() to fetch PFN from swap entry
mm/thp: Carry over dirty bit when thp splits on pmd
mm: Remember young/dirty bit for page migrations
mm/swap: Cache maximum swapfile size when init swap
mm/swap: Cache swap migration A/D bits support

arch/arm64/mm/hugetlbpage.c | 2 +-
arch/x86/include/asm/pgtable-3level.h | 8 +-
arch/x86/mm/init.c | 2 +-
fs/proc/task_mmu.c | 20 +++-
include/linux/swapfile.h | 5 +-
include/linux/swapops.h | 145 +++++++++++++++++++++++---
mm/hmm.c | 2 +-
mm/huge_memory.c | 27 ++++-
mm/memory-failure.c | 2 +-
mm/migrate.c | 6 +-
mm/migrate_device.c | 6 ++
mm/page_vma_mapped.c | 6 +-
mm/rmap.c | 5 +-
mm/swapfile.c | 15 ++-
14 files changed, 214 insertions(+), 37 deletions(-)