Re: [PATCH v6 0/3] Implement IOCTL to get and/or the clear info about PTEs

[Peter Xu]

On Wed, Nov 09, 2022 at 03:23:00PM +0500, Muhammad Usama Anjum wrote:
> Soft-dirty PTE bit of the memory pages can be read by using the pagemap
> procfs file. The soft-dirty PTE bit for the whole memory range of the
> process can be cleared by writing to the clear_refs file. There are other
> methods to mimic this information entirely in userspace with poor
> performance:
> - The mprotect syscall and SIGSEGV handler for bookkeeping
> - The userfaultfd syscall with the handler for bookkeeping

Userfaultfd is definitely slow in this case because it needs the messaging
roundtrip that happens in two different threads synchronously, so at least
more schedule effort even than mprotect.

I saw the other patch on vma merging with SOFTDIRTY, didn't look deeper
there but IIUC it won't really help much if the other commit (34228d47)
can't be reverted then it seems to help nothing.  And, it does looks risky
to revert that because in the same commit it mentioned the case where one
can clear ref right before a vma merge, so definitely worth more thoughts
and testings, which I agree with you.

I'm thinking whether the vma issue can be totally avoided.  For example by
providing an async version of uffd-wp.

Currently uffd-wp must be synchronous and it'll be slow but it services
specific purposes.  And this is definitely not the 1st time any of us
thinking about uffd-wp being async, it's just that we need to solve the
problem of storage on the dirty information.

Actually we can also use other storage form but so far I didn't think of
anything that's easy and clean.  Current soft-dirty bit also has its
defects (e.g. the need to take mmap lock and walk the pgtables), but that
part will be the same as soft-dirty for now.

Now I'm wildly thinking whether we can just reuse the soft-dirty bit in the
ptes already defined.  The GET interface could be similar as proposed here,
or at least a separate issue.

So _maybe_ we can have a feature (bound to the uffd context) for uffd that
enables async uffd-wp, in which case the wr-protect fault is not sending
any message anymore (nor enqueuing) but instead setting the soft-dirty then
quickly resolving the write bit immediately and continue the fault.

Clearing of the soft-dirty bit needs to be done in UFFDIO_WRITEPROTECT
alongside of clearing uffd-wp bit.  So on that part the current GET+CLEAR
interface for pagemap may need to be replaced.  And frankly, it feels weird
to me to allow change mm layout in pagemap ioctls..  With this we can keep
the pagemap interface to only fetch information, like before.

A major benefit of using uffd is that uffd is by nature pte-based, so no
fiddling with vma needed at all.  Firstly, no need to worry about merging
vmas with tons of false positives.  Meanwhile, one can wr-protect in
page-size granule easily.  All the wr-protect is not governed by vma flag
anymore but based on uffd-wp flag, so no extra overhead too on any page
that the monitor is not interested.  There's already infrastructure code
for persisting uffd-wp bit, so it'll naturally work similarly for an async
mode if to come to the world.

It's just that we'll also need to consider exclusive use of the bit, so
we'll need to fail clear_refs on vmas where we have VM_UFFD_WP and also the
async feature enabled.  I would hope that's very rare, but worth thinking
about its side effect.  The same will need to apply to UFFDIO_REGISTER on
async wp mode when soft-dirty enabled, we'll need to bailout too.

Said that, this is not a suggestion of a new design, but just something I
thought about when reading this, and quickly writting this down.

Thanks,

-- 
Peter Xu