[LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

Mon Aug 11 14:42:24 CEST 2025

Hi Cyril,

On Wed, Aug 6, 2025 at 5:26 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > Currently, tst_umount() in lib/tst_device.c tries to repeatedly umount()
> > a mount point, with a retry loop if it gets EBUSY. However, after
> umount()
> > reports success, devices (especially loop devices) can still be held open
> > by another process or kernel, delaying their actual release. This can
> lead
> > to race conditions when the next operation tries to reuse the device like
> > mkfs.ext3 error:
>
> This looks like working around a kernel bug to me. The umount() without
> MNT_DETACH should make the device reusable once the syscall returns back
> to the userspace. At least I wouldn't add this code to LTP unless kernel
> devs tells us that this broken behavior is to be expected.

Perhaps not yet fall into the kernel issue.

Looking into the details for many days, there seems to be something in
userspace briefly reopens the block device right after unmounting it,
I suspect that “something” is likely systemd-udevd running its blkid
builtin.

To prove this, I did some debugging work:

1. Confirm the tested system enables rule:
# cat /usr/lib/udev/rules.d/60-persistent-storage.rules | grep blkid
  KERNEL!="sr*|mmcblk[0-9]boot[0-9]", IMPORT{builtin}="blkid"

2. Check the core functions of udev-builtin-blkid:

https://github.com/systemd/systemd/blob/main/src/udev/udev-builtin-blkid.c#L353

3. Run udev monitor in a separate terminal, and get output with running
./close_range01 test simultaneously:

# udevadm monitor --kernel --udev
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[283886.216245] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.271087] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.277479] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.280656] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.299308] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.303885] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.307296] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.313293] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.317716] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.321284] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.358952] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.364841] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.369347] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.376379] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.378823] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.382704] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.463900] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.467966] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.471508] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.475901] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.491129] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.495081] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.498525] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.502380] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.504077] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.507968] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.558605] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.559145] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.560932] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.562050] change   /devices/virtual/block/loop0 (block)

Looks like udev-builtin-blkid indeed probes via libblkid to populate
ID_FS_* properties.

Then, I use strace tool to trace the openat on systemd-udevd, but weirdly
I got all opens with 'O_PATH',  which opens do not conflict with mkfs.ext*’s
O_EXCL and should not cause “apparently in use by the system”.

4. #sudo strace -f -e trace=openat,close -p $(pidof systemd-udevd) -o log
# cat log | grep '/dev/loop0'
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
702   openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH
<unfinished ...>
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17
702   openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 13
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
...

However, switch to another bpftrace tool gives more suggestive clues,
it catches not only the udev-worker but also udeskd operations on the
/dev/loop0, after checking the flags I doubt that race probably comes
from udiskd operation.

5. Run bpftrace in a separate terminal and get output with running
./close_range01 test simultaneously:

# sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat
/str(args->filename)=="/dev/loop0"/ { printf("%s pid=%d flags=0x%x %s\n",
comm, pid, args->flags, str(args->filename)); }'
Attaching 1 probe...
close_range01 pid=298948 flags=0x2 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
close_range01 pid=298948 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80800 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x82 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0        <--------------
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80800 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x82 /dev/loop0
...

>From the bpfstrace logs, there seems udisksd listening to the same
udevevents and probes the device via libblockdev/blkid, performing
real O_RDONLY opens (flags=0x0). These O_RDONLY opens from
udisksd race with mkfs.ext3’s O_EXCL open, intermittently causing
“apparently in use by the system” (EBUSY).

Does this analysis sound reasonable to you?

-- 
Regards,
Li Wang