[LTP] [PATCH 1/2] lib: multiply the timeout if detect slow kconfigsD

[Petr Vorel]

Hi all,

[ Cc Andrea due aiodio ]

> On Fri, Jan 3, 2025 at 11:48 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> > Hi!
> > > Yes, the benefit of multiplying TIMEOUT (on a slow system) is not only
> > > to avoid increasing the actual execution time of the test, but also to
> > give
> > >  the system more time to wait for the test to complete the final work.

> > > Original:
> > >   |  -- timeout -- | -- max_runtime -- |

> > > Previous:
> > >   |  -- timeout -- | -------- max_runtime * 4 -------- |

> > > Now:
> > >   |  -------- timeout * 4 -------- | -- max_runtime --  |

> > The problems I see here:

> > There are tests, I think this covers mostly the IO stress tests, where
> > the max runtime may be significantly larger than the timeout, so
> > multiplying only the timeout may not be enough there.


> I did a quick grep that some ltp-aiodio tests set it to 1800 sec, which
> only 8/91 occupation in the LTP testcases/, I'm not sure if it's worth
> adding a new field for those few stress tests.

> And with the previous method, the multiple 4 max_runtime for 1800s
> is 2hours per test up limit, I can't imagine how long we will get eventually
> in the whole test time.

Agree. This solution is quite simple. I would also hesitate to add new field for
few tests.

It's questionable whether .max_runtime = 1800 is reasonable for aiodio tests at
all (see below). I'll try to have look and send a patch to shorten it.  For some
tests it'd be nice to have some calculation instead of blindly expecting they
run 30 min (because we have many runs of these tests and vast majority runs
slowly).

OTOH it's questionable how much time we should spent with these tests,
see Jan Kara [1]:

	So the performance of this test is irrelevant because combining buffered
	reads with direct IO writes was always in "better don't do it" territory.
	Definitely not if you care about performance.

NOTE: Jan's comment is only for aiodio_sparse.c and dio_sparse.c, but I suppose
this is related to all LTP aiodio tests (at least dio_append.c does also
buffered reads with O_DIRECT, dio_read.c combines buffered writes with O_DIRECT).

[1] https://lore.kernel.org/ltp/20240722090012.mlvkaenuxar2x3vr@quack3/

> Maybe another way is to create a separate function in a header
> like aio_common.h (or in high-level dir) for handling that significantly
> larger runtime tests.

> BTW, we have TST_UNLIMITED_RUNTIME choice or, invoke
> tst_set_max_runtime() in somehow.



> > I wanted to eventually move to a shorter default timeout, e.g. 10s once
> > we have enough max_runtime anotation in the testcases.


> That's fine, but most tests do not have .max_runtime. I guess what
> we should consider now is making them have enough timeout time
> on the slow kernel, isn't it?


> $ git grep .max_runtime | wc -l
> 91

> $ git grep .max_runtime
> cve/cve-2014-0196.c:    .max_runtime = 60,
> cve/cve-2015-3290.c:    .max_runtime = 180,
> cve/cve-2016-7117.c:    .max_runtime = 60,
> cve/cve-2017-2671.c:    .max_runtime = 40,
> kernel/controllers/cgroup/cgroup_core03.c:      .max_runtime = 20,
> kernel/crypto/af_alg02.c:       .max_runtime = 20,
> kernel/crypto/af_alg07.c:       .max_runtime = 150,
> kernel/crypto/pcrypt_aead01.c:  .max_runtime = 300,
> kernel/fs/fs_fill/fs_fill.c:    .max_runtime = 300,
> kernel/fs/fsplough/fsplough.c:  tst_set_max_runtime(bufsize * loop_count /
> (8 * 1024 * 1024));
> kernel/fs/fsx-linux/fsx-linux.c:        .max_runtime = 1800,

Just looking into some aiodio tests.

We have 22 variants of these tests (none without params). The longest is fsx22
(-N 100000) due much higher -N than other tests and the default (1000).
The highest runtime I see on this test is the current mainline (6.12): 1m 31s.

> kernel/fs/read_all/read_all.c:          tst_res(TINFO, "Worker timeout set
> to 10%% of max_runtime: %dms",
> kernel/fs/read_all/read_all.c:  .max_runtime = 100,
> kernel/io/ltp-aiodio/aio-stress.c:      .max_runtime = 1800,
> kernel/io/ltp-aiodio/aiocp.c:   .max_runtime = 1800,
> kernel/io/ltp-aiodio/aiodio_append.c:   .max_runtime = 1800,

For example aiodio_append runs on 1 sec on Tumbleweed.

> kernel/io/ltp-aiodio/aiodio_sparse.c:   .max_runtime = 1800,

This is a valid one, it runs without parameters 9m 39s (ADSP000).

But we have several parameters (it runs 38x with a different parameters)
Most of them run below 7s.

These are running longer, obviously combination of high -w and -n:
ADSP037 44s (-w 18192k -n 512)
ADSP038 1m 44s (-w 18192k -n 1000)

We simplified our life with blindly saying everything runs 30 min,
but vast majority don't.

Also this is a long run on the current mainline kernel (slow down in 6.6 [1]).
It runs on older kernels ~1 or 2 min.

> kernel/io/ltp-aiodio/dio_append.c:      .max_runtime = 1800,

This runs ~ 32 sec on Tumbleweed.

> kernel/io/ltp-aiodio/dio_read.c:        .max_runtime = 1800,
> kernel/io/ltp-aiodio/dio_sparse.c:      .max_runtime = 1800,

The only long running is ADSP039 (dio_sparse without params): 9m 37s 

> kernel/io/ltp-aiodio/dio_truncate.c:    .max_runtime = 1800,
> kernel/mem/ksm/ksm02.c:         tst_set_max_runtime(32 * (size /
> DEFAULT_MEMSIZE));
> kernel/mem/ksm/ksm02.c: .max_runtime = 32,
> kernel/mem/ksm/ksm04.c:         tst_set_max_runtime(32 * (size /
> DEFAULT_MEMSIZE));
> kernel/mem/ksm/ksm04.c: .max_runtime = 32,
> kernel/mem/mmapstress/mmapstress01.c:   .max_runtime = 12,
> kernel/mem/mtest01/mtest01.c:   .max_runtime = 300,
> kernel/mem/mtest06/mmap1.c:     .max_runtime = 180,
> kernel/mem/mtest06/mmap3.c:     .max_runtime = 60,
> kernel/mem/mtest07/mallocstress.c:      .max_runtime = 600,
> kernel/mem/oom/oom01.c: .max_runtime = TST_UNLIMITED_RUNTIME,
> kernel/mem/oom/oom02.c: .max_runtime = TST_UNLIMITED_RUNTIME,
> kernel/mem/oom/oom03.c: .max_runtime = TST_UNLIMITED_RUNTIME,
> kernel/mem/oom/oom04.c: .max_runtime = TST_UNLIMITED_RUNTIME,
> kernel/mem/oom/oom05.c: .max_runtime = TST_UNLIMITED_RUNTIME,
> kernel/mem/swapping/swapping01.c:       .max_runtime = 600,
> kernel/mem/thp/thp04.c: .max_runtime = 150,
> kernel/mem/tunable/min_free_kbytes.c:   .max_runtime =
> TST_UNLIMITED_RUNTIME,
> kernel/pty/pty03.c:     .max_runtime = 30,
> kernel/pty/pty05.c:     .max_runtime = 150,
> kernel/pty/pty06.c:     .max_runtime = 150,
> kernel/pty/pty07.c:     .max_runtime = 150,
> kernel/sched/cfs-scheduler/cfs_bandwidth01.c:   .max_runtime = 20,
> kernel/sched/cfs-scheduler/starvation.c:
>  tst_set_max_runtime(timeout);
> kernel/security/dirtyc0w_shmem/dirtyc0w_shmem.c:        .max_runtime = 120,
> kernel/security/kallsyms/kallsyms.c:    .max_runtime = 60,
> kernel/sound/snd_seq01.c:       .max_runtime = 60,
> kernel/sound/snd_timer01.c:     .max_runtime = 150,
> kernel/syscalls/bind/bind06.c:  .max_runtime = 300,
> kernel/syscalls/copy_file_range/copy_file_range01.c:    .max_runtime = 5
> kernel/syscalls/fallocate/fallocate06.c:        .max_runtime = 120,
> kernel/syscalls/fork/fork13.c:  .max_runtime = 600,
> kernel/syscalls/fsync/fsync02.c:        .max_runtime = 300,
> kernel/syscalls/gettimeofday/gettimeofday02.c:  .max_runtime = 10,
> kernel/syscalls/inotify/inotify06.c:    .max_runtime = 600,
> kernel/syscalls/inotify/inotify09.c:    .max_runtime = 150,
> kernel/syscalls/inotify/inotify12.c:    .max_runtime = 10,
> kernel/syscalls/ioctl/ioctl_sg01.c:     .max_runtime = 3600,

FYI Martin set 1 hour due long setup on systems with big RAM.
Real runtime is 1 sec on 8GB RAM.

Kind regards,
Petr

> kernel/syscalls/ipc/msgstress/msgstress01.c:    .max_runtime = 180,
> kernel/syscalls/ipc/semget/semget05.c:  tst_set_max_runtime(maxsems / 200);
> kernel/syscalls/ipc/shmctl/shmctl05.c:  .max_runtime = 10,
> kernel/syscalls/keyctl/keyctl02.c:      .max_runtime = 60,
> kernel/syscalls/landlock/landlock04.c:  .max_runtime = 360,
> kernel/syscalls/madvise/madvise06.c:    .max_runtime = 60,
> kernel/syscalls/madvise/madvise11.c:    .max_runtime = 30,
> kernel/syscalls/migrate_pages/migrate_pages03.c:        .max_runtime = 300,
> kernel/syscalls/move_pages/move_pages12.c:      .max_runtime = 240,
> kernel/syscalls/nice/nice05.c:  .max_runtime = 3,
> kernel/syscalls/perf_event_open/perf_event_open02.c:    .max_runtime = 72
> kernel/syscalls/perf_event_open/perf_event_open03.c:    .max_runtime = 300,
> kernel/syscalls/preadv2/preadv203.c:    .max_runtime = 60,
> kernel/syscalls/readahead/readahead02.c:         * speed, sometime test
> timeout when the default max_runtime is used up.
> kernel/syscalls/readahead/readahead02.c:
>  tst_set_max_runtime(test.max_runtime + (usec + usec_ra) / 1000000);
> kernel/syscalls/readahead/readahead02.c:
>  tst_set_max_runtime(1 + testfile_size / (DEFAULT_FILESIZE/32));
> kernel/syscalls/readahead/readahead02.c:        .max_runtime = 30,
> kernel/syscalls/request_key/request_key03.c:    .max_runtime = 20,
> kernel/syscalls/sendfile/sendfile09.c:  .max_runtime = 120,
> kernel/syscalls/sendmsg/sendmsg03.c:    .max_runtime = 150,
> kernel/syscalls/set_mempolicy/set_mempolicy01.c:
>  tst_set_max_runtime(test.max_runtime * (1 << nodes->cnt/16));
> kernel/syscalls/set_mempolicy/set_mempolicy01.c:        .max_runtime = 600,
> kernel/syscalls/setsockopt/setsockopt06.c:      .max_runtime = 270,
> kernel/syscalls/setsockopt/setsockopt07.c:      .max_runtime = 150,
> kernel/syscalls/swapoff/swapoff01.c:    .max_runtime = 60,
> kernel/syscalls/swapon/swapon01.c:      .max_runtime = 60,
> kernel/syscalls/timerfd/timerfd_settime02.c:    .max_runtime = 150,
> kernel/syscalls/writev/writev03.c:      .max_runtime = 75,
> network/can/cve/can_bcm01.c:    .max_runtime = 30,
> network/netstress/netstress.c:  .max_runtime = 300,
> network/nfs/nfs_stress/nfs05_make_tree.c:       .max_runtime = 300,
> network/packet/fanout01.c:      .max_runtime = 180,
> network/sockets/vsock01.c:      .max_runtime = 60,