[LTP] [Automated-testing] [PATCH v2 00/30] Introduce runtime and conver tests
Richard Palethorpe
rpalethorpe@suse.de
Thu May 5 10:13:09 CEST 2022
Hello, Cyril,
"Cyril Hrubis" <chrubis@suse.cz> writes:
> This patchset introduces a concept of runtime, which is described well
> enough in the first patch of the series and converts all tests that were
> previously using timeout API to runtime API.
>
> The main change to the v1 of the patchset is that the runtime is now per
> iteration, e.g. if we run the test for all_filesystem the runtime
> applies to a single filesystem only. And at the same time the overall
> test timeout is a sum of static timeout and runtime. The static part of
> the timeout is meant to cover test teardown and any inaccuracy in the
> test runtime accounting.
>
> The patches that convert tests to runtime API fall into these classes:
>
> - fzsync tests
> - faster timeout tests
> - I/O tests
> - timer precision tests
> - oom tests
> - misc tests that run for some time (controlled by command line
> parameter)
>
> fzsync tests
> ============
>
> These tests previously defined their runtime based on scale factor that
> used slice of the default timeout (300s) for the test runtime.
>
> Now the runtime is explicitly encoded in the test, which I find much
> more user friendly because it's clear how long will the test run.
>
> faster timeout tests
> ====================
>
> There were several tests that set low timeout becuase the test may lock
> up. In these cases the timeout was simply removed because the default
> value (30s) should be small enough.
>
> I/O tests
> =========
>
> The aiodio tests were modified so that their runtime is capped to
> 30 minutes by default. That means that even on very slow machine they
> finish in a reasonable time.
>
> timer precision tests
> =====================
>
> These test simply set the test timeout to be 110% of the sum of the
> sleep_time * samples. That is to accomodate for the context switches and
> upkeeping between the calls in the timer library.
>
>
> oom tests
> =========
>
> Here we just set the .max_iteration_runtime to TST_RUNTIME_UNLIMITED,
> which has the very same effect as .timeout = -1 previously had.
>
> misc test
> =========
>
> In this case the test code was simplified quite a lot since the runtime
> is now passed in a unified way and parsed in the test library.
>
> TODO:
>
> This patchset is reasonably complete in a sense that it removes the
> timeout API at the end. Still there are a few things to consider:
>
> - is reusing the -I parameter a good idea? Wouldn't adding new parameter
> (-r) be better?
Perhaps we could deprecate -I, but convert it to use the new mechanism
while trying to keep the behaviour similar.
I'm not sure we need a global -r option, but unlike -i it's not much
effort to support. It would not surprise me if there are existing tests
which don't work with -i.
>
> - there are quite likely tests that run for more than a second or a two
> and should be made runtime aware
>
> - anything else?
>
> Cyril Hrubis (30):
> Introduce a concept of max runtime
> mtest06/mmap1: Convert to runtime
> mtest06/mmap3: Convert to runtime
> mtest01/mtest01: Convert to runtime
> cve/cve-2015-3290: Convert to runtime
> crypto/af_alg02: Convert to runtime.
> crypto/pcrypt_aead01: Convert to runtime
> syscalls/clock_gettime01: Remove useless timeout
> syscalls/fanotify22: Remove useless timeout
> syscalls/gettimeofday02: Convert to runtime.
> syscalls/inotify06: Convert to runtime.
> syscalls/inotify01: Remove now useless timeout
> syscalls/perf_event_open03: Convert to runtime
> syscalls/readv01: Remove now useless timeout
> syscalls/tgkill03: Remove now unused timeout
> syscalls/setsockopt09: Remove now useless timeout
> syscalls/userfaultfd01: Remove now useless timeout
> syscalls/move_pages12: Convert to runtime
> syscalls/rt_sigqueueinfo01: Remove now useless timeout
> mem/mallocstress: Convert to runtime
> mem/{oom,min_free_kbytes}: Convert to runtime
> crypto/af_alg02: Convert to runtime
> fuzzy_sync: Convert to runtime
> ltp-aiodio/dio_sparse,aiodio_sparse: Convert to runtime.
> ltp-aiodio/read_checkzero: Remove
> ltp-aiodio/dio_{truncate,append}: Convert to runtime
> ltp-aiodio/dio_read: Convert to runtime
> ltp-aiodio/aiodio_append: Convert to runtime
> timer_test: Convert to runtime
> tst_test: Remove timeout stubs
>
> doc/user-guide.txt | 4 +
> include/tst_fuzzy_sync.h | 19 +--
> include/tst_test.h | 31 +++-
> lib/newlib_tests/.gitignore | 5 +-
> lib/newlib_tests/test10.c | 22 ---
> lib/newlib_tests/test12.c | 21 ---
> lib/newlib_tests/test13.c | 1 -
> lib/newlib_tests/test_children_cleanup.c | 1 -
> .../{test18.c => test_runtime01.c} | 10 +-
> lib/newlib_tests/test_runtime02.c | 31 ++++
> lib/tst_test.c | 144 +++++++++++++-----
> lib/tst_timer_test.c | 12 +-
> runtest/ltp-aiodio.part4 | 2 -
> testcases/cve/cve-2015-3290.c | 8 +-
> testcases/kernel/crypto/af_alg02.c | 4 +-
> testcases/kernel/crypto/af_alg07.c | 1 +
> testcases/kernel/crypto/pcrypt_aead01.c | 3 +-
> testcases/kernel/io/ltp-aiodio/.gitignore | 1 -
> .../kernel/io/ltp-aiodio/aiodio_append.c | 25 ++-
> .../kernel/io/ltp-aiodio/aiodio_sparse.c | 26 +++-
> testcases/kernel/io/ltp-aiodio/common.h | 12 +-
> testcases/kernel/io/ltp-aiodio/dio_append.c | 4 +
> testcases/kernel/io/ltp-aiodio/dio_read.c | 9 ++
> testcases/kernel/io/ltp-aiodio/dio_sparse.c | 6 +-
> testcases/kernel/io/ltp-aiodio/dio_truncate.c | 6 +
> .../kernel/io/ltp-aiodio/read_checkzero.c | 84 ----------
> testcases/kernel/mem/mtest01/mtest01.c | 14 +-
> testcases/kernel/mem/mtest06/mmap1.c | 24 +--
> testcases/kernel/mem/mtest06/mmap3.c | 12 +-
> testcases/kernel/mem/mtest07/mallocstress.c | 11 +-
> testcases/kernel/mem/oom/oom01.c | 2 +-
> testcases/kernel/mem/oom/oom02.c | 2 +-
> testcases/kernel/mem/oom/oom03.c | 2 +-
> testcases/kernel/mem/oom/oom04.c | 2 +-
> testcases/kernel/mem/oom/oom05.c | 2 +-
> testcases/kernel/mem/thp/thp04.c | 1 +
> .../kernel/mem/tunable/min_free_kbytes.c | 2 +-
> testcases/kernel/pty/pty03.c | 2 +-
> testcases/kernel/pty/pty05.c | 1 +
> testcases/kernel/pty/pty06.c | 1 +
> testcases/kernel/pty/pty07.c | 1 +
> testcases/kernel/sound/snd_seq01.c | 2 +-
> testcases/kernel/sound/snd_timer01.c | 1 +
> testcases/kernel/syscalls/bind/bind06.c | 2 +-
> .../syscalls/clock_gettime/clock_gettime01.c | 1 -
> .../kernel/syscalls/fanotify/fanotify22.c | 1 -
> .../syscalls/gettimeofday/gettimeofday02.c | 15 +-
> testcases/kernel/syscalls/inotify/inotify06.c | 7 +-
> testcases/kernel/syscalls/inotify/inotify09.c | 1 +
> testcases/kernel/syscalls/inotify/inotify11.c | 1 -
> .../kernel/syscalls/ipc/shmctl/shmctl05.c | 2 +-
> .../kernel/syscalls/move_pages/move_pages12.c | 4 +-
> .../perf_event_open/perf_event_open03.c | 15 +-
> testcases/kernel/syscalls/readv/readv01.c | 1 -
> .../rt_sigqueueinfo/rt_sigqueueinfo01.c | 1 -
> testcases/kernel/syscalls/sendmsg/sendmsg03.c | 1 +
> .../kernel/syscalls/setsockopt/setsockopt06.c | 2 +-
> .../kernel/syscalls/setsockopt/setsockopt07.c | 1 +
> .../kernel/syscalls/setsockopt/setsockopt09.c | 1 -
> testcases/kernel/syscalls/tgkill/tgkill03.c | 1 -
> .../syscalls/timerfd/timerfd_settime02.c | 1 +
> .../syscalls/userfaultfd/userfaultfd01.c | 1 -
> testcases/kernel/syscalls/writev/writev03.c | 2 +-
> 63 files changed, 339 insertions(+), 296 deletions(-)
> delete mode 100644 lib/newlib_tests/test10.c
> delete mode 100644 lib/newlib_tests/test12.c
> rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
> create mode 100644 lib/newlib_tests/test_runtime02.c
> delete mode 100644 testcases/kernel/io/ltp-aiodio/read_checkzero.c
--
Thank you,
Richard.
More information about the ltp
mailing list