[LTP] [PATCH] syscalls: unlimit the process's virtual memory (address space)

[Li Wang]

Recently our test environment changes RLIMIT_AS before LTP runs, so
these tests hit the per-process virtual-memory ceiling long before
they reach the kernel limits they are intended to exercise.

munmap04 exits after only ~36k VMAs and the partial munmap() succeeds:

  ==== munmap04 ====
  command: munmap04
  tst_test.c:1846: TINFO: Overall timeout per run is 0h 05m 24s
  munmap04.c:59: TINFO: Mapped 36280 regions
  munmap04.c:26: TFAIL: munmap(maps[2] + page_sz, page_sz) succeeded

cve-2017-17052’s MAP_POPULATE loop hits ENOMEM and TBROKs:

  ==== cve-2017-17052 ====
  command: cve-2017-17052
  cve-2017-17052.c:48: TBROK: mmap((nil),16777216,PROT_READ(1),32802,-1,0) failed: ENOMEM (12)
  tst_test.c:479: TINFO: Child process reported TBROK killing the test
  tst_test.c:1909: TINFO: Killed the leftover descendant processes

cve-2017-1000364 (stack_clash) fails to allocate the guard gap region:

  ==== cve-2017-1000364 ====
  command: stack_clash
  tst_kconfig.c:629: TINFO: stack_guard_gap is not found in /proc/cmdline
  stack_clash.c:293: TINFO: STACK_GROWSDOWN = 1 == 0x7ffc2455d7c0 > 0x7ffc2455d730
  stack_clash.c:243: TINFO: Stack:0x7ffc2445d000+0x103000 mmap:0x7ffc2435c000+0x1000
  stack_clash.c:127: TBROK: mmap(0x1326000,140444440563712,PROT_NONE(0),1048610,-1,0) failed: ENOMEM (12)
  tst_test.c:479: TINFO: Child process reported TBROK killing the test
  tst_test.c:1909: TINFO: Killed the leftover descendant processes

Explicitly set RLIMIT_AS=RLIM_INFINITY inside each test via struct
tst_test.ulimit, ensuring they always have unlimited virtual address
space regardless of test env ulimit settings.

Follow-up: c075f6c073 ("cve-2017-17052: tolerate ENOMEM during test")
Signed-off-by: Li Wang <liwang@redhat.com>
Cc: Xiubo Li <xiubli@redhat.com>
---

Notes:
    When I debugging munmap04 and stack_clash today, I suddenly realized
    that the cve-2017-17052 error was also closely related to the rlimit
    change, which was intentionally changed by Xiubo (my colleague) to
    stabilize some memory pressure scenarios.
    
    However, this change unexpectedly exposed the need to pay attention to
    RLIMIT_AS in these tests, which also should be fixed in LTP upstream.
    
    I think this patch is valuable to be merged into the new version.

 testcases/cve/cve-2017-17052.c              | 4 ++++
 testcases/cve/stack_clash.c                 | 4 ++++
 testcases/kernel/syscalls/munmap/munmap04.c | 4 ++++
 3 files changed, 12 insertions(+)

diff --git a/testcases/cve/cve-2017-17052.c b/testcases/cve/cve-2017-17052.c
index f1bff3db2..4d9448c27 100644
--- a/testcases/cve/cve-2017-17052.c
+++ b/testcases/cve/cve-2017-17052.c
@@ -130,6 +130,10 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.setup = setup,
 	.test_all = run,
+	.ulimit = (const struct tst_ulimit_val[]) {
+		{RLIMIT_AS, RLIM_INFINITY},
+		{}
+	},
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "2b7e8665b4ff"},
 		{"CVE", "2017-17052"},
diff --git a/testcases/cve/stack_clash.c b/testcases/cve/stack_clash.c
index 0584b2f7a..e9dd0736a 100644
--- a/testcases/cve/stack_clash.c
+++ b/testcases/cve/stack_clash.c
@@ -331,6 +331,10 @@ static struct tst_test test = {
 	.needs_root = 1,
 	.setup = setup,
 	.test_all = stack_clash_test,
+	.ulimit = (const struct tst_ulimit_val[]) {
+		{RLIMIT_AS, RLIM_INFINITY},
+		{}
+	},
 	.tags = (const struct tst_tag[]) {
 		{"CVE", "2017-1000364"},
 		{"linux-git", "58c5d0d6d522"},
diff --git a/testcases/kernel/syscalls/munmap/munmap04.c b/testcases/kernel/syscalls/munmap/munmap04.c
index e1b67aeee..6ccf25b3b 100644
--- a/testcases/kernel/syscalls/munmap/munmap04.c
+++ b/testcases/kernel/syscalls/munmap/munmap04.c
@@ -75,6 +75,10 @@ static struct tst_test test = {
 	.cleanup = cleanup,
 	.needs_root = 1,
 	.min_kver = "4.17",
+	.ulimit = (const struct tst_ulimit_val[]) {
+		{RLIMIT_AS, RLIM_INFINITY},
+		{}
+	},
 	.save_restore = (const struct tst_path_val[]){
 		{ "/proc/sys/vm/max_map_count", TST_TO_STR(MAP_MAX_COUNT), TST_SR_SKIP },
 		{},
-- 
2.52.0