[LTP] [PATCH] mmap22: Bound child memory allocation to avoid OOM

[Attila Fazekas]

Currently, stress_child() allocates memory in an infinite loop without
bounds. Under memory pressure, once the cgroup memory limit is reached
and the parent's droppable pages are reclaimed, the child continues
allocating non-droppable heap memory. This eventually exhausts the cgroup
limit and triggers the cgroup OOM killer, terminating either the child or
the test parent process.

Fix this by bounding the child's allocation to CHILD_ALLOC_MAX
(ALLOC_SIZE + 32MB). The total requested memory (128MB parent + 160MB
child = 288MB) exceeds the 256MB cgroup limit by 32MB, which reliably
forces the kernel to reclaim MAP_DROPPABLE pages. Once CHILD_ALLOC_MAX is
reached, the child calls pause() to keep memory resident. Since 160MB is
well below the 256MB cgroup limit, non-droppable memory alone cannot
trigger an OOM.

Assisted-by: Gemini CLI
Signed-off-by: Attila Fazekas <afazekas@redhat.com>
---
 testcases/kernel/syscalls/mmap/mmap22.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/syscalls/mmap/mmap22.c b/testcases/kernel/syscalls/mmap/mmap22.c
index 0b95c578f..16fcb6dd8 100644
--- a/testcases/kernel/syscalls/mmap/mmap22.c
+++ b/testcases/kernel/syscalls/mmap/mmap22.c
@@ -26,6 +26,7 @@
 
 #define MEM_LIMIT (256 * TST_MB)
 #define ALLOC_SIZE (128 * TST_MB)
+#define CHILD_ALLOC_MAX (ALLOC_SIZE + 32 * TST_MB)
 
 static struct tst_cg_group *cg_child;
 static pid_t child;
@@ -33,13 +34,15 @@ static int page_size;
 
 static void stress_child(void)
 {
-	for (;;) {
+	for (size_t sz = 0; sz < CHILD_ALLOC_MAX; sz += page_size) {
 		char *buf = malloc(page_size);
 
 		if (!buf)
-			exit(1);
+			break;
 		memset(buf, 'B', page_size);
 	}
+
+	pause();
 }
 
 static void test_mmap(void)
-- 
2.56.0