[LTP] [PATCH] Add documentation for KVM testing

[Martin Doucha]

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---
 doc/kvm-test-api.txt | 384 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 384 insertions(+)
 create mode 100644 doc/kvm-test-api.txt

diff --git a/doc/kvm-test-api.txt b/doc/kvm-test-api.txt
new file mode 100644
index 000000000..25b72172d
--- /dev/null
+++ b/doc/kvm-test-api.txt
@@ -0,0 +1,384 @@
+LTP KVM Test API
+================
+
+Testing KVM is more complex than other Linux features. Some KVM bugs allow
+userspace code running inside the virtual machine to bypass (emulated) hardware
+access restrictions and elevate its privileges inside the guest operating
+system. The worst types of KVM bugs may even allow the guest code to crash or
+compromise the physical host. KVM tests therefore need to be split into two
+components – a KVM controller program running on the physical host and a guest
+payload program running inside the VM. The cooperation of these two components
+allows testing even of bugs that somehow cross the virtualization boundary.
+
+NOTE: See also
+      https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines],
+      https://github.com/linux-test-project/ltp/wiki/C-Test-Case-Tutorial[C Test Case Tutorial],
+      https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API].
+
+1. Basic KVM test structure
+---------------------------
+
+KVM tests are simple C source files containing both the KVM controller code
+and the guest payload code separated by `#ifdef COMPILE_PAYLOAD` preprocessor
+condition. The file will be compiled twice. Once to compile the payload part,
+once to compile the KVM controller part and embed the payload binary inside.
+The result is a single self-contained binary that'll execute the embedded
+payload inside a KVM virtual machine and print results in the same format as
+a normal LTP test.
+
+A KVM test source should start with `#include "kvm_test.h"` instead of the
+usual `tst_test.h`. The `kvm_test.h` header file will include the other basic
+headers appropriate for the current compilation pass. Everything else in the
+source file should be enclosed in `#ifdef COMPILE_PAYLOAD ... #else ... #endif`
+condition, including any other header file includes. Note that the standard
+LTP headers are not available in the payload compilation pass, only the KVM
+guest library headers can be included.
+
+.Example KVM test
+[source,c]
+-------------------------------------------------------------------------------
+#include "kvm_test.h"
+
+#ifdef COMPILE_PAYLOAD
+
+/* Guest payload code */
+
+void main(void)
+{
+	tst_res(TPASS, "Hello, world!");
+}
+
+#else /* COMPILE_PAYLOAD */
+
+/* KVM controller code */
+
+static struct tst_test test = {
+	.test_all = tst_kvm_run,
+	.setup = tst_kvm_setup,
+	.cleanup = tst_kvm_cleanup,
+};
+
+#endif /* COMPILE_PAYLOAD */
+-------------------------------------------------------------------------------
+
+The KVM controller code is a normal LTP test and needs to define an instance
+of `struct tst_test` with metadata and the usual setup, cleanup, and test
+functions. Basic implementation of all three functions is provided by the KVM
+host library.
+
+On the other hand, the payload is essentially a tiny kernel that'll run
+on bare virtual hardware. It cannot access any files, Linux syscalls, standard
+library functions, etc. except for the small subset provided by the KVM guest
+library. The payload code must define a `void main(void)` function which will
+be the VM entry point of the test.
+
+2. KVM host library
+-------------------
+
+The KVM host library provides helper functions for creating and running
+a minimal KVM virtual machine.
+
+2.1 Data structures
+~~~~~~~~~~~~~~~~~~~
+
+[source,c]
+-------------------------------------------------------------------------------
+struct tst_kvm_instance {
+	int vm_fd, vcpu_fd;
+	struct kvm_run *vcpu_info;
+	size_t vcpu_info_size;
+	struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS];
+	struct tst_kvm_result *result;
+};
+-------------------------------------------------------------------------------
+
+`struct tst_kvm_instance` holds the file descriptors and memory buffers
+of a single KVM virtual machine:
+
+- `int vm_fd` is the main VM file descriptor created by `ioctl(KVM_CREATE_VM)`
+- `int vcpu_fd` is the virtual CPU filedescriptor created by
+  `ioctl(KVM_CREATE_VCPU)`
+- `struct kvm_run *vcpu_info` is the VCPU state structure created by
+  `mmap(vcpu_fd)`
+- `size_t vcpu_info_size` is the size of `vcpu_info` buffer
+- `struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS]` is the list
+  of memory slots defined in this VM. Unused memory slots have zero
+  in the `userspace_addr` field.
+- `struct tst_kvm_result *result` is a buffer for passing test result data
+  from the VM to the controller program, mainly `tst_res()`/`tst_brk()` flags
+  and messages.
+
+[source,c]
+-------------------------------------------------------------------------------
+struct tst_kvm_result {
+	int32_t result;
+	int32_t lineno;
+	uint64_t file_addr;
+	char message[0];
+};
+-------------------------------------------------------------------------------
+
+`struct tst_kvm_result` is used to pass test results and synchronization data
+between the KVM guest and the controller program. Most often, it is used
+to pass `tst_res()` and `tst_brk()` messages from the VM, but special values
+can also be used to send control flow requests both ways.
+
+- `int32_t result` is the message type, either one of the `TPASS`, `TFAIL`,
+  `TWARN`, `TBROK`, `TINFO` flags or a special control flow value. Errno flags
+  are not supported.
+- `int32_t lineno` is the line number for `tst_res()`/`tst_brk()` messages.
+- `uint64_t file_addr` is the VM address of the filename string for
+  `tst_res()`/`tst_brk()` messages.
+- `char message[0]` is the buffer for arbitrary message data, most often used
+  to pass `tst_res()`/`tst_brk()` message strings.
+
+2.2 Working with virtual machines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The KVM host library provides default implementation of the setup, cleanup
+and test functions for `struct tst_test` in cases where you do not need
+to customize the VM configuration. You can either assign these functions
+to the `struct tst_test` instance directly or call them from your own function
+that does some additional steps. All three function must be used together.
+
+- `void tst_kvm_setup(void)`
+- `void tst_kvm_run(void)`
+- `void tst_kvm_cleanup(void)`
+
+Note: `tst_kvm_run()` calls `tst_free_all()`. Calling it will free all
+previously allocated guarded buffers.
+
+- `void tst_kvm_validate_result(int value)` – Validate whether the value
+  returned in `struct tst_kvm_result.result` can be safely passed
+  to `tst_res()` or `tst_brk()`. If the value is not valid, the controller
+  program will be terminated with error.
+
+- `uint64_t tst_kvm_get_phys_address(const struct tst_kvm_instance *inst,
+  uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual
+  machine `inst` to the corresponding physical address. Returns 0 if
+  the virtual address is not mapped to any physical address. If virtual memory
+  mapping is not enabled in the VM or not available on the arch at all, this
+  function simply returns `addr` as is.
+
+- `int tst_kvm_find_phys_memslot(const struct tst_kvm_instance *inst,
+  uint64_t paddr)` – Returns index of the memory slot in KVM virtual machine
+  `inst` which contains the physical address `paddr`. If the address is not
+  backed by a memory buffer, returns -1.
+
+- `int tst_kvm_find_memslot(const struct tst_kvm_instance *inst,
+  uint64_t addr)` – Returns index of the memory slot in KVM virtual machine
+  `inst` which contains the virtual address `addr`. If the virtual address
+  is not mapped to a valid physical address backed by a memory buffer,
+  returns -1.
+
+- `void *tst_kvm_get_memptr(const struct tst_kvm_instance *inst,
+  uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual
+  machine `inst` to host-side pointer.
+
+- `void *tst_kvm_alloc_memory(struct tst_kvm_instance *inst, unsigned int slot,
+  uint64_t baseaddr, size_t size, unsigned int flags)` – Allocates a guarded
+  buffer of given `size` in bytes and installs it into specified memory `slot`
+  of the KVM virtual machine `inst` at base address `baseaddr`. The buffer
+  will be automatically page aligned at both ends. See the kernel
+  documentation of `KVM_SET_USER_MEMORY_REGION` ioctl for list of valid
+  `flags`. Returns pointer to page-aligned beginning of the allocated buffer.
+  The actual requested `baseaddr` will be located at
+  `ret + baseaddr % pagesize`.
+
+- `struct kvm_cpuid2 *tst_kvm_get_cpuid(int sysfd)` – Get a list of supported
+  virtual CPU features returned by `ioctl(KVM_GET_SUPPORTED_CPUID)`.
+  The argument must be an open file descriptor returned by `open("/dev/kvm")`.
+
+- `void tst_kvm_create_instance(struct tst_kvm_instance *inst,
+  size_t ram_size)` – Creates and fully initializes a new KVM virtual machine
+  with at least `ram_size` bytes of memory. The VM instance info will be
+  stored in `inst`.
+
+- `void tst_kvm_run_instance(struct tst_kvm_instance *inst)` – Executes
+  the program installed in KVM virtual machine `inst`. Any result messages
+  returned by the VM will be automatically printed to controller program output.
+
+- `void tst_kvm_destroy_instance(struct tst_kvm_instance *inst)` – Deletes
+  the KVM virtual machine `inst`. Note that the guarded buffers assigned
+  to the VM by `tst_kvm_create_instance()` or `tst_kvm_alloc_memory()` will
+  not be freed.
+
+The KVM host library does not provide any way to reset a VM instance back
+to initial state. Running multiple iterations of the test requires destroying
+the old VM instance and creating a new one. Otherwise the VM will exit
+without reporting any results on the second iteration and the test will fail.
+The `tst_kvm_run()` function handles this issue correctly.
+
+3. KVM guest library
+--------------------
+
+The KVM guest library provides a minimal implementation of both the LTP
+test library and the standard C library functions. Do not try to include
+the usual LTP or C headers in guest payload code, it will not work.
+
+3.1 Standard C functions
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+`#include "kvm_test.h"`
+
+The functions listed below are implemented according to the C standard:
+
+- `void *memset(void *dest, int val, size_t size)`
+- `void *memzero(void *dest, size_t size)`
+- `void *memcpy(void *dest, const void *src, size_t size)`
+- `char *strcpy(char *dest, const char *src)`
+- `char *strcat(char *dest, const char *src)`
+- `size_t strlen(const char *str)`
+
+3.2 LTP library functions
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`#include "kvm_test.h"`
+
+The KVM guest library currently provides the LTP functions for reporting test
+results. All standard result flags except for `T*ERRNO` are supported
+with the same rules as usual. However, the printf-like formatting is not
+implemented yet.
+
+- `void tst_res(int result, const char *message)`
+- `void tst_brk(int result, const char *message) __attribute__((noreturn))`
+
+A handful of useful macros is also available:
+
+- `TST_TEST_TCONF(message)` – Generates a test program that will simply print
+  a `TCONF` message and exit. This is useful when the real test cannot be
+  built due to missing dependencies or arch limitations.
+
+- `ARRAY_SIZE(arr)` – Returns the number of items in statically allocated
+  array `arr`.
+
+- `LTP_ALIGN(x, a)` – Aligns integer `x` to be a multiple of `a`, which
+  must be a power of 2.
+
+3.3 Arch independent functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`#include "kvm_test.h"`
+
+Memory management in KVM guest library currently uses only primitive linear
+buffer for memory allocation. There are no checks whether the VM can allocate
+more memory and the already allocated memory cannot be freed.
+
+- `void *tst_heap_alloc(size_t size)` – Allocates a block of memory on the heap.
+
+- `void *tst_heap_alloc_aligned(size_t size, size_t align)` – Allocates
+  a block of memory on the heap with the starting address aligned to given
+  value.
+
+3.4 x86 specific functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`#include "kvm_test.h"` +
+`#include "kvm_x86.h"`
+
+- `struct kvm_interrupt_frame` – Opaque arch-dependent structure which holds
+  interrupt frame information. Use the functions below to get individual values:
+
+- `uintptr_t kvm_get_interrupt_ip(const struct kvm_interrupt_frame *ifrm)` –
+  Get instruction pointer value from interrupt frame structure. This may be
+  the instruction which caused an interrupt or the one immediately after,
+  depending on the interrupt vector semantics.
+
+- `int (*tst_interrupt_callback)(void *userdata,
+  struct kvm_interrupt_frame *ifrm, unsigned long errcode)` – Interrupt handler
+  callback prototype. When an interrupt occurs, the assigned callback function
+  will be passed the `userdata` pointer that was given
+  to `tst_set_interrupt_callback()`, interrupt frame `ifrm` and the error
+  code `errcode` defined by the interrupt vector semantics. If the interrupt
+  vector does not generate an error code, `errcode` will be set to zero.
+  The callback function must return 0 if the interrupt was successfully
+  handled and test execution should resume. Non-zero return value means that
+  the interrupt could not be handled and the test will terminate with error.
+
+- `void tst_set_interrupt_callback(unsigned int vector,
+  tst_interrupt_callback func, void *userdata)` – Register new interrupt
+  handler callback function `func` for interrupt `vector`. The `userdata`
+  argument is an arbitrary pointer that will be passed to `func()` every time
+  it gets called. The previous interrupt handler callback will be removed.
+  Setting `func` to `NULL` will remove any existing interrupt handler
+  from `vector` and the interrupt will become fatal error.
+
+[source,c]
+-------------------------------------------------------------------------------
+struct page_table_entry_pae {
+	unsigned int present: 1;
+	unsigned int writable: 1;
+	unsigned int user_access: 1;
+	unsigned int write_through: 1;
+	unsigned int disable_cache: 1;
+	unsigned int accessed: 1;
+	unsigned int dirty: 1;
+	unsigned int page_type: 1;
+	unsigned int global: 1;
+	unsigned int padding: 3;
+	uint64_t address: 40;
+	unsigned int padding2: 7;
+	unsigned int prot_key: 4;
+	unsigned int noexec: 1;
+} __attribute__((__packed__));
+
+struct kvm_cpuid {
+	unsigned int eax, ebx, ecx, edx;
+};
+
+struct kvm_cregs {
+	unsigned long cr0, cr2, cr3, cr4;
+};
+-------------------------------------------------------------------------------
+
+`struct page_table_entry_pae` is the page table entry structure for PAE and
+64bit paging modes. See Intel(R) 64 and IA-32 Architectures Software
+Developer's Manual, Volume 3, Chapter 4 for explanation of the fields.
+
+- `uintptr_t kvm_get_page_address_pae(const struct page_table_entry_pae *entry)`
+  – Returns the physical address of the memory page referenced by the given
+  page table `entry`. Depending on memory mapping changes done by the test,
+  the physical address may not be a valid pointer. The caller must determine
+  whether the address points to another page table entry or a data page, using
+  the known position in page table hierarchy and `entry->page_type`. Returns
+  zero if the `entry` does not reference any memory page.
+
+- `void kvm_get_cpuid(unsigned int eax, unsigned int ecx,
+  struct kvm_cpuid *buf)` – Executes the CPUID instruction with the given
+  `eax` and `ecx` arguments and stores the results in `buf`.
+
+- `void kvm_read_cregs(struct kvm_cregs *buf)` – Copies the current values
+  of control registers to `buf`.
+
+- `uint64_t kvm_rdmsr(unsigned int msr)` – Returns the current value
+  of model-specific register `msr`.
+
+- `void kvm_wrmsr(unsigned int msr, uint64_t value)` – Stores `value`
+  into model-specific register `msr`.
+
+- `void kvm_exit(void) __attribute__((noreturn))` – Terminate the test.
+  Similar to calling `exit(0)` in a regular LTP test, although `kvm_exit()`
+  will terminate only one iteration of the test, not the whole host process.
+
+See Intel(R) 64 and IA-32 Architectures Software Developer's Manual
+for documentation of standard and model-specific x86 registers.
+
+4. KVM guest environment
+------------------------
+
+KVM guest payload execution begins with bootstrap code which will perform
+the minimal guest environment setup required for running C code:
+
+- Activate the appropriate CPU execution mode (IA-32 protected mode
+  on 32-bit x86 or the 64-bit mode on x86_64).
+- Create indentity mapping (virtual address = physical address) of the lower
+  2GB memory region, even if parts of the region are not backed by any host
+  memory buffers. The memory region above 2GB threshold is left unmapped
+  except for one memory page reserved for the `struct tst_kvm_result` buffer.
+- Initialize 8KB stack.
+- Install default interrupt handlers for standard CPU exception vectors.
+
+When the environment setup is complete, bootstrap will call `void main(void)`
+function implemented by the test program. To finish execution of guest payload,
+the test can either return from the `main()` function or call `kvm_exit()`
+at any point.
-- 
2.36.1