diff --git a/README.md b/README.md index dcaa6ec2..5ee5a379 100644 --- a/README.md +++ b/README.md @@ -10,32 +10,24 @@ These are common instructions for applications in the extended maintainers catal Each application gets their own directory, used for documenting, setting up, configuring, building, running, testing and debugging the application. Common information, configuration scripts are collected together in other directories such as `utils/`. -## Instructions for Bincompat Apps +Instructions below assume you are running commands in the application directory. -The following are common information for examples of applications running in binary compatibility (i.e. **bincompat**) mode. -Instructions assume you are running commands in the application example subdirectory. +## Application Directory Contents -Applications running in binary compatibility mode use [`app-elfloader`](https://github.com/unikraft/app-elfloader) images to load and run native Linux ELFs (*Executable and Linking Format* files). -The images are generated using `library/base/`. -We call them `base` images or `base` kernels. - -The images are also stored in the Unikraft registry and can be pulled from there. - -### Directory Contents - -A typical directory for a bincompat app contains: +A typical directory for an application contains: * `Kraftfile`: build / run rules, including pulling the `base` image -* `Dockerfile`: filesystem, including binary and libraries -* `Makefile.docker`: used to generate the root filesystem from the `Dockerfile` rules +* `Dockerfile` (optional): filesystem specification, including binary and libraries +* `Makefile.docker` (optional): used to generate the root filesystem from the `Dockerfile` rules * `README.md`: specific application instructions, such as starting and testing an application * `config.yaml`: configuration file to generate script files to run the application -* specific application files, such as configuration files and source code files +* `single_test.sh`: basic application test +* specific application files, such as configuration files and source code files, typically part of the `rootfs/` directory -### Common Setup +## Common Setup -The following are required before building and running a bincompat app. -In case you have already done them for other bincompat runs, you can skip them: +The following are required before building and running an app. +In case you have already done them for other application runs, you can skip them: 1. [Install Unikraft's companion command-line toolchain `kraft`](https://unikraft.org/docs/cli). @@ -52,29 +44,37 @@ In case you have already done them for other bincompat runs, you can skip them: 1. Install Docker following the [official instructions](https://docs.docker.com/engine/install/). This can be either [Docker Engine](https://docs.docker.com/engine/) or [Docker Desktop](https://docs.docker.com/desktop/). -1. Start a [BuildKit](https://docs.docker.com/build/buildkit/) container to be used by KraftKit to build the filesystem from the Dockerfile. +1. Start a [BuildKit](https://docs.docker.com/build/buildkit/) container to be used by KraftKit to build the filesystem from the `Dockerfile`. While in the top-level directory of this repository, source the `utils/start-buildkit.sh` directory: ```console source utils/start-buildkit.sh ``` -### Basic Build, Run and Use +## Basic Build, Run and Use You build and run the application using [KraftKit](https://github.com/unikraft/kraftkit). Follow the specific instructions in the `README.md` file in the application directory for specifics on building and running with KraftKit and on using the application. -### Scripted Build, Run and Use +## Scripted Build, Run and Use For testing, debugging and, generally, for finer-grained control over the build and run process, we generate scripts that wrap KraftKit, Make, QEMU, Firecracker and other commands. -The `config.yaml` file basic configuration for building and runing the application. +The `config.yaml` file basic configuration for building and running the application. It is used to generate the scripts and configuration files required to build and run the application. -To generate these files, run the generator script: +To generate these files, run one of the generator scripts, depending on the application type: -```console -../../../utils/bincompat/generate.einitrd.py -``` +* If the application is running in binary-compatibility (*bincompat*) mode, then run: + + ```console + ../../../utils/bincompat/generate.einitrd.py + ``` + +* If the application is running in native mode, then run: + + ```console + ../../../utils/native/generate.py + ``` The generator script uses the `config.yaml` file to generates the `scripts/` directory and the `Makefile` file used to to configure, build and run the application with Unikraft. @@ -84,51 +84,43 @@ Note that the output may differ on your system, depending on the compilers and c ```text scripts/ |-- build/ +| |-- kraft-fc-arm64.sh* | |-- kraft-fc-x86_64.sh* +| |-- kraft-qemu-arm64.sh* | |-- kraft-qemu-x86_64.sh* -| |-- make-clang-13-fc-x86_64.sh* -| |-- make-clang-13-qemu-x86_64.sh* -| |-- make-clang-15-fc-x86_64.sh* -| |-- make-clang-15-qemu-x86_64.sh* -| |-- make-gcc-10-fc-x86_64.sh* -| |-- make-gcc-10-qemu-x86_64.sh* -| |-- make-gcc-11-fc-x86_64.sh* -| |-- make-gcc-11-qemu-x86_64.sh* -| |-- make-gcc-12-fc-x86_64.sh* -| |-- make-gcc-12-qemu-x86_64.sh* -| |-- make-gcc-9-fc-x86_64.sh* -| `-- make-gcc-9-qemu-x86_64.sh* +| |-- make-fc-arm64.sh* +| |-- make-fc-x86_64.sh* +| |-- make-qemu-arm64.sh* +| `-- make-qemu-x86_64.sh* |-- defconfig/ +| |-- fc-arm64 | |-- fc-x86_64 +| |-- qemu-arm64 | `-- qemu-x86_64 -|-- kernel/ |-- run/ -| |-- clang-13-fc-x86_64-nofs.json -| |-- clang-13-fc-x86_64-nofs.sh* -| |-- clang-13-qemu-x86_64-nofs.sh* -| |-- clang-15-fc-x86_64-nofs.json -| |-- clang-15-fc-x86_64-nofs.sh* -| |-- clang-15-qemu-x86_64-nofs.sh* -| |-- gcc-10-fc-x86_64-nofs.json -| |-- gcc-10-fc-x86_64-nofs.sh* -| |-- gcc-10-qemu-x86_64-nofs.sh* -| |-- gcc-11-fc-x86_64-nofs.json -| |-- gcc-11-fc-x86_64-nofs.sh* -| |-- gcc-11-qemu-x86_64-nofs.sh* -| |-- gcc-12-fc-x86_64-nofs.json -| |-- gcc-12-fc-x86_64-nofs.sh* -| |-- gcc-12-qemu-x86_64-nofs.sh* -| |-- gcc-9-fc-x86_64-nofs.json -| |-- gcc-9-fc-x86_64-nofs.sh* -| |-- gcc-9-qemu-x86_64-nofs.sh* +| |-- fc-arm64-initrd.json +| |-- fc-arm64-initrd.sh* +| |-- fc-x86_64-initrd.json +| |-- fc-x86_64-initrd.sh* +| |-- kraft-fc-arm64-nofs.sh* | |-- kraft-fc-x86_64-nofs.sh* -| `-- kraft-qemu-x86_64-nofs.sh* -`-- setup.sh* +| |-- kraft-qemu-arm64-nofs.sh* +| |-- kraft-qemu-x86_64-nofs.sh* +| |-- qemu-arm64-9pfs.sh* +| |-- qemu-arm64-initrd.sh* +| |-- qemu-x86_64-9pfs.sh* +| `-- qemu-x86_64-initrd.sh* +|-- setup.sh* +`-- test/ + |-- common.sh + |-- config + `-- test.sh* ``` The contents of the `scripts/` directory are: -- `setup.sh`: script to clone and set up required repositories in the `workdir/` directory +- `setup.sh`: script to clone and set up required repositories in the `workdir/` directory; + `workdir/` will be symlinked to `.unikraft/` for KraftKit-based builds - `defconfig/`: generated configuration files required by the Make-based build; they are generated from `Kraftfile` - `build/`: generated build scripts; @@ -136,7 +128,7 @@ The contents of the `scripts/` directory are: - `run/`: generated run scripts; there is a corresponding run script for each build output from the build phase; Firecracker builds also generate a corresponding `.json` file; -- `kernels/`: directory where to store generated builds (kernels). +- `test/`: generated test scripts To use the scripts, follow the steps below: @@ -150,40 +142,46 @@ To use the scripts, follow the steps below: ```text workdir/ - |-- apps/ - | `-- elfloader/ |-- libs/ - | |-- libelf/ - | `-- lwip/ + | |-- lwip/ + | |-- musl/ + | `-- nginx/ `-- unikraft/ ``` 1. Run a build script from the `build/` directory, such as: ```console - ./scripts/build/make-clang-13-fc-x86_64.sh + ./scripts/build/make-qemu-x86_64.sh ``` - The resulting kernel image is placed in the `kernels/` directory: + The resulting kernel image is located in the `workdir/build/` directory and is copied in the application root directory: ```console - $ ls -lh scripts/kernel/clang-13-nginx_fc-x86_64 - -rwxr-xr-x 2 razvand razvand 15M Jan 27 13:42 scripts/kernel/clang-13-nginx_fc-x86_64 + $ ls -lh nginx_qemu-x86_64 + -rwxr-xr-x 2 razvan razvan 1,8M mai 1 23:41 nginx_qemu-x86_64 + + $ ls -lh workdir/build/nginx_qemu-x86_64 + -rwxr-xr-x 2 razvan razvan 1,8M mai 1 23:41 workdir/build/nginx_qemu-x86_64 ``` 1. Use the corresponding run script in the `run/` directory, such as: ```console - ./scripts/run/clang-13-fc-x86_64-nofs.sh + ./scripts/run/qemu-x86_64-initrd.sh ``` A successful run shows an output such as: ```text - 1: Set IPv4 address 172.44.0.2 mask 255.255.255.0 gw 172.44.0.1 - en1: Added en1: Interface is up - Powered by Unikraft Telesto (0.16.1~a922af77) + Powered by + o. .o _ _ __ _ + Oo Oo ___ (_) | __ __ __ _ ' _) :_ + oO oO ' _ `| | |/ / _)' _` | |_| _) + oOo oOO| | | | | (| | | (_) | _) :_ + OoOoO ._, ._:_:_,\_._, .__,_:_, \___) + Telesto 0.16.3~3facdb22-custom ``` Depending on the running environment, use the following commands to close the virtual machine (if it's a server or something that keeps running): @@ -201,3 +199,74 @@ Depending on the running environment, use the following commands to close the vi ```console pkill -f firecracker ``` + +## Scripted Testing + +You can test the current kernel and libraries work for the application by running the testing scripts. +These scripts consist of: + +* The `single_test.sh` script in the application root directory. This is used to run a full test for the application. +* The scripts in the `scripts/test/` directory. + There are three files: + * `common.sh` contains the basic test functions - it is sourced by `single_test.sh` + * `test.sh` is the actual test script + * `config` is a generated configuration file to run the tests + +The steps to run the scripted tests are: + +1. View and edit the `scripts/test/config` file to remove tests that are not usable or relevant. + Typically, this means removing the `...-fc-arm64.sh` entries in the `RUN_SCRIPTS` array for tests done on x86_64 machines. + Because Firecracker (`fc`) requires full virtualization support, this cannot be achieved on an x86_64 machine when targeting `arm64`. + + Something else may be skipping build scripts and only using run scripts by using `SKIP_BUILD=1`. + +1. Run the `test.sh` script: + + ```console + ./scripts/test/test.sh + ``` + + You will get results such as: + + ```console + [build] kraft-fc-arm64 ... PASSED + [build] kraft-qemu-arm64 ... PASSED + [run] kraft-qemu-arm64-nofs ... PASSED + [build] make-qemu-arm64 ... PASSED + [run] qemu-arm64-9pfs ... PASSED + [run] qemu-arm64-initrd ... PASSED + [build] kraft-fc-x86_64 ... PASSED + [run] kraft-fc-x86_64-nofs ... FAILED + [build] make-fc-x86_64 ... PASSED + [run] fc-x86_64-initrd ... PASSED + [build] make-qemu-x86_64 ... PASSED + [run] qemu-x86_64-9pfs ... PASSED + [run] qemu-x86_64-initrd ... PASSED + [build] kraft-qemu-x86_64 ... PASSED + [run] kraft-qemu-x86_64-nofs ... PASSED + [build] make-fc-arm64 ... PASSED + ``` + +1. Check detailed information in the testing output directory in `scripts/test/`: + + ```console + $ ls scripts/test/2024-05-01-18:42:38/ + build-kraft-fc-arm64.log build-kraft-qemu-x86_64.log build-make-qemu-arm64.log run-kraft-fc-x86_64-nofs.log run-qemu-arm64-9pfs.log run-qemu-x86_64-initrd.log + build-kraft-fc-x86_64.log build-make-fc-arm64.log build-make-qemu-x86_64.log run-kraft-qemu-arm64-nofs.log run-qemu-arm64-initrd.log + build-kraft-qemu-arm64.log build-make-fc-x86_64.log run-fc-x86_64-initrd.log run-kraft-qemu-x86_64-nofs.log run-qemu-x86_64-9pfs.log + ``` + + There is a `.log` file for each test. + +1. To investigate failing tests, apart from checking the log output, you can run a single test by using the `single_test.sh` script: + + ```console + ./single_test.sh ./scripts/run/qemu-x86_64-9pfs.sh - + ``` + +### Notes + +For KraftKit-based runs, the `build` step must happen before the corresponding `run` step. +This is because KraftKit `run` command relies on the directory contents filled by the `build` command. + +You can view and edit different scripts for your own needs.