Running Tests

Running Tests

List available tests

Quicktest can list all the available tests.

quicktest --list-tests

./quicktest --list-tests
Available test cases
OS             Release / Edition    Test case                                     Short description
alpine         v3.11                test_boot_to_login                            # Test we can boot to the login screen and login and out then shutdown
alpine         v3.19                test_boot_to_login                            # Test we can boot to the login screen and login and out then shutdown
ubuntu         24.04                test_boot_to_live_environment                 # Boot to live environment
ubuntu         24.04                test_install_entire_disk_with_defaults        # Install Ubuntu 24.04 using defaults
ubuntu         daily-live           test_boot_to_live_environment                 # Boot to welcome screen
ubuntu         daily-live           test_install_entire_disk_with_defaults        # Install Ubuntu daily live (oracular) using defaults
ubuntu         daily-live           test_install_entire_disk_with_defaults_fde    # Install Ubuntu daily live (oracular) using defaults and encryption
ubuntu-mate    24.04                test_boot_to_live_environment                 # Boot to live environment
ubuntu-mate    24.04                test_install_entire_disk_with_defaults        # Install Ubuntu MATE 24.04 using defaults
ubuntu-mate    daily-live           test_boot_to_live_environment                 # Boot to live environment
ubuntu-mate    daily-live           test_install_entire_disk_with_defaults        # Install Ubuntu MATE daily live (oracular) using defaults

Testing Ubuntu 24.04

Currently, quicktest runs tests for Ubuntu 24.04 by default (this is subject to change). So the following command will run this test in an Ubuntu 24.04 virtual machine.

quicktest test_install_entire_disk_with_defaults

Testing other releases

The same test name is also available for other flavours and releases. To override the default, specify on the command line. For example, this will run a full installation from start to finish, on Ubuntu MATE 24.10 - using the daily-live image.

quicktest test_install_entire_disk_with_defaults ubuntu-mate daily-live

Running on a server

By default Quicktest is expected to run on a laptop or desktop of an enthusiast QA contributor. As such it does things like send desktop notifications, and open the results in the desktop file manager. These are less sensible on a remote headless server. So these options can be switched off.

For example:

QUICKEMU_DISPLAY=none QT_OPEN_RESULTS=false QT_NOTIFY=false ./quicktest test_boot_to_login alpine v3.19

Options

Quicktest features a number of settings which can be changed either by editing the source, or preferably, by setting appropriate environment variables.

To change an option, specify or export it in the environment. For example, to change the screen resolution of the quickemu virtual machine:

QUICKEMU_WIDTH=1368 QUICKEMU_HEIGHT=768 ./quicktest test_install_entire_disk_with_defaults

Quicktest Settings

• QT_TESTCASES_DIR - path to the top level directory where testcases are stored. This can be useful to change if you're working on a folder of test cases that are not shipped with Quicktest.
• QT_LOG - the filename of the log file created as a copy of the terminal output.
• QT_SUMMARY_LOG - the path and file name of the summary log file, which contains one line per test, pass or fail.
• QT_KEYMAPS_DIR - the path to the top level directory where keymaps are stored. These contain mappings for keys (such as shift-f3 for £ or #) to inject characters into Qemu.
• QT_NOTIFY - boolean determining if Quicktest will send a desktop notification at the end of a test to indicate pass or fail. This is useful if running Quicktest in 'headless' mode.
• QT_TEST_LANG - the language used when running tests. There are separate files in the testcases folder under i18n to enable tests to be run in different languages, and thus look for different text labels.
• QT_CREATE_TIMELAPSE - boolean (default: true) - whether or not to generate a timelapse of the screenshots taken during the test case execution. ffmpeg is required for this.
• QT_TIMELAPSE_VIDEO - filename of the mp4 video created after a test is complete.
• QT_SCREENSHOT_COUNT - default: 0 - every screenshot is indexed with a unique number, which starts from 0. If, for some reason there's a requirement to start from another number, use this to change that initial value.
• QEMU_INSERT_DELAY - default: 0.2 - the pause between simulated 'keypresses' when Quicktest is 'typing' in Qemu.
• QT_KEEP_SCREENSHOTS - boolean (default: false) - whether to keep the screenshot ppm files in the log directory, after making a timelapse. They tend to take up a lot of space, so by default they're removed. During debugging it's useful to make this true.
• QT_KEEP_TESSERACT_TEXT - boolean (default: false) - whether to keep the text files tesseract generates from screenshots. Useful to enable when debugging to see what tesseract "sees".
• QT_DISABLE_VM_RUNNING_ERROR - boolean (default: false) - if Quicktest detects that the virtual machine we're launching is already running, it errors out. However, sometimes, a test may need the vm to be up and running already - so we can disable this error.
• QT_VM_WARNING_TIMEOUT - default: 10 - if we set QT_DISABLE_VM_RUNNING_ERROR to true, then there's a final reminder to the user that the VM is running, followed by a short delay. This setting configures that delay. It's there so the user can CTRL+C and abandon the test if they accidentally left a VM running, and don't wish to ruin it with the next test which is about to run.
• QT_DISABLE_DISK_WARNING - boolean (default: false) - most tests (currently) expect the hard disk of the virtual machine to be empty (as they are mostly 'install' type tests). Sometimes a test will be run after another, and will expect there to be data on the disk. This setting allows the warning notice to be disabled.
• QT_OPEN_RESULTS - boolean (default: true) - at the end of the test case run, an open $RESULTS command is run, to launch the desktop file manager on the host, so the user can examine the results. This option disables that activity, which may be useful when running Quicktest on a server or in a CI/CD environment.

Quickemu Settings

• QUICKEMU_VM_DIR - path to top level folder where Quickemu will store virtual machine ISOs, disks and configuration. This may be useful to change if it is desired to put the large disk images on a different disk.
• QUICKEMU_DISPLAY - Quickemu supports 'spice', 'gtk', 'sdl' as options for the graphical frontend used to view and control the virtual machine. Also supported is 'none', which enables so-called 'headless' mode.
• QUICKEMU_WIDTH - horizontal resolution of the virtual machine.
• QUICKEMU_HEIGHT - vertical resolution of the virtual machine.
• QUICKEMU_NETWORK - a boolean determining whether a network connection will be present in the virtual machine. It is useful to set 'false' if testing whether an installer behaves differently when no network is present.
• QUICKEMU_OPTS - an empty variable which can be used to pass additional parameters to the quickemu command-line.

Tesseract OCR Settings

• TESSERACT_OCR_OPTIONS - options sent to the tesseract command-line.
• QT_SCALE_IMAGE - how much to scale screenshot images by (default: 300%) before attempting to OCR them. This has been found to improve OCR performance of Tesseract.
• QT_SCALE_COMMAND - the binary used to do the scaling. Typically that's just /usr/bin/convert but may differ in other environments.
• TESSERACT_LANG - the language tesseract is looking for in screenshots. Tesseract ships datafiles, usually packaged as tesseract-ocr-eng (for example) which must be installed to match this setting.