diff --git a/docs/.wordlist.txt b/docs/.wordlist.txt index 36bafd7d..72731fdf 100644 --- a/docs/.wordlist.txt +++ b/docs/.wordlist.txt @@ -26,15 +26,20 @@ Grafana IAM init installable +Instantiation Intel +IoT Jira JSON Juju Kubeflow Kubernetes +lenovo Lenovo lifecycle logfile +LTS +LVM maas MaaS MAAS @@ -57,7 +62,9 @@ oemscript OLM passwordless Permalink +preconfigured preloaded +Pre provisionable provisioner ReadMe @@ -71,10 +78,13 @@ runtime SDWire SecureBoot SKU +SQA SSID subdirectories subfolders subtree +SUT +Tegra templating Testflinger TPM @@ -85,14 +95,11 @@ UI URI USB UUID +virtualenv VM webhook WPA xenial +xz yaml -LTS -virtualenv -IoT -Tegra zstd -xz diff --git a/docs/custom_conf.py b/docs/custom_conf.py index 03109a89..b12124f6 100644 --- a/docs/custom_conf.py +++ b/docs/custom_conf.py @@ -62,7 +62,7 @@ # Change to the discourse instance you want to be able to link to # using the :discourse: metadata at the top of a file # (use an empty value if you don't want to link) - 'discourse': 'https://discourse.ubuntu.com', + 'discourse': 'https://discourse.ubuntu.com/c/certification/165', # Change to the GitHub info for your project 'github_url': 'https://github.com/canonical/testflinger', @@ -110,7 +110,8 @@ # (This list will be appended to linkcheck_anchors_ignore_for_url) custom_linkcheck_anchors_ignore_for_url = [ - ] + r'https://maas\.io/docs/.*#' # Ignore anchors on this page +] ############################################################ ### Additions to default configuration diff --git a/docs/reference/device-connector-types.rst b/docs/reference/device-connector-types.rst index 6a3565c5..14510e1d 100644 --- a/docs/reference/device-connector-types.rst +++ b/docs/reference/device-connector-types.rst @@ -15,13 +15,13 @@ To specify the commands to run by the device in each test phase, set the ``testf * - ``dragonboard`` - Qualcomm Dragonboard 410c setup to boot from both a special image on a USB stick when the SD card is erased, as well as an SD card that can be provisioned by booting the stable image on a USB stick and then flashing the new image to the SD card. * - ``maas2`` - - Uses `MaaS `_ to provision supported images on devices that are capable of being controlled by a MaaS server. + - Uses `MAAS `_ to provision supported images on devices that are capable of being controlled by a MAAS server. * - ``multi`` - Experimental device type that is used for provisioning multiple other devices in order to coordinate a job across multiple devices at the same time. * - ``muxpi`` - MuxPi or SDWire device capable of multiplexing the SD card so that it can be written, then control can be switched to the DUT to boot the image, see :ref:`muxpi`. * - ``netboot`` - - Special purpose device connector for a few devices that must be booted and flashed remotely but the image they need is not compatible with MaaS. + - Special purpose device connector for a few devices that must be booted and flashed remotely but the image they need is not compatible with MAAS. * - ``noprovision`` - General device connector that does not support provisioning, but can run tests on a device where provisioning is not needed or not possible to do automatically. * - ``oemrecovery`` @@ -33,6 +33,73 @@ To specify the commands to run by the device in each test phase, set the ``testf * - ``hp_oemscript`` - This device connector is used for HP OEM devices running certain versions of OEM supported images that can use a recovery partition to recover not only the same image, but in some cases, other OEM image versions as well. +.. _cm3: + +cm3 +--- + +The ``cm3`` device connector supports the following ``provision_data`` keys: + +.. list-table:: Supported ``provision_data`` keys for ``cm3`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the device, which will be used to boot up the DUT. + +.. _dragonboard: + +dragonboard +----------- + +The ``dragonboard`` device connector supports the following ``provision_data`` keys: + +.. list-table:: Supported ``provision_data`` keys for ``dragonboard`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the SD card, which will be used to boot up the DUT. + +.. _maas2: + +maas2 +----- + +The ``maas2`` device connector supports the following ``provision_data`` keys: + +.. list-table:: Supported ``provision_data`` keys for ``maas2`` + :header-rows: 1 + + * - Key + - Description + * - ``distro`` + - Name of the image to be used for provisioning. This is the name of the + image as it appears in the MAAS web UI and must already be imported into MAAS. + * - ``kernel`` + - Specify a kernel to use during deployment. This is the name of the + kernel as it appears in the MAAS web UI and must already be imported into MAAS. + For more information, see + `MAAS documentation: Set a specific kernel `_. + on this topic + * - ``user_data`` + - A string containing base64 encoded cloud-init user data to be used for provisioning. + For more information, see + `MAAS documentation: Pre-seed cloud-init `_. + on this topic + * - ``disks`` + - Specify a custom disk configuration for the machine. For more information, see the + :doc:`maas_storage`. + + .. _muxpi: muxpi @@ -83,3 +150,97 @@ Image types recognised for user account creation - Ubuntu Core 20 * - ``ubuntu-cpc`` - Ubuntu Certified Public Cloud + +.. _netboot: + +netboot +------- + +The ``netboot`` device connector supports the following ``provision_data`` keys: + +.. list-table:: Supported ``provision_data`` keys for ``netboot`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the device, which will be used to boot up the DUT. + +.. _noprovision: + +noprovision +----------- + +The ``noprovision`` device connector does not support any ``provision_data`` keys. +However, you can specify any key in this dictionary (example: ``skip: false``) in +order to ensure the provision step is run. The only effect this will have, is to +ensure that the system is reachable with ssh before proceeding to the next step. + +.. _oemrecovery: + +oemrecovery +----------- + +The ``oemrecovery`` device connector does not support any ``provision_data`` keys. +Instead, this device connector uses a preconfigured command to reset the device back +to its original state. In order to ensure that the provision step is run, and the +system is reset back to the original state, you can specify any key in this dictionary +(example: ``skip: false``). If you do not want the provision step to run, you can +simply leave out the ``provision_data`` section. + +.. _dell_oemscript: + +dell_oemscript +-------------- + +The ``dell_oemscript`` device connector does not support any ``provision_data`` keys. + +.. list-table:: Supported ``provision_data`` keys for ``dell_oemscript`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the device, which will be used to boot up the DUT. + +.. _lenovo_oemscript: + +lenovo_oemscript +---------------- + +The ``lenovo_oemscript`` device connector does not support any ``provision_data`` keys. + +.. list-table:: Supported ``provision_data`` keys for ``lenovo_oemscript`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the device, which will be used to boot up the DUT. + +.. _hp_oemscript: + +hp_oemscript +------------ + +The ``hp_oemscript`` device connector does not support any ``provision_data`` keys. + +.. list-table:: Supported ``provision_data`` keys for ``hp_oemscript`` + :header-rows: 1 + + * - Key + - Description + * - ``url`` + - URL to a compressed disk image that is downloaded, decompressed using + ``unzstd`` (**xz** format is recommended, but any format supported by + the ``zstd`` tool is supported) and + flashed to the device, which will be used to boot up the DUT. \ No newline at end of file diff --git a/docs/reference/index.rst b/docs/reference/index.rst index cc32f893..efc8047a 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -12,6 +12,7 @@ Working with device connectors job-schema device-connector-types device-connector-conf + maas_storage Working with Testflinger servers and agents diff --git a/device-connectors/src/testflinger_device_connectors/devices/maas2/doc/maas_storage.rst b/docs/reference/maas_storage.rst similarity index 51% rename from device-connectors/src/testflinger_device_connectors/devices/maas2/doc/maas_storage.rst rename to docs/reference/maas_storage.rst index d398c2c5..1a350d44 100644 --- a/device-connectors/src/testflinger_device_connectors/devices/maas2/doc/maas_storage.rst +++ b/docs/reference/maas_storage.rst @@ -1,70 +1,67 @@ -================= -Preamble -================= -This extension of the Testflinger Maas Testflinger Device Connector to handle a variety of node storage layout configurations. This configuration will be passed to Testflinger via the node job configuration yaml file, as part of the SUT provision data (example below). This functionality is containted in the discreet Python module (‘maas-storage.py’) that sits alongside the Maas Testflinger Device Connector, to be imported and called when this device connector is instantiated, if a storage layout configuration is supplied. +MAAS Custom Storage Configuration +================================= +This extension of the Testflinger MAAS Testflinger Device Connector to handle a variety of node storage layout configurations. This configuration will be passed to Testflinger via the node job configuration yaml file, as part of the SUT provision data (example below). This functionality is contained in the discreet Python module (``maas-storage.py``) that sits alongside the MAAS Testflinger Device Connector, to be imported and called when this device connector is instantiated, if a storage layout configuration is supplied. -These storage layout configurations are to be passed along to MAAS, via the CLI API, when the device connector is created as part of its provision phase. While initial scope and use of this module will be limited to SQA’s testing requirements, the availability of this module implies additional consumers can specify disk layout configurations as part of their Testflinger job definitions. +These storage layout configurations are to be passed along to MAAS, via the CLI API, when the device connector is created as part of its provision phase. While initial scope and use of this module will be limited to SQA testing requirements, the availability of this module implies additional consumers can specify disk layout configurations as part of their Testflinger job definitions. -Of note: the initial scope of storage to be supported will be limited to flat layouts and simple partitions; RAID, LVM or bcache configurations are currently unnsupported by this module. This functionality will be added in the future as the need arises. +Of note: the initial scope of storage to be supported will be limited to flat layouts and simple partitions; RAID, LVM or ``bcache`` configurations are currently unsupported by this module. This functionality will be added in the future as the need arises. -================= Job Configuration -================= +----------------- The storage configuration is traditionally supplied as a node bucket config, so we can duplicate how this is laid out in the SUT job configuration at the end of this document. -As below, the storage configuration is defined under ‘disks’ key in the yaml file. It is composed of a list of storage configuration entries, which are dictionaries with at least these two fields, ‘type’ and ‘id’: +As below, the storage configuration is defined under ``disks`` key in the yaml file. It is composed of a list of storage configuration entries, which are dictionaries with at least these two fields, ``type`` and ``id``: + - **type**: the type of configuration entry. Currently this is one of: - - *disk* - a physical disk on the system - - *partition* - a partition on the system - - *format* - instructions to format a volume - - *mount* - instructions to mount a formatted partition -- id: a label used to refer to this storage configuration entry. Other configuration entries will use this id to refer to this configuration item, or the component it configured. + - ``disk`` - a physical disk on the system + - ``partition`` - a partition on the system + - ``format`` - instructions to format a volume + - ``mount`` - instructions to mount a formatted partition +- **id**: a label used to refer to this storage configuration entry. Other configuration entries will use this id to refer to this configuration item, or the component it configured. -================= Storage Types -================= +------------- Each type of storage configuration entry has additional fields of: -- **Disk**: all storage configuration is ultimately applied to disks. These are referenced by ‘disk’ storage configuration entries. - - *disk* - The key of a disk in the machine's hardware configuration. By default, this is an integer value like ‘0’ or ‘1.’ - - *ptable* - Type of partition table to use for the disk (‘gpt’ or ‘msdos’). - - *name* - Optional. If specified, the name to use for the disk, as opposed to the default one which is usually something like 'sda'. This will be used in ‘/dev/disk/by-dname/’ for the disk and its partitions. So if you make the disk name 'rootdisk', it will show up at . This can be used to give predictable, meaningful names to disks, which can be referenced in juju config, etc. - - *boot* - Optional. If specified, this disk will be set as boot disk in MAAS. - - The disk's ‘id’ will be used to refer to this disk in other entries, such as ‘partition.’ -- **Partition**: A partition of a disk is represented by a ‘partition’ entry. - - *device* - The ‘id’ of the ‘disk’ entry this partition should be created on. - - *number* - Partition number. This determines the order of the partitions on the disk. - - *size* - The minimum required size of the partition, in bytes, or in larger units, given by suffixes (K, M, G, T) - - The partition's ‘id’ will be used to refer to this partition in other entries, such as ‘format.’ - - *alloc_pct* - Percent (as an int) of the parent disk this partition should consume. This is optional, if this is not given, then the ‘size’ value will be the created partition size. If multiple partitions exist on a parent disk, the total alloc_pct between them cannot exceed 100. -- **Format**: Instructions to format a volume are represented by a ‘format’ entry. - - *volume* - the ‘id’ of the entry to be formatted. This can be a disk or partition entry. - - *fstype* - The file system type to format the volume with. See MAAS docs for options. - - *label* - The label to use for the file system. - - The format's ‘id’ will be used to refer to this formatted volume in ‘mount’ entries. -- **Mount:** Instructions to mount a formatted volume are represented by a ‘mount’ entry. - - *device* - The ‘id’ of the ‘format’ entry this mount refers to. - - *path* - The path to mount the formatted volume at, e.g. ‘/boot.’ -================= +- **Disk**: all storage configuration is ultimately applied to disks. These are referenced by ``disk`` storage configuration entries. + - ``disk`` - The key of a disk in the machine's hardware configuration. By default, this is an integer value like ``0`` or ``1``. + - ``ptable`` - Type of partition table to use for the disk (``gpt`` or ``msdos``). + - ``name`` - Optional. If specified, the name to use for the disk, as opposed to the default one which is usually something like ``sda``. This will be used in ``/dev/disk/by-dname/`` for the disk and its partitions. So if you make the disk name ``rootdisk``, it will show up at <``/dev/disk/by-dname/rootdisk``>. This can be used to give predictable, meaningful names to disks, which can be referenced in Juju config, etc. + - ``boot`` - Optional. If specified, this disk will be set as boot disk in MAAS. + - The disk's ``id`` will be used to refer to this disk in other entries, such as ``partition``. +- **Partition**: A partition of a disk is represented by a ``partition`` entry. + - ``device`` - The ``id`` of the ``disk`` entry this partition should be created on. + - ``number`` - Partition number. This determines the order of the partitions on the disk. + - ``size`` - The minimum required size of the partition, in bytes, or in larger units, given by suffixes (K, M, G, T) + - The partition's ``id`` will be used to refer to this partition in other entries, such as ``format``. + - ``alloc_pct`` - Percent (as an int) of the parent disk this partition should consume. This is optional, if this is not given, then the ``size`` value will be the created partition size. If multiple partitions exist on a parent disk, the total ``alloc_pct`` between them cannot exceed 100. +- **Format**: Instructions to format a volume are represented by a ``format`` entry. + - ``volume`` - the ``id`` of the entry to be formatted. This can be a disk or partition entry. + - ``fstype`` - The file system type to format the volume with. See MAAS docs for options. + - ``label`` - The label to use for the file system. + - The format's ``id`` will be used to refer to this formatted volume in ``mount`` entries. +- **Mount**: Instructions to mount a formatted volume are represented by a ``mount`` entry. + - ``device`` - The ``id`` of the ``format`` entry this mount refers to. + - ``path`` - The path to mount the formatted volume at, e.g. ``/boot``. + Storage Configuration Instantiation -================= +----------------------------------- - The existing storage configuration on the SUT is first cleared in order to start with a clean slate. -- We will then fetch the SUT block device config via the Maas API in order to verify and choose the appropriate physical disks which exist on the system. These disks must align with the configuration parameters (size, number of disks) presented in the config to proceed. +- We will then fetch the SUT block device config via the MAAS API in order to verify and choose the appropriate physical disks which exist on the system. These disks must align with the configuration parameters (size, number of disks) presented in the config to proceed. - Disk selection should be performed with the following criteria: - - In instances where all disks meet the space requirements, we can numerically assign the lowest physical disk ID (in Maas block-devices) to the first config disk. Subsequent disks will be assigned in numerical order. + - In instances where all disks meet the space requirements, we can numerically assign the lowest physical disk ID (in MAAS block-devices) to the first config disk. Subsequent disks will be assigned in numerical order. - In instances where the total of a config disk’s partition map (determined by adding all configuration partitions on that disk) will only fit on certain node disks, these disks will only be selected for the parent configuration disk of said partition map. - Disk selection will be done in numerical order as above within any smaller pool of disks that meet configuration partitioning criteria. - Node provisioning will fail if configuration partition maps exist that will not adequately fit on any disk, or if the pool of appropriate disks is exhausted prior to accommodating all configuration partition maps. - - However, dynamic allocation of partition sizes using the alloc_pct field will enable a much more flexible allocation of partitions to parent disks, and one only needs to be able to provide the minimum partition size in order to select the most appropriate disk. -- After disk selection takes place, all configuration elements of each storage type will be grouped together for batch processing. This order is determined by the dependency each type has on the other. The types and the order in which they will be processed will be: [‘disk’, ‘partition’, ‘format’, ‘mount’]. + - However, dynamic allocation of partition sizes using the ``alloc_pct`` field will enable a much more flexible allocation of partitions to parent disks, and one only needs to be able to provide the minimum partition size in order to select the most appropriate disk. +- After disk selection takes place, all configuration elements of each storage type will be grouped together for batch processing. This order is determined by the dependency each type has on the other. The types and the order in which they will be processed will be: [``disk``, ``partition``, ``format``, ``mount``]. - As additional storage types are supported in the future, this order will need to remain consistent with any parent-child relationship that exists between storage types. - The storage configuration will then be written to the node disks in this order. - - If a boot partition exists in the configuration, the parent disk will be flagged as a boot disk via the Maas API. The boot partition will then be created on this disk, including an EFI mount if desired. + - If a boot partition exists in the configuration, the parent disk will be flagged as a boot disk via the MAAS API. The boot partition will then be created on this disk, including an EFI mount if desired. - After the storage configuration is completed and written to the node’s physical disks, node provisioning will proceed to OS installation, in addition to any other provisioning steps outside of the node’s storage subsystem. -================= Job Definition Reference -================= +------------------------ .. code-block:: yaml :caption: job.yaml :linenos: diff --git a/docs/tutorial/index.rst b/docs/tutorial/index.rst index 5185f517..a7f61d54 100644 --- a/docs/tutorial/index.rst +++ b/docs/tutorial/index.rst @@ -118,7 +118,7 @@ If the connection is successful, a list of job queues is returned with their que example-queue-3 - for testing device model-3 ... -In this tutorial, let's assume that the job queue you will use is ``example-queue-1``. +In this tutorial, let's assume that the job queue you will use is ``example-queue-1`` for a MAAS-provisioned device type. Alternatively, you can also visit the Web UI of this server at ``https://testflinger.example.com``, where the list of agents, queues and jobs are displayed. @@ -130,13 +130,13 @@ Test jobs are YAML or JSON files that define the configurations and instructions A test job might contain a very complex command workflow that includes provisioning a system image onto the device, updating the firmware, executing a series test and more. In this tutorial, you will start with a simple test job. -The following example shows a test job, written in YAML, that provisions an Ubuntu Jammy system image on the target device and then prints the distribution information: +The following example shows a test job, written in YAML, that provisions an Ubuntu Jammy system image on a MAAS-provisioned device and then prints the distribution information: .. code-block:: yaml job_queue: example-queue-1 provision_data: - url: https://cdimage.ubuntu.com/ubuntu/releases/jammy/release/example.img.xz + distro: jammy test_data: test_cmds: | ssh -t ubuntu@DEVICE_IP lsb_release -a @@ -144,7 +144,7 @@ The following example shows a test job, written in YAML, that provisions an Ubun In the example job definition file: - ``job_queue``: specifies the queue name to which you will submit the job -- ``provision_data``: specifies the source of the system image to be provisioned on the target device. This example uses a URL of the system image to be downloaded, but the actual format of this section varies on device type. +- ``provision_data``: specifies the source of the system image to be provisioned on the target device. This example uses the ``distro`` key to specify the version of the system image to be downloaded, but this key is specific to MAAS devices. The actual format of this section varies on device type. For more information, refer to :doc:`../reference/device-connector-types`. - ``test_data``: contains a ``test_cmds`` section that specifies the list of commands to be executed on the device after the system is provisioned. In this example, the device is instructed to execute the ``lsb_release -a`` command to print the Linux distribution information. You might have noticed that the command is executed over an SSH connection. This is because the Testflinger system uses agents and device connectors to manage test jobs. The test commands are not executed on the test device itself, but on a host system that can reach your test device via SSH. Devices are set up with an SSH key to allow passwordless SSH connection from the test host at the time the provisioning is finished.