Update resource mechanism for templates

[ckunki]

Current implementation

Currently ITDE uses some configuration files

• file init_db.sh for the operating system inside a Docker Container
• file EXAConf for the contained Exasol database.

Folder docker_db_config_template contains a subfolder for each relevant version of Exasol database, e.g. 7.1.0, 7.1.1, etc.. Some subfolders are symlinks as the respective version of Exasol database may share the templates with another version, e.g. subfolder 7.1.1 is a symlink to 7.1.0.

Git commit hook githooks/update_packaging.sh copies the templates to directory exasol_integration_test_docker_environment/docker_db_config/ as during runtime symlinks are not supported and we need actual files.

ITDE's python implementation uses pkg_resources to locate the template files, read them into a string and substitute placeholders by actual values.

Rating

In summary we have at least two problems and an architecture with significant dependencies and complex interactions:

• Symlinks not available on windows platform
• For pkg_resources python currently displays a DeprecationWarning: pkg_resources is deprecated as an API
• 3 folders, 6 files and 36 symlinks
  • copied into another 39 folders with additional 78 files
  • depending on Git commit hoocks and shell scripts
  • commiting generated files to git repository
  • cluttering file system and wasting space
  • creating the risk of out-dated copies
  • encoding rather simple facts

Proposed change

The current ticket therefore proposes to replace the current implementation:

Acceptance criteria

1. Folder docker_db_config_template is removed
2. Folder exasol_integration_test_docker_environment/docker_db_config/ only contains subfolders 7.0, 7.1, and 8.17.0
3. Deprecated pkg_resources is replaced by importlib.resources
4. There is a python implementation
   • mapping Exasol database versions to template files
   • able to allocate and read a template file
5. File noxfile.py is updated to no longer generate the list of supported versions of Exasol database
   based on the folders in docker_db_config_template but return a plain list

import importlib.resources
from exasol_integration_test_docker_environment.lib import PACKAGE_NAME

# add to file eitde/lib/test_environment/spawn_test_database.py
def resource(version: str, filename: str) -> str:
    def directory(version: str):
        if version.startswith("7.0."):
            return "7.0.x"
        if version.startswith("7.1."):
            return "7.1.x"
        if version == "8.17.0":
            return version
        raise RuntimeError("Unsupported version " + version)
    res = (
        importlib.resources.files(PACKAGE_NAME)
        / "docker_db_config" / directory(version) / filename
    )
    with importlib.resources.as_file(res) as template:
        with open(template, "r") as file:
            return file.read()

# sample usage 
content = resource("7.1.1", "EXAConf")
print(f'{content}')

# noxfile.py
# or eitde/lib/test_environment/db_version.py
def get_db_versions() -> List[str]:
    """
    The ITDE only supports EXAConf templates for docker-db versions in the
    format major.minor.bugfix.

    If the specified version number contains additions, such as "d1", or
    "prerelease", ITDE filters these to abtain the base version number.
    Function get_db_versions however uses raw versio numbers to generate the
    test matrix for GitHub Actions requiring to adapt the list for images
    with special names.
    """
    result = [
        "default", "7.0.0", "7.0.1", "7.0.2", "7.0.3", "7.0.4", "7.0.6",
        "7.0.7", "7.0.8", "7.0.9", "7.0.10", "7.0.11", "7.0.12", "7.0.13",
        "7.0.14", "7.0.15", "7.0.16", "7.0.17", "7.0.18", "7.0.19", "7.0.20",
    ]
    result += [
        "7.1.0", "7.1.1", "7.1.2", "7.1.3", "7.1.4", "7.1.5", "7.1.6",
        "7.1.7", "7.1.8", "7.1.9", "7.1.10", "7.1.11", "7.1.12",
        "7.1.13", "7.1.14", "7.1.15", "7.1.16", "7.1.17",
    ]
    # Until an image file is available on DockerHub version 8.17.0 is replaced
    # by the corresponding pre-release.
    result += [
        "7.1.0-dl",
        "prerelease-8.17.0",
    ]
    return result

[ckunki]

Alternative approach:

# file eitde/lib/test_environment/db_version.py
from typing import (List, Optional)

# eitde/lib/test_environment/db_version.py


class DbVersion:
    LATEST="7.1.17"
    DEFAULT="default"

    @classmethod
    def supports_custom_certificates(version: Optional[str]) -> bool:
        return DbVersion.from_string(version) > DbVersion(7, 0, 5)

    @classmethod
    def ci_build_versions(self) -> List[str]:
        """
        The ITDE only supports EXAConf templates for docker-db versions in the
        format major.minor.bugfix.

        If the specified version number contains additions, such as "d1", or
        "prerelease", ITDE filters these to abtain the base version number.
        Function get_db_versions however uses raw versio numbers to generate the
        test matrix for GitHub Actions requiring to adapt the list for images
        with special names.
        """
        return [
            DbVersion.DEFAULT,
            # 7.0.x
            "7.0.0", "7.0.1", "7.0.2", "7.0.3", "7.0.4", "7.0.6", "7.0.7",
            "7.0.8", "7.0.9", "7.0.10", "7.0.11", "7.0.12", "7.0.13", "7.0.14",
            "7.0.15", "7.0.16", "7.0.17", "7.0.18", "7.0.19", "7.0.20",
            # 7.1.x
            "7.1.0", "7.1.1", "7.1.2", "7.1.3", "7.1.4", "7.1.5", "7.1.6",
            "7.1.7", "7.1.8", "7.1.9", "7.1.10", "7.1.11", "7.1.12",
            "7.1.13", "7.1.14", "7.1.15", "7.1.16", "7.1.17",
            # special
            "7.1.0-dl",
            # Until an image file is available on DockerHub version 8.17.0 is replaced
            # by the corresponding pre-release.
            "prerelease-8.17.0",
        ]


    def __init__(self, major, minor, stable):
        self.major = major
        self.minor = minor
        self.stable = stable

    @classmethod
    def from_string(cls, db_version_str: Optional[str]):
        db_version = db_version_str
        if db_version_str in (None, DEFAULT_VERSION):
            db_version = LATEST_DB_VERSION
        if db_version.endswith("-d1"):
            db_version = "-".join(db_version.split("-")[0:-1])
        if db_version.startswith("prerelease-"):
            db_version = "-".join(db_version.split("-")[1:])
        version = tuple([int(v) for v in db_version.split(".")])
        if len(version) != 3:
            raise ValueError(f"Invalid db version given: {db_version}")
        return DbVersion(version[0], version[1], version[2])

    def __str__(self):
        return f"{self.major}.{self.minor}.{self.stable}"

    def to_tuple(self):
        return self.major, self.minor, self.stable

    def __eq__(self, other) -> bool:
        return self.to_tuple() == other.to_tuple()

    def __lt__(self, other) -> bool:
        return self.to_tuple() < other.to_tuple()

    def __le__(self, other) -> bool:
        return self.to_tuple() <= other.to_tuple()

    def __ge__(self, other) -> bool:
        return self.to_tuple() >= other.to_tuple()

    def __gt__(self, other) -> bool:
        return self.to_tuple() > other.to_tuple()

[tkilias]

We should use a mapping File which is stored as Ressource and don't define the mapping in python. Because this way CI and Code use the same source. Further we can add a fallback mechanism for new versions.

[tkilias]

@ckunki please check for overlapping tickets

[ckunki]

@ckunki please check for overlapping tickets

Hello @tkilias besides #344 neither by searching for "docker_db", "config", nor by "template" I could not find an overlapping ticket.

[ckunki]

How about a mapping file like the following?

version-mapping:
  - version: latest
    - alias: 8.18.1
  - version: default
    - alias: 8.18.1
  - version: 7.0.0
    - exaconf: 7.0.0
  - version: 7.1.17
    - exaconf: 7.1.0
  - version: 7.1.0-dl
  - version: prerelease-8.17.0

[ckunki]

updated proposal for mapping file:

versions:
  - latest: 8.18.1
  - default: 8.18.1
  - mapping:
    # default for entries docker-image or exaconf is value of version
    - version: 7.0.0
    - version: 7.0.1
      exaconf: 7.0.0
    - version: 7.1.0
      docker-image: 7.1.0-dl
    - version: 8.17.0
      docker-image: prerelease-8.17.0