Calculate PSA crypto test case dependencies automatically

[gilles-peskine-arm]

Currently, when you write a test case, you need to declare its dependencies, i.e. which compilation options are necessary for it to run and have the desired outcome. This is a tedious and error-prone process. Many test cases are copy-pasted variations, and it's easy to forget to update one dependency when changing the test case data. We often only find out we got it wrong through CI results, or sometimes through bug reports when a particular combination of compile-time options isn't tested on the CI.

It would be helpful if the test framework could determine test case dependencies automatically. This is not always possible, but #4012 shows that it's doable in practice for PSA cryptographic mechanisms.

#4012 was originally intended as a one-time conversion from classic dependencies to PSA_WANT_ dependencies. But the same logic can be used to determine dependencies in generate_test_code.py.

Goals of this issue:

• Let generate_test_code.py determine PSA_WANT_ dependencies automatically for PSA cryptographic mechanisms.
• Documentation explaining how this mechanism works.

We aren't quite ready to do this yet. There are a few issues we may want to resolve first.

• There are a few cases where automatic determination is not possible, so there needs to be a bypass mechansm.
  • For example Script to determine PSA crypto test dependencies automatically #4012 identified not-supported test cases as a pattern where the (negative) dependencies can't be determined automatically (for example a test that uses key type A and algorithm B, and expects PSA_ERROR_NOT_SUPPORTED, could either be testing what happens if A is not supported or if B is not supported).
• The script in Script to determine PSA crypto test dependencies automatically #4012 has some ad hoc exceptions. That's acceptable for a script that does a one-time rewrite of a source file, but not for something built into the test framework. How will we manage these ad hoc cases? Perhaps an annotation in the .data or .function file? How general would this annotation need to be?
  • Preliminary idea: in depends_on: lines in .data files and .function files, @noauto says to not have automatic depenencies at all. @ignore=3,4 says to ignore arguments 3 and 4.
• If some PSA_WANT_ dependencies are explicitly listed in the .data file (and there's no explicit annotation that says to not have automatic dependencies at all), should generate_test_code.py use them instead of the automatic ones? in addition to the automatic ones? ignore them? complain?
• Script to determine PSA crypto test dependencies automatically #4012 only reworks PSA test suites. The automatic determination is still untested on MBEDTLS_USE_PSA_CRYPTO.
• We need a practical way of debugging the automatic dependency mechanism. Looking at the C file is probably good enough.

Out of scope:

• Automatic determination of classic crypto dependencies. (Maybe that's worth doing too, in a follow-up.)
• Automatic determination of dependencies other than cryptographic mechanisms.
• Automatic determination of dependencies of C code. The scope is only dependencies of test data (test case arguments).

[ronald-cron-arm]

As part of this issue it would be great to also think about the maintenance of the WITHOUT_SYSTEMATIC_DEPENDENCIES and
SPECIAL_SYSTEMATIC_DEPENDENCIES lists. Those lists may have to be updated when new PSA_* symbols are added and used in test case parameters or when existing PSA_* symbols are modified. How do we reduce the risk of forgetting about it ?

[gilles-peskine-arm]

Arguably the logic for the systematic dependencies exceptions should be in config_psa.h. In #4012 I wanted to minimize changes outside the test suites. But it would make sense for config_psa.h to have the logic for all the identifiers.

We can live with this exception list for a while though. Adding a new identifier is rare, and if an identifier requires such an exception, we're likely to notice if the new test cases that are being added for this identifier aren't being run.

[gilles-peskine-arm]

Regarding annotations: I think we don't actually need a new annotation mechansm. The test code generator already parses the type of a test case argument. It currently accepts int, char* and data_t*. We could make it accept psa_key_type_t and psa_algorithm_t, and generate dependencies for those types. The few exceptional cases that shouldn't have automatic (not-supported tests) would keep using int.

[gilles-peskine-arm]

There is now a significant amount of Python code dedicated to deducing test case dependencies from test data. At the time of writing, it is used for automatically generated test cases (generate_psa_tests.py). These entirely rely on the automatic generation since Mbed-TLS/mbedtls-framework#104, in the sense that they do not enumerate any dependency manually. The entry point for automatic dependency generation is mbedtls_framework.psa_test_case.TestCase.

At this point, the dependency calculation code needs a few inputs beyond the test data:

• For ECC, FFDH and RSA keys, the bit-size of the key involved. (Needed to determine PSA_WANT_{ECC,DH}_group_bits dependencies, and MBEDTLS_RSA_GEN_KEY_MIN_BITS >= n.)
• Where key pair types are involved, which operations are requires (BASIC, IMPORT, EXPORT, GENERATE, DERIVE).
• Extra annotations for negative test cases where some mechanism doesn't need to be implemented, or needs to not be implemented.
• Also, the dependency calculation assumes that at most one key is involved.

Some of these could be partially deduced from heuristics (e.g. assume that the bit-size is the argument called bits), but as a first step we could already leverage the existing code for positive test cases that involve at most one key that isn't a key pair.