Release plan for linkerd2-conformance

[Pothulapati]

#11 Adds CI to perform static tests and also kind integration tests to see how the tests run. Once we have the CI setup, we would obviously want to push the docker images of the conformance plugin. This makes me wonder on what kind of a release plan should we have?

I personally think that we should keep pushing after each merge to latest without any release versions, but I'm definitely listening for suggestions.

cc: @ihcsim

[mayankshah1607]

I personally think that we should keep pushing after each merge to latest without any release versions

+1 to this

[ihcsim]

Does that mean latest will always work with all older versions of the control plane and proxy? Any idea how other Sonobuoy plugins maintain their versioning?

[mayankshah1607]

@ihcsim I don't think we need to manage versioning of the plugin here. We need to maintain versioning of the Docker image used by the Sonobuoy plugin.

Does that mean latest will always work with all older versions of the control plane and proxy?

Hmm, I am not sure if I understand your question correctly, but regardless of the version of the Docker image, users can always specify which linkerd version to use in the config.yaml. The latest tag only reflects the latest updates to the tests in this repo.

[ihcsim]

but regardless of the version of the Docker image, users can always specify which linkerd version

If I specify Linkerd 2.7 in the config.yaml, will the conformance test still pass? Are there flags and features that the conformance test expect to test but aren't in 2.7?

[mayankshah1607]

If I specify Linkerd 2.7 in the config.yaml, will the conformance test still pass?

Yes, they will! Before the tests run, the specified version of Linkerd CLI is automatically installed in the container / pod (or even your local machine if the binary doesn't already exist). The tests use this Linkerd binary for running various commands. Not specifying a version would, by default, install the latest edge release.

Are there flags and features that the conformance test expect to test but aren't in 2.7?

None of the flags are hard-coded into the tests. However, if you try to pass certain flags or expect to run some feature that is not supported by the specified version of Linkerd, the tests will fail.
For example, the install tests would occassionally fail on KinD when trying to install 2.7.0, due to clock skew between the nodes. However, this check was later improved in 2.7.1; when specifying this as the Linkerd version, the install tests were no longer failing.

[ihcsim]

However, if you try to pass certain flags or expect to run some feature that is not supported by the specified version of Linkerd, the tests will fail.

So if we always tag the test image as latest, how do we know which Linkerd versions will pass with latest?

[mayankshah1607]

So if we always tag the test image as latest, how do we know which Linkerd versions will pass with latest?

Hmm, in my opinion, the conformance suite as a whole shouldn't be marked as being compatible with certain versions of Linkerd, instead, it should be the features that should correspond to certain versions of Linkerd; because the test suite, or the image, is simply going to house all the features and tests that are there. The user is however given the freedom to choose Linkerd version, or skip tests accordingly to prevent flakiness or failures.
Would it make more sense if we documented which features and tests would work with which versions of Linkerd? Do we have a way to keep track of changes in upstream that may have the potential to break the tests here?

[mayankshah1607]

But we should definitely have some kind of semantics that will help our users understand if a particular version of the image can run tests against the newer versions of Linkerd, i.e, if the tests are kept in sync with latest updates to upstream. But I'm not sure what's the cleanest way to deal with this, because after a certain point, updates to this repo may not be as frequent as the changes in upstream. Would it still make sense to update the test image version without actually having any changes made to the test code?