Add ephemeral containers to policies

[phillebaba]

This change adds ephemeral containers to all policies that currently check the containers and init containers field. It is kind of a quick solution just to bring all the policies to a level where ephemeral containers do not introduce security concerns.

I have on purpose skipped the required probes policy as it currently only looks at the containers field. I have also only updated one of the mutation samples.

One thing that comes to mind when looking at the different policies is that there are a bunch of different ways to solve this problem. A couple of rules have a method which returns the container data from each field, and others duplicate code at different levels. I feel like it would be good to find a standard. Right now it would be easy to make mistakes and not include one of the containers field. At the same time I can see a situation where one would want to require read only root file systems for the init containers and containers fields, but not the ephemeral containers fields. Might be better to solve these challenges in the future.

Fixes #188

[phillebaba]

Currently having a look at the best way to implement tests for both init containers and ephemeral containers.

[ritazh]

LGTM
Thanks for the PR @phillebaba! 🎉

[maxsmythe]

Indeed, TY for the PR! LMK when you think you have the tests sorted out.

Rego tests can make it easy to stamp tests out. Adding them to suite.yaml might have a more uniform experience if you're trying to test them all in bulk.

[fseldow]

may have one question
It is said Pod resource allocations are immutable, so setting resources is disallowed. in
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/#what-is-an-ephemeral-container

So, do we need to add ephemeralContainer check in container limit policy?

[fseldow]

so do the policy contaienrrequests and containerresourceratios

[phillebaba]

@fseldow good catch, you are right. It is not needed so will remove. Going to have a second look to see if there are more of these.

[maxsmythe]

@phillebaba How's the testing looking?

[maxsmythe]

LGTM, thank you for doing this!

[phillebaba]

@maxsmythe sorry I got a bit carried away with the testing, but started facing issues as certain rules have multiple violations and other do not. Maybe something to look more into in the future. I added the easiest test cases that I could.

I just need to figure out why the integration tests are failing, do you have any insight into this?

[mac-chaffee]

@phillebaba Looks like the kind cluster used in testing is using v1.21 where the ephemeralContainers field is just ignored, which is causing "disallowed" pods to still get created since the unknown field is stripped out before gatekeeper sees it.

I tested upgrading the kind cluster to v1.23 where ephemeralContainers are enabled, but that still doesn't work because spec.ephemeralContainers: Forbidden: cannot be set on create.

I think you just can't integration-test this feature until we refactor test/bats/test.bats. may want to leave that work for another PR.

[ritazh]

To unblock this PR, perhaps remove the deployments with the ephemeral containers for now?

[phillebaba]

The issue with removing the deployments using ephemeral containers is that it is part of the actual test. Could we not just update the kind version?

[mac-chaffee]

@phillebaba even if you increase the kind version, the integration tests still fail because you can't "kubectl apply" a Pod with an ephemeralContainer in it: spec.ephemeralContainers: Forbidden: cannot be set on create.

[maxsmythe]

To exempt these ephemeral objects from on-cluster testing, can we rename them to remove the example_ prefix?

gatekeeper-library/test/bats/test.bats

Lines 79 to 107 in 259ad1b

	for allowed in "$sample"/example_allowed*.yaml; do
	if [[ -e "$allowed" ]]; then
	# apply resource
	run kubectl apply -f "$allowed"
	assert_match 'created' "$output"
	assert_success
	# delete resource
	kubectl delete --ignore-not-found -f "$allowed"
	fi
	done
	for inventory in "$sample"/example_inventory*.yaml; do
	if [[ -e "$inventory" ]]; then
	run kubectl apply -f "$inventory"
	assert_match 'created' "$output"
	assert_success
	fi
	done
	for disallowed in "$sample"/example_disallowed*.yaml; do
	if [[ -e "$disallowed" ]]; then
	# apply resource
	run kubectl apply -f "$disallowed"
	assert_match 'denied the request' "${output}"
	assert_failure
	# delete resource
	kubectl delete --ignore-not-found -f "$disallowed"
	fi
	done

Then they will only be subject to OPA and gator verify tests

[ritazh]

@phillebaba are you able to make the suggested changes above and in #197 so we can get this PR merged soon? Please let us know how we can help.