Proposal for enhancement of agent certificate/credential rotation

[jpalermo]

Hello Bosh Community.

We’d like to propose an addition to make the rotation of certificates used by the bosh-agent easier. The current procedure is documented here: https://bosh.io/docs/nats-ca-rotation/

Some of the pain points associated with the current process:

• It involves the recreation of all VMs multiple times, which can be time consuming in large bosh environments.
• If the recreation fails for any reason, you must re-recreate any VMs that have already been recreated (or manually bosh recreate the individual VMs that did not yet get the new certificates)

Part of updating VMs currently involves the agent “update_settings” action. This is currently used to update “trusted_certs” if they have been changed on the director. We’d like to expand the scope of this action to also include NATS and Blobstore certificates and credentials.

The director would track the SHA of the currently deployed certificates and credentials in a similar way that it does currently for “trusted_certs”, and if it detects a change, would send new values via the “update_settings” action.

The agent would then save these new values in the “update_settings.json” it keeps, and use them instead of the values found in “settings.json” if they are both available. The agent would be changed to restart itself as part of “update_settings” so the new certificates and credentials could be used by the internal NATS client as soon as possible.

The end result would be a similar flow as seen in https://bosh.io/docs/nats-ca-rotation/, but the recreates would instead be normal updates. Since this makes it part of a normal update, and the current state of each VM is tracked, it also means that if a deploy fails, the VMs that have already been updated will not be updated again when the deploy is retried.

The intent would be to make this a backwards compatible change. So if you have a director that supports this flow, but agents are still old, the director would send the new data with “update_settings”, but the agents would just ignore the extra data. In this case you would still need to use the old recreate process.

[lnguyen]

Hello Bosh community,
The BOSH team has created two PR cloudfoundry/bosh-agent#248 and #2323 as changes for director and agent. Let us know if ya'll have any feedback on proposed changes.

[bgandon]

This is great work overall.

As a complementary remark, the new certificates update flow would be valid before current NATS certificate expire, but in case of already expired certificates, then the recreation flow would still be required.

One question pops up in my mind: could we explicit here the rationale behind the separated “update_settings.json” and “settings.json” files on Bosh instances? Is this to be able, when inspecting some BOSH instance, to audit which settings have been provided through which mean?

[jpalermo]

Current flow in the agent I believe always tries to grab settings.json from the metadata source on startup, and some IAAS don't allow us to modify that original source. So we want to avoid trying to update that since it can be reverted behind our backs.

We could have looked into guarding that behavior, but update_settings.json already existed (probably for the above reason), so we assumed somebody had already investigated and decided this was the better approach.

[beyhan]

The pull-requests are really good.

Two points came into my mind:

• have your thought to have a restart action on the agent side instead of the AgentKiller util? This action could be triggered by the director after update_settings and could make debugging issue with the cert rotation easier.
• Starting a cert rotation together with the director update to this version wan't detected the cert changes. We should add a release note stating this. Or do you have any other idea?

[jpalermo]

• The AgentKiller was actually a pattern that already existed in the agent, but was removed when the only remaining action that used it was removed. So we mostly just brought it back out of git history. Having a pattern that we know was previously used and worked was an appealing feature of the AgentKiller. Having a restart action probably could work, although it also makes a bigger change to the agent/director contract which would make it harder to make this a backwards compatible change. Right now you can use an old director with the new agent, or a new director with the old agent and everything gracefully falls back to the old behavior.
• Correct, we'll want a release note that makes it clear you shouldn't upgrade if you are in the middle of a certificate rotation, or starting one as part of the upgrade. The good news is that even if you are in one of these situations, you just have to do a recreate to fix everything, which is what you would have had to do before these changes anyway.

[jpalermo]

Bosh Agent 2.388.0 and Bosh Director 271.12.0 have been published with the new behavior.

[thelangley]

This sounds wonderful. Do you know which Bionic stemcell version includes bosh agent 2.388.0? Has it been cut yet?

[beyhan]

Isn't yet there. You can find this information here: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/releases. Usually, stemcells are released weekly or two weekly based and the agent is bumped for the release. It should be included into the next stemcell release.

[bgandon]

Versions 1.35 and later (if released and when released) should ship Bosh Agent v2.389+ after recent agent bump last Thursday by Ramon.

[jpalermo]

Stemcells Ubuntu Bionic 1.35 and Windows 2019.41 both have copies of the Bosh Agent with the new behavior.

We've also updated the nats and blobstore credential rotation docs here:
https://bosh.io/docs/nats-ca-rotation/
https://bosh.io/docs/blobstore-ca-rotation/
They now reflect the new process (although they contain a note that if you are not using compatible versions you will need to create VMs instead of just redeploying them).

We think this track of work is finished at this point, but let us know if you run into any problems or have any enhancements.

[bgandon]

Thanks a lot Joseph!

Reading these two documentation pages though, as modified after cloudfoundry/docs-bosh@4fed093, I only see a mention of the new process in the “exclusions” of the old process.

• Director versions prior to 271.12 and stemcells prior to Bionic 1.36 and Windows 2019.41 need to recreate VMs as part of the redeploy steps.

I mean this is perfectly correct. But I'm afraid this doesn't make things clear enough that with latest Bosh Director and Stemcells you get NATS & Blobstore CA rotations “for free” (at least as I've understood it). Also, unexperienced readers could miss this small bullet point in these big documentation pages.

With the new process, operators need to be able to see that it has properly run, and possibly debug it. So, there is “something” to tell about it in some leading section.

Basically, we need to push forward upfront the nice pieces of tech we put in Bosh 👍