diff --git a/doc/ha/dev-setup-ha-pki.md b/doc/ha/dev-setup-ha-pki.md new file mode 100644 index 000000000..307d63230 --- /dev/null +++ b/doc/ha/dev-setup-ha-pki.md @@ -0,0 +1,69 @@ +# Overview + +This guide walks you through creating the certificates necessary to run a three node HA cluster. + +**NOTE**: This folder contains a script `create-pki.sh` which will perform the steps outlined in +this guide for you. + +## Create a CA + +Create a self-signed certificate authority (CA) for the trust-root of your cluster + +``` +ziti pki create ca --trust-domain ha.test --pki-root ./pki --ca-file ca --ca-name 'HA Example Trust Root' +``` + +## Create Controller Certs + +We are going to create an intermediate CA for each controller. We'll use this intermediate CA for +the following purposes: + +1. To create a cert which will represent the controller. It will be used + 1. On the client side when dialing other controllers in the cluster + 2. On the server side when receiving connections from other controllers + 3. On the server side when receiving connections from routers + 4. On the server when handling REST API requests +2. To create identity certs as part of the identity enrollment process +3. To create router certs as part of the router enrollment process + +### Notes + +#### Client vs Server Certs + +You may use separate certs and keys for client and server connections, but it's not necessary. When +you use a server cert on the client side it exposes information about what IPs and DNS entries the +cert is valid for, but since we're only connecting to other controllers, this should not be a +concern. However, the option to use separate certs is available, should you wish to use it. + +#### REST Endpoint Certs + +You may also use a different set of certs for the REST endpoint. + +#### Sharing Signing Certs + +You could use the same signing cert for all controllers in a cluster. However, if a signing cert is +ever compromised, all certs signed by the signing cert would need to be revoked. By using a separate +cert for each controller we limit the fallout from an individual controller or cert being +compromised. + +### Create the Controller 1 signing and server certs + +```shell +# Create the controller 1 intermediate/signing cert +ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl1 --intermediate-name 'Controller One Signing Cert' + +# Create the controller 1 server cert +ziti pki create server --pki-root ./pki --ca-name ctrl1 --dns localhost --ip 127.0.0.1 --server-name ctrl1 --spiffe-id 'controller/ctrl1' + +# Create the controller 2 intermediate/signing cert +ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl2 --intermediate-name 'Controller Two Signing Cert' + +# Create the controller 2 server cert +ziti pki create server --pki-root ./pki --ca-name ctrl2 --dns localhost --ip 127.0.0.1 --server-name ctrl2 --spiffe-id 'controller/ctrl2' + +# Create the controller 3 intermediate/signing cert +ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl3 --intermediate-name 'Controller Three Signing Cert' + +# Create the controller 3 server cert +ziti pki create server --pki-root ./pki --ca-name ctrl3 --dns localhost --ip 127.0.0.1 --server-name ctrl3 --spiffe-id 'controller/ctrl3' +``` diff --git a/doc/ha/dev-setup.md b/doc/ha/dev-setup.md index 5030478cd..1e86a7aba 100644 --- a/doc/ha/dev-setup.md +++ b/doc/ha/dev-setup.md @@ -1,81 +1,26 @@ # HA Setup for Development -**NOTE: HA is a work in progress and not yet usable for anything other than developing HA** +**NOTE: HA is in alpha. Expect bugs. Bug reports are appreciated** To set up a local three node HA cluster, do the following. -## Create a CA +## Create The Necessary PKI -Create a self-signed certificate authority (CA) for the trust-root of your cluster - -``` -ziti pki create ca --trust-domain ha.test --pki-root ./pki --ca-file ca --ca-name 'HA Example Trust Root' -``` - -## Create Controller Certs - -We are going to create an intermediate CA for each controller. We'll use this intermediate CA -for the following purposes: - -1. To create a cert which will represent the controller. It will be used - 1. On the client side when dialing other controllers in the cluster - 2. On the server side when receiving connections from other controllers - 3. On the server side when receiving connections from routers - 4. On the server when handling REST API requests -2. To create identity certs as part of the identity enrollment process -3. To create router certs as part of the router enrollment process - -### Notes - -#### Client vs Server Certs - -You may use separate certs and keys for client and server connections, but it's not necessary. -When you use a server cert on the client side it exposes information about what IPs and DNS entries -the cert is valid for, but since we're only connecting to other controllers, this should not be -a concern. However, the option to use separate certs is available, should you wish to use it. - -#### REST Endpoint Certs - -You may also use a different set of certs for the REST endpoint. - -#### Sharing Signing Certs - -You could use the same signing cert for all controllers in a cluster. However, if a signing -cert is ever compromised, all certs signed by the signing cert would need to be revoked. By -using a separate cert for each controller we limit the fallout from an individual controller -or cert being compromised. - -### Create the Controller 1 signing and server certs - -```shell -# Create the controller 1 intermediate/signing cert -ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl1 --intermediate-name 'Controller One Signing Cert' - -# Create the controller 1 server cert -ziti pki create server --pki-root ./pki --ca-name ctrl1 --dns localhost --ip 127.0.0.1 --server-name ctrl1 --spiffe-id 'controller/ctrl1' - -# Create the controller 2 intermediate/signing cert -ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl2 --intermediate-name 'Controller Two Signing Cert' - -# Create the controller 2 server cert -ziti pki create server --pki-root ./pki --ca-name ctrl2 --dns localhost --ip 127.0.0.1 --server-name ctrl2 --spiffe-id 'controller/ctrl2' - -# Create the controller 3 intermediate/signing cert -ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl3 --intermediate-name 'Controller Three Signing Cert' - -# Create the controller 3 server cert -ziti pki create server --pki-root ./pki --ca-name ctrl3 --dns localhost --ip 127.0.0.1 --server-name ctrl3 --spiffe-id 'controller/ctrl3' -``` +Either run the `create-pki.sh` script found in the folder, or follow the steps in +the [HA PKI Guide](./dev-setup-ha-pki.md) ## Running the Controllers -1. The controller configuration files have relative paths, so make sure you're running things from this directory. +1. The controller configuration files have relative paths, so make sure you're running things from + this directory. 2. Start all three controllers 1. `ziti controller run ctrl1.yml` 2. `ziti controller run ctrl2.yml` 3. `ziti controller run ctrl3.yml` - 4. All three are configured with `minClusterSize` of 3, so they will wait to be joined to a raft cluster - 5. The ctrl1.yml config file has the other two controllers as bootstrap members, so when it starts the first controller will start trying form the raft cluster. + 4. All three are configured with `minClusterSize` of 3, so they will wait to be joined to a raft + cluster + 5. The ctrl1.yml config file has the other two controllers as bootstrap members, so when it + starts the first controller will start trying form the raft cluster. 3. Initialize the edge using the agent 1. `ziti agent controller init admin admin 'Default Admin'` 2. You can of course use different values if you desire @@ -89,14 +34,22 @@ You should now have a three node cluster running. You can log into each controll You could then create some model data on any controller: ``` +# This will create the client side identity and policies ziti demo setup echo client + +# This will create the server side identity and policies ziti demo setup echo single-sdk-hosted ``` Any view the results on any controller ``` +ziti edge login localhost:1280 ziti edge ls services + +ziti edge login -i ctrl2 localhost:1380 ziti edge -i ctrl2 ls services + +ziti edge login -i ctrl3 localhost:1480 ziti edge -i ctrl3 ls services ``` diff --git a/doc/ha/overview.md b/doc/ha/overview.md index 7cc7ac31d..3d7404bac 100644 --- a/doc/ha/overview.md +++ b/doc/ha/overview.md @@ -3,13 +3,15 @@ This document gives a brief overview of how OpenZiti HA works and how it differs from running OpenZiti in non-HA mode. +To set up a developer HA network see the [HA Developer Setup Guide](./dev-setup.md). + ## Operational Considerations ### System of Record -In controller that's not configured for HA, the bolt database is the system of record. In -an HA setup, the raft journal is the system of record. The raft journal is stored in two places, -a snapshot directory and a bolt database of raft journal entries. +In controller that's not configured for HA, the bolt database is the system of record. In an HA +setup, the raft journal is the system of record. The raft journal is stored in two places, a +snapshot directory and a bolt database of raft journal entries. So a non-HA setup will have: @@ -18,7 +20,8 @@ So a non-HA setup will have: An HA setup will have: * raft.db - the bolt database containing raft journal entries -* snapshots/ - a directory containing raft snapshots. Each snapshot is snapshot of the controller bolt db +* snapshots/ - a directory containing raft snapshots. Each snapshot is snapshot of the controller + bolt db * ctrl.db - the controller bolt db, with the current state of the model The location of all three is controlled by the raft/dataDir config property. @@ -28,9 +31,9 @@ raft: dataDir: /var/ziti/data/ ``` -When an HA controller starts up, it will first apply the newest snapshot, then any newer journal entries -that aren't yet contained in a snapshot. This means that an HA controller should start with a -blank DB that can be overwritten by snapshot and/or have journal entries applied to it. So an HA +When an HA controller starts up, it will first apply the newest snapshot, then any newer journal +entries that aren't yet contained in a snapshot. This means that an HA controller should start with +a blank DB that can be overwritten by snapshot and/or have journal entries applied to it. So an HA controller will delete or rename the existing controller database and start with a fresh bolt db. ### Bootstrapping @@ -39,9 +42,9 @@ controller will delete or rename the existing controller database and start with Initial cluster setup can be configured either in the config file or via agent commands. -The controller will not fully start until the raft cluster has bootstrapped. The minimum -number of cluster members is set in the config file. Recommended cluster sizes are -3, 5 or 7. A cluster size of 1 is mostly useful for testing and development. +The controller will not fully start until the raft cluster has bootstrapped. The minimum number of +cluster members is set in the config file. Recommended cluster sizes are 3, 5 or 7. A cluster size +of 1 is mostly useful for testing and development. **Config File Example** @@ -54,9 +57,9 @@ raft: - tls:192.168.1.101 ``` -Note that `bootstrapMembers` can only be used when setting the cluster up for the first time -and should only be used on a single node. It cannot be used later to add additional nodes -to an existing cluster. +Note that `bootstrapMembers` can only be used when setting the cluster up for the first time and +should only be used on a single node. It cannot be used later to add additional nodes to an existing +cluster. **Agent Comands** @@ -80,11 +83,10 @@ ziti agent cluster transfer-leadership [new leader id] #### Edge Admin Initialization -Because RAFT is now the system of record, the previous pattern for configuring the default -admin won't work. In a non-HA system you initialized the controller raft DB with an admin -user directly. If you do this with an HA system, the changes you made directly to the DB -will be lost and replaced by whatever is in raft. To initialize an HA controller cluster -use the new agent command. +Because RAFT is now the system of record, the previous pattern for configuring the default admin +won't work. In a non-HA system you initialized the controller raft DB with an admin user directly. +If you do this with an HA system, the changes you made directly to the DB will be lost and replaced +by whatever is in raft. To initialize an HA controller cluster use the new agent command. ```shell ziti agent controller init @@ -94,36 +96,36 @@ The controller will not fully start until the edge admin has been initialized. ### Snapshot Application and Restarts -If a controller receives a snapshot to apply after starting up, it will apply the snapshot and -then terminate. This assumes that there is a restart script which will bring the controller -back up after it terminates. +If a controller receives a snapshot to apply after starting up, it will apply the snapshot and then +terminate. This assumes that there is a restart script which will bring the controller back up after +it terminates. -This should only happen if a controller is connected to the cluster and then gets disconnected -for long enough that a snapshot is created while it's disconnected. Because applying a snapshot -requires replacing the underlying controller bolt DB, the easiest way to do that is restart. -That way we don't have to worry about replacing the bolt DB underneath a running system. +This should only happen if a controller is connected to the cluster and then gets disconnected for +long enough that a snapshot is created while it's disconnected. Because applying a snapshot requires +replacing the underlying controller bolt DB, the easiest way to do that is restart. That way we +don't have to worry about replacing the bolt DB underneath a running system. ### Metrics -In an HA system, routers will send metrics to all controllers to which they are connected. -There is a new `doNotPropagate` flag in the metrics message, which will be set to false -until the router has successfully delivered the metrics message to a controller. The flag -will then be set to true. So the first controller to get the metrics message is expected -to deliver the metrics message to the events system for external integrators. The other -controllers will have `doNotPropage` set to true, and will only use the metrics message -internally, to update routing data. +In an HA system, routers will send metrics to all controllers to which they are connected. There is +a new `doNotPropagate` flag in the metrics message, which will be set to false until the router has +successfully delivered the metrics message to a controller. The flag will then be set to true. So +the first controller to get the metrics message is expected to deliver the metrics message to the +events system for external integrators. The other controllers will have `doNotPropage` set to true, +and will only use the metrics message internally, to update routing data. ### Certificates There are many ways to set up certificates, so this will just cover a recommended configuration. -The primary thing to ensure is that controllers have a shared root of trust. A configuration -that works would be as follows: +The primary thing to ensure is that controllers have a shared root of trust. A configuration that +works would be as follows: 1. Create a self-signed root CA 2. Create an intermediate signing cert for each controller 3. Create a server cert using the signing cert for each controller -4. Make sure that the CA bundle for each server includes both the root CA and the intermediate CA for that server +4. Make sure that the CA bundle for each server includes both the root CA and the intermediate CA + for that server Note that controller server certs must contain a SPIFFE id of the form @@ -140,7 +142,8 @@ spiffe://example.com/controller/ctrl1 **SPIFFE ID Notes:** -* This ID must be set as the only URI in the `X509v3 Subject Alternative Name` field in the certificate. +* This ID must be set as the only URI in the `X509v3 Subject Alternative Name` field in the + certificate. * These IDs are used to allow the controllers to identify each during the mTLS negotiation. * The OpenZiti CLI supports creating SPIFFE IDs in your certs * Use the `--trust-domain` flag when creating CAs @@ -158,19 +161,18 @@ Controllers now establish connections with each other, for two purposes. Both kinds of traffic flow over the same connection. -These connections do not require any extra open ports as we are using the control -channel listener to listen to both router and controller connections. As part of -the connection process the connection type is provided and the appropriate -authentication and connection setup happens based on the connection type. If no -connection type is provided, it's assumed to be a router. +These connections do not require any extra open ports as we are using the control channel listener +to listen to both router and controller connections. As part of the connection process the +connection type is provided and the appropriate authentication and connection setup happens based on +the connection type. If no connection type is provided, it's assumed to be a router. ## Distributed Model -When looking at how to make the OpenZiti controller model distributed, we first looked at -what characteristics we needed for the model data. +When looking at how to make the OpenZiti controller model distributed, we first looked at what +characteristics we needed for the model data. -Model data is the information the controller needs to figure out what it can do. This -includes things like: +Model data is the information the controller needs to figure out what it can do. This includes +things like: * Services * Routers @@ -186,12 +188,14 @@ includes things like: * Reads happen all the time, from every client and as well as admins * Speed is very important. They affect how every client perceives the system. * Availability is very important. Without reading definitions, can’t create new connections - * Can be against stale data, if we get consistency within a reasonable timeframe (seconds to minutes) + * Can be against stale data, if we get consistency within a reasonable timeframe (seconds to + minutes) * Write characteristics * Writes only happen from administrators * Speed needs to be reasonable, but doesn't need to be blazing fast * Write availability can be interrupted, since it primarily affects management operations - * Must be consistent. Write validation can’t happen with stale data. Don’t want to have to deal with reconciling concurrent, contradictory write operations. + * Must be consistent. Write validation can’t happen with stale data. Don’t want to have to deal + with reconciling concurrent, contradictory write operations. * Generally involves controller to controller coordination Of the distribution mechanisms we looked at, RAFT had the best fit. @@ -217,23 +221,27 @@ So the OpenZiti controller uses RAFT to distribute the data model. Specifically The basic flow for model updates is as follows: 1. A client requests a model update via the REST API. -2. The controller checks if it is the raft cluster leader. If it is not, it forwards the request to the leader. -3. Once the request is on the leader, it applies the model update to the raft log. This involves getting a quorum of the controllers to accept the update. -4. One the update has been accepted, it will be executed on each node of the cluster. This will generate create one or more changes to the bolt database. -5. The results of the operation (success or failure) are returned to the controller which received the original REST request. +2. The controller checks if it is the raft cluster leader. If it is not, it forwards the request to + the leader. +3. Once the request is on the leader, it applies the model update to the raft log. This involves + getting a quorum of the controllers to accept the update. +4. One the update has been accepted, it will be executed on each node of the cluster. This will + generate create one or more changes to the bolt database. +5. The results of the operation (success or failure) are returned to the controller which received + the original REST request. 6. The controller waits until the operation has been applied locally. 7. The result is returned to the REST client. ### Reads -Reads are always done to the local bolt database for performance. The assumption is that if something -like a policy change is delayed, it may temporarily allow a circuit to be created, but as soon as -the policy update is applied, it will make changes to circuits as necessary. +Reads are always done to the local bolt database for performance. The assumption is that if +something like a policy change is delayed, it may temporarily allow a circuit to be created, but as +soon as the policy update is applied, it will make changes to circuits as necessary. ## Runtime Data -In addition to model data, the controller also manages some amount of runtime data. This data is -for running OpenZiti's core functions, i.e. managing the flow of data across the mesh, along with +In addition to model data, the controller also manages some amount of runtime data. This data is for +running OpenZiti's core functions, i.e. managing the flow of data across the mesh, along with related authentication data. So this includes things like: * Links @@ -250,37 +258,36 @@ Runtime data has different characteristics than the model data does. * Reads **and** writes must be very fast * Generally involves sdk to controller or controller to router coordination -Because writes must also be fast, RAFT is not a good candidate for storing this data. -Good performance is critical for these components, so they are each evaluated individually. +Because writes must also be fast, RAFT is not a good candidate for storing this data. Good +performance is critical for these components, so they are each evaluated individually. ### Links -Each controller currently needs to know about links so that it can make routing decisions. -However, links exist on routers. So, routers are the source of record for links. -When a router connects to a controller, the router will tell the controller about any -links that it already has. The controller will ask to fill in any missing links and -the controller will ensure that it doesn't create duplicate links if multiple -controllers request the same link be created. If there are duplicates, the router -will inform the controller of the existing link. +Each controller currently needs to know about links so that it can make routing decisions. However, +links exist on routers. So, routers are the source of record for links. When a router connects to a +controller, the router will tell the controller about any links that it already has. The controller +will ask to fill in any missing links and the controller will ensure that it doesn't create +duplicate links if multiple controllers request the same link be created. If there are duplicates, +the router will inform the controller of the existing link. -The allows the routers to properly handle link dials from multiple routers and keep -controllers up to date with the current known links. +The allows the routers to properly handle link dials from multiple routers and keep controllers up +to date with the current known links. ### Circuits Circuits were and continue to be stored in memory for both standalone and HA mode -controllers.Circuits are not distributed. Rather, each controller remains responsible -for any circuits that it created. - -When a router needs to initiate circuit creation it will pick the one with the lowest -response time and send a circuit creation request to that router. The controller -will establish a route. Route tables as well as the xgress endpoints now track -which controller is responsible for the associated circuit. This way when failures -or other notifications need to be sent, the router knows which controller to talk to. - -This gets routing working with multiple controllers without a major refactor. Future -work will likely delegate more routing control to the routers, so routing should -get more robust and distributed over time. +controllers.Circuits are not distributed. Rather, each controller remains responsible for any +circuits that it created. + +When a router needs to initiate circuit creation it will pick the one with the lowest response time +and send a circuit creation request to that router. The controller will establish a route. Route +tables as well as the xgress endpoints now track which controller is responsible for the associated +circuit. This way when failures or other notifications need to be sent, the router knows which +controller to talk to. + +This gets routing working with multiple controllers without a major refactor. Future work will +likely delegate more routing control to the routers, so routing should get more robust and +distributed over time. ### Api Sessions, Sessions, Posture Data