diff --git a/README.md b/README.md index 3fd8e4c..d3317e4 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,13 @@ # Sart -Sart is the Kubernetes network load-balancer using BGP in Rust. +Sart is the Kubernetes network load-balancer and CNI plugin for Kubernetes using BGP in Rust. -This project is inspired by [Metallb](https://github.com/metallb/metallb). +This project is inspired by [Metallb](https://github.com/metallb/metallb) and [Coil](https://github.com/cybozu-go/coil). > [!WARNING] > This project is under experimental. +> CNI feature is not implemented yet. ## Features @@ -36,7 +37,7 @@ Sart can run on the container based environment using [kind](https://kind.sigs.k And we also need to [install Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html). -First, we have to create the test environment. +First, we have to create the test environment. ```console sart$ make build-image sart$ make certs diff --git a/docs/bgp.md b/docs/bgp.md new file mode 100644 index 0000000..af7177b --- /dev/null +++ b/docs/bgp.md @@ -0,0 +1,126 @@ +# How to Use + +Sart has two CLI interfaces, sartd is for daemon and sart-cli is for controlling sartd. + +### sartd bgp + +We can run the BGP daemon with `sartd bgp` command. +And we can specify some parameters such as AS number and router id. +To integrate Fib manager, we have to give the path to the gRPC endpoint for fib manager. + +```console +Usage: sartd bgp [OPTIONS] + +Options: + -f, --file Config file path for BGP daemon + -a, --as Local AS Number + -d, --format Log display format [default: plain] [possible values: plain, json] + -r, --router-id Local router id(must be ipv4 format) + --fib Fib endpoint url(gRPC) exp) localhost:5001 + --table-id Target fib table id(default is main(254)) + --exporter Exporter endpoint url + -l, --level Log level(trace, debug, info, warn, error) [default: info] + -h, --help Print help +``` + +We also can configure BGP with a configuration file written by Yaml. +This is the example of config files. + +```yaml +asn: 65000 +router_id: 10.0.0.2 +multi_path: true +neighbors: + - asn: 65010 + router_id: 10.0.0.3 + address: 10.0.0.3 + - asn: 65020 + router_id: 10.0.1.3 + address: 10.0.1.3 +``` + + +## sart Commands + +To control running daemons, we can use sart-cli commands. +Now we support following subcommands + +- bgp + +### sart bgp + +`sart bgp` command accepts `global` level and `neighbor` level subcommands. + +`global` level can get and set AS number or router id of the local daemon. +And it also can configure RIB(Loc-RIB) information. + +```console +root@233eff855e37:/# sart bgp global rib -h +Usage: sart bgp global rib [OPTIONS] + +Commands: + get + add + del + help Print this message or the help of the given subcommand(s) + +Options: + -d, --format Display format [default: plain] [possible values: plain, json] + -e, --endpoint Endpoint to API server [default: localhost:5000] + -a, --afi Address Family Information [default: ipv4] [possible values: ipv4, ipv6] + -s, --safi Sub Address Family Information [default: unicast] [possible values: unicast, multicast] + -h, --help Print help +``` + +For example, to add a prefix to running BGP daemon, run this command. +```console +root@233eff855e37:/# sart bgp global rib add 10.0.10.0/24 -a ipv4 -t origin=igp +``` + +`neighbor` level can get and set neighbor information. + +**Some commands are not implemented yet.** + +```console +root@233eff855e37:/# sart bgp neighbor -h +Usage: sart bgp neighbor [OPTIONS] + +Commands: + get + list + add + del + rib + policy + help Print this message or the help of the given subcommand(s) +``` + +For example, to add a neighbor, run this. + +```console +root@233eff855e37:/# sart bgp neighbor add 10.10.0.1 65000 +``` + +## gRPC Interfaces + +Sartd has gRPC interfaces. +Sart-cli calls this interface internally. + +For detail, please see [proto/](https://github.com/teassyi/sart/blob/main/proto). + +## Integrate with FIB Manager + +To install paths received from other peers, we need to a FIB manager. +For example, [FRR](https://frrouting.org/) needs to run [zebrad](https://docs.frrouting.org/en/latest/zebra.html) to install paths bgpd gets. + +For `sartd-bgp`, we can run `sartd-fib` as the FIB manager. +`Sartd-fib` has gRPC interface to interact with other component. + +> [!NOTE] +>`Sartd-fib` is in the PoC phase. + +To integrate with `sartd-fib`, we have to specify the gRPC endpoint to FIB manager as a parameter like below. + +```console +$ sartd bgp --fib localhost:5010 +``` diff --git a/docs/controller/design.md b/docs/controller/design.md index ddf9b50..5ae6ba5 100644 --- a/docs/controller/design.md +++ b/docs/controller/design.md @@ -1,6 +1,6 @@ -# Design of sart-controller +# Design of sart kubernetes custom controller -Sart-controller is a self made Kubernetes network load balancer for announcing [externalIP of Service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). +Sart-controller is a self made Kubernetes network load balancer for announcing [addresses Service type LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). [metallb](https://github.com/metallb/metallb) has a similar feature. @@ -8,10 +8,9 @@ To announce load balancer IP addresses to the external system, sart-controller u So we need BGP speakers outside the Kubernetes cluster and also BGP speaker inside the cluster. Local(inside) BGP speakers are on the each nodes as DaemonSet. And sart-controller controls these speakers via API. -Now, we can only use sartd-bgp as local speakers. > **Note** -> Sart-controller is an alpha system. +> Sart is an alpha system. ## CRD based architecture diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..ecde495 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,367 @@ +# Design of sart + +## Overview + +Sart is Kubernetes network load-balancer and CNI plugin for Kubernetes using BGP in Rust. +This project is inspired by [Metallb](https://github.com/metallb/metallb) and [Coil](https://github.com/cybozu-go/coil). + +> [!WARNING] +> CNI feature is not implemented yet. + +## Programs + +Sart has following progarams. + +- `sart` + - CLI tool to control and describe `sartd`. +- `sartd` + - Daemon program that has following features. + - bgp: BGP daemon + - fib: FIB daemon + - agent: Kubernetes agent running as DaemonSet + - controller: Kubernetes custom controller running as Deployment +- `sartcni` + - TBD + +## BGP + +Sart's BGP feature(sartd-bgp) is based on [RFC 4271 - A Border Gateway Protocol 4 (BGP-4)](https://datatracker.ietf.org/doc/html/rfc4271). +And sartd-bgp is configurable via gRPC interface. + +### Features + +Sartd-bgp has following features. +Now sartd-bgp support minimal features to work as BGP speaker. + +- [x] eBGP +- [x] iBGP +- [x] 4 Bytes ASN +- [ ] Policy injection +- [x] Multiple paths +- [ ] IPv6 features +- [ ] BGP unnumbered +- [ ] Route Reflector + +### Archietecture + +This figure shows a basic model of sartd-bgp. + +![model](./img/model.drawio.svg) + +Sartd-bgp is the event driven model. + + +#### Peer handling + +A peer has one event handler to wait for given events. +And peer has channels to handle some events. +Events include the following. + +- Message +- Timer +- Rib +- Admin + +Once an event is generated, it is queued to the channel. +Then the event handler retrieves and processes its event. +After that, according to the result, it move the state of BGP FSM. + +##### Finite State Machine + +BGP has an independent FSM for each peer. +This diagram shows how BGP states move by given events. +Please refer [RFC 4271: Section 8.2.2](https://datatracker.ietf.org/doc/html/rfc4271#section-8.2.2) for details such as event numbers. + +```mermaid +stateDiagram-v2 + Idle --> Connect: 1 + Idle --> Idle: others + Connect --> Connect: 1,9,14 + Connect --> Active: 18 + Connect --> OpenSent: 16,17 + Connect --> Idle: others + Active --> OpenSent: 16,17 + Active --> OpenConfirm: 19 + Active --> Connect: 9 + Active --> Active: 1,14 + Active --> Idle: others + OpenSent --> OpenSent: 1,14,16,17 + OpenSent --> Active: 18 + OpenSent --> OpenConfirm: 19 + OpenSent --> Idle: others + OpenConfirm --> OpenConfirm: 1,11,14,16,17 + OpenConfirm --> Established: 26 + OpenConfirm --> Idle: others + Established --> Established: 1,11,26,27 + Established --> Idle: others +``` + +#### RIB(Routing Information Base) + +This figure shows how sart-dbgp handle routing information. +RIB-Manager has the event handler and wait for RIB event from peers. + +When Adj-RIB-In gets some paths, its peer filters by Input policies, and publishes the event to store into the Loc-RIB. +And Loc-RIB gets paths, Loc-RIB selects new best path and if the best path is changed, publishes the event to all peers. +If a peer receives its event from RIB-Manager, it filter paths by Output policies, and store into Adj-RIB-Out. + +![rib-model](./img/rib-model.drawio.svg) + +## FIB + +TBD + +## Kubernetes + +Sart has Kubernetes network load-balancer feature. +This is named `sartd-kubernetes`. + +Sartd-kubernetes is Kubernetes custom controller. + +### Components + +Sartd-kubernetes consists of following two components. + +- `controller` +- `agent` + +`controller` runs as `Deployment` to manage non node local resources such as `Service` and `EndpointSlice`. +And this also manages admission webhooks. +This is responsible to control load-balancer addresses. + +`agent` runs as `DaemonSet` on each node to control node local resources. +This interact with the node local BGP speaker to announce load-balancer addresses to external networks. + +### Custom Resource Definition + +`Sartd-kubernetes` defines following custom resources. + +- `AddressPool` +- `AddressBlock` +- `ClusterBGP` +- `NodeBGP` +- `BGPPeer` +- `BGPPeerTemplate` +- `BGPAdvertisement` + +These CRD's API group and version is `sart.terassyi.net`, `v1alpha2` + +This figure shows the relationships between CRDs. + +![kubernetes-model](./img/kubernetes-model.drawio.svg) + +#### AddressPool + +`AddressPool` defines a CIDR usable in itself. +This has `type` parameter to specify for which this pool will be use. +This also has `allocType` parameter to choose a method to allocate. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: AddressPool +metadata: + name: default-lb-pool +spec: + cidr: 10.0.1.0/24 + type: service + allocType: bit + blockSize: 24 + autoAssign: true +``` + +#### AddressBlock + +`AddressBlock` defines a subset of the `AddressPool` and its CIDR must be contained by AddressPool's CIDR. +The allocator entity is tied to `AddressBlock`. + +This is automatically generated by `AddressPool`. + +If the parent `AddressPool`'s type is `service`, `AddressPool` must be one for its pool. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: AddressBlock +metadata: + name: non-default-lb-pool +spec: + autoAssign: false + cidr: 10.0.100.0/24 + nodeRef: null + poolRef: non-default-lb-pool + type: service +``` + +#### ClusterBGP + +`ClusterBGP` defines a cluster wide BGP configurations. +We can select nodes to be configurable by `nodeSelector`. + +This can template node local ASN and Router-Id settings. +And this also provide a way to template BGP peer settings. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: ClusterBGP +metadata: + name: clusterbgp-sample-a +spec: + nodeSelector: + bgp: a + asnSelector: + from: label + routerIdSelector: + from: internalAddress + speaker: + path: 127.0.0.1:5000 + peers: + - peerTemplateRef: bgppeertemplate-sample + nodeBGPSelector: + bgp: a +``` + +#### NodeBGP + +`NodeBGP` defines a node local BGP configuration. + +This is automatically generated by `ClusterBGP`. +This is managed by `agent`. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: NodeBGP +metadata: + labels: + bgp: a + name: sart-worker +spec: + asn: 65000 + peers: + - name: bgppeertemplate-sample-sart-worker-65000-9.9.9.9 + ... + routerId: 172.18.0.2 + speaker: + path: 127.0.0.1:5000 +``` + +#### BGPPeer + +`BGPPeer` defines a local BGP peer abstraction. +This resource will be created by `NodeBGP` and we can also create manually. + +This is manged by `agent`. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: BGPPeer +metadata: + labels: + bgp: a + bgppeer.sart.terassyi.net/node: sart-worker + name: bgppeer-sample-sart-worker +spec: + addr: 9.9.9.9 + asn: 65000 + nodeBGPRef: sart-worker + speaker: + path: 127.0.0.1:5000 +``` + +#### BGPPeerTemplate + +`BGPPeerTemplate` defines a template BGP peer configuration. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: BGPPeerTemplate +metadata: + name: bgppeertemplate-sample +spec: + asn: 65000 + addr: 9.9.9.9 + groups: + - to-router0 +``` + +#### BGPAdvertisement + +`BGPAdvertisement` defines a prefix announced by BGPPeers. +This is automatically generated when the `controller` allocates new addresses. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: BGPAdvertisement +metadata: + labels: + kubernetes.io/service-name: app-svc-cluster + name: app-svc-cluster-hv4l5-10.0.1.0 + namespace: test +spec: + attrs: null + cidr: 10.0.1.0/32 + protocol: ipv4 + type: service +``` + +### IP Address Management(IPAM) + +#### Service type LoadBalancer + +`Sartd-kubernetes` has IPAM for Service type LoadBalancer. + +##### AddressPool + +To allocate IP addresses to LoadBalancers, we can create pools of IP addresses. +To create pools for LoadBalancer, we have to apply `AddressPool` resource with `type: service`. + +We can create multiple pools. + +##### Choosing pools + +We create multiple pools in one cluster. +If there are multiple pools, we can choose the pool from which the controller allocates to the LoadBalancer. +Choosing the pool, we have to add the annotation. + +When we specify multiple pools in the annotation, we can assign multiple addresses from specified pools. + +We also define a default assignable pool to specify `autoAssign: true` in the spec. +If we want to use the pool its `autoAssign` is true, we can omit to specify its pool name in the annotation. +We cannot create multiple default assignable pools. + +When we specify multiple pools, and one of specified pools is default assignable, we must specify its name. + +##### Requesting specific addresses + +We can assign specific addresses to the LoadBalancer. +To specify it, we also use the annotation. + +We can give multiple addresses from one pool or multiple pools. + +##### AddressBlock + +`AddressBlock` is automatically generated by `AddressPool` and is a subset of a pool. +In case of a pool for LoadBalancer(`type: service`), `AddressBlock` is created only one by the pool. +Therefore, CIDR field of `AddressBlock` must be equal to `AddressPool`'s CIDR. + +And its allocator is managed by the `controller`. + +##### Allocating and Releasing LoadBalancer addresses + +Sartd-kubernetes's `controller` watches `Service` and `EndpointSlice` resources. +When it detects the event for LoadBalancer, it triggers the reconciliation logic. + +First, when the LoadBalancer is created, the controller picks addresses from desired pools if addresses are requested, it picks its addresses or if not, automatically. +If its addresses are valid, the controller allocates these. +And it updates the status(`status.loadBalancer.ingress[].ip`) of given `Service` resource. + +### Announcing addresses + +The controller creates `BGPAdvertisement` resources depending on allocated addresses and the `externalTrafficPolicy` of the LoadBalancer. + +When `externalTrafficPolicy` is `Cluster`, advertisements must have all peers which matches group label. +If group label is not specified, all existing peers are picked. + +When `externalTrafficPolicy` is `Local`, advertisements must have peers on the node where backend pods exist. +And target peers must be changed depending on the change of backend pods scheduling. + + +## CNI for Kubernetes diff --git a/docs/fib.md b/docs/fib.md new file mode 100644 index 0000000..58de96e --- /dev/null +++ b/docs/fib.md @@ -0,0 +1,18 @@ +# How to Use + +Sart has two CLI interfaces, sartd is for daemon and sart-cli is for controlling sartd. + +### sartd fib + +We can run the Fib manager to install or uninstall routing information in kernel with `sartd fib`. + +```console +root@233eff855e37:/# sartd fib -h +Usage: sartd fib [OPTIONS] + +Options: + -e, --endpoint Fib manager running endpoint url [default: 127.0.0.1:5001] + -l, --level Log level(trace, debug, info, warn, error) [default: info] + -d, --format Log display format [default: plain] [possible values: plain, json] + -h, --help Print help +``` diff --git a/docs/img/kubernetes-model.drawio.svg b/docs/img/kubernetes-model.drawio.svg new file mode 100644 index 0000000..d3ac36d --- /dev/null +++ b/docs/img/kubernetes-model.drawio.svg @@ -0,0 +1,512 @@ + + + + + + + + +
+
+
+ Service(LoadBalancer) +
+
+
+ + + Service(LoadBalancer) + + + + + + + + + + + + + + + + + +
+
+
+ + sartd-bgp + +
+
+
+ + + sartd-bgp + + + + + + + + + + + +
+
+
+ + sartd-agent + +
+
+
+ + + sartd-agent + + + + + + + + + + + + + + + +
+
+
+ + sartd-bgp + +
+
+
+ + + sartd-bgp + + + + + + + + + + + +
+
+
+ + sartd-agent + +
+
+
+ + + sartd-agent + + + + + + + + + + + + + + + +
+
+
+ + sartd-bgp + +
+
+
+ + + sartd-bgp + + + + + + + + + + + + + +
+
+
+ + sartd-agent + +
+
+
+ + + sartd-agent + + + + + + + +
+
+
+ + External BGP speaker 1 + +
+
+
+ + + External BGP speaker 1 + + + + + + + + + + + + + + + + + + + +
+
+
+ + NodeBGP + +
+
+
+ + + NodeBGP + + + + + + + + + +
+
+
+ + + BGPPeer + + +
+
+
+ + + BGPPeer + + + + + + + + + +
+
+
+ + + BGPPeer + + +
+
+
+ + + BGPPeer + + + + + + + + + +
+
+
+ + NodeBGP + +
+
+
+ + + NodeBGP + + + + + + + + + +
+
+
+ + NodeBGP + +
+
+
+ + + NodeBGP + + + + + + + + + +
+
+
+ + + BGPPeer + + +
+
+
+ + + BGPPeer + + + + + + + + + +
+
+
+ + + BGPPeer + + +
+
+
+ + + BGPPeer + + + + + + + +
+
+
+ + External BGP speaker 2 + +
+
+
+ + + External BGP speaker 2 + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ BGPAdvertisements +
+
+
+ + + BGPAdvertisements + + + + + + + + + +
+
+
+ sart-controller +
+
+
+ + + sart-controll... + + + + + + + + + + + + + + +
+
+
+ + ClusterBGPs + +
+
+
+ + + ClusterB... + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ AddressPool/AddressBlock +
+
+
+ + + AddressPool/AddressBlock + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/sartd/model.drawio.svg b/docs/img/model.drawio.svg similarity index 100% rename from docs/sartd/model.drawio.svg rename to docs/img/model.drawio.svg diff --git a/docs/img/rib-model.drawio.svg b/docs/img/rib-model.drawio.svg new file mode 100644 index 0000000..61ee70f --- /dev/null +++ b/docs/img/rib-model.drawio.svg @@ -0,0 +1,366 @@ + + + + + + + +
+
+
+ + peer1 + +
+
+
+ + + peer1 + + + + + + + + + +
+
+
+ + RIB Manager + +
+
+
+ + + RIB Manager + + + + + + + + + + +
+
+
+ + Adj-RIB-In + +
+
+
+ + + Adj-R... + + + + + + + + + +
+
+
+ + Input policy + +
+
+
+ + + Input policy + + + + + + + + +
+
+
+ + Adj-RIB-Out + +
+
+
+ + + Adj-RI... + + + + + + + + + +
+
+
+ + Output policy + +
+
+
+ + + Output policy + + + + + + + +
+
+
+ + peer1 + +
+
+
+ + + peer1 + + + + + + + + + + +
+
+
+ + Adj-RIB-In + +
+
+
+ + + Adj-R... + + + + + + + + + +
+
+
+ + Input policy + +
+
+
+ + + Input policy + + + + + + + + +
+
+
+ + Adj-RIB-Out + +
+
+
+ + + Adj-RI... + + + + + + + + + +
+
+
+ + Output policy + +
+
+
+ + + Output policy + + + + + + + +
+
+
+ + peer1 + +
+
+
+ + + peer1 + + + + + + + + + + +
+
+
+ + Adj-RIB-In + +
+
+
+ + + Adj-R... + + + + + + + + + +
+
+
+ + Input policy + +
+
+
+ + + Input policy + + + + + + + + +
+
+
+ + Adj-RIB-Out + +
+
+
+ + + Adj-RI... + + + + + + + + + +
+
+
+ + Output policy + +
+
+
+ + + Output policy + + + + + + + + + + + +
+
+
+ + Loc-RIB + +
+
+
+ + + Loc-RIB + + + + + + + + + Text is not SVG - cannot display + + + + diff --git a/docs/kubernetes.md b/docs/kubernetes.md new file mode 100644 index 0000000..c33825a --- /dev/null +++ b/docs/kubernetes.md @@ -0,0 +1,324 @@ +# How to Use as Kubernetes network load-balancer + +To use kubernetes network load-balancer, we need to run two components: `sartd controller` and `sartd agent`. + +The `controller` runs as `Deployment`. +And the `agent` runs as `DaemonSet`. +`agent` must run in host network namespace. + +For installation, please read [setup for kubernetes](./setup-kubernetes.md). + +To know sart's CRDs, please see [design #Kubernetes](./design.md#kubernetes) + +## Address pools + +`AddressPool` is a cluster-scope resource. + +LoadBalancer services are assigned IP addresses from an address pool. +And pods is also assigned their IP address from this. + +The `type` field specifies for which use the address pool is to be used. + +This is an example address pool for LoadBalancer. +To use for LoadBalancer, we have to set `type: service`. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: AddressPool +metadata: + name: default-lb-pool +spec: + cidr: 10.0.1.0/24 + type: service + allocType: bit + blockSize: 24 + autoAssign: true +``` + +`cidr` is the subnet of assignable addresses. +Now, we support only IPv4. + +`allocType` specifies a method how to pick an address from a pool. +We can only use `bit`. + +Bit allocator type choose the address with the lowest number. +For example, `10.0.0.1` and `10.0.0.10` are assignable, the bit allocator always chooses `10.0.0.1`. + +`blockSize` is used to divide the address pool into address blocks. +In case of `type: service`, `blockSize` must be equal to a cidr's prefix length. + +`autoAssign` specifies its pool is used as default pool. +If `autoAssign` is true, we can omit specifying the name of the pool in the annotation(described below). +We cannot create multiple auto assignable pools in each `type`. + +Please note that we cannot change `AddressPool`'s spec fields except for `autoAssign` once it is created. + +## BGP configurations + +To announce assigned addresses to external network, Sart use BGP. + +Before configuring BGP, we have to confirm that `sartd-bgp` runs well. + +### ClusterBGP + +To configure the cluster wide BGP configurations, we can create `ClusterBGP` resources. +`ClusterBGP` is a cluster-scope resource. + +`ClusterBGP` resource templates local BGP speaker settings and peer information for each node. + +This is an example of `ClusterBGP`. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: ClusterBGP +metadata: + name: clusterbgp-sample-a +spec: + nodeSelector: + bgp: a + asnSelector: + from: label + routerIdSelector: + from: internalAddress + speaker: + path: 127.0.0.1:5000 + peers: + - peerTemplateRef: bgppeertemplate-sample + nodeBGPSelector: + bgp: a +``` + +If `nodeSelector` is specified, `ClusterBGP` creates `NodeBGP` resources on only nodes that has labels matching its selector. +If `nodeSelector` is not specified, this creates `NodeBGP` resources on all nodes. +a `NodeBGP` resource is created with the same name as the node. + + +`asnSelector` and `routerIdSelector` is used to configure `NodeBGP` resources. + +`asnSelector.from` can have `asn` or `label`. +If `asnSelector.from` is `asn`, `asnSelector.asn` must be set as valid ASN value. +If it is `label`, ASN value is got from the label named `sart.terassyi.net/asn` added to the node. + +`routerIdSelector.from` can have `internalAddress`, `label` and `routerId`. +In case of `internalAddress`, `NodeBGP`'s router id is got from the node resource's address. +When `label` is specified, router id is got from the label named `sart.terassyi.net/router-id` on the node. +And when `routerId` is specified, router id is got from `routerIdSelector.routerId`. + +`speaker` specifies the endpoint to the local BGP speaker on the node. +When `NodeBGP` resource is created and local speaker is configured on the node, `agent` on the same node interacts with the local BGP speaker via its endpoint. + +`peers` can have the array of `BGPPeerTemplate`(described details in the following section). +We can specify a name of existing `BGPPeerTemplate` resource in a `peerTemplateRef`. +And we can also specify a spec field of `BGPPeerTemplate` inline. +We can choose target node with `nodeBGPSelector`. + +### BGPPeerTemplate + +We can define `BGPPeerTemplate` as a template of BGP peers. +If a `ClusterBGP` resource is created and `peerTemplateRef` is specified, `BGPPeer` resources are created based on this template. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: BGPPeerTemplate +metadata: + name: bgppeertemplate-sample +spec: + asn: 65000 + addr: 9.9.9.9 + groups: + - to-router0 +``` + +### BGPPeer + +`BGPPeer` represents BGP peer in the local BGP speaker. +`BGPPeer` is a cluster-scope resource. + +This is an example. + +```yaml +apiVersion: sart.terassyi.net/v1alpha2 +kind: BGPPeer +metadata: + labels: + bgp: b + bgppeer.sart.terassyi.net/node: sart-control-plane + name: bgppeer-sample-sart-cp +spec: + addr: 9.9.9.9 + asn: 65000 + groups: + - to-router0 + nodeBGPRef: sart-control-plane + speaker: + path: 127.0.0.1:5000 +``` + +`asn` and `addr` specifies the remote BGP speaker configuration. +`speaker` is for the endpoint to local BGP speaker. + +`BGPPeer` is related to the specific `NodeBGP`. +So we need to give the node name in `nodeBGPRef`. + +`BGPPeer`'s `status` field represents the actual state of BGP peer and we can check the status by `kubectl get bgppeer` like below. + +```console +$ kubectl get bgppeer +NAME ASN ADDRESS NODEBGP STATUS AGE +bgppeer-sample-sart-cp 65000 9.9.9.9 sart-control-plane Established 8s +bgppeertemplate-sample-sart-worker-65000-9.9.9.9 65000 9.9.9.9 sart-worker Established 2s +bgppeertemplate-sample-sart-worker2-65000-9.9.9.9 65000 9.9.9.9 sart-worker2 Established 2s +bgppeertemplate-sample-sart-worker3-65000-9.9.9.9 65000 9.9.9.9 sart-worker3 OpenConfirm 1s +``` + +## LoadBalancer Service + +If we complete to prepare BGP related resources and `AddressPools`, we can use LoadBalancer services. + +After adding pools to the cluster, it appears like so. +Sample manifests are [here](../manifests/sample/lb_address_pool.yaml). + +```console +$ kubectl get addresspool +NAME CIDR TYPE BLOCKSIZE AUTO AGE +default-lb-pool 10.0.1.0/24 service 24 true 13s +non-default-lb-pool 10.0.100.0/24 service 24 false 13s +``` + +### Basic usage + +A basic Service type LoadBalancer looks like this. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: app-svc-cluster + namespace: test + annotations: +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + selector: + app: app-cluster + ports: + - name: http + port: 80 + targetPort: 80 +``` + +After creating this, we can confirm that `EXTERNAL-IP` is assigned from the `default-lb-pool`. +In this case, there isn't requested pools. +So the LoadBalancer address is assigned from the `default-lb-pool`. + +```console +$ kubectl -n test get svc app-svc-cluster +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +app-svc-cluster LoadBalancer 10.101.205.218 10.0.1.0 80:31012/TCP 11m +``` + +### Choosing AddressPool + +LoadBalancer services can choose address pools. +To specify it, we use the annotation named `sart.terassyi.net/addresspool` like this. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: app-svc-cluster2 + namespace: test + annotations: + sart.terassyi.net/addresspool: non-default-lb-pool +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + selector: + app: app-cluster2 + ports: + - name: http + port: 80 + targetPort: 80 +``` + +And we can confirm that the `non-default-lb-pool` is used. + +```console +$ kubectl -n test get svc app-svc-local +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +app-svc-local LoadBalancer 10.101.47.224 10.0.100.0 80:31291/TCP 34s +``` + +### Requesting addresses + +LoadBalancer services can also request specific addresses. +To request it, we use the annotation named `sart.terassyi.net/loadBalancerIPs` like this. + +In this example, this service requests `10.0.100.20` from the `non-default-lb-pool`. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: app-svc-cluster2 + namespace: test + annotations: + sart.terassyi.net/addresspool: non-default-lb-pool + sart.terassyi.net/loadBalancerIPs: "10.0.100.20" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + selector: + app: app-cluster2 + ports: + - name: http + port: 80 + targetPort: 80 +``` + +```console +$ kubectl -n test get svc app-svc-cluster2 +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +app-svc-cluster2 LoadBalancer 10.101.186.67 10.0.100.20 80:31857/TCP 17m +``` + +### Assigning multiple addresses + +We can assign multiple addresses to one LoadBalancer from multiple pools. + +This manifest requests to assign multiple addresses from `default-lb-pool` and `non-default-lb-pool`. +To specify multiple value in the annotation, we have to specify values separated by commas. +And it also requests `10.0.100.22`. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: app-svc-cluster3 + namespace: test + annotations: + sart.terassyi.net/addresspool: default-lb-pool,non-default-lb-pool + sart.terassyi.net/loadBalancerIPs: "10.0.100.22" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + selector: + app: app-cluster3 + ports: + - name: http + port: 80 + targetPort: 80 +``` + +In this case, `default-lb-pool` assigns an address automatically and `non-default-lb-pool` assigns the specified address(`10.0.100.22`). + + +```console +$ kubectl -n test get svc app-svc-cluster3 +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +app-svc-cluster3 LoadBalancer 10.101.38.202 10.0.1.1,10.0.100.22 80:31980/TCP 5m45s +``` + +When we choose multiple pools and even if one of these pool is auto assignable, we cannot omit specifying its name in the annotation. + +We also can request multiple addresses from one pool. diff --git a/docs/kubernetes/design.md b/docs/kubernetes/design.md new file mode 100644 index 0000000..74a0c79 --- /dev/null +++ b/docs/kubernetes/design.md @@ -0,0 +1,58 @@ +# Design and implementation of sart kubernetes controller + +Sart has two components related to Kubernetes. +One is `controller` for controlling [kubernetes custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) described below. +`controller` is deployed only one in a cluster as `Deployment`. +Another is `agent` for bridging between its custom resources and BGP speaker running on the node. +`agent` is deployed on each node in a cluster as `DaemonSet`. + +And we also need to run `bgp` as `DaemonSet` to work sart kubernetes controller well. + +Sart is implemented by pure Rust using [kube-rs](https://github.com/kube-rs/kube). + +## Feature + +Sart is a Kubernetes load-balancer to announce addresses of [Service type LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). +[Metallb](https://github.com/metallb/metallb) has a similar feature and sart is inspired by Metallb. + +### Allocating addresses for the LoadBalancer + +### Announcing LoadBalancer addresses to outside of the cluster + +## CRD based architecture + +Sart defines some Custom Resource Definitions to represent . +This figure shows how sart's `controller` and `agent` control load balancer addresses and BGP using custom resources. + +### CRDs + +#### ClusterBGP + +ClusterBGP resource controls peering policies in the cluster. +This resource is required one per cluster. + +#### NodeBGP + +NodeBGP resource exists per nodes. +And this controls a BGP speaker pod on the node such as speaker status and basic information. +This is created by sart-controller when the Node resource is detected. +So we shouldn't create this manually. + +#### BGPPeer + +BGPPeer resource controls a BGP peer between local speaker on the node and the external speaker. +This has information of the peering state and advertisements. + +#### BGPAdvertisement + +BGPAdvertisement resource controls the routing information of the load balancer service. +This has the prefix of the announcing address and node names that should advertise its prefix. + +This resource should be owned by Service resource. +So we shouldn't create this manually. + +#### AddressPool + +AddressPool resource is a CIDR for assignable to load balancers. +This resource is the cluster wide resource. +And we can deploy multiple address pools. diff --git a/docs/setup-kubernetes.md b/docs/setup-kubernetes.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..6a5a181 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,9 @@ +# User Manual + +This document describes how to use Sart. + +Please refer to following links for details of each feature. + +- [bgp](./bgp.md) +- [fib](./fib.md) +- [kubernetes](./kubernetes.md)