feat(fabric): Update playbooks and charts to support add orderer oper…

…ation to an existing network

Signed-off-by: alvaropicazo <[email protected]>

docs/source/guides/fabric/add-new-orderer-peer.md

@@ -3,66 +3,127 @@
 [//]: # (SPDX-License-Identifier: Apache-2.0)
 [//]: # (##############################################################################################)
 
-<a name = "adding-new-orderer-to-existing-organization-in-a-running-fabric-network"></a>
-# Adding a new RAFT orderer to existing Orderer organization in Hyperledger Fabric
+# Add Orderer Node to an existing organization
 
-- [Prerequisites](#prerequisites)
-- [Modifying Configuration File](#modifying-configuration-file)
-- [Run playbook](#run-playbook)
+This guide explains how to add an orderer node to an existing Hyperledger Fabric network using two methods:
 
+1. Using the `add-orderer.yaml` playbook: This method involves running an Ansible playbook that automates the process of adding an orderer node to the network.
+
+1. Using `helm install`: This method involves using the helm install command to directly install the orderer node chart.
 
-<a name = "prerequisites"></a>
 ## Prerequisites
-To add a new Orderer node, a fully configured Fabric network must be present already, i.e. a Fabric network which has Orderers, Peers, Channels (with all Peers already in the channels) and the organization to which the peer is being added. The corresponding crypto materials should also be present in their respective Hashicorp Vault. 
 
----
-**NOTE**: Addition of a new Orderer node has been tested on an existing network which is created by Bevel. Networks created using other methods may be suitable but this has not been tested by Bevel team.
-This works only for RAFT Orderer.
+- A fully configured Fabric network with Orderers and Peers. 
+- Corresponding crypto materials present in Hashicorp Vault or Kubernetes secrets.
+- Hyperledger Bevel configured.
+
+## Method 1: Using the `add-cli.yaml` playbook
+
+1. **Update Configuration File**
+
+      To add a new Orderer node, a fully configured Fabric network must be present already, i.e. a Fabric network which has Orderers, Peers, Channels (with all Peers already in the channels) and the organization to which the peer is being added. The corresponding crypto materials should also be present in their respective Hashicorp Vault. 
+
+      ---
+      **NOTE**: Addition of a new Orderer node has been tested on an existing network which is created by Bevel. Networks created using other methods may be suitable but this has not been tested by Bevel team.
+      This works only for RAFT Orderer.
+
+      ---
+
+1. **Update Configuration File**
+
+      A Sample configuration file for adding new orderer is available [here](https://github.com/hyperledger/bevel/blob/main/platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml). Please go through this file and all the comments there and edit accordingly.
+
+      For generic instructions on the Fabric configuration file, refer [this guide](../networkyaml-fabric.md).
+
+      While modifying the configuration file(`network.yaml`) for adding new peer, all the existing orderers should have `status` tag as `existing` and the new orderers should have `status` tag as `new` under `network.organizations` as
+
+      ```yaml
+
+      --8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:126:135"
+            ..
+            ..
+      --8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:174:174"
+      --8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:185:220"
+
+      ```
+      and under `network.orderers` the new orderer must be added.
+
+      ```yaml
+      --8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:42:66"
+      ```
+
+      The `network.yaml` file should contain the specific `network.organization` details.
+
+      Ensure the following is considered when adding the new orderer on a different cluster:
+      - The CA server is accessible publicly or at least from the new cluster.
+      - The CA server public certificate is stored in a local path and that path provided in network.yaml.
+      - There is a single Hashicorp Vault and both clusters (as well as ansible controller) can access it.
+      - Admin User certs have been already generated and store in Vault (this is taken care of by deploy-network.yaml playbook if you are using Bevel to setup the network).
+      - The `network.env.type` is different for different clusters.
+      - The GitOps release directory `gitops.release_dir` is different for different clusters.
+
+1. **Run playbook**
+
+      The [add-orderer.yaml](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/configuration/add-orderer.yaml) playbook is used to add a new peer to an existing organization in the existing network. This can be done using the following command
+
+      ```
+      ansible-playbook platforms/hyperledger-fabric/configuration/add-orderer.yaml --extra-vars "@path-to-network.yaml"
+      ```
+
+      ---
+      **NOTE:** The `orderer.status` is not required when the network is deployed for the first time but is mandatory for addition of new orderer.
+
 
----
+## Method 2: Using `helm install`
 
-<a name = "modifying-configuration-file"></a>
-## Modifying Configuration File
+1. **Update the orderernode values.yaml file**
 
-A Sample configuration file for adding new orderer is available [here](https://github.com/hyperledger/bevel/blob/main/platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml). Please go through this file and all the comments there and edit accordingly.
+    Following changes are must in the `values.yaml` file for a new orderer node to be added to the network:
 
-For generic instructions on the Fabric configuration file, refer [this guide](../networkyaml-fabric.md).
+    - `certs.settings.createConfigMaps: false` as the ConfigMaps for certs are already generated in the same namespace.
 
-While modifying the configuration file(`network.yaml`) for adding new peer, all the existing orderers should have `status` tag as `existing` and the new orderers should have `status` tag as `new` under `network.organizations` as
+    Refer to the [fabric-orderernode chart documentation](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/charts/fabric-orderernode) for a complete list of available configuration options.
 
-```yaml
+1. **Install the orderernode chart**
+
+    Execute the following command to install the Peer chart:
+	```bash
+	helm dependency update ./fabric-orderernode
+	helm install <release-name> ./fabric-orderernode --namespace <namespace> --values <values-file.yaml>
+	```
+	Replace the following placeholders:
 
---8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:126:135"
-      ..
-      ..
---8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:174:174"
---8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:185:220"
+	- `<release-name>`: The desired name for the orderer node release.
+	- `<namespace>`: The Kubernetes namespace where the Peer should be deployed.
+	- `<values-file.yaml>`: The path to a YAML file containing the new peer configuration values.
 
-```
-and under `network.orderers` the new orderer must be added.
+1. **Update the osnadmin-channel-create values.yaml file**
 
-```yaml
---8<-- "platforms/hyperledger-fabric/configuration/samples/network-fabricv2-raft-add-orderer.yaml:42:66"
-```
-          
-The `network.yaml` file should contain the specific `network.organization` details.
+    Following changes are must in the `values.yaml` file for a new orderer node to be added to the network:
+      ```
+      orderer:
+            addOrderer: true
+            name: orderer5
+            localMspId: orgNameMSP
+            ordererAddress: orderer1.orgname-net:443
+      ```
 
-Ensure the following is considered when adding the new orderer on a different cluster:
-- The CA server is accessible publicly or at least from the new cluster.
-- The CA server public certificate is stored in a local path and that path provided in network.yaml.
-- There is a single Hashicorp Vault and both clusters (as well as ansible controller) can access it.
-- Admin User certs have been already generated and store in Vault (this is taken care of by deploy-network.yaml playbook if you are using Bevel to setup the network).
-- The `network.env.type` is different for different clusters.
-- The GitOps release directory `gitops.release_dir` is different for different clusters.
+    Refer to the [fabric-osn-channel-create chart documentation](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/charts/fabric-osn-channel-create) for a complete list of available configuration options.
 
-<a name = "run-playbook"></a>
-## Run playbook
+1. **Install the osnadmin-channel-create chart**
+
+    Execute the following command to install the fabric-osnadmin-channel-create chart:
+	```bash
+    cd ../..
+	helm install <release-name> ./fabric-osnadmin-channel-create --namespace <namespace> --values <values-file.yaml>
+	```
+	Replace the following placeholders:
 
-The [add-orderer.yaml](https://github.com/hyperledger/bevel/tree/main/platforms/hyperledger-fabric/configuration/add-orderer.yaml) playbook is used to add a new peer to an existing organization in the existing network. This can be done using the following command
+	- `<release-name>`: The desired name for the Peer release.
+	- `<namespace>`: The Kubernetes namespace where the Peer should be deployed.
+	- `<values-file.yaml>`: The path to a YAML file containing the new peer configuration values.
 
-```
-ansible-playbook platforms/hyperledger-fabric/configuration/add-orderer.yaml --extra-vars "@path-to-network.yaml"
-```
 
----
-**NOTE:** The `orderer.status` is not required when the network is deployed for the first time but is mandatory for addition of new orderer.
+## Additional Notes
+- The `add-orderer.yaml playbook` and `helm install` method has been tested on networks created by Bevel. Networks created using other methods may be suitable, but this has not been tested by the Bevel team.
+- Ensure that the network.yaml file contains the specific network.organization details along with the orderer information.

platforms/hyperledger-fabric/charts/fabric-cli/templates/deployment.yaml

@@ -117,6 +117,26 @@ spec:
             fi     
           }
 
+          function getAdminTlsSecret {
+            KEY=$1
+
+            echo "Getting TLS certificates from Vault."
+            vaultBevelFunc "readJson" "${VAULT_SECRET_ENGINE}/${VAULT_SECRET_PREFIX}/users/${KEY}"
+            if [ "$SECRETS_AVAILABLE" == "yes" ]
+            then
+              OSN_TLS_CA_ROOT_CERT=$(echo ${VAULT_SECRET} | jq -r '.["ca_crt"]')
+              ADMIN_TLS_SIGN_CERT=$(echo ${VAULT_SECRET} | jq -r '.["client_crt"]')
+              ADMIN_TLS_PRIVATE_KEY=$(echo ${VAULT_SECRET} | jq -r '.["client_key"]')
+
+              echo "${OSN_TLS_CA_ROOT_CERT}" > ${OUTPUT_TLS_PATH}/tlsca.crt
+              echo "${ADMIN_TLS_SIGN_CERT}" > ${OUTPUT_TLS_PATH}/server.crt
+              echo "${ADMIN_TLS_PRIVATE_KEY}" > ${OUTPUT_TLS_PATH}/server.key
+              ADMIN_TLS_SECRET=true
+            else
+              ADMIN_TLS_SECRET=false
+            fi     
+          }
+
 {{- else }}
 
           function getAdminMspSecret {
@@ -140,21 +160,41 @@ spec:
             fi            
           }
 
+          function getAdminTlsSecret {
+            KEY=$1
+            KUBENETES_SECRET=$(kubectl get secret ${KEY} --namespace {{ .Release.Namespace }} -o json)
+            if  [ "$KUBENETES_SECRET" = "" ]; then
+              ADMIN_TLS_SECRET=false
+            else
+              OSN_TLS_CA_ROOT_CERT=$(echo ${KUBENETES_SECRET} | jq -r '.data.ca_crt'  | base64 -d)
+              ADMIN_TLS_SIGN_CERT=$(echo ${KUBENETES_SECRET} | jq -r '.data.client_crt'  | base64 -d)
+              ADMIN_TLS_PRIVATE_KEY=$(echo ${KUBENETES_SECRET} | jq -r '.data.client_key'  | base64 -d)
+
+              echo "${OSN_TLS_CA_ROOT_CERT}" > ${OUTPUT_TLS_PATH}/tlsca.crt
+              echo "${ADMIN_TLS_SIGN_CERT}" > ${OUTPUT_PATH}/cacerts/server.crt
+              echo "${ADMIN_TLS_PRIVATE_KEY}" > ${OUTPUT_PATH}/keystore/server.key 
+              ADMIN_TLS_SECRET=true
+            fi            
+          }
+
 {{- end }} 
 
           COUNTER=1
           while [ "$COUNTER" -le {{ .Values.healthCheck.retries }} ]
           do
 
             OUTPUT_PATH="${MOUNT_PATH}/admin/msp"
+            OUTPUT_TLS_PATH="${MOUNT_PATH}/tls/msp"
+            mkdir -p ${OUTPUT_TLS_PATH}
             mkdir -p ${OUTPUT_PATH}/admincerts
             mkdir -p ${OUTPUT_PATH}/cacerts
             mkdir -p ${OUTPUT_PATH}/keystore
             mkdir -p ${OUTPUT_PATH}/signcerts
             mkdir -p ${OUTPUT_PATH}/tlscacerts
-            getAdminMspSecret admin-msp  
+            getAdminMspSecret admin-msp
+            getAdminTlsSecret admin-tls
 
-            if [ "$ADMIN_MSP_SECRET" = "true" ]
+            if [ "$ADMIN_MSP_SECRET" = "true" ] && [ "$ADMIN_TLS_SECRET" = "true" ]
             then
               echo "Peer certificates have been obtained correctly"
               break
@@ -214,6 +254,8 @@ spec:
           value: "{{ .Values.ordererAddress }}"
         - name: CORE_PEER_MSPCONFIGPATH
           value: /opt/gopath/src/github.com/hyperledger/fabric/crypto/admin/msp
+        - name: CORE_PEER_TLSCONFIGPATH
+          value: /opt/gopath/src/github.com/hyperledger/fabric/crypto/admin/tls
         volumeMounts:
         - name: certificates
           mountPath: /opt/gopath/src/github.com/hyperledger/fabric/crypto

platforms/hyperledger-fabric/configuration/add-orderer.yaml

@@ -24,25 +24,6 @@
       path: "./build"
       state: absent
 
-  # Create CA Tools helm-value files and check-in
-  - name: Create CA tools for each organization
-    include_role:
-      name: "create/ca_tools/orderer"
-    vars:
-      component_name: "{{ item.name | lower}}-net"
-      component: "{{ item.name | lower}}"
-      component_type: "{{ item.type | lower}}"
-      component_services: "{{ item.services }}"
-      sc_name: "{{ component }}-bevel-storageclass"
-      kubernetes: "{{ item.k8s }}"
-      vault: "{{ item.vault }}"
-      ca: "{{ item.services.ca }}"
-      docker_url: "{{ network.docker.url }}"
-      gitops: "{{ item.gitops }}"
-      values_dir: "{{playbook_dir}}/../../../{{item.gitops.release_dir}}/{{ item.name | lower }}"
-    loop: "{{ network['organizations'] }}"
-    when: item.type == 'orderer'
-
   # Generate script to modify the fetched configuration block
   - name: "Modify the system channel with tls information of new orderer"
     include_role:
@@ -65,18 +46,22 @@
       name: "create/orderers"
     vars:
       build_path: "./build"
-      namespace: "{{ item.name | lower}}-net"
-      component_type: "{{ item.type | lower}}"
-      component_services: "{{ item.services }}"
-      vault: "{{ item.vault }}"
-      git_protocol: "{{ item.gitops.git_protocol }}"
-      git_url: "{{ item.gitops.git_url }}"
-      git_branch: "{{ item.gitops.branch }}"
+      namespace: "{{ org.name | lower}}-net"
+      component_type: "{{ org.type | lower}}"
+      org_name: "{{ org.name | lower }}"
+      component_services: "{{ org.services }}"
+      kubernetes: "{{ org.k8s }}"
+      vault: "{{ org.vault }}"
+      git_protocol: "{{ org.gitops.git_protocol }}"
+      git_url: "{{ org.gitops.git_url }}"
+      git_branch: "{{ org.gitops.branch }}"
       docker_url: "{{ network.docker.url }}"
-      charts_dir: "{{ item.gitops.chart_source }}"
-      values_dir: "{{playbook_dir}}/../../../{{item.gitops.release_dir}}/{{ item.name | lower }}"
+      charts_dir: "{{ org.gitops.chart_source }}"
+      values_dir: "{{playbook_dir}}/../../../{{org.gitops.release_dir}}/{{ org.name | lower }}"
     loop: "{{ network['organizations'] }}"
-    when: item.type == 'orderer'
+    loop_control:
+        loop_var: org
+    when: org.services.orderers is defined and org.services.orderers | length > 0
 
   # Generate script to modify the fetched configuration block
   - name: "Modify the system channel with endpoint information of new orderer"
@@ -112,10 +97,15 @@
     vars:
       build_path: "./build"
       channel_name: "{{ channel.channel_name | lower }}"
+      org_peer: "{{ channel.participants | first}}"
+      vault: "{{ org.vault }}"
+      existingOrdererAddress: "{{ org_peer.ordererAddress }}"
+      peer: "{{ org_peer.peers | first }}"
       docker_url: "{{ network.docker.url }}"
     loop: "{{ network.channels }}"
     loop_control:
       loop_var: channel
+    when: '2.5.' not in network.version
 
   vars: #These variables can be overriden from the command line
     add_new_org: 'false'                #Default for this playbook is false

platforms/hyperledger-fabric/configuration/roles/create/new_orderer/create_appchannel_block/tasks/nested_create_appchannel_block.yaml

@@ -38,14 +38,24 @@
     path: "{{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls"
     state: touch
 
+# Copy tls server.crt from vault to the build directory
+- name: Fetch the tls server.crt from vault
+  shell: |
+    vault kv get -field=server_crt {{ vault.secret_path | default('secretsv2') }}/{{ network.env.type }}{{ org.name }}/orderers/{{ orderer.name }}-tls > server.crt
+    mkdir -p {{ build_path }}/tls-cert/{{ orderer.name }}
+    mv server.crt {{ build_path }}/tls-cert/{{ orderer.name }}
+  environment:
+    VAULT_ADDR: "{{ vault.url }}"
+    VAULT_TOKEN: "{{ vault.root_token }}"
+
 # Create orderer-tls for tls certificate information
 - name: "adding tls certificate information"
   shell: |
     if [ {{ network.env.proxy }} == 'none' ]
     then
       echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\",\"host\":\"{{ orderer.name|lower }}.{{ component_ns }}\",\"port\":{{ orderer.grpc.port }},\"server_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\"}"  >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls 
     else
-      echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\",\"host\":\"{{ orderer.ordererAddress.split(":")[0] | to_json }}\",\"port\":\"{{ orderer.ordererAddress.split(":")[1] | to_json }}\",\"server_tls_cert\":\"$(cat {{ build_path }}/crypto-config/ordererOrganizations/{{ component_ns }}/orderers/{{ orderer.name|lower }}.{{ component_ns }}/tls/server.crt | base64 -w 0)\"}"  >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls 
+      echo -n "{\"client_tls_cert\":\"$(cat {{ build_path }}/tls-cert/{{ orderer.name }}/server.crt | base64 -w 0))\",\"host\":\"{{ orderer.ordererAddress.split(":")[0] | to_json }}\",\"port\":{{ orderer.ordererAddress.split(":")[1] | to_json }},\"server_tls_cert\":\"$(cat {{ build_path }}/tls-cert/{{ orderer.name }}/server.crt | base64 -w 0))\"}"  >> {{ build_path }}/channel-artifacts/{{ channel_name | lower}}-orderer-tls 
     fi
 
 # Create orderer file for adding new endpoint information
@@ -71,6 +81,8 @@
     dest: "{{ build_path }}/update-channel-script.sh"
   vars:
     component_name: "{{ org.name | lower }}"
+    ordererAddress: "{{ orderer.ordererAddress }}"
+    ordererOrg: "{{ org.name | lower }}MSP"
     os: "{{ fabric.os }}"
     arch: "{{ fabric.arch }}"
     version: "{{ network.version }}"