How to import complex YAML structure in etcd

[irfanjs]

We have below YAML structure which consists of multiple key / values pair . these are basically dictionary kind of structure used in kubernetes . Instead of keeping these configuration items in YAML file , we thought to use the key/value store database to maintain these config values.

after analysis , we laned into etcd . however, from the docs , it appears that , single key / value can be easily stored in etcd , but what about dictionary/array kind of structures where single key might have multiple values . please suggest

below is the YAML file

region:
    us-east-1:
        physicalclusters:
        -    abc:
                - lc1:
                    - tenant1
                    - tenant2
                - lc2:
                    - tenant3
                    - tenant4
        -    xyz:
                - lc3:
                    - tenant5
                    - tenant6
                - lc4:
                    - tenant7
                    - tenant8
    us-east-2:
       physicalclusters:
       -    test1:
                - testtenant1
       -    test2:
                - testtenant2

[jmhbnz]

Hey @irfanjs - Thanks for your question, below is a very simplistic example just using etcdctl command line client.

Example file

region:
  us-east-1:
    physicalclusters:
      - abc:
          - lc1:
              - tenant1
              - tenant2
          - lc2:
              - tenant3
              - tenant4
      - xyz:
          - lc3:
              - tenant5
              - tenant6
          - lc4:
              - tenant7
              - tenant8
  us-east-2:
    physicalclusters:
      - test1:
          - testtenant1
      - test2:
          - testtenant2

Create a new etcd key storing the yaml file contents

 ~  Documents  etcd  bin   main  ./etcdctl put example-yaml "$(cat example.yaml)"

OK

Retrieve the keys value

 ~  Documents  etcd  bin   main  ./etcdctl get example-yaml                              example-yaml
region:
  us-east-1:
    physicalclusters:
      - abc:
          - lc1:
              - tenant1
              - tenant2
          - lc2:
              - tenant3
              - tenant4
      - xyz:
          - lc3:
              - tenant5
              - tenant6
          - lc4:
              - tenant7
              - tenant8
  us-east-2:
    physicalclusters:
      - test1:
          - testtenant1
      - test2:
          - testtenant2

As you can see the entire yaml document/structure can be successfully stored and retrieved from an etcd key.

You can of course do something similar via the etcd clientv3 libraries. You may want to refer to other implementations for example kubernetes: https://github.com/kubernetes/kubernetes/blob/6e0cb243d57592c917fe449dde20b0e246bc66be/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L175

[irfanjs]

OK thanks . @jmhbnz
How to get the value of specific key ?
for example : how to retrieve the value , "tenant8" for key lc4 in xyz physical cluster. Thanks

[jmhbnz]

etcd does not have any complex value query features, it is purely a key value store. So you will need to think very carefully about how you structure your keys (they can be nested) versus just storing fully raw documents as values.

If we stick with the example above, and in reference to your question, an alternative structure might be something like

 ~  Documents  etcd  bin   main  ./etcdctl put region/us-east-1/physicalclusters/xyz/lc4/tenant7 hello-tenant-7-details
OK

 ~  Documents  etcd  bin   main  ./etcdctl put region/us-east-1/physicalclusters/xyz/lc4/tenant8 hello-tenant-8-details
OK

We could then retrieve values specific to lc4 or tenant8 as follows using --prefix (refer https://etcd.io/docs/v3.5/tutorials/how-to-get-key-by-prefix). Obviously these values could me more detailed if you passed a more interesting payload when creating the key for each tenant.

 ~  Documents  etcd  bin   main  ./etcdctl get --prefix region/us-east-1/physicalclusters/xyz/lc4
region/us-east-1/physicalclusters/xyz/lc4/tenant7
hello-tenant-7-details
region/us-east-1/physicalclusters/xyz/lc4/tenant8
hello-tenant-8-details

 ~  Documents  etcd  bin   main  ./etcdctl get --prefix region/us-east-1/physicalclusters/xyz/lc4/tenant8
region/us-east-1/physicalclusters/xyz/lc4/tenant8
hello-tenant-8-details

[irfanjs]

Thanks @jmhbnz
Im trying these options . will update you asap

[irfanjs]

@jmhbnz
Hope you are doing well .
Does etcd supports list of values for specific Key ?

[jmhbnz]

Hi @irfanjs - Thanks for your follow-up question. I recommend installing etcd and exploring the capabilities of etcd via etcdctl.

[irfanjs]

Thanks. I did some search and found this one .
https://groups.google.com/g/salt-users/c/bHEuIaiiRTM
The post from Steve Hajducko mentions that ETCD does not support multiple values for specific key. Does this hold true in latest version as well. ? If not, I will setup my keys accordingly. Thanks