[Bug] Setting bootstrap key on startup not working

[rilyw]

We would like to be able to set the bootstrap key, as it removes the step of decoding the generated key and adding it to some keystore/vault. It would be easier to set some key in a vault and start typesense with that key. But we cannot get the changes from #13 working.

apiVersion: v1
kind: Secret
metadata:
  labels:
    app.kubernetes.io/name: typesense-operator
    app.kubernetes.io/managed-by: kustomize
  name: typesense-common-bootstrap-key
  namespace: typesense
type: Opaque
data:
 typesense-api-key: eHl6Cg==

Naturally the key is just a placeholder and is just used for debugging, as the key is "xyz".

spec:
  adminApiKey:
    name: typesense-common-bootstrap-key

The sample is the same with a different key value.

apiVersion: v1
kind: Secret
metadata:
  labels:
    app.kubernetes.io/name: typesense-operator
    app.kubernetes.io/managed-by: kustomize
  name: typesense-common-bootstrap-key
type: Opaque
data:
  typesense-api-key: SXdpVG9CcnFYTHZYeTJNMG1TS1hPaGt0dlFUY3VWUloxc1M5REtsRUNtMFFwQU93R1hoanVIVWJLQnE2ejdlSQ==
---
apiVersion: ts.opentelekomcloud.com/v1alpha1
kind: TypesenseCluster
metadata:
  labels:
    app.kubernetes.io/name: typesense-operator
    app.kubernetes.io/managed-by: kustomize
  name: c-otc-1
spec:
  adminApiKey:
    name: typesense-common-bootstrap-key
.
.
.

We would assume this set the bootstrap key to our defined value, but when sending requests we are unauthorized with our key. The problem is that there is only one secret and that has our value, so we cannot get the actual value of the bootstrap key for our pods. We somehow overwrote the secret that was used.

$ kubectl get secret -n typesense 
NAME                             TYPE     DATA   AGE
typesense-common-bootstrap-key   Opaque   1      6m7s

I then changed the name of our secret from typesense-common-bootstrap-key to typesense-bootstrap-key and it was still the only secret, but now the containers could not start as they did not find typesense-common-bootstrap-key secret, even though the name value for the key was typesense-bootstrap-key.

spec:
  adminApiKey:
    name: typesense-bootstrap-key

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  41s               default-scheduler  Successfully assigned typesense/ts-server-sts-1 to cluster-...
  Normal   Pulled     9s (x4 over 26s)  kubelet            Container image "typesense/typesense:27.1" already present on machine
  Warning  Failed     9s (x4 over 26s)  kubelet            Error: secret "typesense-common-bootstrap-key" not found

Is there anything wrong with our configuration?

[akyriako]

the bootstrap key is bootstraping the cluster once in the beginning and for all - no way back or replacements, you cannot replace it at will (unless if you create another admin key via typesense itself, encode it to base64 and then replace it in the Secret , then recreate the StatefulSet Pods to take the new value in their envvars - but i guess that might be an "emergency" situation and has nothing to do with this operator) which sounds like your case:

We somehow overwrote the secret that was used.

In your second question, where you altered the name of the Secret, the error that you get is fully legit - no malfuction of the operator. The STS Pods were created by the operator to point to that Secret and they're still looking for the original name.

[rilyw]

The secret is in our helm chart in the same file as the TypesenseCluster. It gets deployed at the same time. We are not trying to change the bootstrap key after the pods are already deployed.

We somehow overwrote the secret that was used.

The situation I meant with this is as follows:

1. The TypesenseCluster gets installed via helm.
   1.1. The secret gets installed at the same time.
2. There is only one secret and it has our defined value
   2.1 The TYPESENSE_API_KEY variable in the pod has some generated value and not our value

My deduction of the situation was:

1. the operator creates a secret with the same name as we defined
2. The pods take this value as bootstrap key
3. The secret gets "overwritten" by our installed helm chart
4. There is only one secret and the key in the pods is different as the key in the only secret

Changing the name was a futile step and has nothing to do with the problem at hand. But my thinking was to see if the problem layed in the check of the operator that looks at the TypesenseCluster and if the spec.adminApiKey variable is set or not. If after the name change there were 2 secrets I would know that the operator has not processed the given api key and instead created a new one that was then used by the typesense pods. But if there is only 1 secret after the name change, that would mean there is nothing wrong with the processing but something else.

I then changed the name of our secret from typesense-common-bootstrap-key to typesense-bootstrap-key and it was still the only secret, but now the containers could not start as they did not find typesense-common-bootstrap-key secret, even though the name value for the key was typesense-bootstrap-key.

This was just, because I did not completely reinstall our helm chart, but just used upgrade after the name of the secret was changed in the chart. After reinstalling the chart the secret with the other name is found, yet the TYPESENSE_API_KEY variable in the pod is not the decoded value of the secret, but some other generated value. Naturally sending requests with our defined key will not work in this situation.

$ kubectl get secret -n typesense typesense-bootstrap-key -o yaml
apiVersion: v1
data:
  typesense-api-key: eHl6Cg==

env | grep TYPESENSE_
TYPESENSE_API_KEY=S0OagYMvis7oIe3vxa1ge+fD+lzOweqtvraFhwEogW029BGCQzaqOGwZx/UUdiQTuj63w1vm4IGQs8mEkj7C4gac7HRokTHW1NmgQH9m2RevXJ6jzJCHUuHpz5tJeqsYHPTU8DPZiOpCpfYKoSToufQJsDme4R4DiV86ptYZt8vtBxbzTxCOLM445bxR8Z+JjI8Ijh2G4Yok5sNH3ND9oM8p5vfuuA7aAfSp6aYhFEVSnicdegKjtV/2pAeu6mQtDHJEKuzdoLdkAYfzen2oAkztVvsgode+4/oL5La/44K31WSf5/mhS/NjlY6gPIp+7GPPh52HV+x2/KSmV9a4Lg==

After updating the crd, as it was the fix for problem #26, the situation is different but the problem still the same. Now the correct value is in the pod, but any request is still unauthorized.

$ env | grep TYPESENSE_
TYPESENSE_API_KEY=xyz

$ curl 'http://localhost:8108/keys' \
    -X GET \
    -H "X-TYPESENSE-API-KEY: xyz"
{"message": "Forbidden - a valid `x-typesense-api-key` header must be sent."}

Now I do not know where the differing bootstrap key is coming from or how the pod has a bootstrap key that is different from the TYPESENSE_API_KEY variable.

[akyriako]

Ok I found the bug; your deduction is somehow (under conditions) correct => The Secret was deployed after the Helm Chart (I don't imply you should follow a deployment order; the contrary) and the Operator falsely creates in this case a new Secret with the exact very name you provided in spec.adminApiKey! Definitely this shouldn't happen but it should "fail/wait" till the external Secret is present in the cluster. I will fix it, thanks for digging up the bug.