Unable to verify the first certificate when install vscode extension

[huonguyenlt]

Describe the bug

After setup the che cluster to use the embedded Open VSX registry instance in the plugin-registry, I can see the list of available extensions. However, when trying to install them, I get the error

2024-09-09 10:32:58.654 [warning] [eclipse-che.port]: View container 'endpoints' does not exist and all views registered to it will be added to 'Explorer'.
2024-09-09 10:32:59.570 [info] Updating additional builtin extensions cache
2024-09-09 10:33:01.067 [error] [Extension Host] Unable to create telemetry client: "DEVWORKSPACE_TELEMETRY_BACKEND_PORT" is not set.
2024-09-09 10:33:23.206 [error] Error: unable to verify the first certificate

Che version

7.89

Steps to reproduce

1. config checluster to use the embedded open vsx

components:
    pluginRegistry:
      openVSXURL: ''

2. avaible extenstions shows
   

3. click the install button to install any extension, then got the certificate error log
   

Expected behavior

Should be install successfully

Runtime

other (please specify in additional context)

Screenshots

Installation method

other (please specify in additional context)

Environment

Amazon

Eclipse Che Logs

2024-09-09 10:32:56.914 [info] Resolving connection token (che.stengg-devcheworkspaces.com)...
2024-09-09 10:32:56.916 [info] Resolved connection token (che.stengg-devcheworkspaces.com) after 3 ms
2024-09-09 10:32:56.918 [info] Creating a socket (renderer-Management-ea220a7f-39b8-41e6-87aa-c685dd1dabd9)...
2024-09-09 10:32:56.939 [warning] Via 'product.json#extensionEnabledApiProposals' extension 'genuitecllc.codetogether' wants API proposal 'terminalNameChangeEvent' but that proposal DOES NOT EXIST. Likely, the proposal has been finalized (check 'vscode.d.ts') or was abandoned.
2024-09-09 10:32:57.062 [info] Creating a socket (renderer-ExtensionHost-61f7fbe7-93c0-45b1-bc39-a9ba5c4e0a5d)...
2024-09-09 10:32:57.184 [info] Creating a socket (renderer-Management-ea220a7f-39b8-41e6-87aa-c685dd1dabd9) was successful after 266 ms.
2024-09-09 10:32:57.399 [info] Creating a socket (renderer-ExtensionHost-61f7fbe7-93c0-45b1-bc39-a9ba5c4e0a5d) was successful after 337 ms.
2024-09-09 10:32:58.654 [warning] [eclipse-che.port]: View container 'endpoints' does not exist and all views registered to it will be added to 'Explorer'.
2024-09-09 10:32:59.570 [info] Updating additional builtin extensions cache
2024-09-09 10:33:01.067 [error] [Extension Host] Unable to create telemetry client: "DEVWORKSPACE_TELEMETRY_BACKEND_PORT" is not set.
2024-09-09 10:33:23.206 [error] Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
    at TLSSocket.emit (node:events:518:28)
    at TLSSocket._finishInit (node:_tls_wrap:1085:8)
    at ssl.onhandshakedone (node:_tls_wrap:871:12)
2024-09-09 10:33:23.222 [error] unable to verify the first certificate: Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
    at TLSSocket.emit (node:events:518:28)
    at TLSSocket._finishInit (node:_tls_wrap:1085:8)
    at ssl.onhandshakedone (node:_tls_wrap:871:12)

Additional context

runtime: eks
installation method: helm (che-operator)

[huonguyenlt]

When trying to accss the plugin registry url ( https://che.stengg-devcheworkspaces.com/plugin-registry/v3), the browser redirect me to https://che.stengg-devcheworkspaces.com/v3/ and the page is not working. Is it expected behavior?

[tolusha]

That's a bug. I need to investigate

[tolusha]

@huonguyenlt

Error: unable to verify the first certificate

Could you try in the terminal of a user workspace:

curl --cacert /tmp/che/secret/ca.crt https://che.stengg-devcheworkspaces.com
openssl verify -verbose -CAfile /tmp/che/secret/ca.crt /tmp/che/secret/ca.crt

[tolusha]

When trying to access the plugin registry url

Dashboard doesn't use anymore pluginregistry to fetch editors definitions.
That's some redundant that we need to cleanup.
In general it is expected

[huonguyenlt]

@huonguyenlt

Error: unable to verify the first certificate

Could you try in the terminal of a user workspace:

curl --cacert /tmp/che/secret/ca.crt https://che.stengg-devcheworkspaces.com openssl verify -verbose -CAfile /tmp/che/secret/ca.crt /tmp/che/secret/ca.crt

@tolusha the cert you mentioned does not exist. Here is the log from my empty workspace

projects $ curl --cacert /tmp/che/secret/ca.crt  https://che.stengg-devcheworkspaces.com
curl: (77) error setting certificate verify locations:
  CAfile: /tmp/che/secret/ca.crt
  CApath: none
 
projects $ ls /tmp/che/secret/ca.crt
ls: cannot access '/tmp/che/secret/ca.crt': No such file or directory

projects $ ls /tmp/
composer-installer.php                                node-extra-certificates/                              vscode-git-86ded4a1df.sock
containers-user-1234/                                 podman-run-1234/                                      vscode-ipc-73c8eb01-9350-43f2-a8ec-b8fe198b3c3f.sock
ks-script-eio8pz76                                    poststart-stderr.txt                                  vscode-ipc-b6454629-be1a-41b7-9f39-43ddd5a66847.sock
ks-script-ep7jq71d                                    poststart-stdout.txt

I found instead this cert /public-certs/kube-root-ca.crt.ca.crt in the mount volume of the workspace pod, any chance that the cert you asked for?

 Mounts:
      /.git-credentials/ from devworkspace-merged-git-credentials (ro)
      /checode from claim-devworkspace (rw,path="workspace296409eef9fb4a92/checode")
      /config/user/profile from user-profile (ro)
      /devworkspace-metadata from workspace-metadata (ro)
      /etc/gitconfig from devworkspace-gitconfig (ro,path="gitconfig")
      /etc/ssh/dwo_ssh_key from git-ssh-key (ro,path="dwo_ssh_key")
      /etc/ssh/dwo_ssh_key.pub from git-ssh-key (ro,path="dwo_ssh_key.pub")
      /etc/ssh/ssh_config from git-ssh-key (ro,path="ssh_config")
      /projects from claim-devworkspace (rw,path="workspace296409eef9fb4a92/projects")
      /public-certs from che-trusted-ca-certs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xgxlc (ro)

projects $ curl --cacert /public-certs/kube-root-ca.crt.ca.crt  https://che.stengg-devcheworkspaces.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

Not sure if this info relevant but I use the self signed cert for the che domain (https://eclipse.dev/che/docs/stable/administration-guide/configuring-che-with-self-signed-certificate/). So I the workspace should use this cert rather than some temp cert, right?
I named the secret che.tls3 and provide to checluster as below. Is there anywhere else I have to add this cert?

spec:
  networking:
    domain: che.stengg-devcheworkspaces.com
    tlsSecretName: che.tls3

[tolusha]

So I the workspace should use this cert rather than some temp cert, right?

Yes, but for some reason the certificate is not propagated into workspace.
Let's do it manually:

HOST=che.stengg-devcheworkspaces.com
NAMESPACE=$(kubectl get checluster -A -o "jsonpath={.items[0].metadata.namespace}")
CERTS=$(openssl s_client -showcerts -connect $HOST:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p')

kubectl create configmap custom-certificate --from-literal registry.crt="${CERTS}" -n $NAMESPACE
kubectl label configmap custom-certificate app.kubernetes.io/component=ca-bundle app.kubernetes.io/part-of=che.eclipse.org -n $NAMESPACE

[huonguyenlt]

@tolusha I created the config map as you suggested in the eclipse-che namespace. I see that The certificate is then added to the configmap che-trusted-ca-certs under user workspace namespace. And the configmap che-trusted-ca-certs is used in the workspace pod. So the pod should see the certificate. Still the error when installing extension persists.

List of cm and secert use by workspace pod

Volumes:
  devworkspace-merged-git-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  devworkspace-merged-git-credentials
    Optional:    false
  devworkspace-gitconfig:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      devworkspace-gitconfig
    Optional:  false
  che-trusted-ca-certs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      che-trusted-ca-certs
    Optional:  false
  git-ssh-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  git-ssh-key
    Optional:    false
  user-profile:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  user-profile
    Optional:    false
  claim-devworkspace:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  claim-devworkspace
    ReadOnly:   false
  workspace-metadata:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      workspace7e08c73f453d4f8b-metadata
    Optional:  true
  che-gateway:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      workspace7e08c73f453d4f8b-route
    Optional:  false
  kube-api-access-bp6nb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

che-trusted-ca-certs config map

apiVersion: v1
data:
  custom-certificate.registry.crt: |-
    -----BEGIN CERTIFICATE-----
    MIIGVTCCBT2gAwIBAgITHQAAM0gLriknEHbtxwAAAAAzSDANBgkqhkiG9w0BAQsF
    
    7E4VFyOJphicizPSryPFMqFrEcVvW8mlJuwWXUzTvGHDprkbic/hkqI=
    -----END CERTIFICATE-----
  kube-root-ca.crt.ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
    
    mhTlqHd5Jy0DSn7ARQjZzAElrpdpBeYlO29uDmLDVeRNwpDJrcO926CL0NYL71nc
    ATw=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  annotations:
    che.eclipse.org/included-configmaps: kube-root-ca.crt-247215612.custom-certificate-256823490
    controller.devfile.io/mount-as: file
    controller.devfile.io/mount-path: /public-certs
  creationTimestamp: "2024-08-26T13:33:53Z"
  labels:
    app.kubernetes.io/component: user-settings
    app.kubernetes.io/name: eclipse-che
    app.kubernetes.io/part-of: che.eclipse.org
    controller.devfile.io/mount-to-devworkspace: "true"
    controller.devfile.io/watch-configmap: "true"
  name: che-trusted-ca-certs
  namespace: lethienhuong-nguyen-stengg-com-che-0tv1zl
  resourceVersion: "256823528"
  uid: 0277541c-201a-4578-9c18-4f3fe1ce3eb4

custom-certificate config map

apiVersion: v1
data:
  registry.crt: |-
    -----BEGIN CERTIFICATE-----
    MIIGVTCCBT2gAwIBAgITHQAAM0gLriknEHbtxwAAAAAzSDANBgkqhkiG9w0BAQsF
    
    7E4VFyOJphicizPSryPFMqFrEcVvW8mlJuwWXUzTvGHDprkbic/hkqI=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  creationTimestamp: "2024-09-11T08:56:32Z"
  labels:
    app.kubernetes.io/component: ca-bundle
    app.kubernetes.io/part-of: che.eclipse.org
  name: custom-certificate
  namespace: eclipse-che
  resourceVersion: "256823490"
  uid: 4fc8976e-829f-41fc-ba37-ea7b3b95e16c

error message

2024-09-11 17:14:19.444 [warning] [eclipse-che.port]: View container 'endpoints' does not exist and all views registered to it will be added to 'Explorer'.
2024-09-11 17:14:19.969 [info] Updating additional builtin extensions cache
2024-09-11 17:14:21.350 [error] [Extension Host] Unable to create telemetry client: "DEVWORKSPACE_TELEMETRY_BACKEND_PORT" is not set.
2024-09-11 17:15:03.682 [error] Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
    at TLSSocket.emit (node:events:518:28)
    at TLSSocket._finishInit (node:_tls_wrap:1085:8)
    at ssl.onhandshakedone (node:_tls_wrap:871:12)
2024-09-11 17:15:03.683 [error] unable to verify the first certificate: Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1674:34)
    at TLSSocket.emit (node:events:518:28)
    at TLSSocket._finishInit (node:_tls_wrap:1085:8)
    at ssl.onhandshakedone (node:_tls_wrap:871:12)

[tolusha]

Could you try in the terminal of a user workspace:

curl --cacert /public-certs/custom-certificate.registry.crt https://che.stengg-devcheworkspaces.com

[tolusha]

@vitaliy-guliy
Do you happen to know the cause of the error?

[huonguyenlt]

@tolusha the problem fixed. It is because the root and intermediate certs is missing in the ca chain cert. Thank alot for your help

[tolusha]

Thank you for letting me know.
Could provide more details, how did you figure that out?
That would be helpful for others to detect the same problem.

[huonguyenlt]

@tolusha I tried 2 commands as below. I then checked again the ca chain cert and saw that the intermediate and root certificates are not included.
So I created the new ca chain certs with all the certs included, and the I created new config map with required labels. As soon as the cert was added to the workspace pod, I could install extension with no error
$ openssl verify -verbose -CAfile /public-certs/custom-certificate.registry.crt /public-certs/custom-certificate.registry.crt

error 20 at 0 depth lookup: unable to get local issuer certificate
error /public-certs/custom-certificate.registry.crt: verification failed

$ openssl s_client -connect che.stengg-devcheworkspaces.com:443 -showcerts

depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = [email protected]
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = [email protected]
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 C = SG, ST = SG, L = Singapore, O = Singapore Technologies Engineering Ltd, OU = Corporate, CN = che.stengg-devcheworkspaces.com, emailAddress = [email protected]
verify return:1
---
---
SSL handshake has read 2185 bytes and written 409 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---