Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upstream connect error or disconnect/reset before headers. reset reason: connection termination #127

Open
kumy opened this issue Oct 23, 2024 · 9 comments
Open

upstream connect error or disconnect/reset before headers. reset reason: connection termination #127

kumy opened this issue Oct 23, 2024 · 9 comments

Comments

@kumy
Copy link

kumy commented Oct 23, 2024

Hi, we're trying to deploy the stack. We followed https://openunison.github.io/deployauth/#alternate-deployment-methods page.

We can successfully login on orchestra using GitHub and were presented with the landing page
image

When we try to access https://k8sou.xxx/k8stoken/ we get error

upstream connect error or disconnect/reset before headers. reset reason: connection termination

We didn't find any error in the logs from the pods in the openunison namespace when we hit/refresh that page.

However, we found this one on startup, could it be related 🤷 ?

openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra [2024-10-23 08:25:04,042][main] WARN  UrlHolder - Could not process url : ''
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra java.net.MalformedURLException: no protocol: 
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at java.base/java.net.URL.<init>(URL.java:772) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at java.base/java.net.URL.<init>(URL.java:654) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at java.base/java.net.URL.<init>(URL.java:590) ~[?:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.config.util.UrlHolder.<init>(UrlHolder.java:125) [unison-sdk-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.config.util.UnisonConfigManagerImpl.addAppInternal(UnisonConfigManagerImpl.java:882) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.config.util.UnisonConfigManagerImpl.addApplication(UnisonConfigManagerImpl.java:799) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.proxy.dynamicconfiguration.LoadApplicationsFromK8s.addObject(LoadApplicationsFromK8s.java:476) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.k8s.watch.K8sWatcher.initalRun(K8sWatcher.java:161) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.proxy.dynamicconfiguration.LoadApplicationsFromK8s.loadDynamicApplications(LoadApplicationsFromK8s.java:447) [unison-applications-k8s-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.config.util.UnisonConfigManagerImpl.initialize(UnisonConfigManagerImpl.java:587) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.filter.UnisonServletFilter.init(UnisonServletFilter.java:369) [unison-server-core-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.openunison.OpenUnisonServletFilter.init(OpenUnisonServletFilter.java:118) [open-unison-classes-1.0.41.jar:?]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:111) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.ManagedFilter.createFilter(ManagedFilter.java:86) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:598) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:605) [undertow-servlet-2.3.15.Final.jar:2.3.15.Final]
openunison-orchestra-6c68b44f9d-bdvvl openunison-orchestra 	at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.main(OpenUnisonOnUndertow.java:357) [openunison-on-undertow-1.0.41.jar:?]

Any hint on what we should check, or how to enable more debug logs?
(We're using Istio as Ingress controller.)
Thanks for your help 🙏

Startup logs:
openunison-orchestra.txt
@kumy
Copy link
Author

kumy commented Oct 23, 2024

BTW the oulogin plugin works fine to configure our kube config

# ~
$ kubectl oulogin --host=k8sou.ou.xxxxxx
oulogin 0.0.7
Checking for existing issuer https://k8sou.ou.xxxxxx/auth/idp/k8sIdp
Invalid context or does not exist, launching browser to login
Starting OpenID Connect for host k8sou.ou.xxxxxx
kubectl configuration created


# ~
$ k get po
NAME                                            READY   STATUS      RESTARTS   AGE
kube-oidc-proxy-orchestra-6989c8ffbb-wlglv      2/2     Running     0          39m
openunison-operator-6f5d4b7cc6-dn9w7            2/2     Running     0          45m
openunison-orchestra-b459ffb77-tb7vz            2/2     Running     0          39m
ouhtml-orchestra-login-portal-5fd446c89-l2k74   2/2     Running     0          36m
test-orchestra-orchestra                        0/1     Completed   0          39m

@mlbiam
Copy link
Contributor

mlbiam commented Oct 23, 2024

Does the dashboard work?

@kumy
Copy link
Author

kumy commented Oct 23, 2024

We didn't deployed the kube dashboard if it's what you mean

@mlbiam
Copy link
Contributor

mlbiam commented Oct 23, 2024

We didn't deployed the kube dashboard if it's what you mean

Understood, what happens if you go to https://k8s.ou.XXX/k8stoken/token/user, do you get a bunch of JSON? Also, do you have network policies enabled in your values.yaml?

@kumy
Copy link
Author

kumy commented Oct 23, 2024

Yes we see json, and no network policies enabled

network_policies:
  enabled: false
  ingress:
    enabled: true
    labels: []
  monitoring:
    enabled: true
    labels: []
  apiserver:
    enabled: true
    labels: []
  namespace_label: kubernetes.io/metadata.name

image

@mlbiam
Copy link
Contributor

mlbiam commented Oct 24, 2024

That's so odd. The original error comes usually from the virtual gateway not being able to connect or the certificate isn't setup. but you're getting to the OpenUnison service and pod. The "slow" part, generating tokens, works. It appears the issue is just the conection from OpenUnison --> html pod, but you're able to login to the openunison portal. is it possible that the service mesh is blocking the URL?

@mlbiam
Copy link
Contributor

mlbiam commented Oct 24, 2024

oh, i just saw your screen shot, that's REALLY old. What version of OpenUnison are you deploying (charts, images, etc)

@kumy
Copy link
Author

kumy commented Oct 24, 2024

We're using helm as:

  • helm template openunison tremolo/openunison-operator --namespace openunison | tee openunison-base.yaml
  • helm template orchestra tremolo/orchestra --namespace openunison -f values.yaml | tee orchestra-base.yaml
  • helm template orchestra-login-portal tremolo/orchestra-login-portal --namespace openunison -f values.yaml | tee orchestra-login-portal-base.yaml

from

That deploys images:

$ k get po -o yaml|grep image:|sort -u
      image: ghcr.io/openunison/openunison-k8s:1.0.41
      image: ghcr.io/openunison/openunison-k8s-html:1.0.0
      image: ghcr.io/openunison/openunison-kubernetes-operator:1.0.6
      image: ghcr.io/tremolosecurity/kube-oidc-proxy:1.0.7
      image: ghcr.io/tremolosecurity/python3:1.0.0
      image: upm-istio/proxyv2:1.19.7-distroless

is it possible that the service mesh is blocking the URL?

we were suspecting our base istio rules to block something, but we didn't have any to what/where to start our investigation

@mlbiam
Copy link
Contributor

mlbiam commented Oct 24, 2024

ghcr.io/openunison/openunison-k8s-html:1.0.0

odd, where did that come from? Try setting openunison.html.image to ghcr.io/openunison/openunison-k8s-react:1.0.0 in your values.yaml. also, make sure that openunison.html.legacy is false.

we were suspecting our base istio rules to block something, but we didn't have any to what/where to start our investigation

Do you have Kiali deployed? That's usually a good place. What version of istio?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kumy @mlbiam