Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pod Crashloopbackoff #114

Open
ariretiarno opened this issue Aug 8, 2024 · 7 comments
Open

Pod Crashloopbackoff #114

ariretiarno opened this issue Aug 8, 2024 · 7 comments

Comments

@ariretiarno
Copy link

ERROR

E0808 03:54:09.855425       1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(0x1f663c0), concrete:(*abi.Type)(nil), asserted:(*abi.Type)(0x228c220), missingMethod:""} (interface conversion: interface {} is nil, not *v1.Pod)
goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1ff20e0, 0xc00a219d70})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc009d10a20?})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x1ff20e0?, 0xc00a219d70?})
	/opt/hostedtoolcache/go/1.22.1/x64/src/runtime/panic.go:770 +0x132
github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc003128340?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
	/home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3d4
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc000552730)
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(...)
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001e441f0, {0x2602100, 0xc0044a46c0}, 0x1, 0xc0004d62a0)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001e441f0, 0x3b9aca00, 0x0, 0x1, 0xc0004d62a0)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run in goroutine 136
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x375
panic: interface conversion: interface {} is nil, not *v1.Pod [recovered]
	panic: interface conversion: interface {} is nil, not *v1.Pod

goroutine 226 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc009d10a20?})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x1ff20e0?, 0xc00a219d70?})
	/opt/hostedtoolcache/go/1.22.1/x64/src/runtime/panic.go:770 +0x132
github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc003128340?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
	/home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3d4
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc000552730)
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(...)
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001e441f0, {0x2602100, 0xc0044a46c0}, 0x1, 0xc0004d62a0)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001e441f0, 0x3b9aca00, 0x0, 0x1, 0xc0004d62a0)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run in goroutine 136
	/home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x375
@laszlocph
Copy link
Contributor

Can you retest with the latest version?

@an00pl
Copy link

an00pl commented Oct 24, 2024 •
edited
Loading

We still have the error on v0.4.8
Looks like error is different?


panic: interface conversion: interface {} is nil, not *v1.Pod [recovered]

        panic: interface conversion: interface {} is nil, not *v1.Pod

 

goroutine 234 [running]:

[k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2a1a968](http://k8s.io/apimachinery/pkg/util/runtime.handleCrash(%7B0x2a1a968), 0x4049440}, {0x2327b40, 0xc00b72b800}, {0x4049440, 0x0, 0x10000c004f6f6e0?})

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:89](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:89) +0xee

[k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0](http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash(%7B0x0), 0x0, 0xc004f6f6e0?})

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59) +0x108

panic({0x2327b40?, 0xc00b72b800?})

        /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/panic.go:785 +0x132

[github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc00b725300](http://github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1(%7B%7B0xc00b725300)?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)

        /home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3b4

[github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc0003e5a40)](http://github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc0003e5a40))

        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465

[github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(..](http://github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(..).)

        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243

[k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30](http://k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30)?)

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226) +0x33

[k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002bbb340](http://k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002bbb340), {0x29f96e0, 0xc005416000}, 0x1, 0xc0000b85b0)

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227) +0xaf

[k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bbb340](http://k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bbb340), 0x3b9aca00, 0x0, 0x1, 0xc0000b85b0)

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204) +0x7f

[k8s.io/apimachinery/pkg/util/wait.Until(..](http://k8s.io/apimachinery/pkg/util/wait.Until(..).)

        [/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161](mailto:/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161)

created by [github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run](http://github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run) in goroutine 97

        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x2d2

@elvishsu-kr
Copy link

I have the same error - unfortunately with v0.4.8

E1111 11:00:04.612463       1 iface.go:275] "Observed a panic" panic="interface conversion: interface {} is nil, not *v1.Pod" panicGoValue="&runtime.TypeAsserti
    goroutine 135 [running]:
    k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2a1a968, 0x4049440}, {0x2327b40, 0xc00166ce70})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0xbc
    k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2a1a968, 0x4049440}, {0x2327b40, 0xc00166ce70}, {0x4049440, 0x0, 0x10000c000157000?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:82 +0x5e
    k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000157000?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59 +0x108
    panic({0x2327b40?, 0xc00166ce70?})
        /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/panic.go:785 +0x132
    github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc001985f80?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
        /home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3b4
    github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc0001beeb0)
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465
    github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(...)
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243
    k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
    k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000cb19c0, {0x29f96e0, 0xc000ba0de0}, 0x1, 0xc000112150)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
    k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000cb19c0, 0x3b9aca00, 0x0, 0x1, 0xc000112150)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
    k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
    created by github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run in goroutine 157
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x2d2
 >
panic: interface conversion: interface {} is nil, not *v1.Pod [recovered]
    panic: interface conversion: interface {} is nil, not *v1.Pod

@Tiana125
Copy link

Same here at v0.4.8, it appears the issue is at line 40 of capacitor/pkg/controllers/podController.go.

pod, ok := obj.(*v1.Pod)

@mohan-alpheya
Copy link

Same here at v0.4.8. Any work around for the issue?

E1218 13:20:05.525913       1 iface.go:275] "Observed a panic" panic="interface conversion: interface {} is nil, not *v1.Pod" panicGoValue="&runtime.TypeAssertionError{_interface:(*abi.Type)(0x2290bc0), concrete:(*abi.Type)(nil), asserted:(*abi.Type)(0x26536e0), missingMethod:\"\"}" stacktrace=<
        goroutine 279 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2a1a968, 0x4049440}, {0x2327b40, 0xc007a2a870})
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0xbc
        k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2a1a968, 0x4049440}, {0x2327b40, 0xc007a2a870}, {0x4049440, 0x0, 0x10000c0008e4a80?})
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:82 +0x5e
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0008e4a80?})
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59 +0x108
        panic({0x2327b40?, 0xc007a2a870?})
                /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/panic.go:785 +0x132
        github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc007a05600?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
                /home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3b4
        github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc00062c500)
                /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465
        github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(...)
                /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003eef7c0, {0x29f96e0, 0xc003dc9140}, 0x1, 0xc000248070)
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003eef7c0, 0x3b9aca00, 0x0, 0x1, 0xc000248070)
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
        k8s.io/apimachinery/pkg/util/wait.Until(...)
                /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
        created by github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run in goroutine 145
                /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x2d2
 >
panic: interface conversion: interface {} is nil, not *v1.Pod [recovered]
        panic: interface conversion: interface {} is nil, not *v1.Pod

goroutine 279 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2a1a968, 0x4049440}, {0x2327b40, 0xc007a2a870}, {0x4049440, 0x0, 0x10000c0008e4a80?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0008e4a80?})
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59 +0x108
panic({0x2327b40?, 0xc007a2a870?})
        /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/panic.go:785 +0x132
github.com/gimlet-io/capacitor/pkg/controllers.PodController.func1({{0xc007a05600?, _}, {_, _}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
        /home/runner/work/capacitor/capacitor/pkg/controllers/podController.go:40 +0x3b4
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).processNextItem(0xc00062c500)
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:185 +0x465
github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).runWorker(...)
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:243
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003eef7c0, {0x29f96e0, 0xc003dc9140}, 0x1, 0xc000248070)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003eef7c0, 0x3b9aca00, 0x0, 0x1, 0xc000248070)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/gimlet-io/capacitor/pkg/controllers.(*Controller).Run in goroutine 145
        /home/runner/work/capacitor/capacitor/pkg/controllers/controller.go:235 +0x2d2

@Tiana125
Copy link

I have tried to disable readinessProbe but it didn't help, still restarting

@mohan-alpheya
Copy link

Any finding or fix so far? We have disabled it in our environment and waiting for fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@laszlocph @ariretiarno @Tiana125 @mohan-alpheya @elvishsu-kr @an00pl