Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky behaviour with SKAFFOLD_TIMESTAMPS="true" #7947

Open
pre opened this issue Oct 17, 2022 · 1 comment
Open

Flaky behaviour with SKAFFOLD_TIMESTAMPS="true" #7947

pre opened this issue Oct 17, 2022 · 1 comment
Labels
area/logging kind/bug Something isn't working priority/p2 May take a couple of releases

Comments

@pre
Copy link

pre commented Oct 17, 2022

Expected behavior

Set export SKAFFOLD_TIMESTAMPS="true"

See timestamps and everything working as expected.

Actual behavior

See timestamps and see Skaffold crashing with:

SKAFFOLD_TIMESTAMPS="true" # "false" no problem


2022-10-17 17:00:51 Build [registry.redacted-runtime.redacted.com/redactedname/redacted-runner] succeeded
2022-10-17 17:00:51 Tags used in deployment:
2022-10-17 17:00:51  - registry.redacted-runtime.redacted.com/redactedname/redacted-runner -> 2022-10-17 17:00:51 registry.redacted-runtime.redacted.com/redactedname/redacted-runner:latest@sha256:5953caadff4ca5c639a14ce5c585ac43167ddfdb07e33c011c7101086c836eb9
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:031870b73b70581edb210c7bbf9c326204644133d2c5b35298b489f4969cd79f
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:2b211c8be115b0e394b07cbfca652312d36f20b51203aca241dfbed2521a77ee
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:48ac621202b207ef4618be4d934d6b5bf93f62a3db2572f32d81c91614887dc2
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:4ff700a760ddad183abd063aa0ec9585f80fb00193d4dd3bf4ed523ad990be51
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:083c33ffcd6c15bf5d215039c281dc486d11ce24e4d9137661bf4d50b11f741f
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:1d265b9a2564335d616aa1a1a2ebc150160f3ca7b9aa27f87008e17c42829b86
2022-10-17 17:00:51  - redactedname-redacted -> 2022-10-17 17:00:51 redactedname-redacted:e64526b88b0c1630c36e338cb544a9515fb2f704902ff20597a17587e232e7dc
2022-10-17 17:00:51 Starting deploy...
2022-10-17 17:00:52  - 2022-10-17 17:00:52 namespace/redacted-namespace-name unchanged
2022-10-17 17:00:52  - 2022-10-17 17:00:52 d-functions-runner-tm572tf94d unchanged
2022-10-17 17:00:52  - 2022-10-17 17:00:52 eleases-auth configured
panic: runtime error: slice bounds out of range [161:141]

goroutine 341 [running]:
github.com/segmentio/textio.(*PrefixWriter).discard(...)
	/Users/kbuilder/go/pkg/mod/github.com/segmentio/[email protected]/prefix.go:125
github.com/segmentio/textio.(*PrefixWriter).Write.func1({0x14000f18000, 0x2c, 0x8000})
	/Users/kbuilder/go/pkg/mod/github.com/segmentio/[email protected]/prefix.go:55 +0x280
github.com/segmentio/textio.forEachLine({0x14000f18000, 0x2c, 0x8000}, 0x14000ccfe40)
	/Users/kbuilder/go/pkg/mod/github.com/segmentio/[email protected]/prefix.go:148 +0xb8
github.com/segmentio/textio.(*PrefixWriter).Write(0x140006d41e0, {0x14000f18000, 0x2c, 0x8000})
	/Users/kbuilder/go/pkg/mod/github.com/segmentio/[email protected]/prefix.go:47 +0x84
io.copyBuffer({0x1029689c0, 0x140006d41e0}, {0x10296e580, 0x14000a0a208}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.17/libexec/src/io/io.go:425 +0x224
io.Copy(...)
	/usr/local/Cellar/go/1.17/libexec/src/io/io.go:382
os/exec.(*Cmd).writerDescriptor.func1()
	/usr/local/Cellar/go/1.17/libexec/src/os/exec/exec.go:311 +0x54
os/exec.(*Cmd).Start.func1(0x14000b9e160, 0x14000c241e0)
	/usr/local/Cellar/go/1.17/libexec/src/os/exec/exec.go:441 +0x28
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.17/libexec/src/os/exec/exec.go:440 +0x794

There's a double timestamp in here:

2022-10-17 17:00:51 Starting deploy...
2022-10-17 17:00:52  - 2022-10-17 17:00:52 namespace/redacted-namespace-name unchanged
2022-10-17 17:00:52  - 2022-10-17 17:00:52 d-functions-runner-tm572tf94d unchanged
2022-10-17 17:00:52  - 2022-10-17 17:00:52 eleases-auth configured
panic: runtime error: slice bounds out of range [161:141]

Here

  • eleases-auth should be - secret/artifactory-releases-auth configured
  • d-functions-runner-tm572tf94d unchanged should be - configmap/image-redacted-functions-runner-tm572tf94d unchanged
  • namespace/redacted-namespace-name unchanged should be - namespace/redacted-namespace-name unchanged

Crash happens everytime with SKAFFOLD_TIMESTAMPS="true" but does not happen without SKAFFOLD_TIMESTAMPS or with SKAFFOLD_TIMESTAMPS="false"

Information

  • Skaffold version: v1.39.1
  • Operating system: MacOS
  • Installed via: skaffold.dev
  • Contents of skaffold.yaml:
can provide, but is our company's internal 300 line configuration with multiple modules so can't really provide a redacted version easily.
@gsquared94 gsquared94 added kind/bug Something isn't working priority/p2 May take a couple of releases area/logging labels Oct 24, 2022
@MartinLoeper
Copy link

I do not know if this is related but my skaffold dev crashes terribly at the end of the helm install phase when timestamps are enabled with something like:

2025-01-07 16:38:49 WARN[2025-01-07T16:38:49Z] Failed to deploy :deploying "core": install: signal: broken pipe, please fix the error and press any key to continue. subtask=-1 task=DevLoop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/logging kind/bug Something isn't working priority/p2 May take a couple of releases
Projects
None yet
Development

No branches or pull requests

3 participants