kube-context / kubeconfig not applied

[MarcoSchaar]

Expected behavior

commands should be equal or at least one should be stable

skaffold run --profile <profile> --kube-context='clustername' --kubeconfig='/<path>/.kube/_config'
skaffold run --profile <profile> --kube-context clustername --kubeconfig /<path>/.kube/_config
skaffold run --profile <profile> --kube-context 'clustername' --kubeconfig '/<path>/.kube/_config'
skaffold run --profile <profile> --kube-context "clustername" --kubeconfig "/<path>/.kube/_config"

Actual behavior

kube-context is rarely taken and deployment is not always working as expected

Information

• Skaffold version: v1.11.0
• Operating system: debian-stable
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta5
kind: Config
metadata:
  name: clustername

profiles:
  - name: <profiles>
    deploy:
      kubeContext: clustername
      kubectl:
        manifests:
        - kubernetes-manifests/namespace.yaml
      helm:
        releases:
        - name: releases-name
          chartPath: https://github.<path>.tar.gz
          namespace: name-space
          valuesFiles: 
          - helm-values/values.yaml
          remote: true

Steps to reproduce the behavior

1. rename .kube/config file
2. skaffold run --profile <profile> --kube-context <clustername> --kubeconfig <path to renamed kube config>

debug log output

level=info msg="Skaffold &{Version:v1.11.0 ConfigVersion:skaffold/v2beta5 GitVersion: GitCommit:b1346ef1caded079c5abf11e5c0daae2322c9c6b GitTreeState:clean BuildDate:2020-06-11T20:54:36Z GoVersion:go1.14.2 Compiler:gc Platform:linux/amd64}"
level=info msg="applying profile: <profile>"
level=debug msg="overlaying profile on config for field Build"
level=debug msg="overlaying profile on config for field artifacts"
level=debug msg="overlaying profile on config for field insecureRegistries"
level=debug msg="overlaying profile on config for field tagPolicy"
level=info msg="no values found in profile for field TagPolicy, using original config values"
level=debug msg="overlaying profile on config for field BuildType"
level=info msg="no values found in profile for field BuildType, using original config values"
level=debug msg="overlaying profile on config for field Test"
level=debug msg="overlaying profile on config for field Deploy"
level=debug msg="overlaying profile on config for field DeployType"
level=debug msg="overlaying profile on config for field helm"
level=debug msg="overlaying profile on config for field kubectl"
level=debug msg="overlaying profile on config for field kustomize"
level=debug msg="overlaying profile on config for field statusCheckDeadlineSeconds"
level=debug msg="overlaying profile on config for field kubeContext"
level=debug msg="overlaying profile on config for field PortForward"
level=info msg="Activated kube-context \"clustername\""
level=debug msg="Defaulting build type to local build"
level=debug msg="validating yamltags of struct SkaffoldConfig"
level=debug msg="validating yamltags of struct Metadata"
level=debug msg="validating yamltags of struct Pipeline"
level=debug msg="validating yamltags of struct BuildConfig"
level=debug msg="validating yamltags of struct TagPolicy"
level=debug msg="validating yamltags of struct GitTagger"
level=debug msg="validating yamltags of struct BuildType"
level=debug msg="validating yamltags of struct LocalBuild"
level=debug msg="validating yamltags of struct DeployConfig"
level=debug msg="validating yamltags of struct DeployType"
level=debug msg="validating yamltags of struct HelmDeploy"
level=debug msg="validating yamltags of struct HelmRelease"
level=debug msg="validating yamltags of struct HelmOverrides"
level=debug msg="validating yamltags of struct HelmImageStrategy"
level=debug msg="validating yamltags of struct HelmImageConfig"
level=debug msg="validating yamltags of struct HelmDeployFlags"
level=debug msg="validating yamltags of struct KubectlDeploy"
level=debug msg="validating yamltags of struct KubectlFlags"
level=info msg="Using kubectl context: clustername"
level=debug msg="Using builder: local"
level=debug msg="setting Docker user agent to skaffold-v1.11.0"
level=debug msg="push value not present, defaulting to true because localCluster is false"
level=info msg="Tags generated in 6.339µs"
level=info msg="Cache check complete in 19.783µs"
level=debug msg="getting client config for kubeContext: `clustername`"
unable to connect to Kubernetes: getting client config for Kubernetes client: error creating REST client config for kubeContext "clustername": context "clustername" does not exist
Generating tags...
Checking cache...
Tags used in deployment:

[tejal29]

@MarcoSchaar i was able to reproduce your issue on master

skaffold run --kube-context minikube --kubeconfig ~/.kube/config1 -d gcr.io/tejal-test --tail
Generating tags...
 - skaffold-example -> gcr.io/tejal-test/skaffold-example:v1.11.0-58-g6b57f7198
Checking cache...
 - skaffold-example: Found Remotely
Tags used in deployment:
 - skaffold-example -> gcr.io/tejal-test/skaffold-example:v1.11.0-58-g6b57f7198@sha256:d3f2911879fe442816886da8fc8d496da1a867b93cc64afb6a2ddee3fe8e2a20
unable to connect to Kubernetes: getting client config for Kubernetes client: error creating REST client config for kubeContext "minikube": context "minikube" does not exist

I did some debugging and looks like this line returns a empty config in spite of path being set

skaffold/pkg/skaffold/kubernetes/context/context.go

Line 105 in 85cfd0b

kubeConfig = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, &clientcmd.ConfigOverrides{

This will take some debugging effort.

[dgageot]

The code that handles kube-context and kubeconfig is very convoluted. And for sure, it's buggy. I'm taking a look at how we can fix it

[dgageot]

It's a giant mess.

Those parameters can be set with env var, command line arguments, in skaffold.yaml and by profiles. Profiles can be activated based on kubecontext... So they could in theory change the kubecontext based on the kubecontext...

Also, the survey link is displayed based on kubecontext. But this often ignores the values passed on the command line and in the skaffold.yaml...

[dgageot]

I think we should deprecated the kubecontext field in skaffold.yaml. This can't really work because we rely on the actual kubecontext for many things BEFORE we even parse skaffold.yaml

[dgageot]

@nkubala I think this is actually a P1. wdyt?

[nkubala]

@dgageot sorry missed this, but yes I agree this is P1, this is quite an annoying UX

[briandealwis]

Reducing priority given current commitments.

[MarcoSchaar]

Hi @briandealwis
is there a chance for any next release to get a fix for this issue?

[nkubala]

@MarcoSchaar realistically i don't think we'll have a fix shipped with our 1.14.0 release, so we'll try and target 1.15.0.

i'm going to close this issue as a dupe of #4477 since we moved the conversation there, but i promise we're not ignoring you!

[briandealwis]

Re-opening: #4477 is a different issue: there, we're not validating the provided kube-context and as '' is not matched as a local cluster then builds are happening in the local docker daemon rather than the minikube hosted daemon.

[briandealwis]

Problem here is that context.GetCurrentConfig() — which sets a global singleton — before context.ConfigureKubeConfig() is initialized by the runner. We should move the update check into the runner?

github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/context.getCurrentConfig.func1()
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/kubernetes/context/context.go:117 +0x53
sync.(*Once).doSlow(0x6b5c978, 0x5c27458)
	/usr/local/go/src/sync/once.go:66 +0xec
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:57
github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/context.getCurrentConfig(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/kubernetes/context/context.go:116 +0x132
github.com/GoogleContainerTools/skaffold/pkg/skaffold/config.getConfigForCurrentKubectx.func1()
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/config/util.go:118 +0xab
sync.(*Once).doSlow(0x6b5c928, 0xc000096c38)
	/usr/local/go/src/sync/once.go:66 +0xec
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:57
github.com/GoogleContainerTools/skaffold/pkg/skaffold/config.getConfigForCurrentKubectx(0x0, 0x0, 0x4816231, 0x59d43c0, 0xc000522240)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/config/util.go:112 +0xa5
github.com/GoogleContainerTools/skaffold/pkg/skaffold/config.IsUpdateCheckEnabled(0x0, 0x0, 0x0)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/config/util.go:293 +0x3c
github.com/GoogleContainerTools/skaffold/pkg/skaffold/update.isUpdateCheckEnabled(0x0, 0x0, 0x0)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/update/update.go:78 +0x82
github.com/GoogleContainerTools/skaffold/pkg/skaffold/update.checkVersion(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/update/update.go:53 +0x50
github.com/GoogleContainerTools/skaffold/pkg/skaffold/update.CheckVersion(...)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/pkg/skaffold/update/update.go:44
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.NewSkaffoldCommand.func2.1(0xc000692480, 0xc0006dc1c0)
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/cmd/skaffold/app/cmd/cmd.go:104 +0x4a
created by github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.NewSkaffoldCommand.func2
	/Users/bdealwis/Projects/Skaffold/repo-skaffold/cmd/skaffold/app/cmd/cmd.go:103 +0x47b

[briandealwis]

I think we also poorly support KUBECONFIG, which can be a list of :-separated file names.

[1Const1]

Hello. Have the same issue. Is there any way to work with different Kubernetes configs that located not in the one default directory?

if i use just --config /myfolder/.kube/config
than i get an error :
Error creating REST client config in-cluster: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

if i use --kubeconfig /myfolder/.kube/config than skaffold deploy app in cluster (/myfolder/.kube/config) but searching pod from cluster with config located in profile folder /user/.kube/config
So i have the same problem like topic-starter

Skaffold Version:v1.17.2
ConfigVersion:skaffold/v2beta10

[MoSattler]

Same issue over here. I am super new to k8s and skaffold, but please let me know if I can help in any way to reproduce this.

[Gikkman]

I think this is something that should be looked at pretty soon. It is quite confusing as it is right now, and at least for us, it does produce some problems when we try to share deployment procedures between developers. Having to either have people hard-code kubecontext in their skaffold file or have everyone go through the process of adding entries in their .kube/config file is more work that I would like to do.

Edit
With some testing, I suspect the --kubeconfig flag is not picked up at all. No matter where I place my config file, or what I name it, I always get the "context not found" error.

If I place my config file in ~/.kube/config, leave out the context in the skaffold file and then set the context via a commandline flag, then the context is picked up. And even more, if I have a config in ~/.kube/config with the context I want to use, but point the --kubeconfig flag at a path where to config file exists, the context is still correctly applied.

So I suspect that --kubeconfig flag is not respected at all. I also think that if I point --kubeconfig to somewhere were no file exists, skaffold should either refuse to run since it can't find a file or at least print a warning.

Edit 2
So, as a work-around, you can instead set the KUBECONFIG environment variable, as that one seems to work as it should. If I run KUBECONFIG=$HOME/.kube/custom-config skaffold run -p <profile> --kube-context='<context>' the command runs as it should.

So a temporary fix for the issue could just be to update the documentation and instead reference the technique to set KUBECONFIG over using the --kubeconfig flag

[tejal29]

thanks @Gikkman This is the top priority issue for the team and we will looking at in or June Sprint.

[talksik]

Still have an issue related to this change

unable to connect to Kubernetes: getting client config for Kubernetes client: error creating REST client config for kubeContext "{{.KUBE_CONTEXT}}": context "{{.KUBE_CONTEXT}}" does not exist

I set KUBE_CONTEXT in my skaffold.env file

Here is part of my config in skaffold.yaml:

deploy: 
  kubectl: {}
  kubeContext: "{{.KUBE_CONTEXT}}"

Seems like it can't read environment variable for kubecontex?