Deploy and upgrade nebari on GCP

[akansal1]

what is the best way to deploy and upgrade nebari on GCP ? Nebari guide for GCP deployment doesn't work directly

[Adam-D-Lewis]

Hi @akansal1 Please post what issue you are seeing. We expect it to work directly.

[swapkh91]

@Adam-D-Lewis getting this error when running nebari deploy -c nebari-config.yaml

nebari version - 2024.12.1

[tofu]: │ Error: Provider produced inconsistent result after apply
[tofu]: │ 
[tofu]: │ When applying changes to
[tofu]: │ module.registry-jupyterhub.google_container_registry.registry, provider
[tofu]: │ "provider[\"registry.opentofu.org/hashicorp/google\"]" produced an
[tofu]: │ unexpected new value: Root object was present, but now absent.
[tofu]: │ 
[tofu]: │ This is a bug in the provider, which should be reported in the provider's
[tofu]: │ own issue tracker.
[tofu]: ╵
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/subcommands/deploy.py:92 in     │
│ deploy                                                                                           │
│                                                                                                  │
│   89 │   │   │   msg = "Digital Ocean support is currently being deprecated and will be remov    │
│   90 │   │   │   typer.confirm(msg)                                                              │
│   91 │   │                                                                                       │
│ ❱ 92 │   │   deploy_configuration(                                                               │
│   93 │   │   │   config,                                                                         │
│   94 │   │   │   stages,                                                                         │
│   95 │   │   │   disable_prompt=disable_prompt,                                                  │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/deploy.py:55 in                 │
│ deploy_configuration                                                                             │
│                                                                                                  │
│   52 │   │   │   │   s: hookspecs.NebariStage = stage(                                           │
│   53 │   │   │   │   │   output_directory=pathlib.Path.cwd(), config=config                      │
│   54 │   │   │   │   )                                                                           │
│ ❱ 55 │   │   │   │   stack.enter_context(s.deploy(stage_outputs, disable_prompt))                │
│   56 │   │   │   │                                                                               │
│   57 │   │   │   │   if not disable_checks:                                                      │
│   58 │   │   │   │   │   s.check(stage_outputs, disable_prompt)                                  │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:526 in           │
│ enter_context                                                                                    │
│                                                                                                  │
│   523 │   │   except AttributeError:                                                             │
│   524 │   │   │   raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "          │
│   525 │   │   │   │   │   │   │   f"not support the context manager protocol") from None         │
│ ❱ 526 │   │   result = _enter(cm)                                                                │
│   527 │   │   self._push_cm_exit(cm, _exit)                                                      │
│   528 │   │   return result                                                                      │
│   529                                                                                            │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:137 in __enter__ │
│                                                                                                  │
│   134 │   │   # they are only needed for recreation, which is not possible anymore               │
│   135 │   │   del self.args, self.kwds, self.func                                                │
│   136 │   │   try:                                                                               │
│ ❱ 137 │   │   │   return next(self.gen)                                                          │
│   138 │   │   except StopIteration:                                                              │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None                         │
│   140                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/stages/infrastructure/__init__. │
│ py:899 in deploy                                                                                 │
│                                                                                                  │
│   896 │   def deploy(                                                                            │
│   897 │   │   self, stage_outputs: Dict[str, Dict[str, Any]], disable_prompt: bool = False       │
│   898 │   ):                                                                                     │
│ ❱ 899 │   │   with super().deploy(stage_outputs, disable_prompt):                                │
│   900 │   │   │   with kubernetes_provider_context(                                              │
│   901 │   │   │   │   stage_outputs["stages/" + self.name]["kubernetes_credentials"]["value"]    │
│   902 │   │   │   ):                                                                             │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:137 in __enter__ │
│                                                                                                  │
│   134 │   │   # they are only needed for recreation, which is not possible anymore               │
│   135 │   │   del self.args, self.kwds, self.func                                                │
│   136 │   │   try:                                                                               │
│ ❱ 137 │   │   │   return next(self.gen)                                                          │
│   138 │   │   except StopIteration:                                                              │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None                         │
│   140                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/stages/base.py:298 in deploy    │
│                                                                                                  │
│   295 │   │   │   deploy_config["tofu_import"] = True                                            │
│   296 │   │   │   deploy_config["state_imports"] = state_imports                                 │
│   297 │   │                                                                                      │
│ ❱ 298 │   │   self.set_outputs(stage_outputs, opentofu.deploy(**deploy_config))                  │
│   299 │   │   self.post_deploy(stage_outputs, disable_prompt)                                    │
│   300 │   │   yield                                                                              │
│   301                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/provider/opentofu.py:71 in      │
│ deploy                                                                                           │
│                                                                                                  │
│    68 │   │   │   │   )                                                                          │
│    69 │   │                                                                                      │
│    70 │   │   if tofu_apply:                                                                     │
│ ❱  71 │   │   │   apply(directory, var_files=[f.name])                                           │
│    72 │   │                                                                                      │
│    73 │   │   if tofu_destroy:                                                                   │
│    74 │   │   │   destroy(directory, var_files=[f.name])                                         │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/provider/opentofu.py:152 in     │
│ apply                                                                                            │
│                                                                                                  │
│   149 │   │   + ["-var-file=" + _ for _ in var_files]                                            │
│   150 │   )                                                                                      │
│   151 │   with timer(logger, "tofu apply"):                                                      │
│ ❱ 152 │   │   run_tofu_subprocess(command, cwd=directory, prefix="tofu")                         │
│   153                                                                                            │
│   154                                                                                            │
│   155 def output(directory=None):                                                                │
│                                                                                                  │
│ /Users/swapnesh1/nebari-env/lib/python3.12/site-packages/_nebari/provider/opentofu.py:120 in     │
│ run_tofu_subprocess                                                                              │
│                                                                                                  │
│   117 │   logger.info(f" tofu at {tofu_path}")                                                   │
│   118 │   exit_code, output = run_subprocess_cmd([tofu_path] + processargs, **kwargs)            │
│   119 │   if exit_code != 0:                                                                     │
│ ❱ 120 │   │   raise OpenTofuException("OpenTofu returned an error")                              │
│   121 │   return output                                                                          │
│   122                                                                                            │
│   123                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OpenTofuException: OpenTofu returned an error

I noticed then that the google hashicorp version is old in the generate TF scripts 4.83.0 and that the registry being used is set as container registry. As per GCP, container registry is being deprecated and artifact registry has to be used.

So I changed the hashicorp/google version to 6.14.1 wherever needed and also changed the TF for registry (stages/02-infrastructure/gcp/modules/registry/main.tf) to

resource "google_artifact_registry_repository" "registry" {
  location = var.location
  format = "DOCKER"
  repository_id = "my-repository"
}

I then installed through nebari deploy -c nebari-config.yaml --disable-render and everything was successfully created

[dcmcand]

@swapkh91, thanks for the detailed report and troubleshooting. I have created issue #2894 to track this. PR's are welcome if you have time to contribute.

[swapkh91]

@swapkh91, thanks for the detailed report and troubleshooting. I have created issue #2894 to track this. PR's are welcome if you have time to contribute.

@dcmcand i was checking the contribution guide and it mentions to checkout the develop branch but i dont see any such branch in the repo. Which one should i checkout?

Also i'm not sure what would be the tests for this, didn't find any here specific to GCP

https://www.nebari.dev/community/code-contributions#develop-your-contribution

[dcmcand]

Hi @swapkh91, sorry about that, I will get those docs updated. Just checkout main.

[dcmcand]

I haven't seen this one before, but some quick googling shows it might be a permissions issue with your service role.

[swapkh91]

I haven't seen this one before, but some quick googling shows it might be a permissions issue with your service role.

i'll check, maybe the service account is not being used, althrough yesterday it worked fine

[swapkh91]

I haven't seen this one before, but some quick googling shows it might be a permissions issue with your service role.

not sure what the issue was, the service account was getting used
i deleted the file from the bucket and re-ran the deploy command and it was successful
the default.tflock file was created again and gets deleted after deploy command succeeds
i think this lock was created when i ran the command some hours back and it didn't finish due to some connectivity issue so the file remained in the bucket

[viniciusdc]

i think this lock was created when I ran the command some hours back, and it didn't finish due to some connectivity issue, so the file remained in the bucket

That's also my assumption, though usually, this might happen when the Python process abruptly stops and the lock remains. I tend to initialize the stage and run tofu force-unlock manually when developing; as a general rule, I always suggest waiting until a timeout occurs for opentofu/terraform to lift the lock.

Once you open the PR, feel free to ping any of us there :) -- btw, if you do encounter a similar error message as you've encountered previously with the artifact_registry_repository, I suggest rerunning the deployment with TF_LOG=debug to inspect the response sent by the provider, I wasn't able to reproduce the same issue locally, so I am curious as what was causing it.

[swapkh91]

i think this lock was created when I ran the command some hours back, and it didn't finish due to some connectivity issue, so the file remained in the bucket

That's also my assumption, though usually, this might happen when the Python process abruptly stops and the lock remains. I tend to initialize the stage and run tofu force-unlock manually when developing; as a general rule, I always suggest waiting until a timeout occurs for opentofu/terraform to lift the lock.

Once you open the PR, feel free to ping any of us there :) -- btw, if you do encounter a similar error message as you've encountered previously with the artifact_registry_repository, I suggest rerunning the deployment with TF_LOG=debug to inspect the response sent by the provider, I wasn't able to reproduce the same issue locally, so I am curious as what was causing it.

@viniciusdc these are the logs after setting TF_LOG=debug

[tofu]: ╷
[tofu]: │ Error: Provider produced inconsistent result after apply
[tofu]: │ 
[tofu]: │ When applying changes to
[tofu]: │ module.registry-jupyterhub.google_container_registry.registry, provider
[tofu]: │ "provider[\"registry.opentofu.org/hashicorp/google\"]" produced an
[tofu]: │ unexpected new value: Root object was present, but now absent.
[tofu]: │ 
[tofu]: │ This is a bug in the provider, which should be reported in the provider's
[tofu]: │ own issue tracker.
[tofu]: ╵
[tofu]: 2025-01-08T12:31:20.691+0530 [DEBUG] provider.terraform-provider-google: 2025/01/08 12:31:20 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /var/folders/l1/ftbcmjcd3qn4hm2t0p3vqhm40000gn/T/plugin108223358->: use of closed network connection 
[tofu]: 2025-01-08T12:31:20.691+0530 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
[tofu]: 2025-01-08T12:31:20.696+0530 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.opentofu.org/hashicorp/google/4.8.0/darwin_arm64/terraform-provider-google pid=92331
[tofu]: 2025-01-08T12:31:20.696+0530 [DEBUG] provider: plugin exited
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/swapnesh1/nebari/src/_nebari/subcommands/deploy.py:92 in deploy                           │
│                                                                                                  │
│   89 │   │   │   msg = "Digital Ocean support is currently being deprecated and will be remov    │
│   90 │   │   │   typer.confirm(msg)                                                              │
│   91 │   │                                                                                       │
│ ❱ 92 │   │   deploy_configuration(                                                               │
│   93 │   │   │   config,                                                                         │
│   94 │   │   │   stages,                                                                         │
│   95 │   │   │   disable_prompt=disable_prompt,                                                  │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/deploy.py:55 in deploy_configuration                         │
│                                                                                                  │
│   52 │   │   │   │   s: hookspecs.NebariStage = stage(                                           │
│   53 │   │   │   │   │   output_directory=pathlib.Path.cwd(), config=config                      │
│   54 │   │   │   │   )                                                                           │
│ ❱ 55 │   │   │   │   stack.enter_context(s.deploy(stage_outputs, disable_prompt))                │
│   56 │   │   │   │                                                                               │
│   57 │   │   │   │   if not disable_checks:                                                      │
│   58 │   │   │   │   │   s.check(stage_outputs, disable_prompt)                                  │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:526 in           │
│ enter_context                                                                                    │
│                                                                                                  │
│   523 │   │   except AttributeError:                                                             │
│   524 │   │   │   raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "          │
│   525 │   │   │   │   │   │   │   f"not support the context manager protocol") from None         │
│ ❱ 526 │   │   result = _enter(cm)                                                                │
│   527 │   │   self._push_cm_exit(cm, _exit)                                                      │
│   528 │   │   return result                                                                      │
│   529                                                                                            │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:137 in __enter__ │
│                                                                                                  │
│   134 │   │   # they are only needed for recreation, which is not possible anymore               │
│   135 │   │   del self.args, self.kwds, self.func                                                │
│   136 │   │   try:                                                                               │
│ ❱ 137 │   │   │   return next(self.gen)                                                          │
│   138 │   │   except StopIteration:                                                              │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None                         │
│   140                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/stages/infrastructure/__init__.py:915 in deploy              │
│                                                                                                  │
│   912 │   def deploy(                                                                            │
│   913 │   │   self, stage_outputs: Dict[str, Dict[str, Any]], disable_prompt: bool = False       │
│   914 │   ):                                                                                     │
│ ❱ 915 │   │   with super().deploy(stage_outputs, disable_prompt):                                │
│   916 │   │   │   with kubernetes_provider_context(                                              │
│   917 │   │   │   │   stage_outputs["stages/" + self.name]["kubernetes_credentials"]["value"]    │
│   918 │   │   │   ):                                                                             │
│                                                                                                  │
│ /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py:137 in __enter__ │
│                                                                                                  │
│   134 │   │   # they are only needed for recreation, which is not possible anymore               │
│   135 │   │   del self.args, self.kwds, self.func                                                │
│   136 │   │   try:                                                                               │
│ ❱ 137 │   │   │   return next(self.gen)                                                          │
│   138 │   │   except StopIteration:                                                              │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None                         │
│   140                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/stages/base.py:298 in deploy                                 │
│                                                                                                  │
│   295 │   │   │   deploy_config["tofu_import"] = True                                            │
│   296 │   │   │   deploy_config["state_imports"] = state_imports                                 │
│   297 │   │                                                                                      │
│ ❱ 298 │   │   self.set_outputs(stage_outputs, opentofu.deploy(**deploy_config))                  │
│   299 │   │   self.post_deploy(stage_outputs, disable_prompt)                                    │
│   300 │   │   yield                                                                              │
│   301                                                                                            │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/provider/opentofu.py:71 in deploy                            │
│                                                                                                  │
│    68 │   │   │   │   )                                                                          │
│    69 │   │                                                                                      │
│    70 │   │   if tofu_apply:                                                                     │
│ ❱  71 │   │   │   apply(directory, var_files=[f.name])                                           │
│    72 │   │                                                                                      │
│    73 │   │   if tofu_destroy:                                                                   │
│    74 │   │   │   destroy(directory, var_files=[f.name])                                         │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/provider/opentofu.py:152 in apply                            │
│                                                                                                  │
│   149 │   │   + ["-var-file=" + _ for _ in var_files]                                            │
│   150 │   )                                                                                      │
│   151 │   with timer(logger, "tofu apply"):                                                      │
│ ❱ 152 │   │   run_tofu_subprocess(command, cwd=directory, prefix="tofu")                         │
│   153                                                                                            │
│   154                                                                                            │
│   155 def output(directory=None):                                                                │
│                                                                                                  │
│ /Users/swapnesh1/nebari/src/_nebari/provider/opentofu.py:120 in run_tofu_subprocess              │
│                                                                                                  │
│   117 │   logger.info(f" tofu at {tofu_path}")                                                   │
│   118 │   exit_code, output = run_subprocess_cmd([tofu_path] + processargs, **kwargs)            │
│   119 │   if exit_code != 0:                                                                     │
│ ❱ 120 │   │   raise OpenTofuException("OpenTofu returned an error")                              │
│   121 │   return output                                                                          │
│   122                                                                                            │
│   123                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OpenTofuException: OpenTofu returned an error