Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker registry rate-limits during proxy reboot leaves Kamal installation without a proxy #86

Open
marckohlbrugge opened this issue Dec 21, 2024 · 1 comment

Comments

@marckohlbrugge
Copy link

marckohlbrugge commented Dec 21, 2024

After a recent version bump I needed to manually reboot the proxy. This gave me the following error:

ERROR (SSHKit::Command::Failed): Exception while executing on host RE.MO.VE.D: docker exit status: 125
docker stdout: Nothing written
docker stderr: Error response from daemon: No such container: kamal-proxy
Error: failed to start containers: kamal-proxy
cat: .kamal/proxy/options: No such file or directory
Unable to find image 'basecamp/kamal-proxy:v0.8.4' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.

Unfortunately, it appeared that Kamal Proxy was successfully shutdown but without the new version running. Nor was the old Docker container available.

I was unable to get the proxy running again as I kept running into the above error.

The cat: .kamal/proxy/options: No such file or directory error got me thinking there was a problem with the latest Kamal release. But I finally realized that the real problem was with Docker not being able to pull basecamp/kamal-proxy:v0.8.4.

I ssh'd into my server and tried this:

docker pull basecamp/kamal-proxy:v0.8.4
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Logging into Docker Hub using the following command finally fixed the issue:

docker login -u myusername
docker pull basecamp/kamal-proxy:v0.8.4

After this I was able to run kamal proxy reboot without a problem.

I don't know why my server got rate limited as I only use it for one app and don't do many Docker pulls. However, the real issue here is that Kamal is able to get into a state where it shuts down the proxy and is unable to pull an image. I think this should never be allowed to happen.

I would expect a version bump to be more safe than this. For example, by pulling the new image before stopping and deleting the old container.

@marckohlbrugge marckohlbrugge changed the title Docker Hub rate-limits can lead to broken Kamal installation with no running Proxy Docker registry rate-limits during proxy reboot leaves Kamal installation without a proxy Dec 21, 2024
@marckohlbrugge
Copy link
Author

I might be misremembering the command. It might have been kamal proxy upgrade instead of kamal proxy reboot because it did happen during an upgrade.

I did find this:

https://github.com/basecamp/kamal/blob/1547089da044159a934f1ef90006ae10e599d687/lib/kamal/cli/proxy.rb

  desc "upgrade", "Upgrade to kamal-proxy on servers (stop container, remove container, start new container, reboot app)", hide: true
  option :rolling, type: :boolean, default: false, desc: "Reboot proxy on hosts in sequence, rather than in parallel"
  option :confirmed, aliases: "-y", type: :boolean, default: false, desc: "Proceed without confirmation question"
  def upgrade
    invoke_options = { "version" => KAMAL.config.latest_tag }.merge(options)

    confirming "This will cause a brief outage on each host. Are you sure?" do
      host_groups = options[:rolling] ? KAMAL.hosts : [ KAMAL.hosts ]
      host_groups.each do |hosts|
        host_list = Array(hosts).join(",")
        say "Upgrading proxy on #{host_list}...", :magenta
        run_hook "pre-proxy-reboot", hosts: host_list
        on(hosts) do |host|
          execute *KAMAL.auditor.record("Rebooted proxy"), verbosity: :debug
          execute *KAMAL.registry.login

          "Stopping and removing Traefik on #{host}, if running..."
          execute *KAMAL.proxy.cleanup_traefik

          "Stopping and removing kamal-proxy on #{host}, if running..."
          execute *KAMAL.proxy.stop, raise_on_non_zero_exit: false
          execute *KAMAL.proxy.remove_container
          execute *KAMAL.proxy.remove_image
        end

        KAMAL.with_specific_hosts(hosts) do
          invoke "kamal:cli:proxy:boot", [], invoke_options
          reset_invocation(Kamal::Cli::Proxy)
          invoke "kamal:cli:app:boot", [], invoke_options
          reset_invocation(Kamal::Cli::App)
          invoke "kamal:cli:prune:all", [], invoke_options
          reset_invocation(Kamal::Cli::Prune)
        end

        run_hook "post-proxy-reboot", hosts: host_list
        say "Upgraded proxy on #{host_list}", :magenta
      end
    end
  end

I think it first removes the container and image, and then relies on the boot command to pull the image, create the container, etc.

It would be safer to first pull the new image, boot the container, confirm that worked, and then remove the previous container and image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant