You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After a recent version bump I needed to manually reboot the proxy. This gave me the following error:
ERROR (SSHKit::Command::Failed): Exception while executing on host RE.MO.VE.D: docker exit status: 125
docker stdout: Nothing written
docker stderr: Error response from daemon: No such container: kamal-proxy
Error: failed to start containers: kamal-proxy
cat: .kamal/proxy/options: No such file or directory
Unable to find image 'basecamp/kamal-proxy:v0.8.4' locally
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
See 'docker run --help'.
Unfortunately, it appeared that Kamal Proxy was successfully shutdown but without the new version running. Nor was the old Docker container available.
I was unable to get the proxy running again as I kept running into the above error.
The cat: .kamal/proxy/options: No such file or directory error got me thinking there was a problem with the latest Kamal release. But I finally realized that the real problem was with Docker not being able to pull basecamp/kamal-proxy:v0.8.4.
I ssh'd into my server and tried this:
docker pull basecamp/kamal-proxy:v0.8.4
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Logging into Docker Hub using the following command finally fixed the issue:
After this I was able to run kamal proxy reboot without a problem.
I don't know why my server got rate limited as I only use it for one app and don't do many Docker pulls. However, the real issue here is that Kamal is able to get into a state where it shuts down the proxy and is unable to pull an image. I think this should never be allowed to happen.
I would expect a version bump to be more safe than this. For example, by pulling the new image before stopping and deleting the old container.
The text was updated successfully, but these errors were encountered:
marckohlbrugge
changed the title
Docker Hub rate-limits can lead to broken Kamal installation with no running Proxy
Docker registry rate-limits during proxy reboot leaves Kamal installation without a proxy
Dec 21, 2024
desc"upgrade","Upgrade to kamal-proxy on servers (stop container, remove container, start new container, reboot app)",hide: trueoption:rolling,type: :boolean,default: false,desc: "Reboot proxy on hosts in sequence, rather than in parallel"option:confirmed,aliases: "-y",type: :boolean,default: false,desc: "Proceed without confirmation question"defupgradeinvoke_options={"version"=>KAMAL.config.latest_tag}.merge(options)confirming"This will cause a brief outage on each host. Are you sure?"dohost_groups=options[:rolling] ? KAMAL.hosts : [KAMAL.hosts]host_groups.eachdo |hosts|
host_list=Array(hosts).join(",")say"Upgrading proxy on #{host_list}...",:magentarun_hook"pre-proxy-reboot",hosts: host_liston(hosts)do |host|
execute *KAMAL.auditor.record("Rebooted proxy"),verbosity: :debugexecute *KAMAL.registry.login"Stopping and removing Traefik on #{host}, if running..."execute *KAMAL.proxy.cleanup_traefik"Stopping and removing kamal-proxy on #{host}, if running..."execute *KAMAL.proxy.stop,raise_on_non_zero_exit: falseexecute *KAMAL.proxy.remove_containerexecute *KAMAL.proxy.remove_imageendKAMAL.with_specific_hosts(hosts)doinvoke"kamal:cli:proxy:boot",[],invoke_optionsreset_invocation(Kamal::Cli::Proxy)invoke"kamal:cli:app:boot",[],invoke_optionsreset_invocation(Kamal::Cli::App)invoke"kamal:cli:prune:all",[],invoke_optionsreset_invocation(Kamal::Cli::Prune)endrun_hook"post-proxy-reboot",hosts: host_listsay"Upgraded proxy on #{host_list}",:magentaendendend
I think it first removes the container and image, and then relies on the boot command to pull the image, create the container, etc.
It would be safer to first pull the new image, boot the container, confirm that worked, and then remove the previous container and image.
After a recent version bump I needed to manually reboot the proxy. This gave me the following error:
Unfortunately, it appeared that Kamal Proxy was successfully shutdown but without the new version running. Nor was the old Docker container available.
I was unable to get the proxy running again as I kept running into the above error.
The
cat: .kamal/proxy/options: No such file or directory
error got me thinking there was a problem with the latest Kamal release. But I finally realized that the real problem was with Docker not being able to pullbasecamp/kamal-proxy:v0.8.4
.I ssh'd into my server and tried this:
Logging into Docker Hub using the following command finally fixed the issue:
After this I was able to run
kamal proxy reboot
without a problem.I don't know why my server got rate limited as I only use it for one app and don't do many Docker pulls. However, the real issue here is that Kamal is able to get into a state where it shuts down the proxy and is unable to pull an image. I think this should never be allowed to happen.
I would expect a version bump to be more safe than this. For example, by pulling the new image before stopping and deleting the old container.
The text was updated successfully, but these errors were encountered: