API unreliability: regular 504 gateway timeouts #2232

shrink · 2022-10-11T12:37:17Z

shrink
Oct 11, 2022

Hi,

I am working on building out a Hydrogen storefront. I have deployed a preview of the store into a production environment, connected to a paid Shopify store (on the "Basic" plan) and set up monitoring to regularly interact with the storefront to monitor performance over time.

Unfortunately, I am regularly (i.e: multiple times per day) encountering errors from the Shopify API. I have searched the documentation and GitHub but I've been unable to locate any existing discussion of this issue so I am unsure if this is expected and/or if there is a strategy intended to deal with it (i.e: automatic query retries?).

Example API response:

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

The query that causes the error:

query shopInfo {
  shop {
    name
    description
    primaryDomain {
      url
    }
  }
}

I have tried with caching enabled, disabled and set to the default. The error is occurring regardless of the caching configuration. Here's a couple of x-request-id values in case this is helpful to track down the issue:

145583a7-067c-4e2a-ada2-cbaf04389ef8
4b50ea0b-8d10-4768-9f0a-11832b366b01
d6291cdf-95ce-44ea-925c-7ead0a05a7c1
f2ac113f-3d3c-403f-b06a-46d6ba263512

I guess my questions are:

Is the Storefront API production ready, i.e: should gateway timeouts be rare?
Are there any specific configurations (or queries) that can cause gateway timeouts?
Do you expect implementations of Hydrogen to encounter API errors, and thus have fallbacks in place?

Thanks,

frehner · 2022-10-11T15:38:45Z

frehner
Oct 11, 2022
Collaborator

Is the Storefront API production ready, i.e: should gateway timeouts be rare?

It is production ready, as it currently serves, uh, quite a lot of requests every day. 🙂

Are there any specific configurations (or queries) that can cause gateway timeouts?

Nothing expected, though it appears you may have found an edge case.

Do you expect implementations of Hydrogen to encounter API errors, and thus have fallbacks in place?

Yes, generally it's best practice to handle API errors even if you don't expect them. 👍

I've reached out to the Storefront API team to see if they can help diagnose what's going on with those requests. Thanks for including the headers, that helps things immensely! I'll follow up here once I get some answers.

1 reply

shrink Oct 11, 2022
Author

Thank you for getting back to me so quickly, and the insight :) My experience outside of Hydrogen is that Shopify is far more reliable than anything I'll ever deploy -- I monitor the Shopify hosted site for this store, and that hasn't had any downtime -- so I wasn't sure if there's something Hydrogen unique about the Storefront API (maybe Hydrogen requests are routed differently, or something). I'll keep an eye out for any follow up from the Storefront API team, and log each request so that I can provide every future x-request-id if they need it. Thanks!

kelhusseiny · 2022-10-12T16:01:18Z

kelhusseiny
Oct 12, 2022
Collaborator

Hey 👋🏼 Are we able to replicate this? I'd like to see some newer logs to investigate the lifecycle of the request

9 replies

frehner Oct 23, 2022
Collaborator

I asked the team for an update - I see they made some progress last week with that additional information you provided, but I haven't seen anything since.

Feel free to ping me again if you don't hear from us in a couple of days.

shrink Nov 7, 2022
Author

@frehner just checking in as I haven't heard anything for 2 weeks. The issues continue to occur -- there's been ~70 occurrences so far in November.

Error: 503 Service Unavailable (x-request-id: dfdfe2bb-32df-4696-ac7f-764de95f8b52) Nov 6, 2022 4:00:28 PM UTC
Error: 430 undefined (x-request-id: 8d26990a-e4f3-4390-a34a-b435eb72eb55) Nov 6, 2022 4:57:42 AM UTC
Error: 503 Service Unavailable (x-request-id: f8403cbe-36fc-44bb-a619-e74f5755998f) Nov 4, 2022 4:19:47 PM UTC

I can provide more request IDs if helpful. Thanks!

frehner Nov 7, 2022
Collaborator

Thanks for notifying me; I've reached out again to see if there's any update.

frehner Nov 7, 2022
Collaborator

I heard back that these 503s are the platform protecting itself when the load is high; it sounds like it's likely nothing on your end causing it, but the platform itself just being under heavy load. I was told that the best thing to do right now is to handle these errors and try again if you get a 503.

shrink Nov 7, 2022
Author

@frehner thank you for the information. Follow up questions (not urgent, but any insight you can provide would be greatly appreciated):

Do these protections apply to the whole of the Storefront API or a subset of queries? I can cache quite a few types of queries, like store info and products, so availability for those isn't an issue, but I can't cache things like the cart because they depend on each request. Should I expect that sometimes the platform protections will prevent users from interacting with their cart, and if so, is there a suggestion for how I might handle that in a Shopify/Hydrogen way? Any examples of how other Shopify stores deal with intermittent cart availability would be very helpful!
Does heavy load apply to the system as a whole, or is it due to heavy load on resources that my store is assigned to? Am I less likely to encounter this issue on the "Advanced" plan than I am on the "Basic" plan, or are resources allocated equally regardless of plan? I've looked at the Shopify pricing page and I can't find mention of resources, so I assume allocation is the same on all non-Enterprise plans, but if there is a difference in plans, I'm happy to upgrade :)

Thanks!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API unreliability: regular 504 gateway timeouts #2232

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

API unreliability: regular 504 gateway timeouts #2232

shrink Oct 11, 2022

Replies: 2 comments · 10 replies

frehner Oct 11, 2022 Collaborator

shrink Oct 11, 2022 Author

kelhusseiny Oct 12, 2022 Collaborator

frehner Oct 23, 2022 Collaborator

shrink Nov 7, 2022 Author

frehner Nov 7, 2022 Collaborator

frehner Nov 7, 2022 Collaborator

shrink Nov 7, 2022 Author

shrink
Oct 11, 2022

Replies: 2 comments 10 replies

frehner
Oct 11, 2022
Collaborator

shrink Oct 11, 2022
Author

kelhusseiny
Oct 12, 2022
Collaborator

frehner Oct 23, 2022
Collaborator

shrink Nov 7, 2022
Author

frehner Nov 7, 2022
Collaborator

frehner Nov 7, 2022
Collaborator

shrink Nov 7, 2022
Author