[RTD][Sphinx] Server Error 503: first byte timeout for objects.inv

[amotl]

Problem

This error happens when building the Sphinx documentation. We are hosting the documentation on Read the Docs (readthedocs.org), behind an Nginx for dispatching routes, and Fastly for caching.

intersphinx inventory 'https://cratedb.com/docs/cloud/cli/en/latest/objects.inv' not fetchable due to
<class 'requests.exceptions.HTTPError'>: 503 Server Error: first byte timeout for url:
https://cratedb.com/docs/cloud/cli/en/latest/objects.inv

It's classified as a bug Something isn't working , because it breaks developer and CI workflows. Not constantly, but it seems to happen more often now.

Reports

October 28, Slack

Second time within a week I'm seeing test failures with:

intersphinx inventory 'https://cratedb.com/docs/sqlalchemy-cratedb/objects.inv' not fetchable due to
<class 'requests.exceptions.HTTPError'>: 503 Server Error: first byte timeout for url:
https://cratedb.com/docs/sqlalchemy-cratedb/objects.inv

Do we have some server issues there ?

November 1, Slack

Also on GHA.

intersphinx inventory [...] not fetchable
503 Server Error: first byte timeout for url

-- https://github.com/crate/crate-python/actions/runs/11628806414/job/32384615498?pr=664#step:4:262

December 10, Slack

Do we have any new insights on those errors? Seems to happen more often now. 😞

intersphinx inventory 'https://cratedb.com/docs/cloud/cli/en/latest/objects.inv' not fetchable due to
<class 'requests.exceptions.HTTPError'>: 503 Server Error: first byte timeout for url:
https://cratedb.com/docs/cloud/cli/en/latest/objects.inv

[amotl]

All of those do not reveal any sensible suggestions.

• https://github.com/search?q=readthedocs+first+byte+timeout&type=issues
• https://github.com/search?q=read+the+docs+first+byte+timeout&type=issues
• https://github.com/search?q=sphinx+first+byte+timeout&type=issues
• https://github.com/search?q=objects.inv+first+byte+timeout&type=issues

[amotl]

It might be coming from Fastly. At least, it is Fastly that raises this very error after experiencing a timeout to upstream servers.

bereq.first_byte_timeout

• The amount of time Fastly should wait for a backend server to deliver the first byte of an HTTP response body, before triggering a timeout.
• May be set to any value between 0 and 600s. By default, the timeout is 15s.
• If the timeout is triggered, Fastly will create an error object and raise 503: first byte timeout.

-- Documentation about bereq.first_byte_timeout

@msbt: Can you please check which value is configured in Fastly for the particular /docs route, or otherwise globally? Thanks!

[msbt]

@amotl we have first_byte_timeout = 15s; set for all our services. Do we know if this is actually a timeout (like is it trying for 15s)? @goat-ssh might be able to shed some light, since he has access to the Fastly logs.

[amotl]

access to the Fastly logs.

That would be cool, to get insights if there is something in the logs at those spots where Fastly communicates with our Nginx.

[amotl]

@goat-ssh: Placeholder for log extract that shows there are spurious 503s on different URLs, maybe trimmed to increase the focus on relevant details. Just go ahead and replace this message. Thanks!

[amotl]

Thanks for sharing. That's quite a few 503 responses at different ends, not exclusively about objects.inv, right?

a) It will be nice to have a graph about them, to learn about trends.
b) It will be nice to follow up through them on behalf of the logs of our Nginx web server that is reverse-proxying to readthedocs.org, in order to potentially expand our learning about their origins.

NB: Please note this is a public ticket that wants to be concise about any potential matters of this being possibly related to objects.inv, and should not be used for general extensive debugging. Now that we learn the 503 responses are spurious, and not related to objects.inv at all, other than requests to those will certainly be very frequent, we may continue debugging the root cause on a different spot.