🐛 Bug Report: Failed to parse page JSON data: expected value at line 1 column 1 [RATE LIMITS]

[arch-btw]

Describe the bug

This error is currently being displayed on all instances.

Steps to reproduce the bug

Go to any instance and it will say:

Failed to parse page JSON data: expected value at line 1 column 1 | /r/popular/hot.json?&raw_json=1&geo_filter=GLOBAL

What's the expected behavior?

To display any Reddit page.

Additional context / screenshot

This started happening about 10 minutes ago.
The json file itself is still accessible by browser, so maybe the structure changed?

[khan5ula]

Also experiencing this suddenly.

[Mr3ase]

Same here, I think it’s global somehow. I suppose either they changed something on reddit’s end and/or… they’re onto us?

[babeuh]

I can reproduce, I tried a restart of redlib which somehow fixed it for a short while, about 10 minutes, but then it came back. Maybe a slow rollout of a change on Reddit's side could cause that?

[netrunnereve]

I'm seeing this as well on my own instance and on public instances. I tried restarting Redlib but it didn't help.

[cyhyraethz]

I've also been experiencing this on my own instance. Restarting did seem to fix it, although I'm not sure if it will last.

Edit: Already not working again 15 minutes later.

[r7l]

Same here. But i can call the JSON manually and get a normal response with JSON being returned. Not sure if the JSON itself has changed.

[sigaloid]

Investigating this

[sigaloid]

Is anyone reproducing this locally (not on a public instance)? Or does this require heavy load to trigger?

[Taubin]

Is anyone reproducing this locally (not on a public instance)? Or does this require heavy load to trigger?

It's happening on both my home instance and one on a vps. Both of which I'm the only one using them.

[netrunnereve]

Is anyone reproducing this locally (not on a public instance)? Or does this require heavy load to trigger?

This is happening on my local instance where I'm the only user.

[babeuh]

Is anyone reproducing this locally (not on a public instance)? Or does this require heavy load to trigger?

Happening on my single user instance which doesn't have much load at all.

[SteffenBauer]

Also on my local home server, with only two people accessing it via local network.

Additional information: I run redlib (v0.35.1 compiled from source) as a systemd user service. Restarting the service makes redlib working again for a few minutes, after some minutes again above JSON error. I restarted the service several times, so far reproducible makes redlib always working again for a short time.

[Mr3ase]

Same, on my own instance where I am the only user, and restarting/redeploying from the source image does fix it for a little while, though I am not sure if it lasts for a given duration or a given number of requests.
Also when checking the container logs every JSON error thing is paired with a 403 forbidden from reddit.

Extract from logs of a freshly started container:

Starting Redlib...
Running Redlib v0.35.1 on [::]:8080!
ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden
ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden
ERROR redlib::utils > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1
ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden
ERROR redlib::utils > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1
ERROR redlib::client > Got an invalid response from reddit expected value at line 1 column 1. Status code: 403 Forbidden
ERROR redlib::utils > Error page rendered: Failed to parse page JSON data: expected value at line 1 column 1

[r7l]

Single user here as well. Mine is running on a VPS but i've just tried it locally and it's the same. It works for like 2 or 3 pages and then it breakes until restarted.

[sigaloid]

I've narrowed it down to a network policy error that Reddit's picking up on. This causes a 403 (which we didn't special case previously) and this network violation is detectable via the retry-after header (and various other headers, plus plaintext response, but headers will be fastest).

Going to push out a quick fix to catch the error, refresh the token, and avoid poisoning the cache. You may need to reload the page if you see this error, but it'll ideally be fixed right after.

Would appreciate someone pulling the latest code and testing when this goes in. Docker image will be updated in a few minutes.

[sigaloid]

Leaving open until I get confirmation that you don't see this error repeatedly. It may be spurious but that's the nature of the new (?) server behavior. Future improvements would include a default retry mechanism post-token-refresh (and maybe a better-designed token pool) but for now, if the issue goes away after a reload, I'm happy

[APT37]

Works for me.

Logs still show an error on page load, but that doesn't affect anything in the web app (as far as I can see).

ERROR redlib::utils > Error page rendered: Nothing here

EDIT: just got rate-limited - don't know if it's related.

Couldn't send request to Reddit: Rate limit - try refreshing soon | /r/popular/hot.json?&raw_json=1&geo_filter=GLOBAL

[stoopkidddd]

@sigaloid not entirely related, but might want to look at the failing docker build action, otherwise this fix isn't going to make it out to everyone - https://github.com/redlib-org/redlib/actions/runs/10891397433

See here - #226

[sigaloid]

Done, thanks!

[stoopkidddd]

Thank you for the quick fix 🙏

[r7l]

Thanks allot! If this fix works, could you drop a new release?

[edgar-vincent]

The latest Docker image works for me. Thank you so much for the speedy fix!

[mbestavros]

It sounds like it's specifically a new serverside issue when requesting it from the oauth routes. If we request it normally, I'm not sure if it will give us the same ratelimit bypass as now, since it won't look like a device.

Would it be at all feasible to allow server admins to pick which route they request from?

[sigaloid]

Yeah absolutely, it would likely mean it's reverting to the standard routes' rate limits though, which are worse than OAuth. A retry system would probably be a better option since it really seems to be sporadic/random.

[aurtzy]

Following @thudeous's discovery, I've done a git bisect which seems to point to 3bb5dc5 as the likely first commit with this issue. In my testing, 3bb5dc5 exhibited the behavior within 5 minutes of starting, whereas none of my test runs of redlib on the parent commit 3592728 so far have shown any error message of this sort (one of which had it running for about 2 hours).

[sigaloid]

Shocking. I could not tell you why that would be. But I can reproduce it and can confirm it's fixed.

What I don't know is why. That's just a dep bump to hyper-rustls and matching patch to fix compilation.

I see two possibilities:

• It could possibly be some fingerprinting that certain Reddit edge servers have deployed, and the older version of hyper-rustls doesn't have that issue. I suspect this is it since it just started happening without a corresponding source change.

• Or perhaps it's an issue with sporadic malformed HTTPS requests being generated by that version of hyper-rustls. It is 11 months old at this point and is outdated. But we can't upgrade it until we upgrade all of hyper, tokio, etc. Less likely but still possible and does still fit with the fact that it's a recent change - maybe certain Reddit edge servers have their HTTP parsing library updated to be more strict in what they accept, which would explain it happening all of a sudden.

Looking at hyper-rustls patch for that version, I don't see anything that jumps out at me. Maybe it's one of the dep bumps in this diff. rustls/hyper-rustls@v/0.24.1...v/0.25.0

Either way, you just identified the issue. Thanks so much for bisecting, I didn't think to because I immediately assumed it was a serverside change exclusively so no code change could really cause it (barring the switched endpoints).

[sigaloid]

Fixed in 793047f, new container is out on quay.io: https://github.com/redlib-org/redlib/actions/runs/10925300119

Going to fix a few other things before cutting a new release (such as failing tests). But for now, this issue is done. Thanks to everyone here who helped investigate, especially @aurtzy for finding the issue.

All instance operators should prioritize pulling the new image with at least commit 793047f.

[Tokarak]

That's my commit 😍 I caused all of this

[mirzaakdeniz]

Ahoy mate, I think you updated for Docker, but not the main repo. I installed and running locally, by using:

git clone https://github.com/redlib-org/redlib && cd redlib
cargo run

code. Issue is continuing for me.

[sigaloid]

Docker images are generated from the main repo. If you're seeing the rate limit issue still, it's more likely that it's an IP ban like in #217 (comment). in which case there's not much that can be done

[r7l]

Thanks allot for fixing the issue. Would it be possible to create a new release for those using releases to install Redlib?

[sigaloid]

Yep, once I finish other things on my to do list like the failing tests

[L-Be]

@BugZappa Are you using a VPN by any chance? I had routed my local instance through a VPN using a Gluetun container and I was still getting the error. Once I switched to using the local network there have been no issues. I swapped it back to the VPN to test and got the error again shortly after. I think I was still being rate limited when using the VPN.

[sigaloid]

@BugZappa you're running v0.31.0. Very old.

[BugZappa]

@BugZappa you're running v0.31.0. Very old.

Yeah for some reason Portainers "always pull image" feature sucks ass, so I did it through the CLI and got it now.

Thank you!

[maaaaz]

Hello @sigaloid, thanks for your great work !

Any ETA for the release ?

Maybe release it as "release candidate/alpha/beta" one if you know there are some bugs ?

[maaaaz]

I see that some binaries of the v0.35.1 release (dating back 29th June) were updated 4 days ago:
https://github.com/redlib-org/redlib/releases/tag/v0.35.1

Are these binaries fixing this issue?

If so, why is the "redlib" binary of this release still dating back to 29th June ?

[dave0003]

I tried the redlib-x86_64-unknown-linux-musl version (that was updated 4 days ago) and it seems to be working, unlike the old v0.35.1 "redlib" version.

The Git commit in the "instance information" is also the latest commit: 1838fda

[stag-enterprises]

I'm getting this issue again, using the second-latest commit.

[ButteredCats]

Yep. Both my containers are running into it as well.

[bclouthier]

I'm been getting this issue as well. Sometimes restarting my container gives me one page load.

[sigaloid]

@drewbdoo #301 is the current rate limiting issue, but i can't reproduce it on the latest commit. are you sure you've updated to the latest commit?

[davralin]

fwiw, everything works for me - on the latest commit.
Had some hicups yesterday, but all good today.

[drewbdoo]

Thanks all - I was on the latest but from docker hub, not quay. Working now