IP ban investigation #318
Comments
It should be noted that both are currently fully operational but has previously been experiencing #301. |
|
Can you set the environment variable |
|
Just got the error on both servers. It looks like NYC-1 just became 3 commits too old due to unrelated issues regarding quay.io (it looks like |
|
Hm, the fact that it's near to a token refresh makes me wonder if it's the fact that the token reaches its rate limit too quickly. It currently starts a background task when there's 9 left, which is working:
But... if 9 concurrent requests come in simultaneously, it's possible they exhaust the remaining rate limit count. Is this a high-traffic instance, specifically? |
Definitely not. CAL-1 is one of two secondary servers, only listed on my site. It's primarily used for Invidious. This doesn't appear to be deliberate DDoS either, as my other server, POL-1, has been completely unaffected so far. EDIT: With that being said, Invidious does have the feature to fetch comments from Reddit. But this hasn't been functional since Reddit's API fiasco and only attempts to fetch when the user explicitly clicks the "View Reddit comments" button on a video. Plus it wouldn't make sense why that would start to become a problem now, many months after hosting both. |
|
Hm, yeah, that checks out. So far every IP has been a commercial ASN and I'm hoping that's not the commonality, that they all have flagged some heuristic for being suspicious because they're hosted. |
|
I am noting an unusual amount of requests on NYC-1 (the public server on redlib's list.) I've heavily adjusted rate limits over nginx to be only 5 requests per second (10 burst) but without any further logging, it's difficult to say whether this is regular user activity or possibly scrapers. |
|
I was running the same commit and couldn’t view any pages, either, on a residential IP (which is why I’m not comfortable with sharing it). I was able to get around it by taking my instance offline, requesting an IP unban from Reddit, and bringing it back up. I'm not sure if this is at all helpful information. |
|
That's very helpful. How did you request an unban? And did you have the same IP the whole time (during the ban and after it worked again)? Sometimes residential IPs cycle and change normally. |
|
When I got IP banned, I received a page that looked like this or this when trying to access from my browser. Based on my previous emails, the link to contact support led me to [email protected], who then pointed me to fill out this form. I have had the same IP the entire time. |
So I set up Redlib on one of my homelab machines which uses my residential IP address. It worked for a few hours, then I noticed I get the same error. My residential IP ASN is COMCAST-7922. I'm not sure if this is classified as a "commercial ASN", but if it's not, then hopefully this quells your concern. |
|
Weird. After only a few hours, it should have only requested one token total and presumably not had enough traffic (more than 99 reqs in a 5 minute period) to require a new token rollover. That's really surprising that's all it took to get your IP under their watchful eye. I'm going to think a bit on how to proceed here. I don't know if it makes sense to attempt to identify a single factor that causes this policy ban, at least via trial and error, for a few reasons:
Maybe I need to take a look at the auth flow on a current app since it's been a few months. Not sure if anything that has changed should really cause this, since people who haven't used the app in a few months should still be able to use the app without being IP banned. |
|
OK, right now I can access Redlib from my home IP instance. My public-facing instance is still error. |
|
Seeing the same issue again for some reason, I'm self-hosting it and the only one interacting with the instance (single user) |
|
Getting this as well on my private (single user) self-hosted instance. Issue started at 2:25 PM Eastern Time. I can access the website just fine
|
|
|
I've been self-hosting a private teddit instance for over a year that was blocked this morning, so the issue appears to be widespread FYI. Gave a shot @ redlib just now:
|
|
Got the error since a few hours. Getting on reddit without redlib works fine from same ip. TRACE log: |
|
Same here since about an hour. Private Instance, only used by me. Reddit probably changed something, i restarted my modem to get a new IP to be sure, but still not working, i dont think its ip bans. |
|
same here, cant view any pages but only on SOME public instances |
|
Yeah this is a widespread thing now. This issue was intended to be for actual IP bans (where it would work any other IP except one) but now there's just actually a server side change. Triaging now (3x in the last 2 months. sigh). #324 for the incoming flood of outages 😄 |
|
Given that there seem to be IP-unrelated issues also going on right now, it might also be worth adding a step to try For the other questions:
This was a public instance, with somewhere around 200k requests/day. |
|
Yesterday my instance was working, today not anymore.. Trying to curl reddit.com: The instance seems to be completely blocked.
|
|
I just want to add that I incur in the same problem. Is there a way to specify a proxy server (or even better something like gluetun) for redlib? I am using the standalone built binary |
|
It's not an IP ban issue, it's a widespread problem affecting every Redlib user. Proxy won't help. #324 |
I can confirm that. Just tried redlib from home (so residential ip) and still doesn't work. Seems that they're blocking redlib specifically :( |
|
I use Redlib exclusively from a residential IP, and my instance is also showing this error. |
|
Chiming in to say I'm getting this issue on a residential instance as well. |
|
for all commenting about the JSON error; that's a different issue #324 which is now fixed. Update here if you're experiencing the same error AFTER pulling the LATEST docker image. |
|
I can confirm updating to the recently pushed version works for me too. Specifically, from My step by step solution (I use redlib as a binary for a systemd service): |
|
I have the latest version on my server but I get these errors always: Attaching to redlib So, the container doesn't even start correctly and therfore the site shows nothing but a HTTP ERROR 502 |
|
Okay it seems I was wrong about direct requests to Also just to echo everyone else: thank you so much @sigaloid for all your hard work 😄 |
wait, that's huge! You mean that Redlib works even though in the browser doesn't? I think that means that Redlib no longer triggers the IP bans directly (or are being blocked by them), it's just a residual ban that will eventually expire. |
|
That's what I'm seeing, yes. Specifically:
But to be clear, it seems to me that Redlib on this host was never affected by the IP ban in the first place -- I just didn't realize that it could be blocked when requesting @maxysoft mentioned that their instance host has never been able to curl |
|
My rpi4 at home works again with the latest arm fix. (public ip from home) Public Server hosted at Hetzner (ARM) with latest version still does not work. The docker container doesn't even start at all. redlib | Starting Redlib... |
Confirming this here as well. If I route redlib with the latest commit through a VPN with a blocked IP it still works. |
|
Everything is working for me too on my private residential instance on the latest image. 👍 |
|
Might be helpful for some, use Cloudflare WARP to proxy the redlib instance: |
|
Okay, it's come to my attention that there were some deeper server-side changes that are causing these IP bans. This has caused some serious issues for the popular instances as there's no rigorous rate limit bypass mechanism. The main issue is this rate limiting effect so popular instances are hit hard by them. Unfortunately it looks like a lot of work is necessary to get around this restriction. It may be radio silence (and current status quo - local instances will probably work, but instances that reach rate limit (100 every 10 minutes) may get constant errors) until I can get a real overhauled solution up and running. I can't put a timeline on it, unfortunately. It will be possible that public instances will work, while local instances built from the repo's head won't bypass the rate limiting. Once I'm more sure of its stability, this new version will of course be published here, but only once it's in a place where I'm actually sure it's resilient to the issues it is facing now. |
|
Sheesh. They really don't want us using this. As always, thanks for your continued efforts. |
|
Issue turned out to be a lot easier to fix than my initial thought... 😨 oops. Wanted to ensure that's all that went wrong before pushing it. We should be at a point where IP bans aren't correlated to IP's. If it still is, it's probably a unique case with your IP and it shouldn't be as widespread. If I get tons of reports about it, I can open a new issue. Public instance operators, please update! |
|
Works fine so far on my public instance. Which did not work at all before, only from my homelab it worked. Thanks for the hard work and effort. Appreciate it. Users gonna be happy, I guess. |
|
Unfortunately it looks like Reddit has blocked my NYC-1 instance again - unsure if it's an IP block or what, but CAL-1 also occasionally gets it. |
This comment has been minimized.
This comment has been minimized.
Right now (11/18) there's breakages. Comment in #324. This is reserved for IP bans specifically.
Update (11/19): #324 is solved. Please do not comment here about a JSON error unless you are certain that you're on the latest version; the widespread outage was unrelated to the issue of IP bans.
There is a number of reports of continued JSON errors despite the fact that most instance operators running on clean IPs do not get these. If you are impacted by this, please ensure you can reproduce this on certain IPs by confirming you can visit the Redlib home page on one IP (perhaps try your home IP, VPN to another place, etc) but cannot on the IP in question. Then, comment or email (ipban @ my domain, linked in profile) the following info:
Reminder: Do not comment on this post if you're getting errors, unless you've CONFIRMED it works on some IP that isn't yours.
The text was updated successfully, but these errors were encountered: