[WordPress] Memory Exhaustion

[dagohbah]

Platform

AWS

bndiagnostic ID know more about bndiagnostic ID

a2ba5599-e678-9a76-a4b1-9d93dd6fe5f5

bndiagnostic output

===== Begin of bndiagnostic tool output =====

✓ Processes: No issues found
✓ Mariadb: No issues found
? Connectivity: Found possible issues
✓ Php: No issues found
? Apache: Found possible issues
✓ Resources: No issues found
? Wordpress: Found possible issues

[Connectivity]

Server ports 22, 80 and/or 443 are not publicly accessible. Please check the
following guide to open server ports for remote access:

https://docs.bitnami.com/general/faq/administration/use-firewall/

[Apache]

Press [Enter] to continue:
Found recent error or warning messages in the Apache error log.

[Mon Jul 29 19:05:33.623631 2024] [access_compat:error] [pid 13987:tid 
139920915990208] [client **ip_address**:30556] AH01797: client denied by server 
configuration: /opt/bitnami/wordpress/messages
 [Mon Jul 29 19:05:37.560023 2024] [access_compat:error] [pid 13473:tid 
139920890812096] [client **ip_address**:2196] AH01797: client denied by server 
configuration: /opt/bitnami/wordpress/messages
 [Mon Jul 29 19:05:41.518025 2024] [access_compat:error] [pid 13473:tid 
139920882419392] [client **ip_address**:41336] AH01797: client denied by server 
configuration: /opt/bitnami/wordpress/messages

Please check the following guide to troubleshoot server issues:

https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-erro
rs-apache/

[Wordpress]

Some files don't have the expected permissions Please check the following guide:

https://docs.bitnami.com/general/apps/wordpress/administration/understand-
Press [Enter] to continue:
file-permissions/

===== End of bndiagnostic tool output =====

bndiagnostic was not useful. Could you please tell us why?

N/A

Describe your issue as much as you can

Hello,

We've been having major memory exhaustion type of issues over the past few weeks that I've been trying to troubleshoot. I've thought it could be something like the following:

• DOS Attacks
• DDoS Attacks
• Drive/Volume Filling Up
• Lack of a swap file
• Etc.

I've been having to babysit our website and restart the web services at least once a day. When the issue occurs, there's tons of php-fpm daemons running, and checking the memory with top shows that the memory is basically exhausted overall. Additionally, here's the list of troubleshooting items I've tried to go through (in chronological order):

1. Upgraded our WordFence to the premium version when the issues first occurred
2. Cleared up any space possible on our drive
3. Snapshot/backup the original website, create a new server and restored the snapshot to the server (using AWS Lightsail, and upped the resources to 2 vcpus and 4GB of RAM from 2 vcpus and 2GB of RAM). -> the issue still persisted with this method, so...
4. Next, took a backup with All in one WP migration plugin (without media), backed up the media myself separately, created a brand new instance with the same specs as above, and new public IP. Restored the allinone backup, and re-uploaded all backed up media. Issue still persisted, sadly.
5. Created a 2GB swap file
6. Tried to mess with memory.conf/httpd-medium.conf (gave up on this as I wasn't super sure how to make the calculations)
7. Upped the memory limit in php.ini

I've pursued WordFence support...but they don't really help alleviate DDoS or DOS types of attacks (that's the theory I was trying to trace down with them). I stumbled across many issues/tickets here...one about checking about deny connections, and many regarding the odd behavior of php-fpm/memory related posts.

https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/deny-connections-bots-apache/
#907
#27
#1192
#1470
#1501
#927

I really need help unpacking what I'm seeing, and where to point my efforts to. Do we feel like this is a memory.conf/httpd-medium.conf type of issue, or could this be something else?

Thanks in advance for the help!

[dagohbah]

Hey there - just following back up on this. Thanks!

[gongomgra]

Hi @dagohbah,

Thanks for using Bitnami and sorry for the delay on getting back to you. According to the bndiagnostic bundle you shared, there are a few IP address making a lot of requests to your website

-----------------------------------
Check performance issues: Count number of requests for the 10 most active IP addresses in the last 100.000 requests
-----------------------------------
Running: tail -n 100000 access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10 | awk '{print $1}'                                                                                                                                                                       In: /opt/bitnami/apache2/logs/

Output:

5285
5226
4909
3508
1428
1384
1373
1361
1291
950

We recommend you to check our guide about bots and how to block them. The thing is that every connection to your website requires consuming new resources to perform database and website queries, which may lead to the memory exhaustion issues you are facing. I also see in the information you shared that MariaDB is the process consuming more memory on your server. That can be related to bots connections (DDoS attacks).

It can also happen that one of your plugins or custom themes is making multiple requests to your database (like storing temporary data). We recommend you to check if any of your plugins is known to cause these kinds of issues in the official support forums for WordPress.

https://wordpress.org/support/forums/

We also recommend you to take a look at our performance troubleshooting guide linked below

https://docs.bitnami.com/aws/faq/troubleshooting/troubleshoot-server-performance/

[dagohbah]

Hey @gongomgra!

For what it's worth...I did go through and block some of those connections like the doc describes. I also did a lookup on a lot of these - appear to all be AWS crawler types of bots from what I can tell from research online. So, I'm sure they are "OK" and not malicious. I wonder if blocking them is the best thing, but if it improves performance and stops the issues, maybe it's for the best. I did go through the performance doc as well before posting here, too. Plugins/themes...haven't really dug into that at this time.

Since my last post here my server has stabilized. I'll keep an eye on this for a bit and will close out the issue when I feel like it's safe to do so.

[dagohbah]

Hey - just following up. No issues since I've blocked some of the suspected AWS Bot IP's...so for anyone coming across this in the future, follow the bots and how to block them doc first, give it a day or so and see if the behavior improves before going through all the other troubleshooting things I did! Thank you @gongomgra! ❤️

[gongomgra]

@dagohbah happy to know your issue got solved after blocking some IP addresses that were performing lot of requests! 😄 I'm closing this ticket as solved.