Compiling Ok in AlmaLinux 8.10, but not working #336
Comments
|
In all times, I have recompiled libmodsecurity.so and nginx before compile ngx_http_modsecurity_module, and in all times the result is only head of the pages are supressed. No problems are shown in the logs. |
|
Hi @decioaalves,
Sorry, what is "this" version? Could you explain it exactly?
Here too: what are the "previous" and "this" versions? |
I mean the current version of the 'ModSecurity-nginx' module, as you get when you clone with git. After compile, it generates a 'ngx_http_modsecurity_module.so' library normally, with no warnings.
The 'this' version, is the current version. The connector was updated two months ago. The 'previous' version, was a version earlier version, from June, 2024, I was running. I could to make a debug trace, and I can post it, if you want.. Thank you. |
You mean about PR #334? That was committed in 2 months ago (19th of Nov, 2024), but was merged few days ago. To clarify and reproduce this issue we need:
We need to reproduce the issue. |
|
Commit id of 'ModSecurity-nginx' module: 'Older version' (working good version):
'current version' (NOT working good version):
System: AlmaLinux 8.10 (Cerulean Leopard) Kernels tested:
ModSecurity installed pre-requisites
ModSecurity3 commit id installed in hardware #1:
ModSecurity3 commit id installed in machine #2 and #3:
ModSecurity3 installation
Nginx must be customized, due to the need for compatibility with the CWP WebPanel that is installed
Installed with fresh OWASP CRS 4.3.0, original steps, unchanged
nginx -V
cwpsrv -V (original binary built from CWP Web Panel, that was tested too, and didn't work):
changes to main configuration file from nginx/cwp: /usr/local/cwpsrv/conf/cwpsrv.conf :
Content from file /usr/local/cwpsrv/conf/mod-http-modsecurity.conf:
Original installation from OWASP CRS 4.3.0:
Synthoms:
Solution:
(I'm having trouble making the request with curl, with a password protected page via POST method, as soon as I can, I'll post it here.) |
|
Hi @decioaalves, thanks, I have these ones: so the newest commits from both repository. I also installed Nginx, but during some tests yesterday I didn't get any problem. The curl request would be important, what's the request - and if you turn off the module, then please show us the response from the backend. If it contains any sensitive info, please send that to [email protected].
I see, but I need to know the configuration to reproduce the issue. There was a PR (#334) which probably can occur this issue, but without details I can't do anything. Under the mentioned PR you can see some comments, a customer who uses heavily this module confirms that this patch works as well. But of course, we need to find a solution - if we see what's the problem. |
|
Hi, |
|
thanks - do you also use the connector with last commit (fb678c5) with Bullseye?
Yes, definitely.
Sorry, I don't use Docker just very rarely, it would be huge help if you could provide the request (with payload) and an example response (that I can set as content). And many thanks for these! |
|
I was unable to create an example using curl, but I was able to generate two *.har files using Chrome, one with the correct page returned, and the other broken. So I then upgraded to version v1.0.3-27-gfb678c5, and generated the second file, 'request-broken.har', which shows these same HTTP calls, but results in the second page being broken, being returned at line 954. Strangely, after doing a scan through the modsecurity audit logs, it shows the second page as if it had been returned to me in its entirety, rather than broken in half. But I had already tested it in two different browsers, Chrome, Firefox and links(text only), and the result is always the same. |
|
thanks - I hope this will help me to investigate the issue. Let me check that in next few days. Cc @Dr-Lazarus-V2 - if you able to check, please help me in this. |
|
Have the same problem, this appeared after the commit fb678c5. |
|
Hi all. I breaked my brains with debugging same issue. |
|
a temporary solution is to set use an older commit of the connector mentioned by @decioaalves, the link ef64996 should be a working version without this issue. |
You mean if the response is bigger than 32 kB, then it will be cut off? |
We came to this through experiments. We have some static on the site, files larger than 32 kb open on the client without the beginning of the file. Although the size is shown the same as on the server. |
Could you try it with Now I tried on my environment (with same versions) but everything was fine: the file: Note that there is no proxy, Nginx serves the files directly. |
Yes, I tried both via curl and wget, the result is the same. Can you check the contents of the file? The size is also fine for me when downloading. This is an example of a downloaded file without mod_sec 
As you can see, the beginning of the file was cut off and some information was added at the end. |
|
Oh, now I see. I tested with a 46kB length JS file, and I got exactly the same result as you (I mean the file is cut off). Thanks.
|
|
Hi @decioaalves, @DaveeHorvath, @promiteyka and @makar112233, I can confirm that ef64996 is the last commit where the connector works as we expected. But I'm afraid the broken commit is 62639fa (part of #326), not fb678c5 . Could you confirm guys this? It's easy to reproduce: if you check out the broken commit, the file cut effect is still reproducible as with last commit. @g00g1 ping, it seems like your PR broke the module's normal behavior. |
|
To @decioaalves, @DaveeHorvath, @promiteyka and @makar112233: I think I found the cause of the mentioned problem: in the current code commit @62639fa2 added this line: chain->buf->pos = chain->buf->last;(in line 180). There is a quick fix but before I merge that I want to review the whole patch again, and I need some help to test that it works as well. So please if anyone who's able to test this, please help me. A quick help: Please try the module with this patch with:
@Dr-Lazarus-V2 could you help us in this (run tests)? @g00g1 could you help me to explain why is that value giving in the mentioned line? |
|
@airween - Thanks for taking care of this issue. I've tested the above mentioned issue336fix as part of a custom docker build and it compiles, but any request to a modsec enabled nginx server results in a client-side timeout after a long wait period. I suspect the process is stuck. |
Many thanks for this reply. It's very interesting, this is what I definitely can't reproduce. Note, that I don't use Docker (very rarely) - the problem is I don't know how is it possible to debug inside of Docker. |
|
@airween However, when I clone the master branch via Downloading and unzipping the master.zip file confirms that the content is different to the cloned repo. Shouldn't the zip archive be the same as the current master? |
|
here:
you mentioned the repository source as But here:
you downloaded from my GH repository. If you take a look at my repository, you can see:
my master branch is behind from the official repository's master.
You mean the issue that explained in this ticket (namely the downloaded files which bigger than 32 kB cut off)? That's the "expected" now.
It must be the same - if you use the same source. Now I downloaded the zip and cloned the git, and got the same content (after I removed the $ diff -rN ModSecurity-nginx-zip ModSecurity-nginx-git
<EMTPY>Could you try to change to the mentioned branch after you cloned my repository? git checkout -b issue336fix
git pull origin issue336fixthen run |
|
Ok, apologies for the confusion I caused. I've just tested again three things:
(repo is fast-forwarded) ... I did some more testing. Not every request results in a timeout. Similarly with a Nextcloud instance. Some URLs work, a simple curl works as well, some absolutely don't. By the way, I'm running a modsec disabled nginx instance in parallel with otherwise the same config to rule out other issues. I'll perform some more tests tomorrow to see if I can pinpoint the issue further if that helps. |
|
The 62639fa is broken for me, with *.html, *.json and *.xml above ~32Kb it breaks. However, with this version 'issue336.fix', I was able to test *.json, *.html files, up to ~64Kb Mb, it worked, and the file is served immediately. I cloned with:
However, for files larger than ~64Kb, the files are not immediately served, and curl gives up after 60 seconds. The server access log (nothing in the error log) records:
However, with Chrome navigator, although the 408 (Server Timeout) error is recorded in the server access logs after 60 seconds, seems Chrome request the file, and sometimes the file is served (as recorded in modsec_audit.log file) the first 65535 bytes of the file after more 60 seconds, even though and then, after another 180 seconds, the keep-alive connection is closed in this browser (possibly browser configuration). Sometimes, in the error logs, is recorded only:
|
I tried to build with |
|
Just for your info, I tried the issue336fix branch and got wget https://waf-update.*******************.com/lczw.poster.png --2025-01-18 08:48:39-- (essai : 2) https://waf-update.*******************.com/lczw.poster.png --2025-01-18 08:49:41-- (essai : 3) https://waf-update.*******************.com/lczw.poster.png no data coming it seems |
Yes, I tryed too at this moment, and I can confirm, this version it's funcional and working. |
|
For the past two weeks, I’ve been deploying a new nginx+modsecurity setup every day. However, strangely, this week it suddenly stopped working without any error logs. I thought I was just being careless and started doubting myself, which really got me down… Until today, I found this post. Based on the timing, I believe the issue could be related to the merge fb678c5. I tested a WordPress site by cloning the repository with the following command: This commit seemed to make some improvements, but it still didn’t work as expected. However, reverting to commit ef64996 seems to fully resolve the issue. |
|
Hi! I retried the patch because I saw in the comment that for some people it was working. But for me, it previously dit not work, on the waf I created 2 files on client : curl -v https://waf-update.DEBUG.com/65536bytes.txt AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA* Connection #0 to host waf-update.DEBUG.com left intact It did download correctly. the 16kb + 1 byte file did not
|
|
With the main branch ( one that does not work ) https://github.com/owasp-modsecurity/ModSecurity-nginx On the WAF, I created a file htdocs# cat gen.sh for i in "${!letters[@]}"; do it is a sequence of 8kb of AAA BBB CCC ... ZZZ md5sum abc.txt -------- NOW ON CLIENT SIDE ------- % % md5 abc.txt since the file is only letter it can be edited without messing the terminal, so when I cat the file abc.txt into the terminal but the file sizes match |
No worries :)
yep,
yes, that's the "old" version, probably without this bug.
yes, I think it depends on CT (Content-Type), see ModSecurity's config. If you could check the types, that would be help.
yes, that's the normal behavior.
Thanks! |
Hmm, that's very interesting.
What happens if you revert only the last commit and / or check out to 62639fa? If it works that confirms the wrong patch is the last one. If not, then probably we should check the last two commits. (Based on your sentence above: I’ve been deploying a new nginx+modsecurity setup every day. However, strangely, this week it suddenly stopped working without any error logs - I mean you mentioned this week, and I merged only the last commit this week. If you haven't faced this issue before, then it means the last commit is wrong.) |
What about other types? What does mean "up to ~64Kb Mb" - have you checked with MB content? |
Thanks. What kind of file types have you checked? I mean |
Thanks - I can confirm, for eg. |
Unfortunately just partially working. We should figure out which files and sizes are timed out. |
Sorry, was a typo. When I did cloned with: ... and the sizes reached up to 512Mb, and everything was ok.
Ok, since you didn't announce it, I actually thought that it would still be a work in progress. |
AlmaLinux 9.5 + nginx-1.26.2 fb678c5 -> Failed However, I am quite certain that around January 3rd, I deployed the latest version of ModSecurity-nginx without encountering any issues. Based on the timeline, it should have been the version ae30826. |
|
Hello everyone, I have the issue when i tried to deploy last week, before that i didn't have the issue. I tried the path ef64996 and 2e4cbf7 and that not work for me i did: i did right ? ======================== After that, i tried to use another "ngx_http_modsecurity_module.so" module from one of my server where is working i'm on the last commit of the main branch and my module date is October
I dont know how work ModSecurity and what is a different between working module and the new module but it's work for me |
Hmm... this is interesting. Could you show me details about how did you check? I mean file (typ), size, used settings, rule set, etc... Now I tried this commit and it worked for me.
Thanks, I'll check that too. |
To facilitate better testing, I’ve set up a temporary virtual server (located in Virginia, USA) and sent you the login details via email ([email protected], 'ModSecurity-nginx Debug Server – ISSUE336'). Feel free to modify and debug as needed. Server Details
Website Information
Currently, the site is running commit fb678c5, and the issue is visible as soon as the site is accessed. Refreshing the page results in different outputs each time. ModSecurity Configuration in Nginxmodsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;Further TestingTo test with a different commit, simply switch the commit and recompile Nginx (a script is available, and the process takes about 3 minutes): cd /usr/local/ModSecurity-nginx
git checkout ef64996
~/install-nginx.shI'm not sure how to create a more comprehensive test case. If possible, please take a look directly on the server. I’d really appreciate it! |
|
Since I don't understand the code, I can't give more help. |
Not yet, it's under investigation.
I don't know. A workaround is expected on next week at least.
Thank you. |
|
Here are some news around this issue: I wasn't able to work on this in last few days, but before that I tried to explore what's the root cause. Unfortunately I couldn't find that. I try it again in next few days, but if I can finish that, I can solve this issue this way:
If anyone has an opinion/better idea, please let me know. |
|
Oh, I recently rebuilt modsecurity with nginx 1.27.3, and it seems there was some change in the modsec source code that caused me to have the same problem. My js files have also been truncated, losing their leading parts. Only when I configured proxy_buffer_size to be 128k and proxy_buffers to be 8 16m was this problem resolved, my js files were loaded normally. |
|
In addition, I am also having a problem with my website failing to decode the website content: Another of my websites using websocket had the following error. I used modsecurity off on that path to stop this problem: I think it's related to this issue so I'll provide more information here in the hope of being of assistance. |
|
Hi @SonNgo2211,
thanks for the suggested workarounds. |
I managed to compile it ok on AlmaLinux 8.10, a dozen times, in different ways, with 7 different kernels, on three different hardware, and in none of them did I have any problems building the new version of this connector. However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.
But compiling the previous version of this connector, it works normally.
Is this problem happening only to me, or has anyone else noticed this problem?
The text was updated successfully, but these errors were encountered: