Hitting Max APache Connections - Am I Being Hacked? [closed] - php

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 9 years ago.
Improve this question
There is some very strange activity happening on my server today. I am hitting Max Apache connections but cannot find anything that could be causing it (I don't think I am being DOS attacked or anything).
I checked my Apache logs and found some weird things.
First:
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] --2013-08-13 09:41:13-- http://heatinasnap.net/gs.txt, referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] Resolving heatinasnap.net... 173.254.28.65, referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] Connecting to heatinasnap.net|173.254.28.65|:80... connected., referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] HTTP request sent, awaiting response... 404 Not Found, referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] 2013-08-13 09:41:13 ERROR 404: Not Found., referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:13 2013] [error] [client 85.76.3.157] , referer: http://example.net/forum/index.php
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] --2013-08-13 09:41:31-- http://heatinasnap.net/gs.txt, referer: http://example.net/members
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] Resolving heatinasnap.net... 173.254.28.65, referer: http://example.net/members
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] Connecting to heatinasnap.net|173.254.28.65|:80... connected., referer: http://example.net/members
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] HTTP request sent, awaiting response... 404 Not Found, referer: http://example.net/members
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] 2013-08-13 09:41:31 ERROR 404: Not Found., referer: http://example.net/members
[Tue Aug 13 09:41:31 2013] [error] [client 112.198.64.88] , referer: http://example.net/members
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] --2013-08-13 09:41:33-- http://heatinasnap.net/gs.txt, referer: http://example.net/forum/viewtopic.php?f=9&t=6747
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] Resolving heatinasnap.net... 173.254.28.65, referer: http://example.net/forum/viewtopic.php?f=9&t=6747
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] Connecting to heatinasnap.net|173.254.28.65|:80... connected., referer: http://example.net/forum/viewtopic.php?f=9&t=6747
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] HTTP request sent, awaiting response... 404 Not Found, referer: http://example.net/forum/viewtopic.php?f=9&t=6747
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] 2013-08-13 09:41:33 ERROR 404: Not Found., referer: http://example.net/forum/viewtopic.php?f=9&t=6747
[Tue Aug 13 09:41:33 2013] [error] [client 141.138.54.172] , referer: http://example.net/forum/viewtopic.php?f=9&t=674
I have no idea what heatinasnap.net is (never heard of it).
And second, some sort of vulnerability scanner:
[Tue Aug 13 09:41:40 2013] [error] [client 220.248.145.30] ModSecurity: Access denied with code 406 (phase 2). Match of "rx ^apache.*perl" against "REQUEST_HEADERS:User-Agent" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "55"] [id "990011"] [msg "Request Indicates an automated program explored the site"] [severity "NOTICE"] [hostname "www.mysite.net"] [uri "/"] [unique_id "UgpFpK339QIAAFT1Y2MAAAAC"]
[Tue Aug 13 09:41:41 2013] [error] [client 220.248.145.30] ModSecurity: Access denied with code 406 (phase 2). Match of "rx ^apache.*perl" against "REQUEST_HEADERS:User-Agent" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "55"] [id "990011"] [msg "Request Indicates an automated program explored the site"] [severity "NOTICE"] [hostname "mysite.net"] [uri "/406.shtml"] [unique_id "UgpFpa339QIAAGfpU5MAAAUD"]
[Tue Aug 13 09:41:41 2013] [error] [client 220.248.145.30] File does not exist: /home/hellohel/public_html/406.shtm
Here is my current apache status:
CPU Usage: u147.51 s128.44 cu2247.28 cs0 - 146% CPU load
147 requests/sec - 2.3 MB/second - 16.4 kB/request
512 requests currently being processed, 0 idle workers
I did not see any MaxClient errors in Apache though. There is definitely something weird going on...can anyone provide some insight?
Update:
The cause of the apache hitting max-clients turned out to be a slowloris DOS attack, which was fixed with the apache Mod_Antiloris. Install instructions here:
http://www.hostingdiscussion.com/hardware-server-configuration/27399-installing-mod_antiloris-mitigate-slowloris-dos-attack.html
Update2:
I am not sure if it was luck or not, but the slowloris thing just solved it for a few minutes. It went back to 512 (max) connections shortly after. I am seeing some very high CPU load on simple scripts so I am wondering if it has something to do with handling large log files. One is just a css file taking up `24.66 CPU`. Check out just a few processes:
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 31154 0/45/45 R 23.85 3 1 0.0 0.47 0.47 ? ? ..reading..
0-0 31154 0/36/36 _ 24.66 0 1 0.0 0.43 0.43 81.152.251.175 mysite.net GET /css/dwn.css HTTP/1.1
0-0 31154 0/33/33 R 23.92 2 179 0.0 0.69 0.69 ? ? ..reading..
0-0 31154 0/1/1 W 0.07 119 0 0.0 0.00 0.00 117.102.163.190 mysite.net POST /includes/offers/ajax.php HTTP/1.1
0-0 31154 1/64/64 C 24.74 0 1 26.8 1.85 1.85 24.127.122.188 mysite.net GET /images/soc.png HTTP/1.1
0-0 31154 0/51/51 _ 24.87 0 899 0.0 0.78 0.78 86.111.144.194 mysite.net GET /includes/offers/window.php?file=57860&tooltip=true HTTP/1.
0-0 31154 0/18/18 R 11.00 77 1 0.0 0.27 0.27 ? ? ..reading..

Answer 1

It looks as though your site is opening remote files because those messages indicate that your Apache server is performing a look-up through DNS.
To look for bad code
You would need to figure out what method they're using to access the box. Then look into that code and try to find something out of the ordinary. They will typically use things like exec() and base64_decode() to hide the code, then you can grep for those. Also grep for things like fopen(), fread(), file_get_contents(), and even curl_init(). If you find any of these scripts in places where you're not expecting them, then that will be your exploit.
You should be able to look for outbound traffic on the box using something like conntrackd, ntop, argus, bro-ids, and sancp.
Attempt at a quick fix
Go into the php.ini file and check the system configuration settings for allow_url_fopen and allow_url_include. It looks as though someone is trying to get your site to open the txt file from their site (where the payload exists).
If those settings allow remote opening, then that's how they're causing this behavior. Someone more than likely opened a file on your server from their server and caused an exploit.
If they have code on your box, then you will need to wipe out the contents of the box and update the code from one of your back-ups once the php.ini file has been fixed. Otherwise they could try to change settings on the front-end with their already hosted code using things like ini_set.
Making no changes to the code or settings and restoring from backup will not prevent the behavior. Additionally, you can use something like IPtables to block all outbound requests to heatinasnap.net and its resolved IP [173.254.28.65].
If you're using something like file_get_contents it will be disabled by making this change. cURL on the other hand uses its own libraries and is not going to be affected by the change. Any code on the server can still use cURL though, (even if it's not yours).
Update for DOS Attack
Since you think this is a DOS instead, you might try using mod_reqtimeout. Good settings would be:
RequestReadTimeout header=10 body=30