Recently, I had a single IP call the same page 15,000 times in quick succession which used up a lot of server resources (with resulting warning email from Host service). I am on a shared host so can't load new modules and therefore have no real options to truly limit bandwidth to an IP.
So, I'm trying to figure out how I can use the least amount of resources in spotting an offending IP and redirecting it to a 403 Forbidden page. I am already checking for common hacks and using Project HoneyPot, but to do this for each of the 15,000 page hits is not efficient (and, like this one, it doesn't catch them all).
I currently log access to each page to a mysql table called visitors. I can imagine a couple of ways to go about this:
Option 1: Using MySql:
1) Query the visitors table for the number of hits from the IP over the last 10 seconds.
2) If greater than a certain number (15?), flag the last entry in visitors as blocked for this IP.
3) With each subsequent page request, the query on the visitors table will show the ip as blocked and I can then redirect to 403 Forbidden page.
Option 2: Modifying an Include File on the fly which contains blacklisted IPs:
1) Include a file which returns an array of blacklisted IPs
2) If the current IP is not on the list, query the visitors table as in Option 1 to see if the number of hits from the IP over the last 10 seconds is greater than a certain number.
3) If the IP is offending, modify the include file to include this IP address as shown below.
In essence, my question is: Which uses more resources (x 15,000): a query to the database, or the code below which uses include to read a file and then array_search(), OR is there a better way to do this?
<?php
$ip = $_SERVER['REMOTE_ADDR'];
$filename='__blacklist.php';
if (file_exists($filename)){
// get array of excluded ip addresses
$array = (include $filename);
// if current address is in list, send to 403 forbidden page
var_dump($array);
if (is_array($array) && array_search($ip, $array) !== false){
blockAccess("Stop Bugging Me!");
}
} else {
echo "$filename does not exist";
}
// evaluate some condition which if true will cause IP to be added to blacklist - this will be a query to a MySql table determining number of hits to the site over a period of time like the last 10 seconds.
if (TRUE){
blockip($ip);
}
// debug - let's see what is blocked
// $array = (include $filename);
// var_dump($array);
// add the ip to the blacklist
function blockip($ip){
$filename='__blacklist.php';
if (! file_exists($filename)){
// create the include file
$handle = fopen($filename, "w+");
// write beginning of file - 111.111.111.111 is a placeholder so all new ips can be added
fwrite($handle, '<?php return array("111.111.111.111"');
} else {
// let's block the current IP
$handle = fopen($filename, 'r+');
// Don't use filesize() on files that may be accessed and updated by parallel processes or threads
// (as the filesize() return value is maintained in a cache).
// use fseek & ftell instead
fseek($handle, 0 ,SEEK_END);
$filesize = ftell($handle);
if ($filesize > 20){
// remove ); from end of file so new ip can be added
ftruncate($handle, $filesize-2);
// go to end of file
fseek($handle, 0 ,SEEK_END);
} else {
// invalid file size - truncate file
$handle = fopen($filename, "w+");
// write beginning of file with a placeholder so a new ip can be added
fwrite($handle, '<?php return array("111.111.111.111"');
}
}
//add new ip and closing of array
fwrite($handle, "," . PHP_EOL . '"' . $ip . '");');
fclose($handle);
}
function blockAccess($message) {
header("HTTP/1.1 403 Forbidden");
echo "<!DOCTYPE html>\n<html>\n<head>\n<meta charset='UTF-8' />\n<title>403 Forbidden</title>\n</head>\n<body>\n" .
"<h1>Forbidden</h1><p>You don't have access to this page.</p>" .
"\n</body>\n</html>";
die();
}
?>
There are a lot of points to address here.
Query vs Include
This is basically going to come down to the server its hosted on. Shared hosting is not known for good IO and you will have to test this. This also depends on how many IPs you will be blacklisting.
Blocking Malicious IPs
Ideally you don't want malicious users to hit PHP once you determine that they are malicious. The only real way to do that in a shared environment on apache is to block them from htaccess. Its not recommended but it is possible to modify htaccess from PHP.
Order Deny,Allow
Deny from xxx.xxx.xxx.xxx
Caching
The main concern I have from reading your question is that you seem to not understand the problem. If you are receiving 15,000 hits in a timeframe of seconds, you should not have 15,000 database connections and you should not have all of those requests hitting PHP. You need to cache these requests. If this is happening your system is fundamentally flawed. It should not be physically possible for 1 user on home internet to spike your resource usage that much.
Shared hosting is a bad idea
I suggest in your situation to get a VPS or something else that will allow you to use a reverse proxy and employ far more caching/blacklisting/resource monitoring.
Related
need to forward all Tor users away from my page, with checking ip in tor lists.
Single check was working with ipv4 but not working with ipv6 and multiple list checking.
Can't understand where i get error.
code:
$torip = file("https://check.torproject.org/torbulkexitlist", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$torexits = file("https://lists.fissionrelays.net/tor/exits-ipv6.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$tornode = file("https://lists.fissionrelays.net/tor/relays-ipv6.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$client_ip = $_SERVER['REMOTE_ADDR'];
if (in_array($client_ip, $torip)){
header('Location: https://www.google.com/');
}
if (in_array($client_ip, $tornode)){
header('Location: https://www.google.com/');
}
if (in_array($client_ip, $torexits)){
header('Location: https://www.google.com/');
}
was trying different way's like
if(in_array($client_ip, $torip) or in_array($client_ip, $tornode) or in_array($client_ip, $torexits))
and if ... elseif .. elseif
same can get inside via tor with ip that is in list and can't understand where is the problem.
Thank You to All for help.
UDP:
code part
$tornode = file("https://lists.fissionrelays.net/tor/relays-ipv6.txt", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$client_ip = $_SERVER['REMOTE_ADDR'];
if (in_array($client_ip, $tornode)){
header('Location: https://www.google.com/');
die();
}
is working 100% - question - how to add other list in checking in the right way?
A few things here...
I hope you aren't downloading those lists every time someone visits your page. You should be caching the results of the list downloads for a short time rather than constantly downloading.
The only fissionrelays list you need is exits.txt. As outlined at https://fissionrelays.net/lists, exits.txt contains IPv4 & IPv6 exit nodes. Download that instead of exits-ipv6.txt and relays-ipv6.txt.
It is incorrect to block Tor relays that are not exits. Hits from a relay IP is not Tor traffic. For example, I run a guard relay at home that does not allow exit traffic. Its IP appears in the relay list, but it does not permit any Tor exit traffic so any hits from this IP is not coming from Tor.
If you want to use multiple lists, that's fine. I would suggest the following steps to meet your needs:
1. Download & combine all lists every 10+ minutes
2. De-duplicate and sort the list
3. Save to a file
4. Write a function to search the cached file.
For 1-3, you could use https://gitlab.com/fissionrelays/lists/-/blob/master/tor.php which is a downloader provided by fission relays to download their lists. It sorts as well.
IF your lists are sorted correctly, you can binary search the list for better results, but this is more advanced and not necessary.
Hint, when downloading lists, don't use file() to download as arrays. Use file_get_contents() instead, and append each list onto the other. Once all lists are downloaded and combined, process them into an array (skipping dupes), and then sort the list.
Here's a binary search function you can use to search the sorted list quicker.
/**
* Perform binary search of a sorted array.
* Credit: http://php.net/manual/en/function.array-search.php#39115
*
* Tested by [VigilanTor](https://wordpress.org/plugins/vigilantor/) for accuracy and efficiency
*
* #param string $needle String to search for
* #param array $haystack Array to search within
* #return boolean|number false if not found, or index if found
*/
protected function arrayBinarySearch($needle, $haystack)
{
$high = count($haystack);
$low = 0;
while ($high - $low > 1){
$probe = ($high + $low) / 2;
if ($haystack[$probe] < $needle){
$low = $probe;
} else{
$high = $probe;
}
}
if ($high == count($haystack) || $haystack[$high] != $needle) {
return false;
} else {
return $high;
}
}
Additionally, make sure to call exit() after sending a header(Location) redirect since you want to terminate the request and redirect immediately without running additional PHP code.
if (in_array($needle, $haystack)) {
header('Location: https://www.google.com/');
exit; // Redirect now and don't run any further PHP code
}
Lastly, if you want to assume all Tor traffic is "bad" that's fine. Consider an error message instead of silently redirecting traffic away without explanation which is bad user-experience. Many people use Tor for casual browsing, so you're effectively booting these users from your site with no reason given.
We have a Magneto store that setup in 2 web servers behind a load balancer, which will take care of the load and send customers to either server.
I have a Magento module thats allows customers to upload files through the website, which then get saved in to, [web_server_ip]/media/[visitor_id]/file_name.ext
Since there are 2 servers web_server_ip has to be consistent throughout a customer session when they upload files. Means all files from a customer need to be in same directory in the same server that they got connected at first, regardless of which server they'll be sent by the load balancer within a single session. To ensure that I have following function to get the a consistent Server IP for file uploads.
protected function getServerIp() {
if (Mage::registry("dlite_server")!=""){
$serverip = Mage::registry("dlite_server");
}else{
if (substr($_SERVER['SERVER_ADDR'],0,strpos($_SERVER['SERVER_ADDR'], '.'))=='10'){
//Private IP to Public IP conversion
if ($_SERVER['SERVER_ADDR'] == '10.999.99.999'){
$serverip = '50.99.999.999';
}else if ($_SERVER['SERVER_ADDR'] == '10.888.888.888'){
$serverip = '50.888.88.888';
}else{
$serverip = $_SERVER['SERVER_ADDR'];
}
}else{
$serverip = $_SERVER['SERVER_ADDR'];
}
}
Mage::register("dlite_server", $serverip);
//I have used customer session as below to hold the IP first
//Mage::getSingleton('customer/session')->setDliteServerIp($serverip);
return $serverip;
}
The issue I'm facing is, I tried to hold the IP throughout the customer session using Magento's customer/session which failed 20% of the time. And as you can see now I'm trying to use Mage::registry to hold the IP, that fails even more, like 30% of the time.
So I was wondering is there any thing that's consistent in Magento that I can use to serve my purpose?
i would like to record the visitor ips in a file. then after second attempt to visit, i dont want that person to visit that website within 10 days. If he visits again then i want banned message to be produced. i can do this with mysql database but i want to do through file handling. Its urgent
I think following code snippet will be helpfull for you:
define('IP_FILE', 'ip.php');//where to save IP data
define('LIMIT_SECONDS', 10*24*3600); // 10 days
//loading IP data
function loadIPFile() {
if (file_exists(IP_FILE)) {
require_once IP_FILE;
if (isset($ipArray) and is_array($ipArray)) {
return $ipArray;
}
}
return array();
}
//saving IP data
function saveIPFile($ipArray) {
$fp = fopen(IP_FILE, 'wb');
fwrite($fp, '<?php'.PHP_EOL);
fwrite($fp, '$ipArray = '.var_export($ipArray, true).';'.PHP_EOL);
fwrite($fp, '?>');
fclose($fp);
}
$ipArray = loadIPFile();//load info into array
$ip = $_SERVER['REMOTE_ADDR'];//visitor ip
//if such ip already exists and 10 days are not behind us redirect visitor
if (isset($ipArray[$ip]) and time() - $ipArray[$ip] < LIMIT_SECONDS) {
header('Location: banned_page.php');
exit;
}
else {
//else record new ip or new time
$ipArray[$ip] = $time;
}
//save IP information
saveIPFile($ipArray);
If you know how to do this with the database, why do you want to do it with a file? A database is basically a file with functionality sitting on top to help you access the data. By using files to directly to store this sort of data you are having to rewrite the minimal access functionality that comes out of the box with MySQL. Why reinvent the wheel?
If for some reason you don't have access to MySQL, then perhaps try SQLite http://php.net/manual/en/book.sqlite.php, gives you a lot of sql type functionality, but based on a file in your local directory rather than sql server.
Hey guys i'm making a website where you submit a server for advertising. When the user goes to the index page of my website it grabs the ip's of all the servers submitted and then tests to see if it is online using fsockopen() like so:
while($row = mysql_fetch_assoc($rs)) {
$ip = $row['ip'];
$info = #fsockopen($ip, 25565, $errno, $errstr, 0.5);
if($info) {
$status = "<div><img width='32px' height='32px'
title='$name is online!' src='images/online.png'/></div>";
$online = true;
} else {
$status = "<div><img width='32px' height='32px'
title='$name is offline!' src='images/offline.png'/></div>";
$online = false;
}
}
}
This way works fine, but the only downside is when you load the site it takes a good 2-4 seconds to start loading the website due to the fsockopen() methods being called. I want to know if there is a better way to do this that will reduce the amount of wait time before the website loads.
Any information will be appreciated, thanks.
Store the online status and last check time in a database, if the last check time is longer than 15 minutes for example, update it. I am pretty sure you don't need to get the status on EVERY pageload? It's the time it takes to connect to each server that slows down the website.
Then again, you would probably wanna move the update process to a cronjob instead of relying on someone visiting your website to update the server statuses.
Looking at your example, I'd make all the $status bits be javascript calls to another php page that checks that individual server.
However, the idea to move the status checks to cron job or use some kind of status caching is very good too. Maybe store statuses in a database only only check the ones that have expired (time limit set by you).
I have a client whose domain seems to be getting hit pretty hard by what appears to be a DDoS. In the logs it's normal looking user agents with random IPs but they're flipping through pages too fast to be human. They also don't appear to be requesting any images. I can't seem to find any pattern and my suspicion is it's a fleet of Windows Zombies.
The clients had issues in the past with SPAM attacks--even had to point MX at Postini to get the 6.7 GB/day of junk to stop server-side.
I want to setup a BOT trap in a directory disallowed by robots.txt... just never attempted anything like this before, hoping someone out there has a creative ideas for trapping BOTs!
EDIT: I already have plenty of ideas for catching one.. it's what to do to it when lands in the trap.
You can set up a PHP script whose URL is explicitly forbidden by robots.txt. In that script, you can pull the source IP of the suspected bot hitting you (via $_SERVER['REMOTE_ADDR']), and then add that IP to a database blacklist table.
Then, in your main app, you can check the source IP, do a lookup for that IP in your blacklist table, and if you find it, throw a 403 page instead. (Perhaps with a message like, "We've detected abuse coming from your IP, if you feel this is in error, contact us at ...")
On the upside, you get automatic blacklisting of bad bots. On the downside, it's not terribly efficient, and it can be dangerous. (One person innocently checking that page out of curiosity can result in the ban of a large swath of users.)
Edit: Alternatively (or additionally, I suppose) you can fairly simply add a GeoIP check to your app, and reject hits based on country of origin.
What you can do is get another box (a kind of sacrificial lamb) not on the same pipe as your main host then have that host a page which redirects to itself (but with a randomized page name in the url). this could get the bot stuck in a infinite loop tieing up the cpu and bandwith on your sacrificial lamb but not on your main box.
I tend to think this is a problem better solved with network security more so than coding, but I see the logic in your approach/question.
There are a number of questions and discussions about this on server fault which may be worthy of investigating.
https://serverfault.com/search?q=block+bots
Well I must say, kinda disappointed--I was hoping for some creative ideas. I did find the ideal solutions here.. http://www.kloth.net/internet/bottrap.php
<html>
<head><title> </title></head>
<body>
<p>There is nothing here to see. So what are you doing here ?</p>
<p>Go home.</p>
<?php
/* whitelist: end processing end exit */
if (preg_match("/10\.22\.33\.44/",$_SERVER['REMOTE_ADDR'])) { exit; }
if (preg_match("Super Tool",$_SERVER['HTTP_USER_AGENT'])) { exit; }
/* end of whitelist */
$badbot = 0;
/* scan the blacklist.dat file for addresses of SPAM robots
to prevent filling it up with duplicates */
$filename = "../blacklist.dat";
$fp = fopen($filename, "r") or die ("Error opening file ... <br>\n");
while ($line = fgets($fp,255)) {
$u = explode(" ",$line);
$u0 = $u[0];
if (preg_match("/$u0/",$_SERVER['REMOTE_ADDR'])) {$badbot++;}
}
fclose($fp);
if ($badbot == 0) { /* we just see a new bad bot not yet listed ! */
/* send a mail to hostmaster */
$tmestamp = time();
$datum = date("Y-m-d (D) H:i:s",$tmestamp);
$from = "badbot-watch#domain.tld";
$to = "hostmaster#domain.tld";
$subject = "domain-tld alert: bad robot";
$msg = "A bad robot hit $_SERVER['REQUEST_URI'] $datum \n";
$msg .= "address is $_SERVER['REMOTE_ADDR'], agent is $_SERVER['HTTP_USER_AGENT']\n";
mail($to, $subject, $msg, "From: $from");
/* append bad bot address data to blacklist log file: */
$fp = fopen($filename,'a+');
fwrite($fp,"$_SERVER['REMOTE_ADDR'] - - [$datum] \"$_SERVER['REQUEST_METHOD'] $_SERVER['REQUEST_URI'] $_SERVER['SERVER_PROTOCOL']\" $_SERVER['HTTP_REFERER'] $_SERVER['HTTP_USER_AGENT']\n");
fclose($fp);
}
?>
</body>
</html>
Then to protect pages throw <?php include($DOCUMENT_ROOT . "/blacklist.php"); ?> on the first line of every page.. blacklist.php contains:
<?php
$badbot = 0;
/* look for the IP address in the blacklist file */
$filename = "../blacklist.dat";
$fp = fopen($filename, "r") or die ("Error opening file ... <br>\n");
while ($line = fgets($fp,255)) {
$u = explode(" ",$line);
$u0 = $u[0];
if (preg_match("/$u0/",$_SERVER['REMOTE_ADDR'])) {$badbot++;}
}
fclose($fp);
if ($badbot > 0) { /* this is a bad bot, reject it */
sleep(12);
print ("<html><head>\n");
print ("<title>Site unavailable, sorry</title>\n");
print ("</head><body>\n");
print ("<center><h1>Welcome ...</h1></center>\n");
print ("<p><center>Unfortunately, due to abuse, this site is temporarily not available ...</center></p>\n");
print ("<p><center>If you feel this in error, send a mail to the hostmaster at this site,<br>
if you are an anti-social ill-behaving SPAM-bot, then just go away.</center></p>\n");
print ("</body></html>\n");
exit;
}
?>
I plan to take Scott Chamberlain's advice and to be safe I plan to implement Captcha on the script. If user answers correctly then it'll just die or redirect back to site root. Just for fun I'm throwing the trap in a directory named /admin/ and of coursed adding Disallow: /admin/ to robots.txt.
EDIT: In addition I am redirecting the bot ignoring the rules to this page: http://www.seastory.us/bot_this.htm
You could first take a look at where the ip's are coming from. My guess is that they are all coming from one country like china or Nigeria, in which case you could set up something in htaccess to disallow all ip's from those two countries, as for creating a trap for bots, i havent the slightest idea