Im implementing a delay system so that any IP i deem abusive will automatically get an incremental delay via Sleep().
My question is, will this result in added CPU usage and thus kill my site anyways if the attacker just keeps opening new instances while being delayed? Or is the sleep() command use minimal CPU/memory and wont be much of a burden on a small script. I dont wish to flat out deny them as i'd rather they not know about the limit in an obvious way, but willing to hear why i should.
[ Please no discussion on why im deeming an IP abusive on a small site, cause heres why: I recently built a script that cURL's a page & returns information to the user and i noticed a few IP's spamming my stupid little script. cURLing too often sometimes renders my results unobtainable from the server im polling and legitimate users get screwed out of their results. ]
The sleep does not use any CPU or Memory which is not already used by the process accepting the call.
The problem you will face with implementing sleep() is that you will eventually run out of file descriptors while the attacker site around waiting for your sleep to time out, and then your site will appear to be down to any other people who tries to connect.
This is a classical DDoS scenario -- the attacker do not actually try to break into your machine (they may also try to do that, but that is a different storry) instead they are trying to harm your site by using up every resource you have, being either bandwidth, file descriptors, thread for processing etc. -- and when one of your resources are used up, then you site appears to be down although your server is not actually down.
The only real defense here is to either not accept the calls, or to have a dynamic firewall configuration which filters out calls -- or a router/firewall box which does the same but off your server.
I think the issue with this would be that you could potentially have a LARGE number of sleeping threads laying around the system. If you detect your abuse, immediately send back an error and be done with it.
My worry with your method is repeat abusers that get their timeout up to several hours. You'll have their threads sticking around for a long time even though they aren't using the CPU. There are other resources to keep in mind besides just CPU.
Sleep() is a function that "blocks" execution for a specific amount of time. It isn't the equivalent of:
while (x<1000000);
As that would cause 100% CPU usage. It simply puts the process into a "Blocked" state in the Operating System and then puts the process back into the "Ready" state after the timer is up.
Keep in mind, though, that PHP has a default of 30-second timeout. I'm not sure if "Sleep()" conforms to that or not (I would doubt it since its a system call instead of script)
Your host may not like you having so many "Blocked" processes, so be careful of that.
EDIT: According to Does sleep time count for execution time limit?, it would appear that "Sleep()" is not affected by "max execution time" (under Linux), as I expected. Apparently it does under Windows.
If you are doing what I also tried, I think you're going to be in the clear.
My authentication script built out something similar to Atwood's hellbanning idea. SessionIDs were captured in RAM and rotated on every page call. If conditions weren't met, I would flag that particular Session with a demerit. After three, I began adding sleep() calls to their executions. The limit was variable, but I settled on 3 seconds as a happy number.
With authentication, the attacker relies on performing a certain number of attempts per second to make it worth their while to attack. If this is their focal point, introducing sleep makes the system look slower than it really is, which in my opinion will make it less desirable to attack.
If you slow them down instead of flat out telling them no, you stand a slightly more reasonable chance of looking less attractive.
That being said, it is security through a "type" of obfuscation, so you can't really rely on it too terribly much. Its just another factor in my overall recipe :)
Related
I have a php script on my website that is designed to give a nice overview of a domain name the user enters. It does this job quite well, however it is very slow. This might have something to do with the fact it's checking an array of 64 possible domain names, and THEN moving on to checking nameservers for A records/MX records/NS records etc.
What i would like to know, is it possible to run multiple threads/child processes of this? So that it will check multiple ellements of the array at once, and generate the output a lost faster?
I've put an example of my code in a pastebin (so to avoid creating a huge and spammy post on here)
http://pastebin.com/Qq9qKtP9
In perl I can do something like this:
$fork = new Parallel::ForkManager($threads);
foreach(Something here){
$fork->start and next;
$fork->finish;
}
And i could make the loop run in as many processes as needed. Is something similar possible in PHP or any other ways you can think of to speed this up? The main issue is, cloudflare has a timeout, and often it will take long enough CF blocks the response happening.
Thanks
* Never Mind Support !! *
You never want to create threads (or additional processes for that matter) in direct response to a web request.
If your frontend is instructed to create 60 threads every time someone clicks on page.php, and 100 people come along and request page.php at once, you will be asking your hardware to create and execute 6000 threads concurrently, to say nothing of the threads used by operating system services and other software. For obvious reasons, this does not, and will never scale.
Rather you want to separate out those parts of the application that require additional threads or processes and communicate with this part of the application via some kind of sane RPC. This means that the backend of the application can utilize concurrency via pthreads or forking, using a fixed number of threads or processes, and spreading work as evenly as possible across all available resources. This allows for an influx of traffic; it allows your application to scale.
I won't write example code, it seems altogether too trivial.
The first thing you want to do is optimze your code to shorten the execution time as much as possible.
For example, instead of making five dns queries:
$NS = dns_get_record($murl, DNS_NS);
$MX = dns_get_record($murl,DNS_MX);
$SRV = dns_get_record($murl,DNS_SRV);
$A = dns_get_record($murl,DNS_A);
$TXT = dns_get_record($murl,DNS_TXT);
You can only call dns_get_record once:
$DATA = dns_get_record($murl, DNS_NS + DNS_MX + DNS_SRV + DNS_A + DNS_TXT);
and parse out the variables from there.
Instead of outright forking processes to handle several parts concurrently, I'd implement a queue that all of the requests would get pushed into. The query processor would be limited as to how many items it could process at once, avoiding the potential DoS if hundreds or thousands of requests hit your site at the same time. Without some sort of limiting mechanism, you'd end up with so many processes that the server might hang.
As for the processor, in addition to the previously mentioned items, you could try pecl/Gearman as your queue processor. I haven't used it, but it appears to do what you're looking for.
Another method to optimize this would be implementing a caching system, that saved the results for, say, a week (or whatever). This would cut down on someone looking up the same site repeatedly in a day (or running a script on your site).
I doubt that it's a good idea to fork with PHP the apache process. But if you really want there is PCNTL (which is not available in the apache module).
You might have more fun with pthread. Nowadays you can even download a PHP which claims to be threadsafe.
And finally you have the possibility to use classic non blocking IO which I would prefer in the case of PHP.
We're building an online test with individually timed questions. The questions are a series of forms which the users must complete within a pre-determined number of seconds. Although the users will see a javascript ticking clock on each question, the actual enforcement of the time limit must happen server-side (otherwise it would be very easy for the user to disable/modify the time limit). However, we are concerned that network latency will unfairly penalize users who submit their answer close to the end of the time limit, close enough for latency to make their response over the time limit.
The sloppy solution seems to be to just allow a few second "grace period" when we check the duration as the answer is received at the server, but this is kind of lame. Is there a secure way to determine precisely when the user clicked "Submit".
If it matters, the web app is written in PHP.
I've actually had to deal with this problem before. What we ended up doing, is marking the time that the request for the page was made, and using a javascript ajax request (on load) from the browser to determine the round-trip time. Averaged out it ends up representing their network latency / computer speed, pretty well.
Obviously if it gets to anything over 15-20 seconds you have a bigger problem, but the most we saw was 3-4 seconds.
You can then use this in a "fairness" evaluation of how much time it actually took them.
Why is HTTP-Polling so laggy?
What I have is a button, and whenever a user clicks it a MySQL database field gets updated and the value is displayed to the user. I'm polling every 800 milliseconds and it's very laggy/glitchy. Sometimes when clicking the button it doesn't register it. And I actually need to be polling quite a bit more frequent than every 800 milliseconds.
This is also with just 1 user on the website at a time... When in the end there is going to be many at once.
HTTP-streaming/Long-polling/Websockets instead of polling
When you need real-time information you should avoid polling(frequently). Below I would try to explain why this is wrong. You could compare it to a child in the back of your car screaming every second "are we there yet" while you are replying "we are not there yet" all the time.
Instead you would like to have something like long-polling/HTTP-streaming or websockets. You could compare this to a child in the back of your car telling you to let him know when "we are there" instead of asking us every second. You could imagine this is way more efficient then the previous example.
To be honest I don't think PHP is the right tool for this kind of applications(yet). Some options you have available are:
hosted solutions:
http://pusherapp.com:
Pusher is a hosted API for quickly,
easily and securely adding scalable
realtime functionality via WebSockets
to web and mobile apps.
Our free Sandbox plan includes up to
20 connections and 100,000 messages
per day. Simply upgrade to a paid plan
when you're ready.
http://beaconpush.com/
Beaconpush is a push service for
creating real-time web apps using
HTML5 WebSockets and Comet.
host yourself:
http://socket.io:
Socket.IO aims to make realtime apps
possible in every browser and mobile
device, blurring the differences
between the different transport
mechanisms
When becoming very big the "host yourself" solution is going to less expensive, but on the other hand using something like pusherapp will get you started easier(friendly API) and also is not that expensive. For example pusherapp's "Bootstrap" can have 100 concurrent connections and 200,000 messages per day for $19 per month(but when small beaconpush is cheaper => do the math :)). As a side-note this plan does not include SSL so can not be used for sensitive data. I guess having a dedicated machine(VPS) will cost you about the same amount of money(for a simple website) and you will also have to manage the streaming solution yourself, but when getting bigger this is probably way more attractive.
Memory instead of Disc
whenever a user clicks it a MySQL
database field gets updated and the
value is displayed to the user
When comparing disc I/O(MySQL in standard mode) to memory it is extremely slow. You should be using an in-memory database like for example redis(also has persistent snapshots) or memcached(completely in memory) to speed up the process. I myself really like redis for it's insane speed, simplicity and persistent snapshots. http://redistogo.com/ offers a free plan with 5MB of memory which will probably cover your needs. If not the mini plan of $5 a month will probably cover you, but when getting even bigger a VPS will be cheaper and in my opinion the prefered solution.
Best solution
The best solution(especially if you are getting big) is to host socket.io/redis yourself using a VPS(cost money). If really small I would use redistogo, if not I would host it myself. I would also start using something like beaconpush/pusherapp because of it's simplicity(getting started immediately). Hosting socket.io(advice to play with it on your own machine for when getting big) is pretty simple, but in my opinion more difficult than beaconpush/pusherapp.
Laggy/glitchy? Sounds like a client-side problem. As does the button thing. I'd get your JavaScript in order first.
As for polling, 0.8 sounds a bit time-critical. I don't know about most countries, but here in the third world simple network packets may get delayed for as long a few seconds. (Not to mention connection drops, packet losses and the speed of light.) Is your application ready to cope with all that?
As for an alternate approach, I agree with #Vern in that an interrupt-driven one would be much better. In HTTP terms, it translates to a long-standing HTTP request that does not receive a response until the server has some actual data to send, minimizing delay and bandwidth. (AFAIK) it's an older technique than AJAX, though has been named more recently. Search for "COMET" and you'll end up with both client- and server-side libraries.
there are many things that might cause the lag that you are experiencing. Your server might be able to process the requests fast enough, but if the connection between your client and the server is slow, then you'll see the obvious lag.
The first thing you should try is to ping the server and see what response time you're getting.
Secondly, rather than poll, you might want to consider an interrupt driven approach. This means that only when your server replies, will you send out your next request. This makes sense, so that many clients won't be flooding the server with requests till the point the server cannot cope. This is especially true, then the RTT (Round-Trip-Time) of your request is pretty long.
Hope it helps. Cheers!
A good place to start would be to use a tool like Firebug in Mozilla Firefox that will allow you to watch the requests being sent to the server and look for bottlenecks.
Firebug will break down each part of the request, so you can see if you are having trouble talking to the server or if it is simply taking a long time to come up with a response.
Along with #Vern's answer I would also say that if at all possible I would have the server cache the data ahead of time and then all of the clients will pull from that same cache and not need separate MySQL calls to reach the same data for every update. Then you just have your PHP update the cache whenever the actual DB data changes.
By cache I mean having php write to a file on the sever side, and then clients will simply look at the contents of that one file to see the most updated info. There might be better ways of caching, but being that I have never done this personally before, this is the first solution that popped into my mind.
I'm a bit confused because the logical/programmer brain in me says that if all things are constant, the speed of a function must be constant.
I am working on a PHP web application with jqGrid as a front end for showing the data. I am testing on my personal computer, so network traffic does not apply. I make an HTTP request to a PHP function, it returns the data, and then jqGrid renders it. What has me befuddled is that sometimes, Firebug reports that this is taking 300-600 milliseconds sometimes, and sometimes, it's taking 3.68 seconds. I can run the request over and over again, with very radically different response times.
The query is the same. The number of users on the system is the same. No network latency. Same code. I'm not running other applications on the computer while testing. I could understand query caching improving performance on subsequent requests, but the speed is just fluctuating wildly with no rhyme or reason.
So, my question is, what else can cause such variability in the response time? How can I determine what's doing it? More importantly, is there any way to get things more consistent?
If you use the Apache Benchmarking tool (ab) you can get a set of statistics based on multiple accesses and multiple concurrent accesses, giving you min, mean, median and max access times (and stddev) broken down by connect, processing and waiting; and percentiles... making it an extremely useful tool for identifying if this is really a problem or an aberration.
While it can't diagnose the cause of performance problems, it can tell you if you really do have a problem
The first thing you should do is profile your code (see Simplest way to profile a PHP script). This will show you where your bottleneck is, and then you can figure out why your response times are fluctuating so much.
If you are using Firebug to determine speed you need to consider that both Firefox and PHP are caching resources. PHP, in particular, has a built-in opcode cache to minimize subsequent runtimes. This shows up as a long run on the first instance, followed by a number of short-runs on refresh. I concur with rfw to go deeper in your analysis.
What is the best way to stop bots, malicious users, etc. from executing php scripts too fast? Is it ok if I use the usleep() or sleep() functions to simply do "nothing" for a while (just before the desired code executes), or is that plain stupid and there are better ways for this?
Example:
function login() {
//enter login code here
}
function logout() {
//enter logout code here
}
If I just put, say, usleep(3000000) before the login and logout codes, is that ok, or are there better, wiser ways of achieving what I want to achieve?
edit: Based on the suggestions below, does then usleep or sleep only cause the processor to disengage from the current script being executed by the current user, or does it cause it to disengage from the entire service? i.e. If one user+script invokes a sleep/usleep, will all concurrent users+scripts be delayed too?
The way most web servers work (Apache for example) is to maintain a collection of worker threads. When a PHP script is executed, one thread runs the PHP script.
When your script does sleep(100), the script takes 100 seconds to execute.. That means your worker thread is tied up for 100 seconds.
The problem is, you have a very finite number of worker-threads - say you have 10 threads, and 10 people login - now your web-server cannot serve any further responses..
The best way to rate-limit logins (or other actions) is to use some kind of fast in-memory storage thing (memcached is perfect for this), but that requires running separate process and is pretty complicated (you might do this if you run something like Facebook..).
Simpler, you could have a database table that stores user_id or ip_address, first_failed and failure_counter.
Every time you get a failed login, you (in pseudo code) would do:
if (first_failed in last hour) and (failure_counter > threshold):
return error_403("Too many authentication failures, please wait")
elseif first_failed in last hour:
increment failure_counter
else:
reset first_failed to current time
increment failure_counter
Maybe not the most efficient, and there is better ways, but it should stop brute-forcing pretty well. Using memcached is basically the same, but the database is replaced with memcached (which is quicker)
to stop bots, malicious users, etc.
from executing php scripts too fast?
I would first ask what you are really trying to prevent? If it is denial-of-service attacks, then I'd have to say there is nothing you can do if you are limited by what you can add to PHP scripts. The state of the art is so much beyond what we as programmers can protect against. Start looking at sysadmin tools designed for this purpose.
Or are you trying to limit your service so that real people can access it but bots cannot? If so, I'd look at some "captcha" techniques.
Or are you trying to prevent users from polling your site every second looking for new content? If so, I'd investigate providing an RSS feed or some other way of notifying them so they don't eat up your bandwidth.
Or is it something else?
In general, I'd say neither sleep() nor usleep() is a good way.
Your suggested method will force ALL users to wait unnecessarily before logging in.
Most LAMP servers (and most routers/switches, actually) are already configured to prevent Denial of Service attacks. They do this by denying multiple consecutive requests from the same IP address.
You don't want to put a sleep in your php. Doing so will greatly reduce the number of concurrent requests your serve can handle since you'll have connections held open waiting.
Most HTTP servers have features you can enable to avoid DoS attacks, but failing that you should just track IP addresses you've seen too many times too recently and send them a 403 Forbidden with a message asking them to wait a second.
If for some reason you can't count on REMOTE_ADDR being user specific (everyone behind the same firewall, etc.) you could prove a challenge in the login form and make the remote browser do an extended calculation on it (say, factor a number) that you can quickly check on the server side (with speedy multiplication).