We have a fairly intensive Drupal installation running on a large EC2 server instance. The system takes registrations for campaigns that we host, and when we send out an invitation email the numbers of responses spike to around 1,000 per minute for the first ten or twenty minutes.
The system is running on a fairly standard LAMP installation, using the latest version of Drupal 7. I guess I have three questions:
1.) Should this amount of load be maxing out this size server? The install has the Organic Groups, Tokens, and Webforms modules running.
2.) Are there Mysql/Apache server tweaks that will minimize the amount of load per connection—shortening 'keep alive' time, etc.?
3.) Are there Drupal tweaks that we should implement to do something similar—maybe consolidating SQL calls?
I know this is a lot. Any ideas, suggestions, or critisisms will help.
Thanks!
For high server load with Drupal, combination of NGINX+PHP-FPM+APC+MySQL+VARNISH will be the best option.
If you are using multiple system behind ELB, you can use MEMCACHE instead of APC. You should use MySQL Replication in your architecture along with load balancer.
You can tune the server configuration of NGINX and PHP-FPM separately. If any process of PHP-FPM is taking time you can terminate that particular request (request_terminate_timeout, request_slowlog_timeout ). You can check all the configuration related info here PHP-FPM configuration
Also if you are using AWS, you should try to utilize their other services too in your architecture like S3 as CDN, ElastiCache, Cloudfront etc.
You can also use Google Page Speed service
Related
I am running a Debian server 7, after preforming load testing using Jmeter on my website. I noticed that MySQL was dying after 50 users, PHP was dying after 100+ users and Apache 2 dying after 200+ users. Now my question is what is the best way to restart these services if they are terminated or froze up?
Restarting a service means killing all its current processes and start a new one. In the meantime you have lost/dropped all the requests from some legitimate users that will eventually see an http error or a timeout when connections are dropped.
I would ask myself, are you happy with 200+ users? Is mysql your bottleneck? Etc..
Use some sort of monitoring service like new relic and as a workaround just restart those services when alerts start coming in, either manually or automatically.
But if you want to improve your site performance deploy your service on a better infrastructure so it can scale up to bigger numbers or improve the code/app architecture used on your site, i.e. put some extra caching between mysql and your application.
Also It would be interesting to know how you have managed to test apache, mysql and php separately especially httpd vs php processes with a tool like JMeter that in my experience sure can test apache separately and mysql separately. But your php and apache scripts are really tightly bound together.
I am creating a web application which will support more than 2000 users.
But 2000 concurrent connection will create problem in apache.
So, I googled and found a way that we can create HTTP request queue on server and handle them one by one.
But how should I achieve it using apache and PHP.
I suggest using NGINX or another event-driven server, as it will do what you're wanting without re-creating the wheel by building an HTTP request queue. Of course, if you really want to scale properly, you may think about a load balancer with more than one web server behind it. 2000 concurrent connections is quite large, and really isn't necessary for 2000 users, as not all users will be sending requests simultaneously. The "connection" only last as long as it takes to serve up a page.
You can also use Apache Benchmark (http://httpd.apache.org/docs/2.2/programs/ab.html) to do some quick, preliminary load testing. I believe you'll find that you don't need near the resources you think you do.
I am trying to build a social networking website that would have a large amount of users logged in at the same time, with an upper limit of around 5000 at a time.
I plan to use the LAMP stack. Apache can have 2 mpms (multi processing modules) - preforker(forks a new process per request) and the worker model (which uses threads per request).
Considering that my server side code is in PHP, and I want to be able to scale up the website on demand, which one would be more preferable. Some third party PHP modules we are using are not thread safe, so ideally preforker is recommended.
Is there any way by which delay between swapping processes can be decreased in preforking
Is there any way by which I can use the worker module, with php as well ?
Social networking sites running on the LAMP stack, normally use which mode?
If you plan to server push services, apache is not a good choice. Take a look at cherokee and nginx, both work much better with php-fpm than mod_php with apache2, since you do not need CGI support (only FastCGI is supported with cherokee and nginx) only PHP.
A few weeks ago I migrated my WordPress network from apache2-mpm-itk with mod_php to nginx+php5-fpm. As a result system load drastically dropped, memory load became predictable and the user experience seems to be a lot faster. I also added some free test from loadimpact.com, the serving times were better with the new config also.
Apache isn't preferred with so much requests. You'd be better off with nginx for example. http://nginx.net/
"10000+ concurrent connections per server" sounds good enough ;-)
You could use Apache in worker mode if you do not use PHP with mod_php. Use PHP in Fastcgi mod, php-fpm is a must-try for that.
With current apache stable version (2.2) you will have problems if you want to use it in chrooted mode. But you may try apache 2.3 or wait for the 2.4 and use mod_proxy_fastcgi instead of mod_fastcgi or others. You could also test the new event basef mpm with this version of Apache. Or of course you could try the other web servers as answered by #petermolnar
I have a Lighttpd(1.4.28) web server running on Centos 5.3 and PHP 5.3.6 in fastcgi mode.
The server itself is a quad core with 1gb ram and is used to record viewing statistics for a video platform.
Each request consists of a very small bit of xml being posted and the receiving php script performs a simple INSERT or UPDATE mysql query. The php returns a very small response to acknowledge the request.
These requests are performed very frequently and i need the system to be able to handle as many concurrent connections as possible at a high rate of requests/second.
I have disabled keep alive as only single requests will be made and so I don't need to keep connections open.
One of the things that concern me is that in server-status I am seeing a lot of connections in the 'read' state. I take it this is controlled by server.max-read-idle which is set to 60 by default? Is it ok to change this to something like 5 as I am seeing the majority of connections being kept open for long periods of time.
Also what else can I do to optimise lighttpd to be able to server lots of small requests
This is my first experience setting up lighttpd as I thought it would be more suitable than apache in this case.
Thanks
Irfan
I believe the problem is not in the webserver, but in your PHP application, especially in MySQL part.
I would replace lighty with apache + mod_php, and mysql with some NoSQL such Redis, which will queue the INSERT requests to the database. Then I would write a daemon / crontab that insert the data in MySQL.
We had such thing before, but instead of Redis, we created TXT files in one directory.
We are using Jmeter to test our Php application running on the Apache 2 web server. I can load up Jmeter to use 25 or 50 threads and the load on the server does not increase, however the response time from the server does. The more threads the slower the response time. It seems like Jmeter or Apache is queuing the requests. I have changed the maxclients value in apache web server configuration file, but this does not change the problem. While Jmeter is running I can use the application and get respectable response times. What gives? I would expect to be able to tax my server down to 0% idle by increase the number of threads. Can anyone help point me in the right direction?
Update: I found that if I remove sessions from my application I am able to simulate a full load on the server. I have tried to re-enable sessions and use an HTTP Cookie Manager for each thread, but it does not seem to make an impact.
You need to identify where the bottleneck is occurring, and then attempt to remediate the problem.
The JMeter client should be running on a well equipted machine. I prefer a Solaris/Unix server running the JVM, but for <200 threads, a modern windows machine will do just fine. JMeter can become a bottleneck, and you won't get any meaningful results once it does. Additionally, it should run on a separate machine to what your testing, and preferable on the same network. The WAN latency can become a problem if your test rig and server are far apart.
The second thing to check is your Apache workers. Apache has a module - mod_status - which will show you the state of every worker. It's possible to have your pool size set too low. From the mod_status, you'll be able to see how many workers are in use. To few, and Apache won't have any workers to process requests, and the requests will queue up. Too many, and Apache may exhaust the memory on the box it's running on.
Next, you should check your database. If it's on a separate machine, the database could have an IO or CPU shortage.
If your hitting a bottleneck, and the server and db are on the same machine, you'll generally hit a CPU, RAM, or IO limit. I listed those in the order in which they are easiest to identify. If you get a CPU bound app, you can easily see you CPU usage go to 100%. If you run out of RAM, your machine will start swapping. On both Windows and unix it's fairly easy to see your available free RAM. Lastly, you may be IO bound. This too can be monitored using various tools or stats, but it's not as obvious as CPU.
Lastly, specifically to your question, the one thing that stands out is it's possible to have a huge number of session files stored in a single directory. Often PHP stores session information in files. If this directory gets large, it will take increasingly long amount of time for PHP to find the session. If you ran your test will cookies turned off, the PHP app may have created thousands of session files for each user request. On a Windows server, it will slow down faster than on a unix server, do to differences in the way directories are stored on the two operating systems.
Are you using a constant throughput timer? If Jmeter can't service the throughput with the threads allocated to it, you'll see this queueing and blowouts in the response time. To figure out if this is the problem, try adding more threads.
I also found a report of this happening when there are javascript calls inside the script. In this instance, try to move javascript calls to the test plan element at the top of the script, or look for ways to pre-calculate the value.
Try checking a static file served by apache and not by PHP to see if the problem is in the Apache config or the PHP config.
Also check your network connections and configuration. Our JMeter testing was progressing nicely until it hit a wall. Eventually realized we only had a 100Mb connection and it was saturated, going to gigabit fixed it. Your network cards or switch may be running at a lower speed than you think, especially if their speed setting is "auto".