I have a quite large website running in centos server. It carries out a lot of shell commands and runs a lot of MySQL queries. Everything seems fine. Running quite well. But at the same moment Apache is utilizing a lot of memory continuously. As far as I know the memory utilized by Apache should be freed once the script execution is completed but it's not like that in my case.
Is this some kind of problem in my code that makes Apache utilize more memory or should I be checking other stuff also?
Unfortunately Apache processes won't return memory used by mod_php. You have to restart/recycle the Apache processes once in a while. Check out the configuration value MaxRequestsPerChild which will do this for you automatically (it's off by default on CentOS if I recall correctly).
Set it to something low, but not too low since recycling is expensive. A few hundreds will probably do.
Related
i have some problem with my VPS.
We are using a VPS to run our CMS and our websites, for now we have 300MB memory limit, and now we are close to reach the limit.
To maintain low cost(i know, increase memory is not to much expensive), but if i find a solution to optimize what we have, will be better.
What can i do?
Thanks!
I would suggest to increase memory - more memory, faster website :-)
But if speed is not important, reduce all cache sizes, set php memory_limit to 8M, disable opcode caching (APC, eAccelerator)
or try Raspberry Pi as server, now comes with 512MB :-)
I have a small 256MB vps which uses
apache + php + mariadb (mysqld)
I found memory was being gobbled by apache at every request 20MB at time for a simple wordpress page. It would settle but over time gobble into swap space and slow everything to a crawl. I suspect there are ways to fine tune mpm_event and mpm_worker to stop entering swap but I didn't figure out how.
When working in such tight environments it is important to know what is using what memory and reduce everything, so that swapping is minimized.
A summary of what I did follows, I managed to get 100MB physical headroom for tweaking, this is not a heavily loaded server but needs to be accessible (and cheap):
Using slackware (the os and default services seem to use less memory than debian, perhaps there is less loaded by default on my providers image)
Switch mysqld innodb tables off and use myisam
configure mysqld to reduce cache sizes if relevant
install cms fresh so they are definitely using myisam or alter all the tables
in apache choose to use the mod_mpm_prefork rather than mpm_event and mpm_worker ("apachectl -V" will tell you which is being used)
set sensible values for maxserver(start from low values for number of servers and requests and work up)
test the server under load whilst doing 'watch free' or 'top' through ssh
you should be able to see memory & the processes being created and destroyed as the load increases
adjust your httpd server settings and retest until happy
I like to see the memory max out at no more than 90% and reduce back down when the load is gone before I am convinced it will stay up without grinding to a halt (start using swap).
check your php.ini memory settings as stated in another answer.
I have also set a cron job to send me an email when swap starts to get used significantly, so I can restart something or even restart the whole server if I can't find what has used the memory, this should happen less and less the more you fine tune.
As stated this is not an environment that will perform well under heavy load but the cost may be more important to you.
Just my two pence worth...
I would look at Nginx like concerto49 recommended, if you have just one website on it also consider Litespeed (www.litespeedtech.com) they have a free version which may be sufficient to power your site.
If its PHP based, then strip out everything you're not using. Use APC/XCache to processing every request. Nginx also has a caching module, that could help so you can avoid hitting PHP for every request if its still fresh.
What type of VPS is it? OpenVZ? Xen? KVM? If it is OpenVZ does it have VSwap or Burst memory?
What type of CMS / Website are you running? Is it PHP based? Are you using Apache? If so have you tried nginx? I would look at optimizing the web server component and removing unused processes/applications to reduce memory and increase performance.
I'm using PHP-FPM to serve dynamic content to nginx through fastcgi. Everything is working fine except that the PHP-FPM proccesses are using so much memory. According to new relic process monitor PHP-FPM have 6 processes running and combined they are using 220+ MB. Doesn't seem normal to me? What could be the problem?
A couple things to check are/ tips to debug in a development environment:
Could you set the php memory_limit to 35M and see where PHP errors out?
Are there any large database queries being ran?
Are you using APC with PHP, what is your shared memory size (shm)?
If you are familiar with xDebug (http://xdebug.org/). You could use the xDebug profiler (http://xdebug.org/docs/profiler), to see what is using memory. That is your best bet IMO.
PHP-FPM allows you to set a max request parameter (pm.max_requests) that will restart a child process after a certain amount of requests. Watch out setting that to low might result in poor performance, and odd behavior under load.
To be honest 37M per process is high, however I have seen applications like Magento get close to that, so there may be no problem. Hope that helps.
I need to use the minimal amount possible of RAM on a VM Server running Ubuntu, using LAMP. Is there known tips for using minimal mem? I already set the MySQL cache/childs but that isn't helping much.
What you should do, is retain Linux, Mysql and PHP, but do away with Apache. Or at least, stop running PHP in-process with Apache.
The chances are you're using the Apache prefork model with an in-process PHP module. This is very bad for memory efficiency on most workloads, because it keeps a heavy PHP process open even for HTTP connections which aren't requesting any dynamic content just now.
What you want to do instead is use another web server (for example Nginx, but Apache would work too) and run PHP as a FastCGI daemon. This is easy to set up and googling for "PHP fastcgi" returns numerous examples.
You can then have a small, fixed number of "heavy" processes running PHP (No more than a couple per core, I reckon), but still have good capacity for running real applications, because "idle" HTTP connections, such as those serving keep-alives or waiting for requests don't use up the "heavy" processes, only the lighter web server processes.
A web server which uses limited forking / few processes is probably better - such as Nginx, or Apache with a different thread model. This is incompatible with mod_php, which is why you need to run it as FastCGI instead.
Make sure you turn off InnoDB in mysql, that will use about 100MB less memory
You can set the memory_limit in php.ini to something smaller. However if your program requests more memory than the limit then the script will crash with a fatal error.
That will only change memory usage in PHP. There may be a similar setting for MySQL in my.cnf but I don't know what it would be off-hand.
Really though the best way to reduce memory usage is to write programs that don't use a lot of memory.
About MySQL, Apache and others: http://library.linode.com/troubleshooting/memory-networking#sph_diagnosing-and-fixing-memory-issues
For PHP: in php.ini you can set memory_limit.
For Linux: use 32-bit versions (until you have more than 4Gb of RAM).
Don't reduce all these settings until you really need it, or it will be reason of performance degradation. Try to give enough memory to your system first.
I was testing my site with Jmeter to see how hundred threads would affect site performance and tested it with apache and mod_php and nginx with fastcgi. I noticed that the bottlebeck was always cpu in both apache and nginx. when I was looking at spu usage in nginx setup I could see that php cgi processes were taking all cpu.
What can I do to decrease php cpu usage?
The reason PHP is taking up more resources than your web servers is that PHP is doing a lot more work, and does so in an intepreted language (Apache and Nginx is both compiled into CPU instructions).
The first step is to implement an opcode cache (I recommend APC since it's easily installed and maintained by the PHP people). This will cache a "compiled" version of your scripts which will remove a big bit of the script execution.
The only other way to decrease CPU usage is to make your code more effective. Try to identify bottlenecks (large arrays and other data structures may be the first to check out) and find ways to make those parts of the code perform better.
We run a medium-size site that gets a few hundred thousand pageviews a day. Up until last weekend we ran with a load usually below 0.2 on a virtual machine. The OS is Ubuntu.
When deploying the latest version of our application, we also did an apt-get dist-upgrade before deploying. After we had deployed we noticed that the load on the CPU had spiked dramatically (sometimes reaching 10 and stopping to respond to page requests).
We tried dumping a full minute of Xdebug profiling data from PHP, but looking through it revealed only a few somewhat slow parts, but nothing to explain the huge jump.
We are now pretty sure that nothing in the new version of our website is triggering the problem, but we have no way to be sure. We have rolled back a lot of the changes, but the problem still persists.
When look at processes, we see that single Apache processes use quite a bit of CPU over a longer period of time than strictly necessary. However, when using strace on the affected process, we never see anything but
accept(3,
and it hangs for a while before receiving a new connection, so we can't actually see what is causing the problem.
The stack is PHP 5, Apache 2 (prefork), MySQL 5.1. Most things run through Memcached. We've tried APC and eAccelerator.
So, what should be our next step? Are there any profiling methods we overlooked/don't know about?
The answer ended up being not-Apache related. As mentioned, we were on a virtual machine. Our user sessions are pretty big (think 500kB per active user), so we had a lot of disk IO. The disk was nearly full, meaning that Ubuntu spent a lot of time moving things around (or so we think). There was no easy way to extend the disk (because it was not set up properly for VMWare). This completely killed performance, and Apache and MySQL would occasionally use 100% CPU (for a very short time), and the system would be so slow to update the CPU usage meters that it seemed to be stuck there.
We ended up setting up a new VM (which also gave us the opportunity to thoroughly document everything on the server). On the new VM we allocated plenty of disk space, and moved sessions into memory (using memcached). Our load dropped to 0.2 on off-peak use and around 1 near peak use (on a 2-CPU VM). Moving the sessions into memcached took a lot of disk IO away (we were constantly using about 2MB/s of disk IO, which is very bad).
Conclusion; sometimes you just have to start over... :)
Seeing an accept() call from your Apache process isn't at all unusual - that's the webserver waiting for a new request.
First of all, you want to establish what the parameters of the load are. Something like
vmstat 1
will show you what your system is up to. Look in the 'swap' and 'io' columns. If you see anything other than '0' in the 'si' and 'so' columns, your system is swapping because of a low memory condition. Consider reducing the number of running Apache children, or throwing more RAM in your server.
If RAM isn't an issue, look at the 'cpu' columns. You're interested in the 'us' and 'sy' columns. These show you the percentage of CPU time spent in either user processes or system. A high 'us' number points the finger at Apache or your scripts - or potentially something else on the server.
Running
top
will show you which processes are the most active.
Have you ruled out your database? The most common cause of unexpectedly high load I've seen on production LAMP stacks come down to database queries. You may have deployed new code with an expensive query in it; or got to the point where there are enough rows in your dataset to cause previously cheap queries to become expensive.
During periods of high load, do
echo "show full processlist" | mysql | grep -v Sleep
to see if there are either long-running queries, or huge numbers of the same query operating at once. Other mysql tools will help you optimise these.
You may find it useful to configure and use mod_status for Apache, which will allow you to see what request each Apache child is serving and for how long it has been doing so.
Finally, get some long-term statistical monitoring set up. Something like zabbix is straightforward to configure, and will let you monitor resource usage over time, such that if things get slow, you have historical baselines to compare against, and a better ieda of when problems started.
Perhaps you where using worker MPM before and now you are not?
I know PHP5 does not work with the Worker MPM. On my Ubuntu server, PHP5 can only be installed with the Prefork MPM. It seems that PHP5 module is not compatible with multithreading version of Apache.
I found a link here that will show you how to get better performance with mod_fcgid
To see what worker MPM is see here.
I'd use dTrace to solve this mystery... if it was running on Solaris or Mac... but since Linux doesn't have it you might want to try their Systemtap, however I can't say anything about its usability since I haven't used it.
With dTrace you could easily sniff out the culprits within a day, and would hope with Systemtap it would be similiar
Another option that I can't assure you will do any good, but it's more than worth the effort. Is to read the detailed changelog for the new version, and review what might have changed that could remotely affect you.
Going through the changelogs has saved me more than once. Especially when some config options have changed and when something got deprecated. Worst case is it'll give you some clues as to where to look next