I have an store running on Prestashop 1.5.4. I keep having a problem with the site behaviour.
Everytime i check this number
user#server:~$ mysql -se "show status like '%threads_connected%'"
If the number gets above 25, my site becomes really slow. Opening a page takes forever and page load can get as high as 1 - 2 minutes.
The only solution (temporary) is for me to restart the apache services.
I am pretty sure 25 is a pretty low number.
In case you need to know, i don't have direct access to my.ini nor to any access to MySQL Server Configuration. My database is stored in a shared (DBaSS).
I do however have full access to my webserver (apache2.conf, php.ini, www.example.com.conf)
A little info that might help:
-se "show variables like '%max%'"
Outputs:
I guess my question is how can i improve performance ? Can i improve this by limiting the number of "threads_connected" ?
footnote:
I am aware that there is probably a bad query somewhere in the code, however, at the moment i need a quick solution for this. As reviewing all queries can take some time.
EDIT #1
Perhaps this information might give some ideas
EDIT #2
Can i improve this by limiting the number of "threads_connected"?
Absolutely not.
All you could do by limiting the number of connections would be to cause errors for users of your site, because their particular Apache process wouldn't be able to connect to the database.
The problem is not the number of threads connected. That is a symptom of the real problem, which is that you have one or more queries that perform poorly, or that you do not have enough memory for Apache to scale up when traffic gets heavy, forcing your machine into heavily swapping, and thereby slowing down Apache to the point that it keeps connections open longer.
I need a quick solution for this
Sorry... but there isn't another solution other than to find the actual problem and fix it.
More useful than the number of connections is what are these connections doing right now?
SHOW FULL PROCESSLIST; in MySQL will answer that question. If they are just sleeping, they aren't hurting anything on the MySQL side, and you may want to limit the number of Apache processes, but if that is a side effect of the level of site traffic, then your server may actually be too small, or you may need to disable HTTP keepalive, or tweak the timeout, if browser connections are holding open Apache children that are idle, consuming memory.
Related
I am running HTTP API which should be called more than 30,000 time per minute simultaneously.
Currently I can call it 1,200 time per minute. If I call 1200 time per minute, all the request are completed and get response immediately.
But if I called 12,000 time per minute simultaneously it take 10 minute to complete all the request. And during that 10 minute, I cannot browse any webpage on the server. It is very slow
I am running CentOS 7
Server Specification
Intel® Xeon® E5-1650 v3 Hexa-Core Haswell,
RAM 256 GB DDR4 ECC RAM,
Hard Drive2 x 480 GB SSD(Software-RAID 1),
Connection 1 Gbit/s
API- simple php script that echo the time-stamp
echo time();
I check the top command, there is no load in the server
please help me on it
Thanks
Sounds like a congestion problem.
It doesn't matter how quick your script/page handling is, if the next request gets done within the execution time of the previous:
It is going to use resources (cpu, ram, disk, network traffic and connections).
And make everything parallel to it slower.
There are multiple things you could do, but you need to figure out what exactly the problem is for your setup and decide if the measure produces the desired result.
If the core problem is that resources get hogged by parallel processes, you could lower connection limits so more connections go in to wait mode, which keeps more resources available for actually handing out a page instead of congesting everything even more.
Take a look at this:
http://oxpedia.org/wiki/index.php?title=Tune_apache2_for_more_concurrent_connections
If the server accepts connections quicker then it can handle them, you are going to have a problem which ever you change. It should start dropping connections at some point. If you cram down French baguettes down its throat quicker then it can open its mouth, it is going to suffocate either way.
If the system gets overwhelmed at the network side of things (transfer speed limit, maximum possible of concurent connections for the OS etc etc) then you should consider using a load balancer. Only after the loadbalancer confirms the server has the capacity to actually take care of the page request it will send the user further.
This usually works well when you do any kind of processing which slows down page loading (server side code execution, large volumes of data etc).
Optimise performance
There are many ways to execute PHP code on a webserver and I assume you use appache. I am no expert, but there are modes like CGI and FastCGI for example. Which can greatly enhance execution speed. And tweaking settings connected to these can also show you what is happening. It could for example be that you use to little number of PHP threats to handle that number of concurrent connections.
Have a look at something like this for example
http://blog.layershift.com/which-php-mode-apache-vs-cgi-vs-fastcgi/
There is no 'best fit for all' solution here. To fix it, you need to figure out what the bottle neck for the server is. And act accordingly.
12000 Calls per minute == 200 calls a second.
You could limit your test case to a multitude of those 200 and increase/decrease it while changing settings. Your goal is to dish that number of requestst out in a shortest amount of time as possible, thus ensuring the congestion never occurs.
That said: consequences.
When you are going to implement changes to optimise the maximum number of page loads you want to achieve you are inadvertently going to introduce other conditions. For example if maximum ram usage by Apache would be the problem, the upping that limit will ensure better performance, but heightens the chance the OS runs out of memory when other processes also want to claim more memory.
Adding a load balancer adds another possible layer of failure and possible slow downs. Yes you prevent congestion, but is it worth the slow down caused by the rerouting?
Upping performance will increase the load on the system, making it possible to accept more concurrent connections. So somewhere along the line a different bottle neck will pop up. High traffic on different processes could always end in said process crashing. Apache is a very well build web server, so it should in theories protect you against said problem, however tweaking settings wrongly could still cause crashes.
So experiment with care and test before you use it live.
I launched a website about a week ago and I sent out an email blast to a mailing list telling everyone the website was live. Right after that the website went down and the general error log was flooded with "exceeded process limit" errors. Since then, I've tried to really clean up a lot of the code and minimize database connections. I will still see that error about once a day in the error log. What could be causing this error? I tried to call the web host and they said it had something to do with my code but couldn't point me in any direction as to what was wrong with the code or which page was causing the error. Can anyone give me any more information? Like for instance, what is a process and how many processes should I have?
Wow. Big question.
Obviously, your maxing out your apache child worker processes. To get a rough idea of how many you can create, use top to get the rough memory footprint of one http process. If you are using wordpress or another cms, it could easily be 50-100m each (if you're using the php module for apache). Then, assuming the machine is only used for web serving, take your total memory, subtract a chunk for OS use, then divide that by 100m (in this example). Thats the max worker processes you can have. Set it in your httpd.conf. Once you do this and restart apache, monitor top and make sure you don't start swapping memory. If you do, you have set too high a number of workers.
If there is any other stuff running like mysql servers, make space for that before you compute number of workers you can have. If this number is small, to roughly quote a great man 'you are gonna need a bigger boat'. Just kidding. You might see really high memory usage for a http process like over 100m. You can tweak your the max requests per child lower to shorten the life of a http process. This could help clean up bloated http workers.
Another area to look at is time response time for a request... how long does each request take? For a quick check, use firebug plugin for firefox and look at the 'net' tab to see how long it takes for your initial request to respond back (not images and such). If for some reason request are taking more than 1 or 2 seconds to respond, that's a big problem as you get sort of a log jam. The cause of this could be php code, or mysql queries taking too long to respond. To address this, make sure if you're using wordpress to use some good caching plugin to lower the stress on mysql.
Honestly, though, unless your just not utilizing memory by having too few workers, optimizing your apache isn't something easily addressed in a short post without detail on your server (memory, cpu count, etc..) and your httpd.conf settings.
Note: if you don't have server access you'll have a hard time figuring out memory usage.
The process limit is typically something enforced by shared webhost providers, and generally has to do with the number of processes executing under your account. This will typically equate to the number of connections made to your server at once (assuming one PHP process per each connection).
There are many factors that come into play. You should figure out what that limit is from your hosting provider, and then find a new one that can handle your load.
My hosting says that apache connections limit is 30. I don't whether its enough or not for an average site with 100 visitors per day. I want to know what are the things I should adapt for this limit while coding the site. Mostly I 'll use php sessions and little ajax . I want to know if there any precautions and recommended practices (if any) to avoid hitting this limit.
Thank you.
Since you will be using AJAX, I can't stress this enough...Do not long poll with Apache! It will hold your connections open and effectively perform a DOS(Denial of Service) on your own site.
Other than that, minimize the time it takes between when Apache receives a request to when it outputs and closes. The big blinking neon sign here is to use caching. Whether it is file based caching or something like Memcached or APC, this can drastically reduce the time Apache holds a connection open.
Taken by itself, the statement "apache connections limit is 30" doesn't actually mean much -- Apache configuration can be fairly involved and there are a lot of numbers/parameters. But if we assume that what this really means is 'MaxClients is 30', then what you need to know is that you have a limit of 30 simultaneous connections. However, connection 31 isn't rejected -- it should just be queued until there's a thread available to respond to the request. There's a lot of specifics according to the config, etc, but I doubt you need to worry much.
This means there are 30 possible concurrent connections possible, if you have 100 visitors per day, it's very unlikely to have about a third at the same time.
As you are growing with your site I'd recommend you another server/hoster.
But as if you don't make long running persistent connections and high frequent AJAX call all the time, this should be enough.
Connection limit is most probably simultaneous requests. So if you're only at the development stage, that is fine. But as for once it has launched, that is a different story. If your expected traffic is only about 100 visitors a day, then you will most probably be fine. I would however recommend to change your VPS host if it is anything over that, as if the server is turning away visitors, then it is not good for business.
But in all honesty you're better off developing locally for now to save your bandwidth for actual visitors, as from your description you don't seem to be using anything that requires a live site.
Recently we've been having problems with our LAMP setup and we started to see the number of MySQL database connections spike up every now and then. We suspect that some mysql operation is taking longer than usual and apache just started to build a backlog of connections to deal with incoming requests.
Question is, is there a way to per page statistics on things like average load time? median load time? max/min load time for each php page (page1.php, page2.php, page3.php etc). So that we can narrow down where the problem is. Is there such thing included as part of apache? Maybe a separate module?
From the log format, you can just log the time taken (%D) in your access logs, and after an incident, sort on time-taken, and check the urls. I'm not aware of any application that checks this out of the box, but a lot of applications can handle apache's access logs, so chances are there are those who can work with it. I seldomly look at page-specific logs, only server totals, so I can't help you there.
If MySQL is busy / the cause:
Close a connection to MySQL if you're done with it, so the connection is released sooner.
Increase the maximum allowed connections if you really need them.
If you still have hanging processes, check the output of SHOW FULL PROCESSLIST to see what queries are being performed.
You can enable the slow_query_log, logging all queries above a certain amount of miliseconds (in newer versions, old versions only supported seconds) or not using indexes. The command line tool mysqldumpslow can accurately group / count the queries.
If you have access to php.ini, you can use Xdebug : http://xdebug.org/
We have an application that is comprised of a couple of off the shelf PHP applications (ExpressionEngine and XCart) as well as our own custom code.
I did not do the actual analysis so I don't know precisely how it was determined, but am not surprised to hear that too many MySQL connections are being left unclosed (I am not surprised because I have been seeing significant memory leakage on our dev server, where over the course of a day or two, starting from 100MB upon initial boot, the entire gig of ram gets consumed, and very little of it is cached).
So, how do we go about determining precisely which PHP code is the culprit? I've got prior experience with XDebug, and have suggested that, when we've gotten our separate, staging environment reasonably stable, that we retrofit XDebug on dev and use that to do some analysis. Is this reasonable, and/or does anybody else have more specific and/or additional suggestions?
You can use the
SHOW PROCESSLIST
SQL command to see what processes are running. That will tell you the username, host, database, etc that are in use by each process. That should give you some idea what's going on, especially if you have a number of databases being accessed.
More here: https://dev.mysql.com/doc/refman/8.0/en/show-processlist.html
This should not be caused by a php code because mysql connections are supposed to be automatically closed.
cf : http://www.php.net/manual/function.mysql-connect.php :
The link to the server will be closed
as soon as the execution of the script
ends, unless it's closed earlier by
explicitly calling mysql_close().
Some suggestions :
does your developper has technically a direct access to your production mysql server ? if yes, then they probably just leave their Mysql Manager open :)
do you have some daily batch process ? if yes, maybe that there are some zombi process in memory
PHP automatically closes any mysql connections when the page ends. the only reason that a PHP web application would have too many unclosed mysql connections is either 1) you're using connection pooling, or 2) there's a bug in the mysql server or the connector.
but if you really want to look at your code to find where it's connecting, see http://xdebug.org/docs/profiler
As others said, PHP terminates MySQL connections created through mysql_connect or the msqli/PDO equivalents.
However, you can create persistent connections with mysql_pconnect. It will look for existing connections open and use those; if it can't find one, it will open a new one. If you had a lot of requests at once, it could have caused loads of connections to open and stay open.
You could lower the maximum number of connections, or lower the timeout for persistent connections. See the comments at the bottom of the man page for more details.
I used to run a script that polled SHOW STATUS for thread count and I noticed that using mysql_pconnect always encouraged high numbers of threads. I found that very disconcerting because then I couldn't tell when my connection rate was actually dropping. So I made sure to centralize all the places where mysql_connect() was called and eliminate mysql_pconnect().
The next thing I did was look at the connection timeouts and adjust them to more like 30 seconds because. So I adjusted my my.cnf with
connect-timeout=30
so I could actually see the number of connections drop off. To determine the number of connections you need open is dependent on how many apache workers you're running times the number of database connections they each will open.
The other thing I started doing was adding a note to my queries in order to spot them in SHOW PROCESSLIST or mytop, I would add a note column to my results like:
$q = "SELECT '".__FILE__.'.'.__LINE__."' as _info, * FROM table ...";
This would show me the file issuing the query when I looked at mytop, and it didn't foil the MySQL query cache like using
/* __FILE__.'.'.__LINE__ */
at the start of my query would.
I suppose another couple of things I can do, with regard to the general memory issue, as opposed specifically to MySQL, and particularly within the context of our own custom code, would be to wrap our code with calls to one or the other of the following PHP built-in functions:
memory_get_usage
memory_get_peak_usage
In particular since I am currently working on logging from some custom code, I can log the memory usage while I'm at it