So last night I had a cpu spike for 100% for 30 minutes which made the server not very usable :-(
The server doesn't seem to be the quickest even today (its running on Amazon cloud!).
The application is a chat application which only has about registered 5 users who's client polls a php script every every 5 seconds for new information ( each request hits mysql).
Running some commands I found on the net it returned that I have 200 thousand connections - is this active or since the server was up?
Can anyon offer any advice if there is anything out of the ordinary in the below.
(Note these stats are from today where there were only 2 users logged in)
120 thosand connections since your mysql running up. About 100% of cpu. Check apache logs in /var/logs/apacheX looking for excessive error messages. It can be the server slow.
Related
I have just set up a new server using Ubuntu 16.04 and php7-fpm. I've set up 3 cronjobs to run: every 5 minutes, every 7 minutes and every 10 minutes. All of the functions are PHP functions, that are connecting to a MySQL database + querying results from a 3rd party server. Looking at the results the cron jobs are always running from beginning to the end. So that can't be the issue - I believe.
So everything works perfectly, however, I can see that the RAM usage keeps increasing gradually and it's not stopping. I'm seeing that there are like 100+ PHP processes open on the server and each of them is using some of the virtual RAM. I guess this is because one PHP worker is being opened every time a CRON job starts.
Here are a few pictures:
I can see that there are 4 PHP processes that are standing out from the others. Looks like that there is a lot of IO there also.
So I believe that something's not alright around there. The other strange thing is that I have more than 100+ PHP processes since yesterday -> which is logical since the cronjobs have run more than 100 times.
What is the issue here and how can I fix it? Is it possible that I have so many processes because I'm not returning anything at the end of the script? Or it must be something that's causing the script to overlap?
I'm using a Cloud Server with 2vCores, 4GB RAM, Ubuntu 16.04 OS, PHP7.0-fpm, Nginx.
I'm trying to figure out what is causing my system to open a large number of PHP threads. This issue has occurred 3 times over the last 2 weeks, and is capable of crashing our application if undetected for several hours, as once it opens up 300 database connections it prevents anyone further from connecting.
The application is based on CakePHP 2.X, is running across multiple EC2 Instances, which share an RDS database.
The primary identifier that something is going wrong is high number of database connections, as shown by this graph:
We have CloudWatch monitoring setup to notify us on slack when average connections go above 40 for more than 5 minutes (normally connections don't go much above 10).
Looking at New Relic I can also see that the number of php processes steadily increased by 1 per minute. This is on our operations server which just handles background processing and tasks, and does not handle any web traffic.
Over the same time the graphs on the web servers appear normal.
In looking at New Relics information on long-running processes there is no information provided that would suggest any php processes ran for 20+ minutes, however, these processes were killed manually which may be why they're not visible within New Relic - I believe it may not record processes which are killed.
While this issue has now occurred 3 times, I'm still unsure what is causing the problem or how to debug what a particular running php thread is doing.
The last time this happened I could see all the php threads running, and could see they had been running for some time, but had no idea what they were doing or how to find out what they were doing, and to prevent the database from becoming overloaded I had to kill them all.
Are there any tools, or other information I am overlooking here which may help me in my search to determine which particular process is causing this issue?
I have a apache 2.4, php, mysql based web application. The application works like charm most of the time with 40ms response times. Already performance tuned with apache cache, php's opcache and apache mod_deflate.
However rarely once every 2 hours or so...when I invoke the application on a browser...it hangs saying waiting ..... for few seconds say 40 seconds or so...and then picks up back to super speed.
what do you think could be possible reasons?
Just for information. it's all localhost. and there is no network.
I have got an PHP application running a report and firing off about 1 million SQL queries to SQL server within 90 seconds. During the period of time no one else can use this web-based application - the egg timer is rolling but nothing loads until the report is timed out or completed. I tested this problem in an isolated environment with just myself in there, ran the report in a browser, then any actions from other browser windows to this application site hung.
Some details about the testing environment:
Windows 2008 R2 x64 - IIS 7.5 - PHP 5.3.8 via FastCGI
Windows 2008 R2 x64 - SQL Server 2008 R2 x64
FastCGI settings in IIS:
Instance MaxRequests = 200
Max Instances = 16
Activity Timeout = 70
Idle Timeout = 300
Queue Length = 1000
Rapid Fails PerMin = 10
Request Timeout = 90
Each of the SQL requests completes less than 60ms on the SQL server side. CPU loads of both of the Web server and the SQL server are less than 10%. The Web server has 16GB RAM and about 60% RAM are available when running the report.
It seems that, PHP's been firing off too many requests to the SQL server and becoming too busy to handle other requests. If this is the case then there should be something I can tweak to make PHP handle more concurrent requests.
Does anyone know? Please help!
I'll just stab in the dark here and assume it's due to session locking.
When you use the standard session handler that comes with PHP, it makes sure that your session files cannot be corrupted by using (advisory) write locks throughout the course of your script's execution (unless session_write_close() is called earlier).
Other scripts that try to access the same session (your browser would pass the same cookie value) will wait for the lock to get released, as long as it takes.
You can verify this by using two completely different browsers to simulate two users (one runs the report, the other accesses the site). If that works, you're pretty sure it's due to session locking.
This shouldn't be a problem, because you will know when you're running a report, but if this might cause issues nonetheless you can consider two things:
don't start a session for the report script (but this also means unauthorized users could try to run your report script)
close the session before your grunt work starts, by using session_write_close() after you have verified the user's identity.
I am developing a big application and i have to load test it. It is a EC2 based cluster with one HighCPU Ex.Large instance for application which runs PHP / NGinx.
This applicaton is responsible for reading data from a redis server which holds some 5k - 10k key values, it then makes the response and logs the data into a mongoDB server and replies back to client.
Whenever i send a request to the app server, it does all its computations in about 20 - 25 ms which is awesome.
I am now trying to do some load testing and i run a php based app on my laptop to send requests to server. Many thousands of them quickly over 20 - 30 seconds. During this load period, whenever i open the app URL in the browser, it replies back with the execution time of around 25 - 35 ms which is again cool. So i am sure that redis and mongo are not causing bottlenecks. But it is taking about 25 seconds to get the response back during load.
The high CPU ex. large instance has 8 GB RAM and 8 cores.
Also, during the load test, the top command shows about 4 - 6 php_cgi processes consuming some 15 - 20% of CPU.
I have 50 worker processes on nginx and 1024 worker connections.
What could be the issue causing the bottleneck ?
IF this doesnt work out, i am seriously considering moving out to a whole java application with an embedded webserver and an embedded cache.
UPDATE - increased PHP_FCGI_CHILDREN to 8 and it halfed the response time during load
50 worker processes is too many, you need only one worker process per CPU core. Using more worker processes will invoke inter-process switching, that will consume many time.
What you can do now:
1. Set worker process to minimum (one worker per CPU, e.g. 4 worker process if you have 4 cpu units), but worker connections - to maximum (10240 for example)
Tune up TCP stack via sysctl. You can reach stack limits if you have many connections
Get statistics from nginx stub_status module (you can use munin + nginx, its easy to setup and gave you enough information about system status).
Check nginx error.log and system messages log for errors.
Tune up nginx (decrease connection timings and max query size).
I hope that helps you.