I have just set up a new server using Ubuntu 16.04 and php7-fpm. I've set up 3 cronjobs to run: every 5 minutes, every 7 minutes and every 10 minutes. All of the functions are PHP functions, that are connecting to a MySQL database + querying results from a 3rd party server. Looking at the results the cron jobs are always running from beginning to the end. So that can't be the issue - I believe.
So everything works perfectly, however, I can see that the RAM usage keeps increasing gradually and it's not stopping. I'm seeing that there are like 100+ PHP processes open on the server and each of them is using some of the virtual RAM. I guess this is because one PHP worker is being opened every time a CRON job starts.
Here are a few pictures:
I can see that there are 4 PHP processes that are standing out from the others. Looks like that there is a lot of IO there also.
So I believe that something's not alright around there. The other strange thing is that I have more than 100+ PHP processes since yesterday -> which is logical since the cronjobs have run more than 100 times.
What is the issue here and how can I fix it? Is it possible that I have so many processes because I'm not returning anything at the end of the script? Or it must be something that's causing the script to overlap?
I'm using a Cloud Server with 2vCores, 4GB RAM, Ubuntu 16.04 OS, PHP7.0-fpm, Nginx.
Related
I have a cloud server in Rackspace with cPanel installed. I have some 16 sites running on it. Out of them 14 sites run under a single account (this is a Drupal multisite installation). Everything has been running fine for last 5 months. Recently my server became unresponsive and had to be rebooted. Later it was found out that the server ran out of memory.
The issue now continue to occur intermittently. At that time I can find a lot of php processes popping out at once, memory usage increases at that time and the comes to nearly 200 MB out of the total 8GB.
/usr/bin/php /home/username/public_html/index.php
All sites become inaccessible. The load average also spikes. After 5-8 minutes the huge number of php processes disappears and then memory usage also comes to normal. This lasts not more than 5-6 minutes
The issue now continue to occur intermittently. And I checked all server logs and could not find any trace of the issue. I checked the server using maldet, rkhunter and could not find any traces of malicious codes or back-doors
The strange issue is that the issue does not occur during the most peak hours. It occurs during off-peak hours as well. There is no pattern in which this issue occurs.
I can find that there were 150 php instances running at once yesterday.
Can someone guide me in the correct direction? Is this a server side issue or has something to do with the internal site functions?
For some reason today my EC2 server keeps hitting 100% CPU and high disk reads. I've turned all of my cronjobs off, and yet it's still happening. My Database is an RDS outside the server. Any initial items I should check? I'm using PHP scripts for my cronjobs (8 seperate scripts that run anywhere from every 5 minutes to twice a day).
Both happen at the same time. I've also attached a screenshot of what shows when I run top.
If it is happening right now, run top to see if it is actually a PHP script: Maybe it's some other process.
If you can't catch it on the act, I suggest you set up atop to provide the same data in retrospect.
I have a PHP script that pulls down a bunch of RSS feeds. To prevent overloading the publishers' servers, I use the PHP Sleep function to slow things down.
The entire script could last for a couple of hours.
If I run this from a Cron job on GoDaddy, it will happily work for 5 - 10 minutes and then return a server error. I checked and the PHP maximum execution time is 30 seconds, so I'm not sure if this is the cause of the problem.
If I run the job on my Mac, my local PHP also has a default maximum execution time of 30 seconds, but this script does work if I run it from the terminal, but I don't understand why.
How do I loop a script that will exceed 30 seconds without running into unreliability problems?
Help appreciated.
Short answer is use set_time_limit(0) to allow for a long-running script. Your terminal (CLI) PHP probably has it set to 0. You could also be running out of memory, especially on PHP 5.2 or older. Log all errors to a file, and inspect it.
You could rewrite your program to be able to work on a subset of the data during one run. The benefit of that approach is you could use it to run 24/7 or to run every five minutes, depending on what the PHP environment supports. You could also run multiple instances at a time, each working on their own data.
It's already several weeks since occasionaly all 8 CPU cores on server machine got almost to 100% load because of the Apache processes (just Apache, there are no other high-load processes except MySQL).
Restarting Apache always works but usually just for couple hours but once it was even a week. There's no visible pattern. Overall load gets to around 10 but the machine is still respondinng very quickly (usually it's floating from 1 to 4 or 6 tops, depends on daytime). But there is no connection between these never-ending spikes and actual server stress caused by visits or cron jobs. It seems pretty random, so my best guess is a faulty PHP script cycling over and over doing god knows what.
If nothing else, such a useless load is heating up the CPU, shortening his lifetime and who knows how many resources are spent by this PHP process I can't identify.
All the PHP processes are wrapped by Apache processes, so I can't actually see which PHP scripts are currently running in the UNIX process list. Is there any other way around to point to specific script? Other than implement some kind of tracking code inside every PHP script? I'm not running webhosting but there is more than one site and I don't see how could I implement a tracking code in such an extensive code spread across the webserver.
Therefore I'm looking for pure unix solution (Debian). In worst case it would point me to PHP script which includes dozens of PHP classes and contollers. In the best case it would be some simple script. Either way, it's a start. I think.
You may not see the php processes in process list, but you can get their PIDs by this PHP function: getmypid();
In case you're using Debian, get the value returned by the above function, then open a terminal and use the 'top' command:
top -p xxxx
where xxxx is the PID returned by "getmypid();"
I have created a php/mysql scraper, which is running fine, and have no idea how to most-efficiently run it as a cron job.
There are 300 sites, each with between 20 - 200 pages being scraped. It takes between 4 - 7 hours to scrape all the sites (depending on network latency and other factors). The scraper needs to do a complete run once daily.
Should I run this as 1 cron job which runs for the entire 4 - 7 hours, or run it every hour 7 times, or run it every 10 minutes until complete?
The script is set up to run from the cron like this:
while($starttime+600 > time()){
do_scrape();
}
Which will run the do_scrape() function, which scrapes 10 urls at a time, until (in this case) 600 seconds has passed. The do_scrape can take between 5 - 60 seconds to run.
I am asking here as I cant find any information on the web about how to run this, and am kind of wary about getting this running daily, as php isnt really designed to be run as a single script for 7 hours.
I wrote it in vanilla PHP/mysql, and it is running on cut down debian VPS with only lighttpd/mysql/php5 installed. I have run it with a timeout of 6000 seconds (100 minutes) without any issue (the server didnt fall over).
Any advice on how to go about this task is appreciated. What should I be watching out for etc..? or am i going about executing this all wrong?
Thanks!
There's nothing wrong with running a well-written PHP script for long periods. I have some scripts that have literally been running continuously for months. Just watch your memory usage, and you should be fine.
That said, your architecture is pretty basic, and is unlikely scale very well.
You might consider moving from a big monolithic script to a divide-and-conquer strategy. For instance, it sounds like your script is making synchronous requests for every URL is scrapes. If that's true, then most of that 7 hour run time is spent idly waiting for a response from some remote server.
In an ideal world, you wouldn't write this kind of thing PHP. Some language that handles threads and can easily do asynchronous http requests with callback would be much better suited.
That said, if I were doing this in PHP, I'd be aiming at having a script that kicks of N children who grab data from URLs, and stick the response data in some kind of work queue, and then another script that pretty much runs all the time, processing any work it finds in the queue.
Then you just cron your fetcher-script-manager to run once an hour, it manages some worker processes that fetch the data (in parellel, so latency doesn't kill you), and stick the work on the queue. Then the queue-cruncher sees the work on the queue and crunches it.
Depending on how you implement the queue, this could scale pretty well. You could have multiple boxes fetching remote data, and sticking it on some central queue box (with a queue implemented in mysql, or memcache, or whatever). You could even conceivably have multiple boxes taking work from the queue and doing the work.
Of course, the devil is in the details, but this design is generally more scalable and usually more robust than a single-threaded fetch-process-repeat script.
You shouldn't have a problem running it once a day to completion. That's the way I would do it. Timeouts are a big issue if php is being served through a web server, but since you are interpreting directly through the php executable this is ok. I would advise you to use python or something else that is more task-friendly, though.