I have a php script which takes an image, processes it and then writes the new image to file. I'm using imagick/imagemagick with php 5.3.8 with fastcgi. After reading around I thought maybe the garbage collecting function might help but it hasn't stopped php's memory usage in TOP from growing to triple digits. I used to run this script in cron.
<?php
var_dump(gc_enabled()); // true
var_dump(gc_collect_cycles()); // number comes out to 0
?>
Not sure what to do. So far the only thing that helps keep php in check is by doing a 'service php-fpm reload' every hour or so. Would using imagick as a shared ext instead of statically compiled one help? Any suggestions or insight is greatly appreciated.
Two options:
Farm out the work through gearman or the like to a script that will die completely. Generally I'll run my workers through a certain number of jobs, then have them die. They'll be restarted by supervisor in my setup so it's not a problem. The death after N requests just avoids memory issues.
As of 5.4 this might help: http://ca3.php.net/manual/en/function.apache-child-terminate.php
A note about built in vs external libraries. I haven't played with this aspect of image magick, but I saw it with GD. You get a much lower memory value from the PHP functions when you're using the external library, but the actual memory usage is nearly equal.
A good start to check for memory leaks is valgrind.
If PHP has lots of available memory to use then it doesn't bother to wipe the memory since it doesn't think it needs to. As it uses more, or if other applications start to use more memory, then it will clear the memory of what it can.
You can force the memory to be cleared for a variable by setting it to NULL, but unset() is recommended because you shouldn't need to force it to use less memory as PHP will clean up by itself.
But otherwise, a snippet of your code is required to answer your question.
My Apache gets killed sometimes because it runs out of memory. This did not use to happen. I have several hundred PHP files in this project.
I suspect that there was an unintentional recursion created -- not just a function calling itself, but function A calling B which then calls A again.
I've tried just reading the code and finding such recursion, but with no luck.
Is there a way I can tell PHP to keep track of ALL recursions, or throw a warning when its internal stack is over a certain size?
My Apache gets killed sometimes because it runs out of memory.
Well that's at least a point to start with. I could read in your question you suspect that is caused by a PHP script.
To find out which one, you need to look further. One way I could think of is that you enable PHP error logging. Then set the memory_limit to a low value so you can provoke errors where much memory consumption happens. You find the line where this happens in the error log (see as well Protocol of some PHP Memory Stretching Fun).
This should give you some potential places.
I'm not specifically sure if recursion must cause this and you wrote yourself that you only assume that. To detect recursion, you can use xdebug and limit recursion depth:
xdebug.max_nesting_level=<your preferred value>
This should give you as well more useful information in the error logs.
If both memory and recursion is not the case, it might either not be related to PHP or PHP segaults and your webserver can't handle that situation. No idea how mod_php deals with segfaults for example but you haven't specified the SAPI you're using anyway.
A typical source of PHP segfaults however are regular expressions that kick out PHP before triggering the pcre recursion limit. If you're making use of regular expressions in your code, you can just reduce the recursion limit to a lower value like and test if that helps. However, from your question it's not clear if that is a cause at all. So you first need to find out more.
I have a PHP script that we run every few minutes through a cron entry, and every now and then (about once a week) instead of ending normally, it stays running, eating up 100% of a CPU core (i'm assuming, looping infinitely)
Looking at the code and "thinking" about it, I can't find any reason for this to happen, but it does. So far, when I get 3 or more of those I kill them, and that solves the CPU issue, but I'd like to do something about this...
Is there any way to dump a process, or attach to it with a debugger so that I can know something, anything about what it's doing? (Just which PHP line it's on would be of huge help). I don't mind if the process dies when I dump, or anything.
This is a PHP script, running from the command line, in a CentOS 5.6 machine, and I'm a big noob when it comes to *nix, so if you can point me to some tutorial for dummies that'd be awesome.
Thank you!
Daniel
There's no way I'm aware of to attach a debugger to a PHP process that hasn't specifically been prepared with a PHP debugging extension (such as xdebug). However, you may be able to make some guess as to what's going on using the more general-purpose utility strace, which can deliver a trace of the system calls being run by a process. This will only tell you what system calls are being executed, but this may be enough (depending on the context) to determine what's going on anyway.
While the idea of a PHP daemon has been covered to death on here, I haven't found anything specifically related to how to do this in PHP 5.3. As I've been told, 5.3 introduced new garbage collection/memory management to allow PHP to more cleanly run as a daemon.
I know PHP's no one's first choice for this kind of thing, but in my circumstances it might have to do.
I know in PHP 4, you would have to use something like the System_Daemon class, but I was wondering if that was still needed with the new version of PHP, and wether I'd need to do anything special to use the new features.
PHP uses reference counting for managing allocated memory. When a cycle exists between objects, their reference count is never decremented and the objects are never freed (until the end of the script).
The only goal of the garbage collector added in PHP5.3 is to kill those cycles. This effectively helps in reducing the memory usage of long running scripts, like daemons.
Other than that, PHP5.3 adds nothing new for long running scripts / daemons.
There has been some efforts in making app servers in PHP lately, you may want to look at them:
https://github.com/indeyets/appserver-in-php
The garbage collector is an internal thing. It does not change how you write a daemon. And there was just a more inefficient form of garbage collection (resource freeing) before PHP 5.3, that's all. http://php.net/manual/en/features.gc.performance-considerations.php
You are supposed to still fork daemon processes, as there is no threading support to use instead. And this implicitly takes care of freeing memory, so it doesn't practically matter.
I wish to create a background process and I have been told these are usually written in C or something of that sort. I have recently found out PHP can be used to create a daemon and I was hoping to get some advice if I should make use of PHP in this way.
Here are my requirements for a daemon.
Continuously check if a row has been
added to MySQL database table
Run FFmpeg commands on what was
retrieved from database
Insert output into MySQL table
I am not sure what else I can offer to help make this decision. Just to add, I have not done C before. Only Java and PHP and basic bash scripting.
Does it even make that much of a performance difference?
Please allow for my ignorance, I am learning! :)
Thanks all
As others have noted, various versions of PHP have issues with their garbage collectors. Of course, if you know that your version does not have such issues, you eliminate that problem. The point is, you don't know (for sure) until you write the daemon and run it through valgrind to see if the installed PHP leaks or not on any given machine. So on that hand, you may write it just to discover that what Zend thinks is fixed might still be buggy, or you are dealing with a slightly older version of PHP or some extension. Icky.
The other problem is somewhat buggy signals. In my experience, signal handlers are not always entered correctly with PHP, especially when the signal is queued instead of merged. That may not be an issue for you, i.e. if you just need to handle SIGINT/SIGUSR1/SIGUSR2/SIGHUP.
So, I suggest:
If the daemon is simple, go ahead and use PHP. If it looks like its going to get rather complex, or allocate lots of memory, you might consider writing it in C after prototyping it in PHP.
I am a pretty die hard C person. However, I see nothing wrong with hammering out something quick using PHP (beyond the cases that I explained). I also see nothing wrong with using PHP to prototype something that may or may not be later rewritten in C. For instance, handling database stuff is going to be much simpler if you use PHP, versus managing callbacks using other interfaces in C. So in that instance, for a 'one off', you will surely get it done much faster.
I would be inclined to perform this task with a cron job, rather than polling the database in a daemon.
It's likely that your FFmpeg command will take a while to do it's thing, right? In that case, is it really necessary to be constantly polling the database? Wouldn't a cronjob running each minute (or every five, ten or twenty minutes for that matter) be a simpler way to achieve the same thing?
Php isn't any better or worse for this kind of thing than any of the other common scripting languages. It has fairly complete access to all of the system calls and library utilities you would need to do this sort of work. If you are most comfortable using PHP for scripting, then php will do the job for you.
The only down side is that php is not quite as ubiquitous as, say, perl or python, which is installed on almost every flavor of unix. Php is only found on systems that are going to be serving dynamic web content. Not that a Php interpreter is too large or costly to install also, but if your biggest concern is getting your program to many systems, that may be a slight hurdle.
I'll be contrary and recommend you try the php daemon. It's apparently the language you know the best. You'll presumably incorporate a timer in any case, so you can duplicate the querying frequency on the database. There's really no penalty as long as you aren't naively looping on a query.
If it's something not executed frequently, you could alternatively run the php from cron, letting youor code drain the queue and then die.
But don't be afraid to stick with what you know best, as a first approximation.
Try not to use triggers. They'll impose unnecessary coupling, and they're no fun to test and debug.
One problem with properly daemonizing a PHP script is that PHP doesn't have interfaces to the dup() or dup2() syscalls, which are needed for detaching the file descriptors.
A cron-job would probably work just fine, if near-instant actions is not required.
I'm just about to put live, a system I've built, based on the queueing daemon 'beanstalkd'. I send various small messages from (in this case, PHP) webpage calls to the daemon, and a PHP script then picks them up from the queue and performs various tasks, such as resizing images or checking databases (often passing info back via a Memcache-based store).
To avoid long-running processes, I've wrapped it in a BASH script, that, depending on the value returned from the script ("exit(1);") will restart the script, for every (say) 50 tasks it's performed. If it's restarting because I plan it to, it will do so instantly, any other exit value (the default is 0, so I don't use that) would pause a few seconds before it was restarted.
Running as a cron job with sensibly determined periodicity, a PHP script can do the job, and production stability is certainly achievable. You might want to limit the number of simultaneous FFMpeg instances, and be sure to have complete application logging and exception handling. I have implemented continuously running polling processes in Java, as well as the every-ten-minute cron'd PHP script, and both do the job nicely.
You might want to consider making a mysql trigger that executes a system command (i.e. FFmpeg) instead of a daemon. If some lag isn't a problem, you could also put something in cron that executes every few minutes to check. Cron would be my choice, if it is an option.
To answer your question, php is perfectly fine to run as a daemon. It does not have to be done in C.
If you combine the answers from Kent Fredric, tokenmacguy and Domster you get something useful.
php is probably not good for long execution times,
so let's keep every execution cycle short and make sure the OS takes care of the cleanup of any memoryleaks.
As a tool to start your php script cron can be a good tool.
And if you do it like that, there is not much difference between languages.
However, the question still stands.
Is php even capable to run as a normal daemon for long times (some years)?
Or will assorted memoryleaks eat up all your ram and kill the system?
/Johan
If you do so, pay attention to memory leaks. PHP 5.2 has some problems with its garbage collector, according to this (fixed in 5.3). Perhaps its better to use cron, so the script starts clean every run.
For what you've described, I would go with a daemon. Make sure that you stick a sleep in the poll loop, so that you don't bombard the database when there are no new tasks. A cronjob works better for workflow/report type of jobs, where there isn't some particular event that triggers the next run.
As mentioned, PHP has some problems with memory management. You need to be sure that you test your code for memory leaks, since these would build up over time, in a long running script. PHP doesn't have real garbage collection - It relies on reference counting, which means that cyclic references will cause leaks. If you're aware of this, you can code around it.
If you do decided to go down the daemon route, there is a great PEAR module called System_Daemon which I've recently used successfully on a PHP v5.3.0 installation. It is documented on the authors blog: http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php
If you have PEAR installed, you can install this module using:
pear install -f System_Daemon
You will also need to create a initialisation script: /etc/init.d/<your_daemon_name>
Then you can:
Start Daemon: /etc/init.d/projNotifMailDaemon start
Stop Daemon: /etc/init.d/projNotifMailDaemon stop
Logs are kept at: /var/log/<your_daemon_name>.log
I wouldn't recommend it. PHP is not designed for longterm execution. Its designed primarily with short lived pages.
In my experience PHP can have problems with leaking memory for some of the larger tasks.
A cron job and a little bit of bash scripting should be everything you need by the sounds of it. You can do things like:
$file=`mysqlquery -h server < "select file from table;"`
ffmpeg $file -fps 50 output.a etc.
so bash would be easier to write, port and maintain IMHO than to use PHP.
If you know what you are doing sure. You need to understand your operating system well. PHP generally isn't suited for most daemons because it isn't threaded and doesn't have a decent event based system for all tasks. However if it suits your needs then no problem. Modern PHP (5.3+) is really stable and doesn't have any memory leaks. As long as you enable the GC and don't implement your own memory leaks, etc you'll be fine.
Here are the stats for one daemon I am running:
uptime 17 days (last restart due to PHP upgrade).
bytes written: 200GB
connections: hundreds
connections handled, hundreds of thousands
items/requests processed: millions
node.js is generally better suited although has some minor annoyances. Some attempts to improve PHP in the same areas have been made but they aren't really that great.
Cron job? Yes.
Daemon which runs forever? No.
PHP does not have a garbage collector (or at least, last time I checked it did not). Therefore, if you create a circular reference, it NEVER gets cleaned up - at least not until the main script execution finishes. In daemon process this is approximately never.
If they've added a GC in new versions, then yes you can.
Go for it. I had to do it once also.
Like others said, it's not ideal but it'll get-er-done. Using Windows, right? Good.
If you only need it to run occasionally (Once per hour, etc).
Make a new shortcut to your firefox, place it somewhere relevant.
Open up the properties for the shortcut, change "Target" to:
"C:\Program Files\Mozilla Firefox\firefox.exe" http://localhost/path/to/script.php
Go to Control Panel>Scheduled Tasks
Point your new scheduled task at the shortcut.
If you need it to run constantly or pseudo-constantly, you'll need to spice the script up a bit.
Start your script with
set_time_limit(0);
ob_implicit_flush(true);
If the script uses a loop (like while) you have to clear the buffer:
$i=0;
while($i<sizeof($my_array)){
//do stuff
flush();
ob_clean();
sleep(17);
$i++;
}