I have a scheduled task that runs a script on a regular basis (every hour). This script does some heavy interaction with the database and filesystem and regularly takes several minutes to run. The problem is, the server's cpu-usage spikes while the script is running and slows down normal operations. Is there a way to throttle this process so that it takes longer but does not consume as many resources?
I've looked at different configuration options for PHP but there does not appear to be any that fit my needs.
Setting memory_limit in php.ini to something lower causes my data objects to overflow quite easily.
I've seen similar posts where people suggested using sleep() at certain points in the script but that does not prevent the script from spiking the server.
The optimal solution would be some way to tell the Lamp (in this case Wamp) stack to only use 10% max cpu utilization. I'm not concerned at all about runtime and would prefer that it take longer if it means saving cpu cycles per second. My alternate solution would be to setup a different server with database replication so the cron could go to town without slowing everything else down.
Environment: Windows Server 2k3, Apache 2.2.11, PHP 5.2.9, MySQL 5.1
I appreciate any insight to this situation.
EDIT: I appreciate all the answers, even the ones that are *nix-specific. It's still early enough in my situation to change the hosting environment. Hopefully this question will help others out regardless of the OS.
This is a tricky problem. If you're running the PHP script via the command line, you can set the process's scheduling priority to low (start /low php.exe myscript.php I believe). If your PHP script itself is actually doing most of the processing that's eating your CPU, this might work. However, you said you are doing some heavy database and filesystem interaction, which this solution will not help. It looks like there is a MySQL hint "LOW_PRIORITY" for INSERT and UPDATE queries that may help you there, but I have not tried those.
You can set processes in Windows to be a lower priority. I'm not sure how the process is being kicked off, but if you set the process to be a low priority, whatever wants CPU resources will get them if you set the priority to be really low.
In UNIX (LAMP) I managed to solve the problem by checking the load of the server before continuing the loop
function get_server_load($windows = 0) {
$os = strtolower(PHP_OS);
if(strpos($os, "win") === false) {
if(file_exists("/proc/loadavg")) {
$load = file_get_contents("/proc/loadavg");
$load = explode(' ', $load);
return $load;
}
elseif(function_exists("shell_exec")) {
$load = explode(' ', `uptime`);
return $load;
}
else {
return "";
}
}
}
for(... ... ...){
$data = get_server_load();
if($data[0] < 0.2){
// continue
}else{
sleep(1);
}
}
This function should work also on windows but I can't guarantee it. On linux it gives you back an array with the load of the last 1 minute, 5 minutes and 15 minutes
Also, consider to start your scripts (if by CLI) with a lower priority (in Linux, use "nice")
You can also use other values before continuing the loop, like the number of Apache active processes (you can parse the page 127.0.0.1/server_status?auto if you enabled the mod_status in httpd.conf), or also the MySQL situation (active connections?)
Can you alter your cron entry to launch your script using nice?
Not a good idea to use a server for serving clients and analyse data.
So if you are looking for a final solution, make a few redesign of your application and offload the data analysis from the frontends and the live database to another system dedicated to this task.
Even if you can successfully throttle the analyzer, it would use up precious resources otherwise would be available to serve the users.
This might be a difficult change but it may worth it refactoring your data structures into iterators. Also, if you have circular references in your code, provide a method like clearReferences() that unsets these objects. This is a problem that is solved in PHP 5.3 by the way.
So if you have:
class Row
{
protected $_table;
public function __construct($table)
{
$this->_table = $table;
}
}
class Table
{
protected $_row;
public function __construct()
{
$this->_row = new Row($this);
}
}
Add a clearReferences() method to the Row class:
class Row
{
public function clearReferences()
{
$this->_table = null;
}
}
That's all I can come up with for the moment.
I have a bunch of scripts that I run from cron in a similar way using nice:
0 * * * * nice -n19 php myscript.php
This won't help the RAM utilization (only changing the way the script is written can do that), but it only uses the CPU that would otherwise be idle.
EDIT: didn't see that the question involved a Windows environment, sorry... leaving this in for any *nix users having the same problem..
Perhaps what your script is simply trying to do too much all at once. Would it do less if it ran three times an hour?
Another solution might be to setup an additional server just for running this sort of 'backend' processing. This would particularly effective if it is not putting undue load in the database, just the web server.
Yet another approach to look at is whether it's work can be divided in a different direction. These sorts of scripts often have a few big SQL statements that generate results used to generate a whole lot of little SQL statements. If the latter could be put aside somewhere, they can be run against the database as a later step. Such an approach might also let you use an unbuffered query to fetch the pre-processing data which could cut down significantly on memory consumption by the PHP code.
If you have it(Apache) running as a service, you can change the priority
settings in the Win control center /services.
Your CPU usage will spike anyway, but other programs will be preferred
by the scheduler.
Also try putting the database/server on a different hd than your
Applications.
Related
I have a daily cron job which takes about 5 minutes to run (it does some data gathering and then various database updates). It works fine, but the problem is that, during those 5 minutes, the site is completely unresponsive to any requests, HTTP or otherwise.
It would appear that the cron job script takes up all the resources while it runs. I couldn't find anything in the PHP docs to help me out here - how can I make the script know to only use up, say, 50% of available resources? I'd much rather have it run for 10 minutes and have the site available to users during that time, than have it run for 5 minutes and have user complaints about downtime every single day.
I'm sure I could come up with a way to configure the server itself to make this happen, but I would much prefer if there was a built-in approach in PHP to resolving this issue. Is there?
Alternatively, as plan B, we could redirect all user requests to a static downtime page while the script is running (as opposed to what's happening now, which is the page loading indefinitely or eventually timing out).
A normal script can't hog up 100% of resources, resources get split over the processes. It could slow everything down intensly, but not lock all resources in (without doing some funky stuff). You could get a hint by doing top -s in your commandline, see which process takes up a lot.
That leads to conclude that something locks all further processes. As Arkascha comments, there is a fair chance that your database gets locked. This answer explains which table type you should use; If you do not have it set to InnoDB, you probally want that, at least for the locking tables.
It could also be disk I/O if you write huge files, try to split it into smaller read/writes or try to place some of the info (e.g. if it are files with lists) to your database (assuming that has room to spare).
It could also be CPU. To fix that, you need to make your code more efficient. Recheck your code, see if you do heavy operations and try to make those smaller. Normally you want this as fast as possible, now you want them as lightweight as possible, this changes the way you write code.
If it still locks up, it's time to debug. Turn off a large part of your code and check if the locking still happens. Continue turning on code untill you notice locking. Then fix that. Try to figure out what is costing you so much. Only a few scripts require intense resources, it is now time to optimize. One option might be splitting it into two (or more) steps. Run a cron that prepares/sanites the data, and one that processed the data. These dont have to run at syncronical, there might be a few minutes between them.
If that is not an option, benchmark your code and improve as much as you can. If you have a heavy query, it might improve by selecting only ID's in the heavy query and use a second query just to fetch the data. If you can, use your database to filter, sort and manage data, don't do that in PHP.
What I have also implemented once is a sleep every N actions.
If your script really is that extreme, another solution could be moving it to a time when little/no visitors are on your site. Even if you remove the bottleneck, nobody likes a slow website.
And there is always the option of increasing your hardware.
You don't mention which resources are your bottleneck; CPU, memory or disk I/O.
However if it is CPU or memory you can do something this in you script:
http://php.net/manual/en/function.sys-getloadavg.php
http://php.net/manual/en/function.memory-get-usage.php
$yourlimit = 100000000;
$load = sys_getloadavg();
if ($load[0] > 0.80 || memory_get_usage() > $yourlimit) {
sleep(5);
}
Another thing to try would be to set your process priority in your script.
This requires SU though, which should be fine for a cronjob?
http://php.net/manual/en/function.proc-nice.php
proc_nice(50);
I did a quick test for both and it work like a charm, thanks for asking I have cronjob like that as well and will implement it. It looks like the proc_nice only will do fine.
My test code:
proc_nice(50);
$yourlimit = 100000000;
while (1) {
$x = $x+1;
$load = sys_getloadavg();
if ($load[0] > 0.80 || memory_get_usage() > $yourlimit) {
sleep(5);
}
echo $x."\n";
}
It really depend of your environment.
If using a unix base, there is built-in tools to limit cpu/priority of a given process.
You can limit the server or php alone, wich is probably not what you are looking for.
What you can do first is to separate your task in a separate process.
There is popen for that, but i found it much more easier to make the process as a bash script. Let''s name it hugetask for the example.
#!/usr/bin/php
<?php
// Huge task here
Then to call from the command line (or cron):
nice -n 15 ./hugetask
This will limit the scheduling. It mean it will low the priority of the task against others. The system will do the job.
You can as well call it from your php directly:
exec("nice -n 15 ./hugetask &");
Usage: nice [OPTION] [COMMAND [ARG]...] Run COMMAND with an adjusted
niceness, which affects process scheduling. With no COMMAND, print the
current niceness. Niceness values range from
-20 (most favorable to the process) to 19 (least favorable to the process).
To create a cpu limit, see the tool cpulimit which has more options.
This said, usually i am just putting some usleep() in my scripts, to slow it down and avoid to create a funnel of data. This is ok if you are using loops in your script. If you slow down your task to run in say 30 minutes, there won't be much issues.
See also proc_nice http://php.net/manual/en/function.proc-nice.php
proc_nice() changes the priority of the current process by the amount
specified in increment. A positive increment will lower the priority
of the current process, whereas a negative increment will raise the
priority.
And sys_getloadavg can also help. It will return an array of the system load in the last 1,5, and 15 minutes.
It can be used as a test condition before launching the huge task.
Or to log the average to find the best day time to launch huge task. It can be susrprising!
print_r(sys_getloadavg());
http://php.net/manual/en/function.sys-getloadavg.php
You could try to delay execution using sleep. Just cause your script to pause between several updates of your database.
sleep(60); // stop execution for 60 seconds
Although this depends a lot on the kind of process you are doing in your script. Maybe or not helpful in your case. Worth a try, so you could
Split your queries
do the updates in steps with sleep inbetween
References
Using sleep for cron process
I could not describe it better than the quote in the above answer:
Maybe you're walking the database of 9,000,000 book titles and updating about 10% of them. That process has to run in the middle of the day, but there are so many updates to be done that running your batch program drags the database server down to a crawl for other users.
So modify the batch process to submit, say, 1000 updates, then sleep for 5 seconds to give the database server a chance to finish processing any requests from other users that have backed up.
Sleep and server resources
sleep resources depend on OS
adding sleep to allevaite server resources
Probably to minimize you memory usage you should process heavy and lengthy operations in batches. If you query the database using an ORM like doctrine you can easily use existing functions
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/batch-processing.html
It's hard to tell what exactly the issue may be without having a look at your code (cron script). But to confirm that the issue is caused by the cron job you can run the script manually and check website responsiveness. If you notice the site being down when running the cron job then we would have to have a look at your script in order to come up with a solution.
Many loops in your cron script might consume a lot of CPU resources.
To prevent that and reduce CPU usage simply put some delays in your script, for example:
while($long_time_condition) {
//Do something here
usleep(100000);
}
Basically, you are giving the processor some time to do something else.
Also you can use the proc_nice() function to change the process priority. For example proc_nice(20);//very low priority. Look at this question.
If you want to find the bottlenecks in your code you can try to use Xdebug profiler.
Just set it up in your dev environment, start the cron manually and then profile any page. Also you can profile your cron script as well php -d xdebug.profiler_enable=On script.php, look at this question.
If you suspect that the database is your bottleneck than import pretty large dataset (or entire database) in your local database and repeat the steps, logging and inspecting all the queries.
Alternatively if it possible setup the Xdebug on the staging server where the server is as close as possible to production and profile the page during cron execution.
I've been looking at asynchronous database requests in PHP using mysqlnd. The code is working correctly but comparing performance pulling data from one reasonable sized table versus the same data split across multiple tables using asynchronous requests I'm not getting anything like the performance I would expect although it does seem fairly changeable according to hardware setup.
As I understand it I should be achieving, rather than:
x = a + b + c + d
Instead:
x = max(a, b, c, d)
Where x is the total time taken and a to d are the times for individual requests. What I am actually seeing is a rather minor increase in performance on some setups and on others worse performance as if requests weren't asynchronous at all. Any thoughts or experiences from others who may have worked with this and come across the same are welcome.
EDIT: Measuring the timings here, we are talking about queries spread over 10 tables, individually the queries take no more than around 8 seconds to complete, combining the time each individual request takes to complete (not asynchronously) it totals around 18 seconds.
Performing the same requests asynchronously total query time is also around 18 seconds. So clearly the requests are not being executed in parallel against the database.
EDIT: Code used is exactly as shown in the documentation here
<?php
$link1 = mysqli_connect();
$link1->query("SELECT 'test'", MYSQLI_ASYNC);
$all_links = array($link1);
$processed = 0;
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) {
$links[] = $errors[] = $reject[] = $link;
}
if (!mysqli_poll($links, $errors, $reject, 1)) {
continue;
}
foreach ($links as $link) {
if ($result = $link->reap_async_query()) {
print_r($result->fetch_row());
if (is_object($result))
mysqli_free_result($result);
} else die(sprintf("MySQLi Error: %s", mysqli_error($link)));
$processed++;
}
} while ($processed < count($all_links));
?>
I'll expand my comments and I'll try to explain why you won't gain any performance using the setup you have currently.
Asynchronous, in your case, means that the process of retrieving data is asynchronous compared to the rest of your code. The two moving parts (getting data) and working with the data are separate and are executed one after another, but only when the data arrives.
This implies that you want to utilize the CPU to its fullest, so you won't invoke PHP code until the data is ready.
In order for that to work, you must seize the control of PHP process and make it use one of operating system's event interfaces (epoll on Linux, or IOCP on Windows). Since PHP is either embedded into a web server (mod_php) or runs as its own standalone FCGI server (php-fpm), that implies the best utilization of asynchronous data fetching would be when you run a CLI php script since it's quite difficult to utilize event interfaces otherwise.
However, let's focus on your problem and why your code isn't faster.
You assumed that you are CPU bound and your solution was to retrieve data in chunks and process them that way - that's great, however since nothing you do yields faster execution, that means you are 100% I/O bound.
The process of retrieving data from databases forces the hard disk to perform seeking. No matter how much you "chunk" that, if the disk is slow and if the data is scattered around the disk - that part will be slow and creating more workers that deal with parts of the data will just make the system slower and slower since each worker will have the same problem with retrieving the data.
I'd conclude that your issue lies in the slow hard disk, too big of a data set that might be improperly constructed for chunked retrieval. I suggest updating this question or creating another question that will help you retrieve data faster and in a more optimal way.
I'm currently working on a scheduled tasks (like cronjobs) system in my CMS application. I would like to know if there's any possibility to terminate a piece of code when it's execution time is getting longer than a specified time limit.
Some scheduled tasks may be heavy, a task may run into a infinite loop, to keep it short; there are lots of reasons why tasks could take up a lot of time. One of the few problems with this is that the task would take up much more resources than it actually needs which decreases the performance of the server it's running on, this would make the CMS and the site it self less reliable.
Think of the system as follows: Once the scheduled tasks manager is triggered, it will execute all tasks that where scheduled for that time. Think of a simple for loop which would execute each task, synchronized, one after the other. Think of this simple example bellow:
// This array would contain an array with all scheduled tasks that should be executed.
$tasks = Array();
// Loop through each task
foreach($tasks as $t) {
// Execute the task
$t->execute();
}
Now, the $task->execute(); method makes PHP's interpreter wait until the code of the task is executed. My question is if there's any possible way to terminate the code running behind the execute(); method if it's execution time is getting longer than a specified time.
For example, if a time limit of 10 seconds is supplied, and a task is taking more than 30 seconds to execute it's code, the running code of the task should be terminated after 10 seconds to allow the task scheduler to continue to the next task.
It is possible to implement this feature inside the code of each scheduled task with a simple if-statement which checks the time each time a part of the code is being executed. The problem with this is that this needs to be implemented in the code of each task, since this system is meant to be used for 3th party developers to used these kind of tasks properly to implement these with their plugins. I don't like to 'force' all the developers that use the tasks to implement such features.
I know about the set_time_limit(); method PHP supplies, the problem is though that this method would terminate the whole session instead of that one method, this is why this doesn't solve the problem.
"Porting" my comment to an answer:
This other question is very similar.
PHP, by its default implementation, runs scripts in a single thread so it is impossible to cancel code execution from the script itself.
Only very, very few PHP functions support asynchronous operation, e.g. you can run an asynchronous query with mysqlnd, however this particular feature is part of the mysql driver, not PHP.
There are extensions which introduce multi-threading to PHP, namely pthread.
To recommend yet another source, read this answer by the author of the extension.
Unfortunately, for a "mainstream" project (which should run as good as possible on as generic setups as possible), using pthreads is not do-able.
The extension is unlikely to be installed on any shared webspaces.
Also, keep in mind that thread-safe programming (or scripting for those who care) is a serious task and you have to care for it. This means preparing your threaded code to not lock up waiting for each other (deadlock), corrupt each other's data, etc.
A detailed answer on the term of thread-safety can be found here.
Thoroughly think about the work involved, the risks of messing something up, and the gains of using threading.
Make sure your desire is really important enough to introduce complex threading.
I've encountered this kind of problem, I have a trick that can solve this. My workaround was that, I used cURL to execute the local script like this:
// This is the curl call which will execute the local script
$execute_script = curl_call('local_script.php');
In the 'local_script.php' file is the code to execute
// In local_script.php
// Set time limit
set_time_limit(10);
// Execute task
$task->execute();
// Print 1 to let the curl call know that the script executed at the right time
echo '1';
Then, in the cURL call part, you can check if the script executed just fine when it returns "1"
if ($execute_script == '1'){
// Successful script
} else {
// Timed out
}
Hope that helps
You can just break the loop if it exceeds the time:
$max = 10; //maximum time in seconds
$start = microtime(true);
foreach($tasks as $t) {
// Execute the task
$t->execute();
if (microtime(true) - $start > $max) {
break;
}
}
I need some implementation advice. I have a MYSQL DB that will be written to remotely for tasks to process locally and I need my application which is written in PHP to execute these tasks imediatly as they come in.
But of course my PHP app needs to be told when to run. I thought about using cron jobs but my app is on a windows machine. Secondly, I need to be constantly checking every few seconds and cron can only do every minute.
I thought of writing a PHP daemon but I am getting consued on hows its going to work and if its even a good idea!
I would appreciate any advice on the best way to do this.
pyCron is a good CRON alternative for Windows:
Since this task is quite simple I would just set up pyCron to run the following script every minute:
set_time_limit(60); // one minute, same as CRON ;)
ignore_user_abort(false); // you might wanna set this to true
while (true)
{
$jobs = getPendingJobs();
if ((is_array($jobs) === true) && (count($jobs) > 0))
{
foreach ($jobs as $job)
{
if (executeJob($job) === true)
{
markCompleted($job);
}
}
}
sleep(1); // avoid eating unnecessary CPU cycles
}
This way, if the computer goes down, you'll have a worst case delay of 60 seconds.
You might also want to look into semaphores or some kind of locking strategy like using an APC variable or checking for the existence of a locking file to avoid race conditions, using APC for example:
set_time_limit(60); // one minute, same as CRON ;)
ignore_user_abort(false); // you might wanna set this to true
if (apc_exists('lock') === false) // not locked
{
apc_add('lock', true, 60); // lock with a ttl of 60 secs, same as set_time_limit
while (true)
{
$jobs = getPendingJobs();
if ((is_array($jobs) === true) && (count($jobs) > 0))
{
foreach ($jobs as $job)
{
if (executeJob($job) === true)
{
markCompleted($job);
}
}
}
sleep(1); // avoid eating unnecessary CPU cycles
}
}
If you're sticking with the PHP daemon do yourself a favor and drop that idea, use Gearman instead.
EDIT: I asked a related question once that might interest you: Anatomy of a Distributed System in PHP.
I'll suggest something out of the ordinary: you said you need to run the task at the point the data is written to MySQL. That implies MySQL "knows" something should be executed.
It sounds like perfect scenario for MySQL's UDF sys_exec.
Basically, it would be nice if MySQL could invoke an external program once something happened to it.
If you use the mentioned UDF, you can execute a php script from within - let's say, INSERT or UPDATE trigger.
On the other hand, you can make it more resource-friendly and create MySQL Event (assuming you're using appropriate version) that would use sys_exec to invoke a PHP script that does certain updates at predefined intervals - that reduces the need for Cron or any similar program that can execute something at predefined intervals.
i would definately not advise to use cronjobs for this.
cronjobs are a good thing and very useful and easy for many purposes, but as you describe your needs, i think they can produce more complications than they do good. here are some things to consider:
what happens if jobs overlap? one takes longer to execute than one minute? are there any shared resources/deadlocks/tempfiles? - the most common method is to use a lock file, and stop the execution if its occupied right at the start of the program. but the program also has to look for further jobs right before it completes. - this however can also get complicated on windows machines because they AFAIK don't support write locks out of the box
cronjobs are a pain in the ass to maintain. if you want to monitor them you have to implement additional logic like a check when the program last ran. this however can get difficult if your program should run only on demand. the best way would be some sort of "job completed" field in the database or delete rows that have been processed.
on most unix based systems cronjobs are pretty stable now, but there are a lot of situatinos where you can break your cronjob system. most of them are based on human error. for example a sysadmin not exiting the crontab editor properly in edit mode can cause all cronjobs to be deleted. a lot of companies also have no proper monitoring system for the reasons stated above and notice as soon as their services experience problems. at this point often nobody has written down/put under version control which cronjobs should run and wild guessing and reconstruction work begins.
cronjob maintaince can be further complicated when external tools are used and the environment is not a native unix system. sysadmins have to gain knowledge of more programs and they can have potential errors.
i honestly think just a small script that you start from the console and let open is just fine.
<?php
while(true) {
$job = fetch_from_db();
if(!$job) {
sleep(10)
} else {
$job->process();
}
}
you can also touch a file (modify modification timestamp) in every loop, and you can write a nagios script that checks for that timestamp getting out of date so you know that your job is still running...
if you want it to start up with the system i recommend a deamon.
ps: in the company i work there is a lot of background activity for our website (crawling, update processes, calculations etc...) and the cronjobs were a real mess when i started there. their were spread over different servers responsible for different tasks. databases were accessed wildly accross the internet. a ton of nfs filesytems, samba shares etc were in place to share resouces. the place was full of single points of failures, bottlenecks and something constantly broke. there were so many technologies involved that it was very difficult to maintain and when something didnt work it needed hours of tracking down the problem and another hour of what that part even was supposed to do.
now we have one unified update program that is responsible for literally everyhing, it runs on several servers and they have a config file that defines the jobs to run. eveyrthing gets dispatched from one parent process doing an infinite loop. its easy to monitor, customice, synchronice and everything runs smoothly. it is redundant, it is syncrhonized and the granularity is fine. so it runs parallel and we can scale up to as many servers as we like.
i really suggest to sit down for enough time and think about everything as a whole and get a picture of the complete system. then invest the time and effort to implement a solution that will serve fine in future and doesnt spread tons of different programs throughout your system.
pps:
i read a lot about the minimum interaval of 1/5 minutes for cronjobs/tasks. you can easily work around that with an arbitrary script that takes over that interval:
// run every 5 minutes = 300 secs
// desired interval: 30 secs
$runs = 300/30; // be aware that the parent interval needs to be a multiple of the desired interval
for($i=0;$i<$runs;$i++) {
$start = time();
system('myscript.php');
sleep(300/10-time()+$start); // compensate the time that the script needed to run. be aware that you have to implement some logic to deal with cases where the script takes longer to run than your interavl - technique and problem described above
}
This looks like a job for a job server ;) Have a look at Gearman. The additional benefit of this approach is, that this is triggered by the remote side, when and only then there is something to do, instead of polling. Especially in intervals smaller than (lets say) 5 min polling is not very effective any more, depending on the tasks the job performs.
The quick and dirty way is to create a loop that continuously checks if there is new work.
Psuedo-code
set_ini("max_execution_time", "3600000000");
$keeplooping = true;
while($keeplooping){
if(check_for_work()){
process_work();
}
else{
sleep(5);
}
// some way to change $keeplooping to false
// you don't want to just kill the process, because it might still be doing something
}
Have you tried windows scheduler(comes with Windows by default)? In this you will need to provide php path and your php file path. It works well
Can't you just write a java/c++ program that will query for you through a set time interval? You can have this included in the list of startup programs so its always running as well. Once a task is found, it can handle it on a separate thread even and process more requests and mark others complete.
The most simple way is to use embed Windows schedule.
Run your script with php-cli.exe with filled php.ini with extensions your script needs.
But I should to say that in practice you don't need so short time interval to run your scheduled jobs. Just make some tests to get best time interval value for yours one. It is not recommended to setup time interval less than a 1 minute.
And another little advise: make some lock file at the beginning of your script (php flock function), check for availability to write into this lock file to prevent working of two or more copies same time and at the end of your script unlink it after unlocking.
If you have to write output result to DB try to use MySQL TRIGGERS instead of PHP. Or use events in MySQL.
I'm attempting to make a php script that can load the current weather forecast and it uses a bit of XML pre-processing to digest the input, however it is accessed quite often and reloaded. The problem begins with my current host, which yes I do understand why, limits the amount of processing power a script takes up.
Currently takes an entire process for ever execution, which is around 3 seconds per execution. I'm limited to 12, yet I get quite a few pings.
My question to you guys is: What methods, if any, can I use to cache the output of a script so that it does not have to pre-process something it already did 5 minutes ago. Since it is weather, I can have a time difference of up to 2 hours.
I am quite familiar with php too, so don't worry xD.
~Thank you very much,
Jonny :D
You could run a cronjob that would generate the weather forecast data and then just display the whole thing from cache. You could use APC so it is always loaded in memory (plus all other added advantages).
The Zend Framework provides the Zend_Cache object with multiple backends (File, memcached, APD). Or you can roll your own with something like:
$cachFile = "/path/to/cache/file";
$ttl = 60; // 60 second time to live
if (!file_exists($cacheFile) || time()-filemtime($cacheFile) > $ttl) {
$data = getWeatherData(); // Go off and get the data
file_put_contents(serialize($cacheFile), $data);
} else {
$data = unserialize(file_get_contents($cacheFile));
}
need a code snippet to see what kind of processing you are doing. consider using xdebug to better optimize your code.
Also you may use a benchmarking tool such as AB to see how many processes your server can handle.
there are several different caching mechanisms available but without seeing what kind of process you are doing it is hard to say...
3 seconds is an extremely long execution time, as already asked, some cold would be nice to see how you process the 'input' and in what format said input is in.
A quick and dirty article about caching out of script to file is found here:
http://codestips.com/?p=153