To give a bit of background to my project first, I have a Wordpress website of which I have turned off the cron by tweaking the wp-config to:
define('DISABLE_WP_CRON', true);
I then set up a cron to the wp-cron file on the server to call the file every minute and this has been working as expected and as I need it to.
I use a plugin called 'Wp-Crontrol' to set up my own cron calls on various functions within my functions file.
Memory Limit: 2048M
PHP max execution time: 2700
The Issue
I have one function in particular that takes a report from a remote source and loops through each line entering each line into the database using the $wpdb class. If i place the function on a php page and go to it the function works perfectly as expected and enters all 6900 rows into the database after some time.
The way I would like it to work is to run a cron on that function (like I do with so many other things with no problems) but the issue is that when I set it up via cron it only seems to insert around 3000 rows before it just stops with no errors logged?
I am struggling to work out why running it manually would work perfectly but a scheduled cron of the same function during the night would only do half the job and not finish, stopping half way through?
I have turned on all error logging I can think of but nothing shows?
WordPress "cron" jobs are not the same thing as a system cron job. For instance, unless you have a high traffic site, it's pretty much impossible to set a gaurenteed 60 second cron with WordPress. WordPress crons are activated when someone puts an HTTP request into the system. A system cron runs off a daemon that runs in system memory. They are just different concepts and work differently.
Issue was processing time but for some reason it would not flag at all in any error messages. After I thinned out long processes it started behaving as expected.
Related
I would like to ask you about differences between wp-cron and server cron. Basically I have a php script which takes some minutes to complete. If I run it through browser, browser shows error something about timeout.
So I put it into server cron (paid hosting) and it is working.
I was trying set_time_limit... when was accessing through browser but it did not work...
Anyway I want to execute this script lets say every 2 weeks. I know that wordpress has wp-cron, but my question is it is good as server cron? Will it work for long time scripts?
Because of my point of view the wp-cron is initialized by user (browser) so it should apply limits in php.ini right?
Because it is paid hosting I can not modify a lot of options...
So what is the difference between these crons?
Thanks
Yes, wp-cron is initialized by the browser. Therefore, it will only execute for 30 seconds by default. Therefore, it is not meant for running scripts that may take longer than a few second.
Server cron runs your script from the command line which does not have same time constraint.
From everything you have said your script is a good candidate to run as a server cron.
I have a script that updates my database with listings from eBay. The amount of sellers it grabs items from is always different and there are some sellers who have over 30,000 listings. I need to be able to grab all of these listings in one go.
I already have all the data pulling/storing working since I've created the client side app for this. Now I need an automated way to go through each seller in the DB and pull their listings.
My idea was to use CRON to execute the PHP script which will then populate the database.
I keep getting Internal Server Error pages when I'm trying to execute a script that takes a very long time to execute.
I've already set
ini_set('memory_limit', '2G');
set_time_limit(0);
error_reporting(E_ALL);
ini_set('display_errors', true);
in the script but it still keeps failing at about the 45 second mark. I've checked ini_get_all() and the settings are sticking.
Are there any other settings I need to adjust so that the script can run for as long as it needs to?
Note the warnings from the set_time_limit function:
This function has no effect when PHP is running in safe mode. There is no workaround other than turning off safe mode or changing the time limit in the php.ini.
Are you running in safe mode? Try turning it off.
This is the bigger one:
The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.
Are you using external system calls to make the requests to eBay? or long calls to the database?
Look for particularly long operations by profiling your php script, and looking for long operations (> 45 seconds). Try to break those operations into smaller chunks.
Well, as it turns out, I overlooked the fact that I was testing the script through the browser. Which means Apache was handling the PHP process, which was executed with mod_fcgid, which had a timeout of exactly 45 seconds.
Executing the script directly from shell and CRON works just fine.
(Our server is Linux based)
I'm an experienced PHP developer but first time i'll develop a bot which always running and fetch some datas.
I'll explain my application with a simple (and sample) scenario. I have about 2000 web site url and my application will visit this url's and record contents of web page's . This application will work 7 days 24 hours. It will start working again when it's finish 2000 web sites.
But i need some suggestions for my server. As you see, my application will be run infinity until i shut down server. I can do this infinity loop with this :
while(true)
{
APPLICATION CODES HERE
}
But i think this will be an evil for server :) Is it possible to doing something like this, on server side?
Also i think using cronjobs but it's not work for my scenario. Because my script start working again asap it's finish working. I have to "start again when you finish your work" , not "start every 30 minutes" . Because i don't know, maybe fetching all 2000 websites, will take more than 30 minutes or less than 30 minutes.
I hope i explained it very well.
Also i'm worried about memory usage. As you know garbage collector cleans memory after every PHP script stop. But as i said, my app won't stop for days (maybe weeks) . So garbage collector won't be triggered. I'm manually unsetting (unset() function) all used variables at end of script. Is it enough?
I need some suggestions from server administrators :)
PS. I'm developing it as console application, not a web application. I can execute it from command line.
Batch processing.. store all the sites in a csv or something, mark them after completion, then work on all the ones non-marked, then work on all the marked.. etc. Only do say 1 or 5 at a time, initiate batch script every minute from cron..
Don't even try to work on all of them at once.. any errors and you won't know what happened..
Could even store the jobs in a database, store processing stats etc.. allows for fine-tuning and better reporting.
You will probably hit time-limits trying to run infinite php scripts, even from the command line.. also your server admin will hate you. Will probably run into Memory limits if you don't release resources properly.. far too easily done with php.
Read: http://www.ibm.com/developerworks/opensource/library/os-php-batch/
Your script could just run through the list once and quit. That way, what ever resources php is holding can be freed.
Then have a shell script that calls the php script in an infinite loop.
As php is not designed for long running task, I am not sure if the garbage collection is up to the task. Quiting after every run will force it to release everything.
I have to run a pretty heavy task on PHP once a week (script that curls to various locations (websites, API's), gathers, sorts data and inserts it into a db). The whole script takes about 10 to 15 mintues to run on my mac (localhost) - guessing it'll run a bit faster on a server. Nevertheless - I'm currently looping through with AJAX, so when each task is finished, next one is launched. Now I need to run it weekly, automatically. So I think I can't do it with AJAX Anymore.
Do I have to just set the php.ini to let a script run for 30 mintues or there is a better way to do it ?
The maximum execution time of the PHP script is determined by the amount of time in which no output has been generated. So writing data into STDOUT (e.g. to a logfile) will keep the script running.
However, if you're running the script from command line, the max-execution-time will be defaulted to zero anyway and as already suggested, I'd start the script with a cronjob instead of an AJAX-Request or similar methods. I actually do that for most of my php-scripts performing administrative tasks like synchronizing data across several databases or similar purposes.
php.ini has nothing to do with scheduling jobs. It's simply definining PHP's startup settings. What you want is a cron job, as your title says.
For OSX cron setup, see http://hintsforums.macworld.com/showthread.php?s=&threadid=39005
I would like to add an intensive task (lets say 5 minutes execution time) into Wordpress using cron job.
I been using this code to add new cron task inside the Wordpress system.
wp_schedule_event(time(), "interval-name", "hook-name");
I read somewhere in the net that cron task will be executed when there is request hit the Wordpress (either in the public site or the admin). Can anybody acknowledge that is true?
If that the case then I should not put my intensive task into cron task, because it will make user wait for long time after the task finished. What should I do now?
Anybody experienced this situation? Any suggestion?
I think to create a new page to be executed by crontab (for example http://example.com/wp-content/plugins/plugin-example/intensive-task.php)
The wordpress documentation says that it will be run when someone visits your site, so yes, you're correct. It will only be one user that gets a slow page load, so it's up to you if you want to avoid that.
If you are running it from a regular con job, there's no need to make it a page on your site though; especially if it's an intensive job, as you say, then this could easily be exploited place a large load on your server. You can easily run php from the command line to execute your job safely and without causing any slow load times on your page.
If you would use regular cronjob that wouldn't be the case
but i suspect that wp does what you said, since that would make it versatile working in different hosts with different setups as long as they have php and mysql running independent from real cronjobs which must be installed by the web host separately