I have a script, lets call it linkchecker, that loops through about 10.000 URLs, checking them for http status codes.
When they are checked, the url is marked as checked in my DB.
It wont output anything until its done, which can take many hours.
So I thought about just having another script that will run the linkchecker in the background, while continually polling the DB about how many URLs are checked, so I can follow the progress, and if any URLs are giving a problem with long connection time and so on.
I tried just running the linkchecker in an iframe, but nothing will load until the linkchecker has finished.
How can i execute this linkchecker in the background while the main script runs normally, executing other tasks?
You have to set a cron job (if you are running Linux) that executes a curl command to access a PHP script (external, like 'curl http://domain.com/php/something.php') or just executing a php command pointing to a internal file.
You can make a scheduler that executes every minute (that's the minimum execution time supported by cron job) and executes a "block" of your work. Of course, you must set PHP to skip the 30 seconds execution limit used by default.
In the DB make a column you call "Checked". Just make the PHP-script update which are checked in the database. Use phpMyAdmin to see the database graphically, just choose to sort after the column "Checked" and then you could see how far it has come.
You have to do it this way, because the webpage will not update until the script is done. However you could make the script run another script that says how far the process is, but that would maybe be too time-consuming?
You also have to go into php.ini to check that max_execution_time is set to several hours. 60*60*24 = 1day = 86 400 seconds.
Hope this helps! :)
I suggest making an AJAX request to new page lets call it ajaxChecker.php
In thuis page its just see if there is unchecked URLs (return number)
If the number of unchecked is zero then echo the output in the new div
function checker()
{
$.post('ajaxChecker.php',function(data){
if(data.length > 0)
$('#result').html(data);
});
}
setInterval( "checker()", 10000 );
And of course make a request through ajax or cron job to start it first
Related
Everything I google tells me this should not be happening, however it is.
I'm building a migration tool to build a 'master database'. I have an admin panel, only accessible to a few select people. There is a merge button that starts an AJAX call to run the migration php function. I'm not positive how long this script takes considering I'm still developing it but none the less I'm expecting a minimum of 20 minutes once pushed to production and populated with the production database. I do NOT need a lecture on best practices telling me not to do it via a GUI. This will become a cron as well, however I want to be able to induce it manually, if the admin desires.
So here's my process. The migration function immediately closes the session session_write_close() allowing me to run multiple php scripts simultaneously. I do this because I start a setInterval that checks to see a session variable. This is my 'progress' which is just an int on what loop iteration I'm on. In my migration script I open sessions, add 1 to that int, and close the sessions again. I do this at the end of each loop.
By doing this I have successfully created a progress for my AJAX. Now I noticed something. If I start my migration, then close out of my tab - or refresh. Once I reload the page my progress continues to grow. This tells me that the migration script is still executing in the background.
If I close 100% out of my browser, or clear my sessions I no longer see progress go up. This however is not because the script stops. This is because my progress indication relies on sessions and once I clear my sessions or close out my browser my session cookie changes. However I know the script is still running because I can query the database manually and see that entries are being added.
NOW to my question:
I do NOT want this. If my browser closes, if I press refresh, if I loose connection, etc I want the script to be TERMINATED. I want it to stop mid process.
I tried ignore_user_abort(false); however I'm pretty sure this is specific to command line and made no difference for me.
I want it to be terminated because I'm building a 'progress resume' function where we can choose where to resume the migration progress again.
Any suggestions?
UPDATE:
I didn't want to go this route but some solution I just though of is I could have another session variable. And it's my 'last time client was validated' which could be a timestamp. In my javascript, on the client side, every like 30 seconds I could hit a php script to 'update last time client was validated'. And in my migration function at the beginning of each loop I could check to make sure that timestamp isn't like 60 seconds old for example. If it IS 60 seconds old, or older, I do a die thus stopping my script. This would locally mean 'if there is no client updating this timestamp then we can assume the user closed out of his browser/tab/refreshed'. And as for the function I can ignore this check if in command line (cron). Not the ideal solution but it is my plan B
I am, and did, go with the solution to ping from the client to indicate if the client is still alive or not.
So essentially this is what I did:
From the client, in javascript, I set up a setInterval to run every 1.5 seconds and that hits a php script via AJAX. This php script updates a session variable with the current timestamp (this could easily be a database value if you needed to, however I didn't want the overhead of another query).
$_SESSION['migration_listsync_clientLastPing_'.$decodedJson['progressKey']] = time();
Then, inside my migration function I run a check to see if the 'timestamp' is over 10 seconds old, and if it is I die - thus killing the script.
if(isset($_SESSION['migration_listsync_clientLastPing_'.$progressKey])){
$calc = time() - $_SESSION['migration_listsync_clientLastPing_'.$progressKey];
if($calc > 10){
die();
}
}
I added a 'progressKey' param which is a random number from 1-100 that is generated when the function is called. This number is generated in javascript and passed into both of my AJAX calls. This way if the user refreshes the page and then immediately pressed the button again we won't have 2 instances of the function running. The 'old' instance will die after a few seconds and the new instance will take over.
This isn't an ideal solution however it is an effective one.
i have a big script written in php, which should import a lot of informations in a prestashop installation, using webservices, this script is written in "sections" I mean, there is a function that import the categories, another one that import products, then manufacturers, and so on, there are about 7 - 10 functions called in the main script. Basically I assume that this script must run for about an hour, passing from a function to the next one and so on since it arrives at the last function, then return some values and stops until the next night.
i would like to understand if it could be better :
1) impose a time limit of 30 minutes everytime i enter a new function (this will prevent the timeout)
2) make a chain of pages, each one with a single function call (and of course the time limit)
or any other idea... i would like to :
know if a function has been called (maybe using a global variable?)
be sure that the server will execute the function in order (so the pages chain)...
i hope to have beeen clear, otherwise i'll update the question.
edits:
the script is executed by another server that will call a page, the other server is "unkown" from me, so I simply know only that this page is called (they could also call the function by going on the page) but anyway i have no controll on it.
For any long running scripts, I would run it through the commandline, probably with a cronjob to kick it off. If it's triggered from the outside, I would create a job queue (for example in the database) where you insert a new row to signify that it should run, along with any variable input params. Then the background job would run - say - every 5 minutes, check if there's a new job in the queue. If there's not, just exit. If there is, mark that it has begun work and start processing. When done, mark that it's done.
1 hour of work is a looooooooong time though. Nothing you can do to optimise that?
You can increase the time limit for execution of a script as much as you want using :
set_time_limit(seconds);
And also for long running scripts you need a more memory. you can increase the memory limit using :
ini_set('memory_limit','20M');
And second other thing you have to make sure is that you are running your script on a dedicated server because if you are using a shared server you server will kill automatically long running scripts.
I run a procedure that takes about 20 minutes to complete, I just wonder if PHP can keep the connection active until the process finishes.
To be clear, this page will have a button which when you press it will call a php page to run a sql query, in the main page I just wait for the Http request to be complete to send a success message.
For queries that are set to take up some time, you should move some automation requests into the mix, preferably cronjobs if you have access to a linux server.
With cronjobs, you can create enteries in the database for specific values, linked to the user. The cronjob will kick in lets say, every 5 minutes to execute a query if a pending query has finished. This will minimize the fact the user will need to sit on the page until completion. Because you should know, the second the user navigates away from the active page; all active connections, queries etc will stop.
Once the script has complete, make a custom message at the end to send to the user letting them know that their process has been completed.
You should also know, PHP works down the script, from line 1 to the end; so if your hang is on line 40 for example; the script will sit on that line until the query has completed then carry on processing.
Edit: This is for example purposes only to point you in the direction that i'm getting at, and should not be used as you see it. This is merely a markup example
<?php
if (isset($_POST['ButtonToExecute']))
{
// Query to update a table in your database which an external PHP script will work with
// Example Table Structure:
/*
Username
State
*/
if ($state == 0)
{
// Then update state to 1 with username to the username for the query to take place on
}
else
{
// Warn user that their process will take longer to complete as their already is an active query in process
// but add them to a secondry pipeline which will be picked up on the next cronjob interval
}
}
?>
On your cronjob, might have:
<?php
if ($state=='1')
{
// Execute your script
// After query execution update state to 2
// if state == 2 then insert custom message/email to send
}
?>
Edit your crontab:
yourpreferrededitor /etc/crontab
^ Where yourpreferrededitor means your text editor, whether nano or other.
and your cronjob line:
* * * * * root /usr/bin/php /var/www/cron.php
^ This is taken from a current cronjob I have constantly running set for every minute of every day
A Cronjob will give the user the ability to navigate away from the page, because as I mentioned above, the second the user navigates away from the script.. All on going processes stop. The cronjob will carry on running throughout without no user interaction needed (apart from they make the initial request)
You could do it in an ordinary php script if you set the timeout limit and ignore user abort but this is a bad idea because your user will have no feedback if the script worked.
At the company I work at we had the same problem
What we did:
Run the query in a separate script.
Start the script WITHOUT waiting for results
Split the query into multiple parts (using limit and offset)
Have the query script write messages to a temp file
Implement a status screen fetching the current state from the log via ajax
BTW an example for all the wise guys asking why it takes so long:
using a transitive hull can speed up your application a lot if you have to deal with a tree with millions of nodes but building it can take hours.
You could try:
set_time_limit(0);
You may want to take a look at Set Time Limit
In your PHP ini you can set a max-execution time for scripts, however you NEVER want to have a user sit on a page loading screen for that long.
One thing that I could suggest would to e increase your max execution time to something around 30 minutes, and then call this in javascript after a page has already been rendered, so the user will not be left in the dark not knowing what the script is doing.
I set up a very small (internal) dedicated web server, and I need to pull some energy data every 10 seconds or so from an XML file. This is the PHP code I have thus far:
<?php
$mydata = simplexml_load_file('http://192.168.x.xx:yyy/data.xml');
echo $mydata->device[0]->name;
echo $mydata->device[0]->value;
?>
I tested similar code out on my web server and PHP is installed and I think this should work, but I'd like to have this run every 10 seconds or so. This way the data displaying on my web page is always up to date. The web page will be left running 24/7 as a sign on the wall. What's the easiest way to refresh the data?
I would simply refresh the portion of the web page that displays the data using Ajax. Trigger the refresh using a JavaScript timer.
The easiest way? Add this line to your page.
<meta http-equiv="refresh" content="10">
This may not be the best way though, especially if you have a lot of stuff on the page that you don't need reloaded every 10 seconds. If that's the case, you should look into AJAX.
If you have a page setup that returns only the data you need (like your example) you can make an asynchronous request every 10 seconds using JavaScript's setInterval() to get the latest data and show it.
If you are running this through a browser then I would go with Eric's answer.
However if you have this running from command line you can do one of the two:
Have your current script say pull_energy_data.php to run from a cron job every 10 seconds. Don't forget to create some sort of locking mechanism. Just in case the job takes more than 10 seconds to run and you'll have more than one script running at the same time.
Another approach is having a script wrapping your pull_energy_data.php running in a loop and executing it every 10 seconds. This is less desirable than the previous approach as you'll need to keep track of when pull_energy_data.php last ran and how to issue a command to stop the wrapper script.
assuming you are running the server in Linux/Unix, perhaps look into writing a cronjob (automated job) for it?
One solution I can think of is to run the PHP script as a cronjob every 10 seconds, writing the output into a file or a database table.
You can then write a separate PHP script that reads the contents of the file/DB entry whenever anyone loads it in a browser.
i have script which must execute after every n minutes. n minutes is dynamic so that i could not set a cron job to call the script (at a specific time).
so what i did was i stored the time after every n minutes in an array so that when the script is executed, it will first check whether the current time is in the array. if it is found in the array, it continues to executes otherwise it exits.
to execute the script, i must use a cron job to run every minute to check the time in the array. unfortunately, my web host only allows 5 minutes as the least interval. so every time the script is called, i check whether the values between $current_time and $current_time + (4*60) // 4 minutes is found in the array. if it is, and if needed, i use time_sleep_until to delay the script until the time reaches the value found in the array.
so if my script executes at 10:05 and the value found in the array is 10:06, i let the script sleep until 10:06 before it continues to execute. however, if the sleep time is more than a minute or so, i get a Mysql server gone away.
how can i prevent this? or is there a better way to do this?
thanks!
A couple choices, which is better I do not know.
One, is make sure your script works with CLI and after that minute is up call it with the http://www.php.net/exec function (if your host allows it).
Two, is setup a script, with a possible hash as a key and use a header redirect after the minute is up, this would call the script brand new so a new MySQL connection is made.
A third option is to set the script up like in two, except setup a schedule task / cron job on your computer that opens that page (it would have to be in the webroot) and calls it every minute or however you want. This is not a set method, but depends on how much your computer is on.
Fourth, similar to the third but use a free cron job hosting service like: http://www.onlinecronjobs.com/en
Hope that helps. If I think of other options I will update.