I'm building a spider which will traverse various sites and data mining them.
Since I need to get each page separately this could take a VERY long time (maybe 100 pages).
I've already set the set_time_limit to be 2 minutes per page but it seems like apache will kill the script after 5 minutes no matter.
This isn't usually a problem since this will run from cron or something similar which does not have this time limit. However I would also like the admins to be able to start a fetch manually via a HTTP-interface.
It is not important that apache is kept alive for the full duration, I'm, going to use AJAX to trigger a fetch and check back once in a while with AJAX.
My problem is how to start the fetch from within a PHP-script without the fetch being terminated when the script calling it dies.
Maybe I could use system('script.php &') but I'm not sure it will do the trick.
Any other ideas?
$cmd = "php myscript.php $params > /dev/null 2>/dev/null &";
# when we call this particular command, the rest of the script
# will keep executing, not waiting for a response
shell_exec($cmd);
What this does is sends all the STDOUT and STDERR to /dev/null, and your script keeps executing. Even if the 'parent' script finishes before myscript.php, myscript.php will finish executing.
if you don't want to use exec you can use a php built in function !
ignore_user_abort(true);
this will tell the script to resume even if the connection between the browser and the server is dropped ;)
Related
I have 2 websites, hosted on 2 different servers. They are kind of interlinked. Sometimes I just do stuff on Website-1 and run a script on Website-2. Like I edited something on Website-1 and now I want to run a script on Website-2 to update accordingly on it's server.
Till now I am using following code on website 1.
$file = file_get_contents('Website-2/update.php');
But the problem with this is that my Website-1 server script stops running and wait for the file to return some data. And I don't wanna do anything with that data. I just wanted to run the script.
Is there a way where I can do this in a better way or tell PHP to move to next line of code.
If you want to call the second site without making your user wait for a response,
I would recommend using a message queue.
Site 1 request would put a message to the queue.
Cron job to check queue and run update on site 2 when message exists.
Common queues apps to look at:
[https://aws.amazon.com/sqs/?nc2=h_m1][1]
[https://beanstalkd.github.io/][2]
[https://www.iron.io/mq][3]
[1]: https://aws.amazon.com/sqs/?nc2=h_m1
[2]: https://beanstalkd.github.io/
[3]: https://www.iron.io/mq
What you're trying to achieve is called a web hook and should be implemented with proper authentication, so that not anybody can execute your scripts at any time and overload your server.
On server 2 you need to execute your script asynchronously via workers, threads, message queues or similar.
You can also run the asynchronous command on your server 1. There are many ways to achieve this. Here are some links with more on this.
(Async curl request in PHP)
(https://segment.com/blog/how-to-make-async-requests-in-php/)
Call your remote server as normal. But, In the PHP script you normally call, Take all the functionality and put it in a third script. Then from the old script call the new one with (on Linux)
exec('php -f "{path to new script}.php" $args > /dev/null &');
The & at the end makes this a background or non-blocking call. Because you call it from the remote sever you don't have to change anything on the calling server. The php -f runs a php file. The > /dev/null sends the output from that file to the garbage.
On windows you can use COM and WScript.Shell to do the same thing
$WshShell = new \COM('WScript.Shell');
$oExec = $WshShell->Run('cmd /C php {path to new script}.php', 0, false);
You may want to use escapeshellarg on the filename and any arguments supplied.
So it will look like this
Server1 calls Server2
Script that was called (on Server2) runs exec and kicks off a background job (Server2) then exits
Server1 continues as normal
Server2 continues the background process
So using your example instead of calling:
file_get_contents('Website-2/update.php');
You will call
file_get_contents('Website-2/update_kickstart.php');
In update_kickstart.php put this code
<?php
exec('php -f "{path}update.php" > /dev/null &');
Which will run update.php as a separate background (non-blocking) call. Because it's non-blocking update_kickstart.php will finish and return to searver1 which can go about it's business and update.php will run on server2 independantly
Simple...
The last note is that file_get_contents is a poor choice. I would use SSH and probably PHPSecLib2.0 to connect to server2 and run the exec command directly with a user that has access only to that file(Chroot it or something similar). As it is anyone can call that file and run it. With it behind a SSH login it's protected, with it Chrooted that "special" user can only run that one file.
This might be a very stupid question so please bear with me.
I have a a php script that makes API calls to Shopify.
The entire point of this php script is to print out statements for each customer.
Now it has to run through about 200 customers.
This entire process takes about 15 minutes.
Ordinarily this runs on a monthly basis with a cron job.
But I need to be able to run it manually as well. I just want the page to execute and do everything in the background with my browser or internet connection playing NO role as to whether the complete execution completes.
The cron job runs header_php.php?run=monthly
Is there anyway I can run it manually, make sure it gets a 200 response from the page, and then close my browser tab and ensure that apache does the rest?
I would be executing it via an AJAX call as well.
Another thing, once each statement is done being processed, the script outputs it to pdf and emails it to the customer. So there's no feedback required from the page when it runs.
Easily doable with simple HTTP headers.
Start output buffering
Output the response, if any
Send Content-length and Connection: close headers
Flush and end output buffers
The browser receives HTTP response
Continue time consuming processing
This SO anwer nails it (the comments are helpful as well).
You can call your script with other script that runs in background the job.
shell_exec('nohup /usr/bin/php /dir/to/your/script.php > /dev/null 2>/dev/null &');
Then you don't need to wait to finish the job
on linux
<?php
exec(''.$command.' > /dev/null 2>&1 &');
?>
on windows
<?php
$shell = new COM("WScript.Shell");
$shell->run($command, 0, false);
?>
where command would be something like
php -f $path/to/filename
if you put that in a page, you can then call it whenever you want and it will spawn a thread that will call apache, but not require the browser to wait for any response.
I have a php file that takes in a couple parameters from the URL and then runs an exec command, in which i want to wait and have the results of the exec displayed. This exec command takes about 20-30 seconds to finish. It never completes because the webpage just gets an nginx 502 bad gateway error (times out).. Instead of extending nginx timeout error, as that's bad practice to have a connection hang for that long, how can i run the php's exec in the back and then have it returned on the page after it's complete?
Or is there a better way to accomplish this without using php?
Have the PHP trigger an exec to a script that is forked so it runs in the background (ends with &). The page should then return some js that periodically polls the server via ajax requests to check the original script's status. The script should output its STDOUT and STDERR to unique file(s) so that the polling script can check the status.
Edit: If you need to know the script's exit code wrap it in another script:
#/bin/bash
uniqId=$1
yourscript $uniqId &
myPid=$!
echo $myPid > $uniqId'.pid'
wait $myPid
echo $? > $uniqId'.returned'
Call like ./wrapper.sh someIdUniqueToClient &. Untested, but you get the gist.
How can I run a php file IN BACKGROUND after submitting a form. The loading has to happen in background since it usually takes a very long time.
Basically it's just like running a cronjob, except I want to trigger it manually and with my browser.
There are several possible ways to do this.
Try setting ignore_user_abort to TRUE in your script.
If changed to TRUE scripts will not be terminated after a client has aborted their connection.
Take a look at popen() and pclose(). You can do something like this:
pclose(popen("start php /path/to/myscript.php", "r"));
You can kick off a separate PHP process with a system() or exec() call. Something like this:
system('php /path/to/myscript.php >/dev/null 2>&1 &');
Start the request using AJAX. The browser will continue running while waiting for a response. You can even show a popup or some information when the request is finished, although you don't have to.
I want initiate one php page as background process from another php page.
Use popen():
$command = 'php somefile.php';
pclose(popen($command,'r'));
This launches somefile.php as a background process.
This is a technique I used to get around restrictions applied by my webhost (who limited cronjobs to 15 minutes of execution time, so my backup scripts would always timeout).
exec( 'php somefile.php | /dev/null &' );
The breakdown of this line is:
exec() - PHP reference Runs the specified command, as if from the Linux Command Line.
php somefile.php: Invokes PHP to open, and run, somefile.php. This is the same behaviour as what would happen if that file was accessed through a web browser.
| ("pipe") - Sends the output of the proceeding command to a specified target. In this instance, it would "pipe" the content which would normally be read by the web browser accessing the file.
/dev/null - A blackhole. No, not kidding. It is a place where you send output if you just want it to disappear.
& - Appending this character to the end of a Linux command means "Do not wait - Send this to the background and continue."
So, in summary, the provided code will execute a PHP script, return no output, and not wait for it to finish before continuing onto the next line.
(And, as always, if any of these assumptions on my part are in error, I would love to be corrected by more knowledgeable members of the community.)
You have to make sure, that the background process is not terminated when the processing of the page finished. If you are on a Linux system, you could try to use the nohup command:
$command = 'nohup php somefile.php';
pclose(popen($command,'r'));
If it still gets terminated, you could try the "daemon" command.