I am creating a pdf report of some 100 pages using shell script mode of wkhtmltopdf. Now, the thing is that the script takes a whole lot of time to create the report and save it in the specified path.
Currently, I am calling the script like this:
$stmt = $sh_script.' > /dev/null 2>/dev/null &';
shell_exec($stmt);
Doing this causes the shell to run in background and the report gets generated, but on the browser end after sometime I see
504 Gateway Time-out
nginx/1.4.6 (Ubuntu)
Which is not a very comfortable message, I want to call the shell_exec statement in an asynchronous way so that it is called and php code moves on. Can someone help me with this?
So before I give you a recommended solution for your question, I want to add really quick that I would probably find a php library which can achieve what you are looking to do, instead of executing shell commands from PHP, which can be a bad idea for many reasons, but if you intend on going that route, please look at this answer for some advice on security when executing shell commands via PHP:
https://stackoverflow.com/a/4535900/4660602
Now, to get back to your answer, anytime you are executing a big task in PHP that requires some extra time, it is never really a good idea to keep the user waiting at the browser. That is where building a Queue can become crucial to your application. You begin to work on some task AFTER you explain to your user via your user interface that the work is being done in the background and they will be notified when it is completed, etc.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, IronMQ or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner, but the user does not have to wait for a response until it finishes working, hence no more 504 timeouts.
OR you could try a much dirtier solution and just increase the script timeout value in php and on your server, but this can have some unexpected results. For nginx & for Apache
Best of luck!
You're close .. But you want nohup (No hang up) and your statment at the end should look more like /dev/null 2>&1 &
That said .. I believe this should work:
exec('nohup' . $sh_script . ' > /dev/null 2>&1 &')
Related
We need to distribute software which should contain a PHP script which will run for some minutes. Therefore I am searching a best practise way to do this in 2017.
It has to be invoked by an HTTP request. There should be no HTTP request waiting for some minutes so the script has to run still AFTER the visitor got his HTTP response.
It has to run periodically (every night). It also should run every night per default (like a cron job). Notice: since the software is going to be distributed to clients there is no way for us to add a cronjob manually (we have no access to our clients servers). Everything should be accomplished within PHP code.
(Please note that I read existing blog posts and Stackoverflow questions myself but I could not find a satisfying answer)
Maybe anyone knows how frameworks like Symfony and Laravel or webshops like Magento accomplish such tasks? Still I want to know how to do it by myself in plain PHP without using frameworks or libraries.
Many solutions exist:
using exec (rather insecure), that triggers a background job (recommended in comments, I would probably prefer symfony process, but insecure nevertheless).
using a cron to trigger a symfony process every so often, not over http so way more secure.
using php-fpm, you can send a response without stopping the process using fastcgi_finish_request
using a queue system (SQS, RabbitMQ, Kafka and so on).
using a cron manager in PHP
using a daemon and something like supervisord to make sure it runs continuously.
The best solutions are definitely queues and cron, then PHP-FPM, rest of it is just rubbish.
There is absolutely no way for you to run on someone's server without doing something that won't work at some point.
Sidenote: you said you did not want libraries in order to know how to do it yourself, I added links to libraries as reading them may give you a deeper knowledge of the technology, these libraries are really high quality.
Magento only runs it cronjobs, if you setup a regular cronjob for Magento. It has a cron.sh, that runs every minute an executes jobs in Magento's queue.
Any Solution to execute long-running tasks via http involves web-server configuration.
Finally I think there are two ways to start a long running process in PHP via an HTTP request (without letting the user wait for a long time):
Using FPM and send the response to the user with fastcgi_finish_request(). After the response is sent you can do whatever you want, for example long running tasks. (Here you don't have to start a new process, just continue with PHP).
fastcgi_finish_request();
longrunningtask();
Create a new process using one of the following functions. Redirect STDOUT and STDERR to null and put it to the background (you have to do both). To still get output of the new process the new process can write to some log file.
exec('php longrunningtask.php >/dev/null 2>/dev/null &');
shell_exec('php longrunningtask.php >/dev/null 2>/dev/null &');
system('php longrunningtask.php >/dev/null 2>/dev/null &');
passthru('php longrunningtask.php >/dev/null 2>/dev/null &');
Or use proc_open() like symfony/process does.
Notes:
symfony/process: on the docs you can read that you can use FPM and fastcgi_finish_request() for long running tasks.
Security: I can see no built in security risk, you just have to do things right then everything is fine (you could use password protection, validate possible inputs to your commands, etc.).
For the cron-job-part of the question there is no answer. I don't think it's possible.
I am trying to start a linux shell script from PHP5 that will run for 24hours, but I want the webpage to return within seconds. I though this could be solved by making a script spawning of the task, but it does not seem to work.
I have been searching around for a solution or a "one shot / fire and forget" option for a couple of days without any luck.
The following example shows the problem.
In PHP 5 I make one of the following call (tried a lot it this point)
passthru("dummy_script.sh");
or
system("dummy_script.sh");
or
shell_exec("dummy_script.sh");
The dummy script look the following:
#!/bin/sh
{
while true
do
sleep 1
done
} &
I can see the that process gets started, but the webpage does not return before I make a 'killall dummy_script.sh'. If I run the script manually in a terminal it return immediately and spawns of the loop.
Does anyone know a way here I can spawn of the task without making the webpage wait it ?
Hope you guys can help me out, it would be most appreciated.
To answer your question:
You may start looking at pcntl_fork. Or you may check this. Basically, you are using the native fork to fork the long running process so your php frontend does not have to wait.
If you're feeling adventurous, you may put your "job" (your request to this long running process) in a DB. A cron job then checks the DB for incoming requests and it is the one that executes that process.
Another method is to use resque, but don't bother at this point.
I'm currently launching an asynchronous job with PHP to perform some tests.
To make it work, I found on SO some tips, like the use of popen and start:
$commande = "testu.bat";
$pid = popen('start /B ' . $commande, 'r');
$status = pclose($pid);
The testu.bat's folder is in my user PATH.
This script performs some task, and to control it's execution, it should generates a log file, but I never get it.
Whereas if I just remove the /B option, it works fine and I get my log file.
Did I miss something about background execution? How can I catch the error informations when it is running in the background?
It appears you are operating under the assumption that the /B switch to the start command means "background". It does not. From the start usage:
B Start application without creating a new window. The
application has ^C handling ignored. Unless the application
enables ^C processing, ^Break is the only way to interrupt
the application.
Processes launched by start are asynchronous by default. Since that appears to be what you want, just run the command without the /B switch.
Interesting one... Ok, here's what I think is going on:
Because you run the task in the background, the PHP script will just carry on, it is not waiting for testu.bat to return anything...
Or put another way, popen does what it was instructed to do, which is starting the task in the background (which it does) but then control is handed immediately back to PHP, whilst the log file is still being created in the background and the php script carries on at the same time...
What I would do in this case is let testu.bat call the php script (or another PHP script) in a callback type fashion once it has done its processing, in a similar way as in Javascript you would use callbacks for asynchromous Ajax calls...
Maybe provide the callback script command as a parameter to testu.bat..?
Hope this is of any help...
I'm not quite sure about your goal here, but here are some info you might use:
for figuring out background errors, you may find these functions useful:
set_exception_handler();
set_error_handler();
register_shutdown_function();
Of course write out the errors they catch into some file.
If you do not need any data back from your requests, you can simply use:
fsockopen()
curl
and give them a short timeout (10 milisec). The scripts will run in the backround.
Alternatively if you do need the data back, you can either put it into a database and set up a loop that checks if the data has already been inserted, or simply output it into a file and check for its existence.
In my opinion start launches the specified command by creating a new prcoess in the "background". Therefore the execution of start itself "just" starts the second process and exists immediately.
However, using the /B switch, the command to be executed will be excuted in the context of the start process. Therefore the execution of the start process takes longer. Now what I suspect is that executing pclose terminates the start process and as a result of this you don't get your log file.
Maybe one solution (not testet though) could be executing something like
start /B cmd "/C testu.bat" where start just tries to execute cmd and cmd gets /C testu.bat as parameter which is the "command" it shall execute.
Another thought:
What happens if you don't call $status = pclose($pid);?
Just for some people seeking this trick to works, in my case it just needs to activate the PHP directive ignore_user_abort in php.ini or by the PHP platform function.
Without this activated, the process is killed by pclose() without finishing the job.
Your problem is most likely properly solved by using a queue system. You insert a job into a queue that a background process picks up and works on. In this way the background task is completely independent of the HTTP request that initiated the task - but you can still monitor its progress.
The two most popular software packages that can help you in your scenario:
Gearman
Check out this gist and this totorial for installation on Windows.
RabbitMQ
Check out this tutorial for installation on Windows.
Note that implementing a queuing solution is not a 5 minute prospect, but it is technically the right approach for this sort of situation. For a company product this is the only viable approach; for a personal project where you just want something to work, there is a level of commitment required to see it through for the first time. If you're looking to expand your development horizons, I strongly suggest you give the queue a shot.
I'm creating a webservice for an Android app in PHP with MySQL. I want to continuously check whether any data is available. I haven't got any idea how to get data as a background process. How can I execute a query without any request or without calling file?
I searched and got some code like
$command = "php -d max_execution_time=50 -f myfile.php '".$param."' >/dev/null &";
exec($command);
But where should I put this code so this query will run continuously?
Yes, the ampersand trick will work. You can use something like supervisord to restart it every few hours, so that any memory leaks are dealt with. This also makes it less fragile if it were to crash or hang.
Also, you can use something like cron to run a task for 10 minutes, and then die off and wait for cron to start it again - bear in mind that with most background tasks, it doesn't matter if there's a short period the task is not running, since it will catch up. It's worth checking in each run whether the previous one is still running, and exit early if it is: that way you don't have two background tasks causing race-conditions when retrieving work from your database.
Finally you can use a job server, such as Gearman. This will allow you to send tasks to it in an asynchronous fashion, and they will be run by worker tasks (in either time or priority order). This is probably the most reliable approach, but it takes a bit more work to set up. There's a PHP module for this, but in my experience it's more of a hassle to use than Net_Gearman, which is available in PEAR.
I'm having a problem where putty gets regularly disconnected. So, when I run a PHP script from the terminal, it always gets interrupted. The script is supposed to run several hours, so I'm not having any luck with it.
How can I completely run this from the server side? I'm reading about cron jobs, but I'm having a hard time understanding at this time. Is there any alternative to cron for what I need?
I have several script PHP files that need to be run, one by one, or perhaps two at a time. Any ideas?
You don't need to leave it run in a cron job - you can just run the php script inside a screen.
Simply type;
screen php /path/to/myphpscript.php
A screen will continue running even after you disconnect from PuTTY. If you need to check up on it, you can use;
screen -r
To re-attach yourself to this process, and view any output.
You need to prevent the process from terminating when the session disconnects.
Something like this would work:
nohup php myscript.php
You can create a cron job to start the php script periodically based on a list of time tasks. More info. You could also start the task in the background from the console. i.e. php-cgi script.php& this would make the script a background task
Take a look at GNU Screen; it allows you to detach and reattach a session later, which is perfect for long-running scripts. Cron is a good option if you want it to happen in a recurring fashion; one-off batch jobs can be scheduled with something like at. For more intense computing needs, you might want to look into a more full-fledged job scheduling system like TORQUE.
You can run your program in background
php ./yourscript.php &