I'm creating a webservice for an Android app in PHP with MySQL. I want to continuously check whether any data is available. I haven't got any idea how to get data as a background process. How can I execute a query without any request or without calling file?
I searched and got some code like
$command = "php -d max_execution_time=50 -f myfile.php '".$param."' >/dev/null &";
exec($command);
But where should I put this code so this query will run continuously?
Yes, the ampersand trick will work. You can use something like supervisord to restart it every few hours, so that any memory leaks are dealt with. This also makes it less fragile if it were to crash or hang.
Also, you can use something like cron to run a task for 10 minutes, and then die off and wait for cron to start it again - bear in mind that with most background tasks, it doesn't matter if there's a short period the task is not running, since it will catch up. It's worth checking in each run whether the previous one is still running, and exit early if it is: that way you don't have two background tasks causing race-conditions when retrieving work from your database.
Finally you can use a job server, such as Gearman. This will allow you to send tasks to it in an asynchronous fashion, and they will be run by worker tasks (in either time or priority order). This is probably the most reliable approach, but it takes a bit more work to set up. There's a PHP module for this, but in my experience it's more of a hassle to use than Net_Gearman, which is available in PEAR.
Related
We need to distribute software which should contain a PHP script which will run for some minutes. Therefore I am searching a best practise way to do this in 2017.
It has to be invoked by an HTTP request. There should be no HTTP request waiting for some minutes so the script has to run still AFTER the visitor got his HTTP response.
It has to run periodically (every night). It also should run every night per default (like a cron job). Notice: since the software is going to be distributed to clients there is no way for us to add a cronjob manually (we have no access to our clients servers). Everything should be accomplished within PHP code.
(Please note that I read existing blog posts and Stackoverflow questions myself but I could not find a satisfying answer)
Maybe anyone knows how frameworks like Symfony and Laravel or webshops like Magento accomplish such tasks? Still I want to know how to do it by myself in plain PHP without using frameworks or libraries.
Many solutions exist:
using exec (rather insecure), that triggers a background job (recommended in comments, I would probably prefer symfony process, but insecure nevertheless).
using a cron to trigger a symfony process every so often, not over http so way more secure.
using php-fpm, you can send a response without stopping the process using fastcgi_finish_request
using a queue system (SQS, RabbitMQ, Kafka and so on).
using a cron manager in PHP
using a daemon and something like supervisord to make sure it runs continuously.
The best solutions are definitely queues and cron, then PHP-FPM, rest of it is just rubbish.
There is absolutely no way for you to run on someone's server without doing something that won't work at some point.
Sidenote: you said you did not want libraries in order to know how to do it yourself, I added links to libraries as reading them may give you a deeper knowledge of the technology, these libraries are really high quality.
Magento only runs it cronjobs, if you setup a regular cronjob for Magento. It has a cron.sh, that runs every minute an executes jobs in Magento's queue.
Any Solution to execute long-running tasks via http involves web-server configuration.
Finally I think there are two ways to start a long running process in PHP via an HTTP request (without letting the user wait for a long time):
Using FPM and send the response to the user with fastcgi_finish_request(). After the response is sent you can do whatever you want, for example long running tasks. (Here you don't have to start a new process, just continue with PHP).
fastcgi_finish_request();
longrunningtask();
Create a new process using one of the following functions. Redirect STDOUT and STDERR to null and put it to the background (you have to do both). To still get output of the new process the new process can write to some log file.
exec('php longrunningtask.php >/dev/null 2>/dev/null &');
shell_exec('php longrunningtask.php >/dev/null 2>/dev/null &');
system('php longrunningtask.php >/dev/null 2>/dev/null &');
passthru('php longrunningtask.php >/dev/null 2>/dev/null &');
Or use proc_open() like symfony/process does.
Notes:
symfony/process: on the docs you can read that you can use FPM and fastcgi_finish_request() for long running tasks.
Security: I can see no built in security risk, you just have to do things right then everything is fine (you could use password protection, validate possible inputs to your commands, etc.).
For the cron-job-part of the question there is no answer. I don't think it's possible.
I am creating a pdf report of some 100 pages using shell script mode of wkhtmltopdf. Now, the thing is that the script takes a whole lot of time to create the report and save it in the specified path.
Currently, I am calling the script like this:
$stmt = $sh_script.' > /dev/null 2>/dev/null &';
shell_exec($stmt);
Doing this causes the shell to run in background and the report gets generated, but on the browser end after sometime I see
504 Gateway Time-out
nginx/1.4.6 (Ubuntu)
Which is not a very comfortable message, I want to call the shell_exec statement in an asynchronous way so that it is called and php code moves on. Can someone help me with this?
So before I give you a recommended solution for your question, I want to add really quick that I would probably find a php library which can achieve what you are looking to do, instead of executing shell commands from PHP, which can be a bad idea for many reasons, but if you intend on going that route, please look at this answer for some advice on security when executing shell commands via PHP:
https://stackoverflow.com/a/4535900/4660602
Now, to get back to your answer, anytime you are executing a big task in PHP that requires some extra time, it is never really a good idea to keep the user waiting at the browser. That is where building a Queue can become crucial to your application. You begin to work on some task AFTER you explain to your user via your user interface that the work is being done in the background and they will be notified when it is completed, etc.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, IronMQ or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner, but the user does not have to wait for a response until it finishes working, hence no more 504 timeouts.
OR you could try a much dirtier solution and just increase the script timeout value in php and on your server, but this can have some unexpected results. For nginx & for Apache
Best of luck!
You're close .. But you want nohup (No hang up) and your statment at the end should look more like /dev/null 2>&1 &
That said .. I believe this should work:
exec('nohup' . $sh_script . ' > /dev/null 2>&1 &')
I am trying to start a linux shell script from PHP5 that will run for 24hours, but I want the webpage to return within seconds. I though this could be solved by making a script spawning of the task, but it does not seem to work.
I have been searching around for a solution or a "one shot / fire and forget" option for a couple of days without any luck.
The following example shows the problem.
In PHP 5 I make one of the following call (tried a lot it this point)
passthru("dummy_script.sh");
or
system("dummy_script.sh");
or
shell_exec("dummy_script.sh");
The dummy script look the following:
#!/bin/sh
{
while true
do
sleep 1
done
} &
I can see the that process gets started, but the webpage does not return before I make a 'killall dummy_script.sh'. If I run the script manually in a terminal it return immediately and spawns of the loop.
Does anyone know a way here I can spawn of the task without making the webpage wait it ?
Hope you guys can help me out, it would be most appreciated.
To answer your question:
You may start looking at pcntl_fork. Or you may check this. Basically, you are using the native fork to fork the long running process so your php frontend does not have to wait.
If you're feeling adventurous, you may put your "job" (your request to this long running process) in a DB. A cron job then checks the DB for incoming requests and it is the one that executes that process.
Another method is to use resque, but don't bother at this point.
I need to write a server-side program that lives on the server, and is checking a database consistently for new entries.
When a new entry shows up in the database, the program should process the data and put the results somewhere else.
It is important to hi-light that the process isn't instigated by new entries showing up, but by the program checking for new entries on its own.
Some people I've spoken to brought up cron jobs, I was curious what if this is the solution for me? I see that it has limitations, it won't run less than every minute. I was hoping for the program to run every 5 seconds, would I be better off writing a shell script or is that a bootleg fix?
I'm not sure if this is conventional (?) but...
Use a database trigger on INSERT that runs an external program (PHP, Python, .. whatever). Which database are you using? I think this post is old but might be of help: http://crazytechthoughts.blogspot.co.uk/2011/12/call-external-program-from-mysql.html
There is a technique I've frequently used when dealing with queues that I've been processing.
#!/bin/sh
php -f checkDBAndAct.php
sleep 5
exec $0
The exec $0 part starts the script running again, replacing itself in memory, so it will run forever without issues. Any memory the PHP script uses is cleaned up whenever it exits, so that's not a problem either.
A simple line will start it, and put it into the background:
cd /x/y/z ; nohup ./loopToProcessDB.sh &
or it can be similarly started when the machine starts with various means (such as Cron's '#reboot ....')
-- from https://stackoverflow.com/a/2686100/6216
An extended version is on http://PHPscaling.com and https://gist.github.com/alister/1386212
Though I'd use an actual queue system, rather than a DB, as there are a number of downsides to bending a database to this task.
I'm having a problem where putty gets regularly disconnected. So, when I run a PHP script from the terminal, it always gets interrupted. The script is supposed to run several hours, so I'm not having any luck with it.
How can I completely run this from the server side? I'm reading about cron jobs, but I'm having a hard time understanding at this time. Is there any alternative to cron for what I need?
I have several script PHP files that need to be run, one by one, or perhaps two at a time. Any ideas?
You don't need to leave it run in a cron job - you can just run the php script inside a screen.
Simply type;
screen php /path/to/myphpscript.php
A screen will continue running even after you disconnect from PuTTY. If you need to check up on it, you can use;
screen -r
To re-attach yourself to this process, and view any output.
You need to prevent the process from terminating when the session disconnects.
Something like this would work:
nohup php myscript.php
You can create a cron job to start the php script periodically based on a list of time tasks. More info. You could also start the task in the background from the console. i.e. php-cgi script.php& this would make the script a background task
Take a look at GNU Screen; it allows you to detach and reattach a session later, which is perfect for long-running scripts. Cron is a good option if you want it to happen in a recurring fashion; one-off batch jobs can be scheduled with something like at. For more intense computing needs, you might want to look into a more full-fledged job scheduling system like TORQUE.
You can run your program in background
php ./yourscript.php &