We need to distribute software which should contain a PHP script which will run for some minutes. Therefore I am searching a best practise way to do this in 2017.
It has to be invoked by an HTTP request. There should be no HTTP request waiting for some minutes so the script has to run still AFTER the visitor got his HTTP response.
It has to run periodically (every night). It also should run every night per default (like a cron job). Notice: since the software is going to be distributed to clients there is no way for us to add a cronjob manually (we have no access to our clients servers). Everything should be accomplished within PHP code.
(Please note that I read existing blog posts and Stackoverflow questions myself but I could not find a satisfying answer)
Maybe anyone knows how frameworks like Symfony and Laravel or webshops like Magento accomplish such tasks? Still I want to know how to do it by myself in plain PHP without using frameworks or libraries.
Many solutions exist:
using exec (rather insecure), that triggers a background job (recommended in comments, I would probably prefer symfony process, but insecure nevertheless).
using a cron to trigger a symfony process every so often, not over http so way more secure.
using php-fpm, you can send a response without stopping the process using fastcgi_finish_request
using a queue system (SQS, RabbitMQ, Kafka and so on).
using a cron manager in PHP
using a daemon and something like supervisord to make sure it runs continuously.
The best solutions are definitely queues and cron, then PHP-FPM, rest of it is just rubbish.
There is absolutely no way for you to run on someone's server without doing something that won't work at some point.
Sidenote: you said you did not want libraries in order to know how to do it yourself, I added links to libraries as reading them may give you a deeper knowledge of the technology, these libraries are really high quality.
Magento only runs it cronjobs, if you setup a regular cronjob for Magento. It has a cron.sh, that runs every minute an executes jobs in Magento's queue.
Any Solution to execute long-running tasks via http involves web-server configuration.
Finally I think there are two ways to start a long running process in PHP via an HTTP request (without letting the user wait for a long time):
Using FPM and send the response to the user with fastcgi_finish_request(). After the response is sent you can do whatever you want, for example long running tasks. (Here you don't have to start a new process, just continue with PHP).
fastcgi_finish_request();
longrunningtask();
Create a new process using one of the following functions. Redirect STDOUT and STDERR to null and put it to the background (you have to do both). To still get output of the new process the new process can write to some log file.
exec('php longrunningtask.php >/dev/null 2>/dev/null &');
shell_exec('php longrunningtask.php >/dev/null 2>/dev/null &');
system('php longrunningtask.php >/dev/null 2>/dev/null &');
passthru('php longrunningtask.php >/dev/null 2>/dev/null &');
Or use proc_open() like symfony/process does.
Notes:
symfony/process: on the docs you can read that you can use FPM and fastcgi_finish_request() for long running tasks.
Security: I can see no built in security risk, you just have to do things right then everything is fine (you could use password protection, validate possible inputs to your commands, etc.).
For the cron-job-part of the question there is no answer. I don't think it's possible.
Related
I am creating a pdf report of some 100 pages using shell script mode of wkhtmltopdf. Now, the thing is that the script takes a whole lot of time to create the report and save it in the specified path.
Currently, I am calling the script like this:
$stmt = $sh_script.' > /dev/null 2>/dev/null &';
shell_exec($stmt);
Doing this causes the shell to run in background and the report gets generated, but on the browser end after sometime I see
504 Gateway Time-out
nginx/1.4.6 (Ubuntu)
Which is not a very comfortable message, I want to call the shell_exec statement in an asynchronous way so that it is called and php code moves on. Can someone help me with this?
So before I give you a recommended solution for your question, I want to add really quick that I would probably find a php library which can achieve what you are looking to do, instead of executing shell commands from PHP, which can be a bad idea for many reasons, but if you intend on going that route, please look at this answer for some advice on security when executing shell commands via PHP:
https://stackoverflow.com/a/4535900/4660602
Now, to get back to your answer, anytime you are executing a big task in PHP that requires some extra time, it is never really a good idea to keep the user waiting at the browser. That is where building a Queue can become crucial to your application. You begin to work on some task AFTER you explain to your user via your user interface that the work is being done in the background and they will be notified when it is completed, etc.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, IronMQ or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner, but the user does not have to wait for a response until it finishes working, hence no more 504 timeouts.
OR you could try a much dirtier solution and just increase the script timeout value in php and on your server, but this can have some unexpected results. For nginx & for Apache
Best of luck!
You're close .. But you want nohup (No hang up) and your statment at the end should look more like /dev/null 2>&1 &
That said .. I believe this should work:
exec('nohup' . $sh_script . ' > /dev/null 2>&1 &')
I'm creating a webservice for an Android app in PHP with MySQL. I want to continuously check whether any data is available. I haven't got any idea how to get data as a background process. How can I execute a query without any request or without calling file?
I searched and got some code like
$command = "php -d max_execution_time=50 -f myfile.php '".$param."' >/dev/null &";
exec($command);
But where should I put this code so this query will run continuously?
Yes, the ampersand trick will work. You can use something like supervisord to restart it every few hours, so that any memory leaks are dealt with. This also makes it less fragile if it were to crash or hang.
Also, you can use something like cron to run a task for 10 minutes, and then die off and wait for cron to start it again - bear in mind that with most background tasks, it doesn't matter if there's a short period the task is not running, since it will catch up. It's worth checking in each run whether the previous one is still running, and exit early if it is: that way you don't have two background tasks causing race-conditions when retrieving work from your database.
Finally you can use a job server, such as Gearman. This will allow you to send tasks to it in an asynchronous fashion, and they will be run by worker tasks (in either time or priority order). This is probably the most reliable approach, but it takes a bit more work to set up. There's a PHP module for this, but in my experience it's more of a hassle to use than Net_Gearman, which is available in PEAR.
I'm having a problem where putty gets regularly disconnected. So, when I run a PHP script from the terminal, it always gets interrupted. The script is supposed to run several hours, so I'm not having any luck with it.
How can I completely run this from the server side? I'm reading about cron jobs, but I'm having a hard time understanding at this time. Is there any alternative to cron for what I need?
I have several script PHP files that need to be run, one by one, or perhaps two at a time. Any ideas?
You don't need to leave it run in a cron job - you can just run the php script inside a screen.
Simply type;
screen php /path/to/myphpscript.php
A screen will continue running even after you disconnect from PuTTY. If you need to check up on it, you can use;
screen -r
To re-attach yourself to this process, and view any output.
You need to prevent the process from terminating when the session disconnects.
Something like this would work:
nohup php myscript.php
You can create a cron job to start the php script periodically based on a list of time tasks. More info. You could also start the task in the background from the console. i.e. php-cgi script.php& this would make the script a background task
Take a look at GNU Screen; it allows you to detach and reattach a session later, which is perfect for long-running scripts. Cron is a good option if you want it to happen in a recurring fashion; one-off batch jobs can be scheduled with something like at. For more intense computing needs, you might want to look into a more full-fledged job scheduling system like TORQUE.
You can run your program in background
php ./yourscript.php &
In the past, I ran a bunch of scripts each as a separate cron job. Now I'd like to run a controller script with one cron job, then have that call the scripts separately (and in parallel, all at the same time), so I don't have to create a new cron job every time I add another script.
I looked up pcntl_fork() but we don't have that installed. Can fsockopen() do this as well?
A few questions:
I saw this example, http://phplens.com/phpeverywhere/?q=node/view/254, that uses fsockopen(). Will this allow me to run PHP scripts in parallel? Note, the scripts don't interact, but I would still like to know if any of them exited prematurely with an error.
Secondly the scripts I'm running aren't externally accessible, they are internal only. The script was previously run like so: php -f /path/to/my/script1.php. It's not a web-accessible path. Would the example in #1 work with this, or only web-accessible paths?.
Thanks for any advice you can offer.
You can use proc_open to run multiple processes without waiting for each process to finish.
You will have a process handle, you can terminate each process at any time and you can read the standard output of each process.
You can also communicate via pipes, which is optional.
Passing 1st param php /your/path/to/script.php param1 "param2 x" means starting a separate PHP process.
proc_open (see Example #1)
Ultimately you will want to use an infinite while loop + usleep (or sleep) to avoid maxing out on the CPU. Break when all processes finish, or after you killed them.
Edit: you can know if a process has exited prematurely.
Edit2: a simpler way of doing the above is popen
Please correct me if I'm wrong, but if I understand things correctly, the solution Tiberiu-Ionut Stan proposed implies that starting the processes with proc_open and waiting for them to finish will not be run as a cron script, but is part of a running program/service, right?
As far as I understand the cron jobs, the controller script user920050 was thinking of using would be started by cron on a schedule and each new instance would launch the processes all over again, do the waiting for them to finish and probably run in parallel with other cron-launched instances of the controller script.
I know about PHP not being multithreaded but i talked with a friend about this: If i have a large algorithmic problem i want to solve with PHP isn't the solution to simply using the "curl_multi_xxx" interface and start n HTTP requests on the same server. This is what i would call PHP style multithreading.
Are there any problems with this in the typical webserver environment? The master request which is waiting for "curl_multi_exec" shouldn't count any time against its maximum runtime or memory length.
I have never seen this anywhere promoted as a solution to prevent a script killed by too restrictive admin settings for PHP.
If i add this as a feature into a popular PHP system will there be server admins hiring a russian mafia hitman to get revenge for this hack?
If i add this as a feature into a
popular PHP system will there be
server admins hiring a russian mafia
hitman to get revenge for this hack?
No but it's still a terrible idea for no other reason than PHP is supposed to render web pages. Not run big algorithms. I see people trying to do this in ASP.Net all the time. There are two proper solutions.
Have your PHP script spawn a process
that runs independently of the web
server and updates a common data
store (probably a database) with
information about the progress of
the task that your PHP scripts can
access.
Have a constantly running daemon
that checks for jobs in a common
data store that the PHP scripts can
issue jobs to and view the progress
on currently running jobs.
By using curl, you are adding a network timeout dependency into the mix. Ideally you would run everything from the command line to avoid timeout issues.
PHP does support forking (pcntl_fork). You can fork some processes and then monitor them with something like pcntl_waitpid. You end up with one "parent" process to monitor the children it spanned.
Keep in mind that while one process can startup, load everything, then fork, you can't share things like database connections. So each forked process should establish it's own. I've used forking for up 50 processes.
If forking isn't available for your install of PHP, you can spawn a process as Spencer mentioned. Just make sure you spawn the process in such a way that it doesn't stop processing of your main script. You also want to get the process ID so you can monitor the spawned processes.
exec("nohup /path/to/php.script > /dev/null 2>&1 & echo $!", $output);
$pid = $output[0];
You can also use the above exec() setup to spawn a process started from a web page and get control back immediately.
Out of curiosity - what is your "large algorithmic problem" attempting to accomplish?
You might be better to write it as an Amazon EC2 service, then sell access to the service rather than the package itself.
Edit: you now mention "mass emails". There are already services that do this, they're generally known as "spammers". Please don't.
Lothar,
As far as I know, php don't work with services, like his concorrent, so you don't have a way for php to know how much time have passed unless you're constantly interrupting the process to check the time passed .. So, imo, no, you can't do that in php :)