Continually running PHP script using Bash - php

I have a long running PHP script that has a memory leak which causes it to fail part way through. The script uses a 3rd party library and I haven't been able to find the source of the leak.
What I would like to do is create a bash script that continually runs the PHP script, processing 1000 records at a time, until the script returns an exit code saying it has finished processing all records. I figure this should help me get around the memory leak because the script would run for 1000 records, exit, and then a new process would be started for another 1000 records.
I'm not that familiar with Bash. Is this possible to do? How do I get the output from the PHP script?
In pseudocode, I was thinking of something along the lines of:
do:
code = exec('.../script.php')
# PHP script would print 0 if all records are processed or 1 if there is more to do
while (code != 0)

$? gives you the exit code of a program in bash
You can do something ilke
while /bin/true; do
php script.php
if [ $? != 0 ]; then
echo "Error!";
exit 1;
fi
done
You can probably even do:
while php script.php; do
echo "script returned success"
done

Do you have to use bash? You could probably do this with PHP:
while (true) {
$output = exec('php otherscript.php', $out, $ret);
}
The $ret variable will contain the exit code of the script.

Use a simple until loop to automatically test the exit status of the PHP script.
#!/bin/sh
until script.php
do
:
done
The colon is merely a null operator, since you don't actually want to do anything else in the loop.
until while execute the command script.php until it returns zero (aka true). If the script returned 0 to indicate not done instead of 1, you could use while instead of until.
The output of the PHP script would go to standard out and standard error, so you could wrap the invocation of the shell script with some I/O re-direction to stash the output in a file. For example, if the script is called loop.sh, you'd just run:
./loop.sh > output.txt
though of course you can control the output file directly in the PHP script; you just have to remember to open the file for append.
You might want to ask a separate question about how to debug the PHP memory leak though :-)

You can write:
#!/bin/bash
/usr/bin/php prg.php # run the script.
while [ $? != 0 ]; do # if ret val is non-zero => err occurred. So rerun.
/usr/bin/php prg.php
done

Implemented solution in PHP instead as:
do {
$code = 1;
$output = array();
$file = realpath(dirname(__FILE__)) . "/script.php";
exec("/usr/bin/php {$file}", $output, $code);
$error = false;
foreach ($output as $line) {
if (stripos($line, 'error') !== false) {
$error = true;
}
echo $line . "\n";
}
} while ($code != 0 && !$error);

Related

php live output for bash script if output is filtered [duplicate]

I'm just experimenting with PHP and shell_exec on my Linux server. It's a really cool function to use and I am really enjoying it so far. Is there a way to view the live output that is going on while the command is running?
For example, if ping stackoverflow.com was run, while it is pinging the target address, every time it pings, show the results with PHP? Is that possible?
I would love to see the live update of the buffer as it's running. Maybe it's not possible but it sure would be nice.
This is the code I am trying and every way I have tried it always displays the results after the command is finished.
<?php
$cmd = 'ping -c 10 127.0.0.1';
$output = shell_exec($cmd);
echo "<pre>$output</pre>";
?>
I've tried putting the echo part in a loop but still no luck. Anyone have any suggestions on making it show the live output to the screen instead of waiting until the command is complete?
I've tried exec, shell_exec, system, and passthru. Everyone of them displays the content after it's finished. Unless I'm using the wrong syntax or I'm not setting up the loop correctly.
To read the output of a process, popen() is the way to go. Your script will run in parallel with the program and you can interact with it by reading and writing it's output/input as if it was a file.
But if you just want to dump it's result straight to the user you can cut to the chase and use passthru():
echo '<pre>';
passthru($cmd);
echo '</pre>';
If you want to display the output at run time as the program goes, you can do this:
while (# ob_end_flush()); // end all output buffers if any
$proc = popen($cmd, 'r');
echo '<pre>';
while (!feof($proc))
{
echo fread($proc, 4096);
# flush();
}
echo '</pre>';
This code should run the command and push the output straight to the end user at run time.
More useful information
Note that if you are using sessions then having one of those running will prevent the user from loading other pages, as sessions enforce that concurrent requests cannot happen. To prevent this from being a problem, call session_write_close() before the loop.
If your server is behind a nginx gateway, then the nginx buffering may be disruptive to the desired behavior. Set the header header('X-Accel-Buffering: no'); to hint nginx that it shouldn't do that. As headers are sent first, this has to be called in the beginning of the script, before any data is sent.
First of all, thanks Havenard for your snippet - it helped a lot!
A slightly modified version of Havenard's code which i found useful.
<?php
/**
* Execute the given command by displaying console output live to the user.
* #param string cmd : command to be executed
* #return array exit_status : exit status of the executed command
* output : console output of the executed command
*/
function liveExecuteCommand($cmd)
{
while (# ob_end_flush()); // end all output buffers if any
$proc = popen("$cmd 2>&1 ; echo Exit status : $?", 'r');
$live_output = "";
$complete_output = "";
while (!feof($proc))
{
$live_output = fread($proc, 4096);
$complete_output = $complete_output . $live_output;
echo "$live_output";
# flush();
}
pclose($proc);
// get exit status
preg_match('/[0-9]+$/', $complete_output, $matches);
// return exit status and intended output
return array (
'exit_status' => intval($matches[0]),
'output' => str_replace("Exit status : " . $matches[0], '', $complete_output)
);
}
?>
Sample Usage :
$result = liveExecuteCommand('ls -la');
if($result['exit_status'] === 0){
// do something if command execution succeeds
} else {
// do something on failure
}
If you're willing to download a dependency, Symfony's processor component does this. I found the interface to working with this cleaner than reinventing anything myself with popen() or passthru().
This was provided by the Symfony documentation:
You can also use the Process class with the foreach construct to get
the output while it is generated. By default, the loop waits for new
output before going to the next iteration:
$process = new Process('ls -lsa');
$process->start();
foreach ($process as $type => $data) {
if ($process::OUT === $type) {
echo "\nRead from stdout: ".$data;
} else { // $process::ERR === $type
echo "\nRead from stderr: ".$data;
}
}
As a warning, I've run into some problems PHP and Nginx trying to buffer the output before sending it to the browser. You can disable output buffering in PHP by turning it off in php.ini: output_buffering = off. There's apparently a way to disable it in Nginx, but I ended up using the PHP built in server for my testing to avoid the hassle.
I put up a full example of this on Gitlab: https://gitlab.com/hpierce1102/web-shell-output-streaming

Executing a php script in order

I have a multiple php scripts that are working currently right now. I created a cron job to execute all the scripts at a given time. But now the client want a trigger/event type so he can execute those scripts. So I thought of using the exec function.
So here is the problem, those script has to be executed in order. E.g: I have 2 scripts namely step1.php & step2.php. How do I run the 2 php script in order and in the backgroud process.
I read that using the 3rd parameter in exec function can return a result but it only always gave me a result: string(0) ""
This is what I want to achieve:
$step1 = exec("php step1.php > /dev/null &", $output, $returnVal);
if($step1 === TRUE) exec("php step2.php > /dev/null &", $output, $returnVal);
Or maybe their another php function that are more suitable than using exec??? Really don't know. Please help
Thanks a many guys
You don't have to run exec() from a php script to handle code in other php scripts. There are probably a dozen ways to do what you want, but one I would probably use:
Functionalize the code in step1.php and step2.php:
Old (Pseudo):
<?php
$var = true;
return $var;
?>
New:
<?php
function foo() {
$var = true;
return $var;
}
?>
Include those scripts (because the code is now functionalized, it doesn't get executed until you call the functions). So, in your script that calls the step scripts:
<?php
include('step1.php');
include('step2.php');
?>
Now call the functions you need with whatever logic you require:
<?php
include('step1.php');
include('step2.php');
if(foo() == true) {
bar(); //bar() is found in step2.php
}
?>
Again, several ways to accomplish this, much depending on your requirements and what the code in the step php scrips are doing. I'm making assumptions about that given the lack of details about what step1 and step2 are trying to execute.
Call the second script on the end of the first one.
if (... == TRUE) {
include('second_script.php');
}
And then you only need to run the first script on cron.
Assuming that you have step1.php outputting true on success
$step1 = exec("php step1.php > /dev/null &", $output, $returnVal);
$step1 is now the last line from the result of the command.
[EDIT: Misread the manual. Depending on what you want, you probably do need to check $step1 for the outputted result, and you are getting an empty string because you're not outputting anything in the step1.php script? Everything else appears to be correct however.]
What you should be checking is $returnVal for return status. This variable is passed by reference (according to the manual), so your code should be:
exec("php step1.php > /dev/null &", $output, $returnVal);
if($returnVal === TRUE) exec("php step2.php > /dev/null &", $output, $returnVal);
if($returnVal === TRUE) exec("php step3.php > /dev/null &", $output, $returnVal);
You could even use a while loop:
$num_steps = 4; //Arbitrary number for example
$step = 1;
while($returnVal === TRUE && $i <= $num_steps) {
exec('php step'.$step.'.php > /dev/null &', $output, $returnVal);
$i++;
}
[Careful, I haven't tested the above and it may not be suitable for what you want to do]
EDIT: islanddave's answer below is 'better'. I assumed that your current process was set, and you cannot change it (e.g. amount of time you have, can't refactor existing code for legacy reasons etc.).

running shell_exec in php causes web server to hang

I am running the following code. What it does is take a text file, splits it into parts that end with '_part' ending and than calls the same script with a flag to process the files - uploading the content to a Drupal system.
What happens is that the script runs and finishes the work, all invoked scripts finish too and I can see the results. but each time after I run it the web server stops responding. Is there anything basic that I am missing or doing wrong?
if(isset($argv[3])){
$isSplit = $argv[3] == 'true' ? true : false;
}
if($isSplit){
$fileSplitter = new CSVFileParts($fileName);
$parts = $fileSplitter->split_file();
echo 'Splited file to '.$parts.' parts'.PHP_EOL;
for($part =0; $part < $parts; $part++){
echo shell_exec('php Service.php u ./partial_files/'.basename($fileName).'.part_'.$part.' false > /dev/null 2>/dev/null &');
}
}else{
$log->lwrite('uploading '.$argv[2]);
$drupalUploader = new DrupalUploader($fileName, $log);
$drupalUploader->upload();
}
shell_exec — Execute command via shell and return the complete output as a string
shell_exec expects the file handle to be open, but you redirect everything to /dev/null and detach it.
As you plan to detach the process and remove all the output, you should use exec() and escapeshellcmd()
see: http://www.php.net/manual/en/function.exec.php

How can I script a 'shutdown -r 1' and return an exit status?

I am writing a program that will at some point call a shell script. I need this shell script (bash, or if necessary PHP 4+ will work) to be called by the program, and return an exit status that I can relay before the 1 minute is reached and the system reboots.
Here's an idea of what I mean, best as I can describe:
Program calls 'reboot' script
Reboot script runs 'shutdown -r 1' and then exits with a status of 0
Program echo's out the exit status
Server reboots
I can get everything to work except the exit status - no matter what I try the program never exits its loop waiting for an exit status, so it never returns anything but the reboot still occurs. This program runs other scripts that return exit statuses, so I need this one to as well to maintain functionality and all that...
Any help is appreciated!
EDIT- The program that calls the reboot script is a PHP script that runs in a loop. When certain events happen, the program runs certain scripts and echos out the exit status. All of them work but this - it never returns an exit status.
Scripts are being called using system($cmd) where $cmd is './scriptname.sh'
Assuming you're opening the process using proc_open, then calling proc_get_status should return an array that has the exit code in it.
You could create a bash script that backgrounds the shutdown process:
#!/bin/bash
shutdown -r 1 &
exit 0
This returns control to the parent shell, which receives "0" as the exit code.
Unfortunately, you can't rely on PHP's system() and exec() functions to retrieve the proper return value, but with a nice little workaround in BASH, it's possible to parse exit code really effectively:
function runthis($command) {
$output = array();
$retcode = -1;
$command .= " &2>1; echo $?";
exec($command, $output, $retcode);
$retcode = intval(array_pop($output));
return $retcode;
}
if (runthis("shutdown -r 1") !== 0) echo "Command failed!\n";
Let me break down what does the code doing:
$command .= " &2>1; echo $?"; - expand the command so we pipe the stderr into stdout, then run echo $?
echo $? - this special bash parameter which expands to the last executed command's exit code.
exec($command, $output, $retcode); - execute the command. ($retcode is just a placeholder here since the returned data isn't trustworthy. We'll overwrite it later.) The command's output will be written in $output as an array. Every element will represent an individual row.
$retcode = intval(array_pop($output)); - parse the last row as an integer. (since the last command will be echo $?, it will be always the actual exitcode.
And that's all you need! Although it's a really crude code, and prone to errors if not used correctly, it's perfect for executing simpler tasks, and it will always give you the proper exit code.
For more professional (and programmatic) approach, you have to dig yourself into PHP's pnctl, posix, stream functions, and also Linux pipe handling.

PHP exec() return value for background process (linux)

Using PHP on Linux, I'd like to determine whether a shell command run using exec() was successfully executed. I'm using the return_var parameter to check for a successful return value of 0. This works fine until I need to do the same thing for a process that has to run in the background. For example, in the following command $result returns 0:
exec('badcommand > /dev/null 2>&1 &', $output, $result);
I have put the redirect in there on purpose, I do not want to capture any output. I just want to know that the command was executed successfully. Is that possible to do?
Thanks, Brian
My guess is that what you are trying to do is not directly possible. By backgrounding the process, you are letting your PHP script continue (and potentially exit) before a result exists.
A work around is to have a second PHP (or Bash/etc) script that just does the command execution and writes the result to a temp file.
The main script would be something like:
$resultFile = '/tmp/result001';
touch($resultFile);
exec('php command_runner.php '.escapeshellarg($resultFile).' > /dev/null 2>&1 &');
// do other stuff...
// Sometime later when you want to check the result...
while (!strlen(file_get_contents($resultFile))) {
sleep(5);
}
$result = intval(file_get_contents($resultFile));
unlink($resultFile);
And the command_runner.php would look like:
$outputFile = $argv[0];
exec('badcommand > /dev/null 2>&1', $output, $result);
file_put_contents($outputFile, $result);
Its not pretty, and there is certainly room for adding robustness and handling concurrent executions, but the general idea should work.
Not using the exec() method. When you send a process to the background, it will return 0 to the exec call and php will continue execution, there's no way to retrieve the final result.
pcntl_fork() however will fork your application, so you can run exec() in the child process and leave it waiting until it finishes. Then exit() with the status the exec call returned.
In the parent process you can access that return code with pcntl_waitpid()
Just my 2 cents, how about using the || or && bash operator?
exec('ls && touch /tmp/res_ok || touch /tmp/res_bad');
And then check for file existence.

Categories