i have my queues set up and working (the jobs get run), however the script doesn't seem to wait for my exec line to run before progressing with the next line of code.
This means i'm getting exceptions in the next few lines (because it's looking for a file that hasn't been produced yet)
My closure is:
Queue::push(function($job) use ($gid,$eid)
{
$phantomLoc = base_path()."/vendor/bin/phantomjs";
$scriptLoc = app_path()."/libraries/makeVideo.js";
$pageAddress = route('image_maker_video', array($gid,$eid));
$imageName = base_path().'/../data/team_images/'.$gid.'/video-sheets/'.$eid."/";
$execString = $phantomLoc.' '.$scriptLoc.' '.$pageAddress.' '.$imageName;
//empty the folder first
Helpers::emptyFolder($imageName);
exec($execString, $return_array, $return_value);
if ($return_value == 0) {
//now convert image sequence to video
$outputPath = base_path().'/../data/team_images/'.$gid.'/video-sheets/'.$eid;
$return_value = Helpers::PNGsToVideo($imageName, $outputPath);
if ($return_value == 0) {
//it worked!!
Helpers::emptyFolder($imageName);
//rmdir($imageName);
return "video in progress";
return Redirect::to('/team_images/'.$gid.'/video-sheets/'.$eid.".mkv");
} else {
Log::error($return_value." - ffmpeg return val");
abort(500, $return_value." - ffmpeg return val");
}
} else {
Log::error($return_value." - video phantom return val");
abort(500, $return_value." - video phantom return val");
}
$job->delete();
});
and it seems to skip straight through the exec line, although i do think it is still being run.
Note, if i change the driver back to sync then it all runs completely fine (but obviously not in a queue)
Any idea how to wait for exec?
It turns out that exec was erroring out, but the error (from phatomjs)gave a return code of 0.
Turns out that the error was because the line
$pageAddress = route('image_maker_video', array($gid,$eid));
Was providing a url that just had localhost rather than localhost:8888
So i've hacked it to do a string replace for localhost.
Not sure why putting it into a queue would cause laravel to provide incorrect urls. But at least it's working!
Related
I made a Symfony console command which uses pngquant to process and compress a long list of images. The images are read from a CSV file.
The batch is working fine until the end in local environment but in the stage environment it works for about 5 minutes and then it starts returning empty result from the shell_exec command. I even made a retry system but it's always returning empty result:
// escapeshellarg() makes this safe to use with any path
// errors are redirected to standard output
$command = sprintf(
'%s --quality %d-%d --output %s --force %s 2>&1',
$this->pngquantBinary,
$minQuality,
$maxQuality,
escapeshellarg($tempPath),
$path
);
// tries a few times
$data = null;
$attempt = 0;
do {
// command result
$data = shell_exec($command);
// error
if (null !== $data) {
$this->logger->warning('An error occurred while compressing the image with pngquant', [
'command' => $command,
'output' => $data,
'cpu' => sys_getloadavg(),
'attempt' => $attempt + 1,
'sleep' => self::SLEEP_BETWEEN_ATTEMPTS,
]);
sleep(self::SLEEP_BETWEEN_ATTEMPTS);
}
++$attempt;
} while ($attempt < self::MAX_NUMBER_OF_ATTEMPTS && null !== $data);
// verifies that the command has finished successfully
if (null !== $data) {
throw new \Exception(sprintf('There was an error compressing the file with command "%s": %s.', $command, $data));
}
The problem is that the same command executed in another shell in the same environment works fine! I mean, when I log the error, if I put the exactly same command in another instance on the same server, works fine.
Even from the Symfony logs I can't read any error, where should I look for a more detailed error?
What can be causing this? Memory and processor are fine during execution!
After many attempts I read this question:
Symfony2 Process component - unable to create pipe and launch a new process
The solution was to add a call to gc_collect_cycles after flush during the loop!
if ($flush || 0 === ($index % self::BATCH_SIZE)) {
$this->em->flush();
$this->em->clear();
// clears the temp directory after flushing
foreach ($this->tempImages as $tempImage) {
unlink($tempImage);
}
$this->tempImages = [];
// forces collection of any existing garbage cycles
gc_collect_cycles();
}
IMPORTANT: also keep an eye on the number of disk inodes.
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 125193 392 124801 1% /dev
tmpfs 127004 964 126040 1% /run
/dev/vda2 5886720 5831604 55116 100% /
I am attempting to call one Artisan (Laravel) command from another command. However, I need to be able to retrieve an array from the command that is called from the "main" command...
i.e
// Command 1
public function handle() {
$returnedValue = $this->call( 'test:command' );
dump( $returnedValue ); // <-- is 5
}
// Command 2
public function handle() {
return $this->returnValue();
}
private function returnValue() {
$val = 5;
return $val;
}
I have looked through the documentation and can't find a way to do this, so I was wondering if there was a way or if I need to re-think my approach.
Thanks!
Artisan Commands don't behave the same way as, for example, Controller functions. They return an exitCode, which in my testing was always 0 (couldn't get anything to return if an error is thrown).
Your approach won't work if you try to get a return value, but you can access \Artisan::output(); to see what exactly is sent by the first artisan command you call.
// FirstCommand.php
public function handle(){
\Artisan::call("second:command");
if(\Artisan::output() == 1){
$this->info("This Worked");
} else {
$this->error("This Didn't Work");
}
}
Note: I used \Artisan::call(); there's some apparent differences between the two where using $this->call() didn't work as expected, but \Artisan::call() did. $this->call() sent both 0 and 1 back, regardless of the actual code being executed; not sure what's up there. Tested on Laravel 5.0, which is quite behind the current, so maybe that's it.
// SecondCommand.php
public function handle(){
try {
$test = 1 / 1;
} catch (\Exception $ex){
$this->error("0");
}
$this->info("1");
}
Running php artisan first:command in my console returns:
$ php artisan first:command
This Worked
Now, if switch the code in $test to
$test = 1 / 0;
I get this in my console:
$ php artisan first:command
This Didn't Work
So, the rule here I guess is to avoid outputting anything in the second command prior to the result you want to check with \Artisan::output().
I have a simple AJAX call that retrieves text from a file, pushes it into a table, and displays it. The call works without issue when testing on a Mac running Apache 2.2.26/PHP 5.3 and on an Ubuntu box running Apache 2.2.1.6/PHP 5.3. It does not work on RedHat running Apache 2.2.4/PHP 5.1. Naturally, the RedHat box is the only place where I need it to be working.
The call returns 200 OK but no content. Even if nothing is found in the file (or it's inaccessible), the table header is echoed so if permissions were a problem I would still expect to see something. But to be sure, I verified the file is readable by all users.
Code has been redacted and simplified.
My ajax function:
function ajax(page,targetElement,ajaxFunction,getValues)
{
xmlhttp=new XMLHttpRequest();
xmlhttp.onreadystatechange=function()
{
if (xmlhttp.readyState===4 && xmlhttp.status===200)
{
document.getElementById(targetElement).innerHTML=xmlhttp.responseText;
}
};
xmlhttp.open('GET','/appdir/dir/filedir/'+page+'_funcs.php?function='+ajaxFunction+'&'+getValues+'&'+new Date().getTime(),false);
xmlhttp.setRequestHeader('cache-control','no-cache');
xmlhttp.send();
}
I call it like this:
ajax('pagename','destelement','load_info');
And return the results:
// Custom file handler
function warn_error($errno, $errstr) {
// Common function for warning-prone functions
throw new Exception($errstr, $errno);
}
function get_file_contents() {
// File operation failure would return a warning
// So handle specially to suppress the default message
set_error_handler('warn_error');
try
{
$fh = fopen(dirname(dirname(__FILE__))."/datafile.txt","r");
}
catch (Exception $e)
{
// Craft a nice-looking error message and get out of here
$info = "<tr><td class=\"center\" colspan=\"9\"><b>Fatal Error: </b>Could not load customer data.</td></tr>";
restore_error_handler();
return $info;
}
restore_error_handler();
// Got the file so get and return its contents
while (!feof($fh))
{
$line = fgets($fh);
// Be sure to avoid empty lines in our array
if (!empty($line))
{
$info[] = explode(",",$line);
}
}
fclose($fh);
return $info;
}
function load_info() {
// Start the table
$content .= "<table>
<th>Head1</th>
<th>Head2</th>
<th>Head3</th>
<th>Head4</th>";
// Get the data
// Returns all contents in an array if successful,
// Returns an error string if it fails
$info = get_file_contents();
if (!is_array($info))
{
// String was returned because of an error
echo $content.$info;
exit();
}
// Got valid data array, so loop through it to build the table
foreach ($info as $detail)
{
list($field1,$field2,$field3,$field4) = $detail;
$content .= "<tr>
<td>$field1</td>
<td>$field2</td>
<td>$field3</td>
<td>$field4</td>
</tr>";
}
$content .= "</table>";
echo $content;
}
Where it works, the response header indicates the connection as keep-alive; where it fails, the connection is closed. I don't know if that matters.
I've looked all over SO and the net for some clues but "no content" issues invariably point to same-origin policy problems. In my case, all content is on the same server.
I'm at a loss as to what to do/where to look next.
file_get_contents() expects a parameter. It does not know what you want, so it returned false. Also, you used get_file_contents() which is the wrong order.
This turned out to be a PHP version issue. In the load_info function I was using filter_input(INPUT_GET,"value"), but that was not available in PHP 5.1. I pulled that from my initial code post because I didn't think it was part of the problem. Lesson learned.
My intention is this.
My client.html calls a php script check.php via ajax. I want check.php to check if another script task.php is already being run. If it is, I do nothing. If it is not, I need to run it in the background.
I have an idea what I want to do, but am unsure how to do it.
Part A. I know how to call check.php via ajax.
Part B. In check.php I might need to run task.php. I think I need something like:
$PID = shell_exec("php task.php > /dev/null & echo $!");
I think the "> /dev/null &" bit tells it to run in the background, but am unsure what the "$!" does.
Part C. The $PID I need as a tag of the process. I need to write this number (or whatever) to a file in the same directory, and need to read it every call to check.php. I can't work out how to do that. Could someone give me a link of how to read/write a file with a single number in to the same directory?
Part D. Then to check if the last launched task.php is still running I am going to use the function:
function is_process_running($PID)
{
exec("ps $PID", $ProcessState);
return(count($ProcessState) >= 2);
}
I think that is all the bits I need, but as you can see I am unsure on how to do a few of them.
I would use an flock() based mechanism to make sure that task.php runs only once.
Use a code like this:
<?php
$fd = fopen('lock.file', 'w+');
// try to get an exclusive lock. LOCK_NB let the operation not blocking
// if a process instance is already running. In this case, the else
// block will being entered.
if(flock($fd, LOCK_EX | LOCK_NB )) {
// run your code
sleep(10);
// ...
flock($fd, LOCK_UN);
} else {
echo 'already running';
}
fclose($fd);
Also note that flock() is, as the PHP documentation points out, portable across all supported operating systems.
!$
gives you the pid of the last executed program in bash. Like this:
command &
pid=$!
echo pid
Note that you will have to make sure your php code runs on a system with bash support. (Not windows)
Update (after comment of opener).
flock() will work on all operating systems (As I mentioned). The problem I see in your code when working with windows is the !$ (As I mentioned ;) ..
To obtain the pid of the task.php you should use proc_open() to start task.php. I've prepared two example scripts:
task.php
$fd = fopen('lock.file', 'w+');
// try to get an exclusive lock. LOCK_NB let the operation not blocking
// if a process instance is already running. In this case, the else
// block will being entered.
if(flock($fd, LOCK_EX | LOCK_NB )) {
// your task's code comes here
sleep(10);
// ...
flock($fd, LOCK_UN);
echo 'success';
$exitcode = 0;
} else {
echo 'already running';
// return 2 to let check.php know about that
// task.php is already running
$exitcode = 2;
}
fclose($fd);
exit($exitcode);
check.php
$cmd = 'php task.php';
$descriptorspec = array(
0 => array('pipe', 'r'), // STDIN
1 => array('pipe', 'w'), // STDOUT
2 => array('pipe', 'w') // STDERR
);
$pipes = array(); // will be set by proc_open()
// start task.php
$process = proc_open($cmd, $descriptorspec, $pipes);
if(!is_resource($process)) {
die('failed to start task.php');
}
// get output (stdout and stderr)
$output = stream_get_contents($pipes[1]);
$errors = stream_get_contents($pipes[2]);
do {
// get the pid of the child process and it's exit code
$status = proc_get_status($process);
} while($status['running'] !== FALSE);
// close the process
proc_close($process);
// get pid and exitcode
$pid = $status['pid'];
$exitcode = $status['exitcode'];
// handle exit code
switch($exitcode) {
case 0:
echo 'Task.php has been executed with PID: ' . $pid
. '. The output was: ' . $output;
break;
case 1:
echo 'Task.php has been executed with errors: ' . $output;
break;
case 2:
echo 'Cannot execute task.php. Another instance is running';
break;
default:
echo 'Unknown error: ' . $stdout;
}
You asked me why my flock() solution is the best. It's just because the other answer will not reliably make sure that task.php runs once. This is because the race condition I've mentioned in the comments below that answer.
You can realize it, using lock file:
if(is_file(__DIR__.'/work.lock'))
{
die('Script already run.');
}
else
{
file_put_contents(__DIR__.'/work.lock', '');
// YOUR CODE
unlink(__DIR__.'/work.lock');
}
Too bad I didn't see this before it was accepted..
I have written a class to do just this. ( using file locking ) and PID, process ID checking, on both windows and Linux.
https://github.com/ArtisticPhoenix/MISC/blob/master/ProcLock.php
I think your are really overdoing it with all the processes and background checks. If you run a PHP script without a session, then you are already essentially running it in the background. Because it will not block any other request from the user. So make sure you don't call session_start();
Then the next step would be to run it even when the user cancels the request, which is a basic function in PHP. ignore_user_abort
Last check is to make sure it's only runs once, which can be easily done with creating a file, since PHP doesnt have an easy application scope.
Combined:
<?php
// Ignore user aborts and allow the script
// to run forever
ignore_user_abort(true);
set_time_limit(0);
$checkfile = "./runningtask.tmp";
//LOCK_EX basicaly uses flock() to prevents racecondition in regards to a regular file open.
if(file_put_contents($checkfile, "running", LOCK_EX)===false) {
exit();
}
function Cleanup() {
global $checkfile;
unlink($checkfile);
}
/*
actual code for task.php
*/
//run cleanup when your done, make sure you also call it if you exit the code anywhere else
Cleanup();
?>
In your javascript you can now call the task.php directly and cancel the request when the connection to the server has been established.
<script>
function Request(url){
if (window.XMLHttpRequest) { // Mozilla, Safari, ...
httpRequest = new XMLHttpRequest();
} else if (window.ActiveXObject) { // IE
httpRequest = new ActiveXObject("Microsoft.XMLHTTP");
} else{
return false;
}
httpRequest.onreadystatechange = function(){
if (httpRequest.readyState == 1) {
//task started, exit
httpRequest.abort();
}
};
httpRequest.open('GET', url, true);
httpRequest.send(null);
}
//call Request("task.php"); whenever you want.
</script>
Bonus points: You can have the actual code for task.php write occasional updates to $checkfile to have a sense of what is going on. Then you can have another ajax file read the content and show the status to the user.
Lets make the whole process from B to D simple
Step B-D:
$rslt =array(); // output from first exec
$output = array(); // output of task.php execution
//Check if any process by the name 'task.php' is running
exec("ps -auxf | grep 'task.php' | grep -v 'grep'",$rslt);
if(count($rslt)==0) // if none,
exec('php task.php',$output); // run the task,
Explanation:
ps -auxf --> gets all running processes with details
grep 'task.php' --> filter the process by 'task.php' keyword
grep -v 'grep' --> filters the grep process out
NB:
Its also advisable to put the same check in task.php file.
If task.php is executed directly via httpd (webserver), it will only be displayed as a httpd process and cannot be identified by 'ps' command
It wouldn't work under load-balanced environment. [Edited: 17Jul17]
You can get an exclusive lock on the script itself for the duration of the script running
Any other attempts to run it will end as soon as the lock() function is invoked.
//try to set a global exclusive lock on the file invoking this function and die if not successful
function lock(){
$file = isset($_SERVER['SCRIPT_FILENAME'])?
realpath($_SERVER['SCRIPT_FILENAME']):
(isset($_SERVER['PHP_SELF'])?realpath($_SERVER['PHP_SELF']):false);
if($file && file_exists($file)){
//global handle stays alive for the duration if this script running
global $$file;
if(!isset($$file)){$$file = fopen($file,'r');}
if(!flock($$file, LOCK_EX|LOCK_NB)){
echo 'This script is already running.'."\n";
die;
}
}
}
Test
Run this in one shell and try to run it in another while it is waiting for input.
lock();
//this will pause execution until an you press enter
echo '...continue? [enter]';
$handle = fopen("php://stdin","r");
$line = fgets($handle);
fclose($handle);
I'm trying to create a PHP file, which wouldn't run if it's already running. Here's the code I'm using:
<?php
class Test {
private $tmpfile;
public function action_run() {
$this->die_if_running();
$this->run();
}
private function die_if_running() {
$this->tmpfile = #fopen('.refresher2.pid', "w");
$locked = #flock($this->tmpfile, LOCK_EX|LOCK_NB);
if (! $locked) {
#fclose($this->tmpfile);
die("Running 2");
}
}
private function run() {
echo "NOT RUNNNING";
sleep(100);
}
}
$test = new Test();
$test->action_run();
The problem is, when I run this from console, it works great. But when I try to run it from browser, many instances can run simultaneously. This is on Windows 7, XAMPP, PHP 5.3.2. I guess OS is thinking that it's the same process and thus the functionality falls. Is there a cross-platform way to create a PHP script of this type?
Not really anything to promising. You can't use flock for that like this.
You could use system() to start another (php) process that does the locking for you. But drawbacks:
You need to do interprocess communication. Think about a way how to tell the other program when to release the lock etc. You can use stdin for messenging und use 3 constants or something. In this case it's still rather simple
It's bad for performance because you keep creating processes which is expensive.
Another way would be to start another program that runs all the time. You connect to it using some means of IPC (probably just use a tcp channel because it's cross-platform) and allow this program to manage file acces. That program could be a php script in an endless loop as well, but it will probably be simpler to code this in Java or another language that has multithreading support.
Another way would be to leverage existing ressources. Create a dummy database table for locks, create an entry for the file and then do table-row-locking.
Another way would be not to use files, but a database.
I had a similar problem a while ago.
I needed to have a counter where the number returned was unique.
I used a lock-file and only if this instance was able to create the lock-file was it allowed to read the file with the current number.
Instead of counting up perhaps you can allow the script to run.
The trick is to let try a few times (like 5) with a small wait/sleep in between.
function GetNextNumber()
{
$lockFile = "lockFile.txt";
$lfh = #fopen($lockFile, "x");
if (!$lfh)
{
$lockOkay = false;
$count = 0;
$countMax = 5;
// Try ones every second in 5 seconds
while (!$lockOkay & $count < $countMax)
{
$lfh = #fopen($lockFile, "x");
if ($lfh)
{
$lockOkay = true;
}
else
{
$count++;
sleep(1);
}
}
}
if ($lfh)
{
$fh = fopen($myFile, 'r+') or die("Too many users. ");
flock($fh, LOCK_EX);
$O_nextNumber = fread($fh, 15);
$O_nextNumber = $O_nextNumber + 1;
rewind($fh);
fwrite($fh, $O_knr);
flock($fh, LOCK_UN);
fclose($fh);
unlink($lockFile); // Sletter lockfilen
}
return $O_nextNumber;
}