I am running VW in daemon mode. As a standalone executable, it runs perfectly fine. In daemon mode, I see something about predictions and options initially but not the end result. Not sure what exactly is going on.
This is how I call VW6
/bin64/vw --daemon --num_children 2 -t -i ~/modelbow.vw6 --min_prediction 0 --max_prediction 1 -p stdout 2>&1
I check vw6 is running fine. I send data using simple php script (removed debug lines for brevity):
$fp = fsockopen("localhost",26542, $errno, $errstr, 3);
$fp_dat = fopen("/tmp/ml.dat", "r");
$mldata = explode("\n", file_get_contents("/tmp/ml.dat"));
$mlstr = implode($mldata);
fwrite($fp, $mlstr);
$result = trim(fgets($fp, 1024));
print $result;
Print $result above prints nothing. The only thing I see in stdout is
num sources = 1
Num weight bits = 28
learning rate = 10
initial_t = 1
power_t = 0.5
decay_learning_rate = 1
predictions = stdout
only testing
average since example example current current current
loss last counter weight label predict features
While in a standalone executable mode, if I run with the same model same dat file just without the -daemon option, it happily gives a result at the end
...
...
predictions = stdout
only testing
average since example example current current current
loss last counter weight label predict features
1.000000 ba66dfc7a135e2728d08010b40586b90
Any idea what could be going wrong here with the daemon mode? I tried using -p /tmp/ option as well...ran the daemon mode with sudo but nothing helped. Is there a debug dump option or verbose option or something else to know what exactly is going on?
thanks
The reason it is not working is not in vw but in the PHP client code.
explode on "\n", strips the newlines out.
implode without a glue-string parameter results in glue-string defaulting to the empty string.
Result: newlines are stripped out.
All examples are merged into one big (and incomplete, since there's no newline at the end) example.
vw needs newlines to separate examples, without them it will be waiting forever for the 1st example to complete.
So I think you need to change the implode line of code to:
$mlstr = implode("\n", $mldata);
for it to work.
You will also need an additional ending newline to get the last line through.
Related
With a PHP script which runs in CLI mode, I want to get the cursor position in a portable way.
With the code :
// Query Cursor Position
echo "\033[6n";
In the terminal, this code reports the cursor position, as
wb ?> ./script.php
^[[5;1R
wb ?> ;1R
But, I can't retrieve the two values (row: 5, column: 1) in the code.
After some tests with output buffering :
ob_start();
echo "\033[6n";
$s = ob_get_contents();
file_put_contents('cpos.txt',$s);
I've "\033[6n" in the cpos.txt file, not the device answer.
And reading STDIN :
$timeout = 2;
$sent = false;
$t = microtime(true);
$buf = '';
stream_set_blocking(STDIN,false);
while(true){
$buf .= fread(STDIN,8);
if(!$sent){
echo "\033[6n";
$sent = true;
}
if($t+$timeout<microtime(true))
break;
}
var_dump($buf);
The buffer is empty but the terminal show the device answer :
wb ?> ./script.php
^[[5;1R
string(0) ""
wb ?>
Is there a way, without curses, to get the cursor position ?
The code you have so far almost works, and you'll find that hitting enter and waiting for your timeout to complete does produce a string containing the answer, but with a \n character on the end. (Note the string length of 7 instead of 0.)
$ php foo.php
^[[2;1R
string(7) "
"
The issue here is that stream_set_blocking does not prevent the terminal from buffering input line-by-line, so the terminal doesn't send anything to stdin of your program until the enter key is pressed.
To make the terminal send characters immediately to your program without line-buffering, you need to set the terminal to "non-canonical" mode. This disables any line-editing features, such as the ability to press backspace to erase characters, and instead sends characters to the input buffer immediately. The easiest way to do this in PHP is to call the Unix utility stty.
<?php
system('stty -icanon');
echo "\033[6n";
$buf = fread(STDIN, 16);
var_dump($buf);
This code successfully captures the response from the terminal into $buf.
$ php foo.php
^[[2;1Rstring(6) ""
However, this code has a couple of issues. First of all, it doesn't re-enable canonical mode in the terminal after it's finished. This could cause issues when trying to input from stdin later in your program, or in your shell after your program exits. Secondly, the response code from the terminal ^[[2;1R is still echoed to the terminal, which makes your program's output look messy when all you want to do is read this into a variable.
To solve the input echoing issue, we can add -echo to the stty arguments to disable input echoing in the terminal. To reset the terminal to its state before we changed it, we can call stty -g to output a list of current terminal settings which can be passed to stty later to reset the terminal.
<?php
// Save terminal settings.
$ttyprops = trim(`stty -g`);
// Disable canonical input and disable echo.
system('stty -icanon -echo');
echo "\033[6n";
$buf = fread(STDIN, 16);
// Restore terminal settings.
system("stty '$ttyprops'");
var_dump($buf);
Now when running the program, we don't see any junk displayed in the terminal:
$ php foo.php
string(6) ""
One last potential improvement we can make to this is to allow the program to be run when stdout is redirected to another process / file. This may or may not be necessary for your application, but currently, running php foo.php > /tmp/outfile will not work, as echo "\033[6n"; will write straight to the output file rather than to the terminal, leaving your program waiting for characters to be sent to stdin as the terminal was never sent any escape sequence so will not respond to it. A workaround for this is to write to /dev/tty instead of stdout as follows:
$term = fopen('/dev/tty', 'w');
fwrite($term, "\033[6n");
fclose($term); // Flush and close the file.
Putting this all together, and using bin2hex() rather than var_dump() to get a listing of characters in $buf, we get the following:
<?php
$ttyprops = trim(`stty -g`);
system('stty -icanon -echo');
$term = fopen('/dev/tty', 'w');
fwrite($term, "\033[6n");
fclose($term);
$buf = fread(STDIN, 16);
system("stty '$ttyprops'");
echo bin2hex($buf) . "\n";
We can see that the program works correctly as follows:
$ php foo.php > /tmp/outfile
$ cat /tmp/outfile
1b5b323b3152
$ xxd -p -r /tmp/outfile | xxd
00000000: 1b5b 323b 3152 .[2;1R
This shows that $buf contained ^[[2;1R, indicating the cursor was at row 2 and column 1 when its position was queried.
So now all that's left to do is to parse this string in PHP and extract the row and column separated by the semicolon. This can be done with a regex.
<?php
// Example response string.
$buf = "\033[123;456R";
$matches = [];
preg_match('/^\033\[(\d+);(\d+)R$/', $buf, $matches);
$row = intval($matches[1]);
$col = intval($matches[2]);
echo "Row: $row, Col: $col\n";
This gives the following output:
Row: 123, Col: 456
It's worth noting that all this code is only portable to Unix-like operating systems and ANSI/VT100-compatible terminals. This code may not work on Windows unless you run the program under Cygwin / MSYS2. I'd also recommend that you add some error handling to this code in case you don't get the response from the terminal that you expect for whatever reason.
(this is really a comment, but it's a bit long)
Using hard coded terminal sequences is a very long way from "portable". While most terminal emulators available currently will support ANSI, vt100 or xterm codes which have a common base there is a very well defined API for accessing interactive terminals known as "curses". A PHP extension is available in pecl. This is just a stub interface to the curses system - present on any Unix/Linux system. While it is possible to set this up on mswindows, using cygwin or pdcurses, it's not an easy fit. You omitted to mention what OS you are working on. (The mswindows console uses ANSI sequences)
There is a toolkit (hoa) based on termcap (predecessor to curses) which might be useful.
To "retrieve" the data you just need to read from stdin (although it would be advisable to uses non-blocking up for this).
I'm having some trouble using the PHP exec() function. Whenever the script I'm attempting to run is short, exec() works just fine. But if the script takes any more than a second or so, it fails. Note, I've attempted run the long script manually on the command line, and it works just fine. It seems as though the PHP interpreter is killing my external script if it takes any longer than a second or so to run. Any thoughts or suggestions? Here is my code:
<?php
$fileName = "foobar.docx";
$argVar = $fileName;
exec("python3 /var/www/html/jan8/alexandrina.py /var/www/html/jan8/$argVar");
echo "$output";
?>
And here is my script:
#!/usr/bin/env python3
import docx
import sys
docxFile = "".join(sys.argv[1:])
# The Three Lines Below Create a New Variable "htmlFile"
# "htmlFile" is the same as "docxFile", except ".docx" is cut off
# and replaced with ".html"
myNumber = len(docxFile) - 5
htmlFile = docxFile[0:myNumber]
htmlFile = htmlFile + '.html'
def generateHTML(filename):
doc = docx.Document(filename)
fullText = []
for para in doc.paragraphs:
fullText.append('<p>')
fullText.append(para.text)
fullText.append('</p>')
fullText.append('\n')
return '\n'.join(fullText)
file = open(htmlFile, "w")
file.write(generateHTML(docxFile))
file.close()
print("end python script")
Additional notes: I've increased the max execution time limits in php.ini, but I don't think that should matter, as the "long script" should only take a few seconds to run. Also, when I refer to the "short script", and the "long script", I'm actually referring to the same script. The difference between the "long script" and the "short script" is just the time to execute as it may vary depending on the size of the file I'm asking the script to process. Anyway... any suggestions would really be appreciated!
Ordinarily, php exec function should block until the command you run has completed. I.e., the PHP script will halt, waiting for the command to finish until continuing with the rest of your script. I was half thinking that your server was experiencing a max_execution_time timeout, but you've clearly stated that even just a couple of seconds is too long and even these fairly short scripts are having trouble.
A couple of solutions occur to me. The simplest one is to alter the python command so that a) any output is routed to a file or output stream and b) the process is run in the background. According to the docs on exec:
If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
I also would like you to make use of the two additional optional parameters for the exec function.
$fileName = "foobar.docx";
$argVar = $fileName;
$cmd = "python3 /var/www/html/jan8/alexandrina.py /var/www/html/jan8/$argVar";
// modify your command to toss output, background the process, and output the process id
$cmd_modified = $cmd . " >/dev/null & echo \$!";
$cmd_output = NULL; // this will be an array of output
$cmd_return_value = NULL; // this will be the return value of the script
exec($cmd_modified, $cmd_output, $cmd_return_value);
echo "exec has completed</br>";
echo "output:<br>" . print_r($cmd_output, TRUE) . "<br>";
echo "return value: " . print_r($cmd_return_value, TRUE);
This may help or may not. If it does not, we still might be able to solve the problem using posix commands.
EDIT: according to crispytx, the long scripts are resulting in a $cmd_return_val of 1 which means an error is happening. Try changing this one line:
$cmd_modified = $cmd . " >/dev/null & echo \$!";
to this
$cmd_modified = $cmd . " & echo \$!";
And let us know what the output of $cmd_output is -- it should at the very least have the process id of the newly spawned process.
Thanks for all the help S. Imp. I had a little trouble debugging using your suggestions because I happened to be using AJAX to call the script. However, I wrote simpler script using your suggestions to try and debug the problem and this is what I found:
Array ( [0] => Traceback (most recent call last): [1] => File "/var/www/html/jan8/alexandrina.py", line 28, in [2] => file.write(generateHTML(docxFile)) [3] => UnicodeEncodeError: 'ascii' codec can't encode character '\u2026' in position 25: ordinal not in range(128) )
So it looks like the problem has to do with ascii encoding! Even though the larger file was just a docx file with the same text as the shorter docx file repeated over and over again for 300 pages. It seems that if a docx file exceeds 1 pages, ascii characters are inserted that aren't present in single page docx files. I have no idea if this post will ever end up helping anyone, but who knows!
[SOLVED]
I have a want to take a File.open('somefile', 'w+') and make it read a file, take one line of text at a time, and visually write it slowly in another file. The reason I ask this question is because I can find nothing that does this already in code, nor can I find anything that actually controls the speed of how fast a program writes on the computer. I know that this can be simulated in a program such as Adobe Aftereffects so long as you provide a cursor after a character and the visual effect doesn't take place too quickly, but I've got 4,000 lines of code that I want to iterate over and can't afford to do this manually. This effect can also be achieved with a Microsoft Macro, but this requires it to be manually entered into the macro with no option of copy and paste.
-solutions preferred in Python, Ruby, and PHP-
If I understood properly, what you are trying to achieve, here you go:
input = File.read('readfrom.txt', 'r')
File.open('writeto.txt', 'w+') do |f|
input.chars.each do |c|
f.print(c) # print 1 char
f.flush # flush the stream
sleep 1 # sleep
end
end
This is one quick and dirty way of doing it in Python.
from time import sleep
mystring= 'My short text with a newline here\nand then ensuing text'
dt = 0.2 #0.2 seconds
for ch in mystring:
with open('fn_out','w+') as f:
f.write(ch)
f.flush()
sleep(dt)
f.flush() will result in updating the file with the changes.
One could make this more elaborate by having a longer pause after each newline, or a variable timestep dt.
To watch the change one has to repeatedly reload the file, as pointed out by #Tom Lord so you could run something like this beforehand to watch it in the terminal:
watch -n 0.1 cat fn_out
After some serious testing, I have finally developed a piece of code that will do the very thing I want. Tom Lord gave me some new words to use in my search terms "simulate typing" and this led me to win32ole with its SendKeys function. Here is a code that will iterate over all the characters in a file and print them out exactly as they were saved while simulating typing. I will see about making this into a gem for future use.
require 'win32ole'
wsh = WIN32OLE.new("WScript.Shell")
wsh.Run("Notepad.exe")
while not wsh.AppActivate("Notepad")
sleep 1
end
def fileToArray(file)
x = []
File.foreach("#{file}") do |line|
x << line.split('')
end
return x.flatten!
end
tests = fileToArray("readfrom.txt")
x = 0
while x <= tests.length
send = tests[x]
wsh.SendKeys("#{send}")
x += 1
sleep 0.1
end
I've written and compiled a command line program (abc.exe) that runs for about 10 mins, makes some calculations and outputs 1 line of results per 10 secs, like this:
C:\>abc.exe
Result 1
Result 2
Result 3
.
.
.
The following php script should theoretically show these results in real time, as they are produced by the program.
<?php
$a = popen('abc.exe', 'r');
while($b = fgets($a)) {
echo $b."<br>\n";
ob_flush();flush();
}
pclose($a);
?>
However, I just get all the results at the end of the execution of the program. Why could this be happening? Please note that if I replace abc.exe with ping 8.8.8.8 or tracert 8.8.8.8, the script works like a charm. Please, help me, I've tried everything that is suggested in similar questions here, but nothing seems to work!
I have a PHP script which executes a shell command:
$handle = popen('python last', 'r');
$read = fread($handle, 4096);
print_r($read);
pclose($handle);
I echo the output of the shell output. When I run this in the command I get something like this:
[root#localhost tester]# python last
[last] ZVZX-W3vo9I: Downloading video webpage
[last] ZVZX-W3vo9I: Extracting video information
[last] ZVZX-W3vo9I: URL: x
[download] Destination: here.flv
[download] 0.0% of 10.09M at ---b/s ETA --:--
[download] 0.0% of 10.09M at 22.24k/s ETA 07:44
[download] 0.0% of 10.09M at 66.52k/s ETA 02:35
[download] 0.1% of 10.09M at 154.49k/s ETA 01:06
[download] 0.1% of 10.09M at 162.45k/s ETA 01:03
However, when I run that same command from PHP I get this output:
[last] ZVZX-W3vo9I: Downloading video webpage
[last] ZVZX-W3vo9I: Extracting video information
[last] ZVZX-W3vo9I: URL: x
[download] Destination: here.flv
As you can see the bottom bit is missing which is the bit I need!! The problem before was that the percentages were being updated on the same line but now I have changed my Python script so that it creates a new line. But this made difference! :(
This question is related to this one.
Thank you for any help.
Update
Needed to redirect output "2>&1". Arul got lucky :P since I missed the deadline to pick the one true answer which belonged to Pax!
You read only the first 4,096 bytes from the pipe, you'll need to place the fread/print_r in a loop and check for the end-of-file using the feof function.
$handle = popen('python last', 'r');
while(!feof($handle))
{
print_r(fread($handle, 4096));
}
pclose($handle);
The first step is to see where the output is going. The first thing I would do is choose a slightly smaller file so that you're not waiting around for seven minutes for each test.
Step 1/ See where things are being written in the shell. Execute the command python last >/tmp/stdout 2>/tmp/stderr then look at those two files. Ideally, everything will be written to stdout but that may not be the case. This gives you the baseline behavior of the script.
Step 2/ Do the same thing when run from PHP by using $handle = popen('python last >/tmp/stdout 2>/tmp/stderr', 'r');. Your PHP script probably won't get anything returned in this case but the files should still be populated. This will catch any changed behavior when running in a non-terminal environment.
If some of the output goes to stderr, then the solution should be as simple as $handle = popen('python last 2>&1', 'r');
Additionally, the doco for PHP states (my bolding):
Returns a file pointer identical to that returned by fopen(), except that it is unidirectional (may only be used for reading or writing) and must be closed with pclose(). This pointer may be used with fgets(), fgetss(), and fwrite().
So I'm not sure you should even be using fread(), although it's shown in one of the examples. Still, I think line-based input maps more to what you're trying to achieve.
Irrespective of all this, you should read the output in a loop to ensure you can get the output when it's more than 4K, something like:
$handle = popen ('python last 2>&1', 'r');
if ($handle) {
while(! feof ($handle)) {
$read = fgets ($handle);
echo $read;
}
pclose ($handle);
}
Another thing to look out for, if you're output is going to a browser and it takes too long, the browser itself may time out since it thinks the server-side connection has disappeared. If you find a small file working and your 10M/1minute file not working, this may be the case. You can try flush() but not all browsers will honor this.
It is much much easier to do this:
$output = `python last`;
var_dump($output);
The 'ticks' (`) will execute the line and capture the output. Here is a test example:
File test.php:
<?php
echo "PHP Output Test 1\n";
echo "PHP Output Test 2\n";
echo "PHP Output Test 3\n";
echo "PHP Output Test 4\n";
echo "PHP Output Test 5\n";
?>
File capture.php:
<?php
$output = `php test.php`;
var_dump($output);
?>
Output from php capture.php:
string(80) "Test PHP Script
Test PHP Script
Test PHP Script
Test PHP Script
Test PHP Script
"
You can then split the output into an array based on line breaks:
$outputArray = explode('\n', $output);
OR use proc_open(), which gives you much more control than popen as you can specify where you want stdin, stdout and stderr to be handled.
OR you can fopen STDIN or php://stdin and then pipe to php:
? python last | php script.php
I would go with option 1 and using the backticks, easiest way.
I would not be surprised to find that the progress report is omitted when the output is not going to a tty. That is, the PHP is capturing everything that is sent, but the progress report is not being sent.
There is ample precedent for commands behaving differently depending on where the output goes - starting with the good old ls command.
Of course, if you wrote the Python script that you're running, this is much less likely to be the cause of the trouble.
How can you verify whether this hypothesis is valid? Well, you could start by running the script at the command line with the standard output going to one file and the standard error going to another. If you see the progress information in one of those files, you know a whole lot more about what is going on. If you see the progress information on the screen still, then the script is probably echoing the progress information to /dev/tty, but when PHP runs it, there is no /dev/tty for the process. If you don't see the progress information at all (on screen or in a file), then my original hypothesis is probably verified.
Try running stream_get_contents() on $handle. It's a better way to work with resources where you don't know the exact size of what you're trying to retrieve.
Could you read in a while loop instead of using fread()?
while( !feof($handle) )
echo fgets($handle);
You may have to flush() also.
What do you see with
python last | less
?
Maybe the output you want is emitted on STDERR. Then you have to start it this way:
$handle = popen('python last 2>&1', 'r');
The 2>&1 directs STDERR into STDOUT, which you are capturing with popen.
If you're just trying to show the progress of the python command inside a terminal window, then I would recommend the following instead:
<?php
system("python last > `tty`");
You won't need to capture the output then, and the user can even Ctrl+C the download without aborting the PHP script.
You are missing a flush call. (In your python app, and possibly your php app aswell)
That is because when you use standard stream stdin/stdout interactively (from cmdline) they are in a line-buffered mode (in short system flushes on each new line), but when you call it from within your program streams are in a fully buffered mode(doesn't output till system buffer is full).
More info on this here buffering in standard streams