I'm trying to test a php file from a C program(...)
Basically I have a filename that I want to check against php -l and store the output for further processing.
A simple solution in that case would be to redirect the output to a file. And then read the file into an array. You then can have your further processing with the array.
Something like this(in C):
system("php -l yourfile.php > myfile");
FILE *f = fopen("myfile", "rb");
fseek(f, 0, SEEK_END);
long pos = ftell(f);
fseek(f, 0, SEEK_SET);
char *array = malloc(pos);
fread(array, pos, 1, f);
fclose(f);
//your processing part here..
free(array); // free allocated memory
Solution #2: Invoke the PHP interpreter, and pipe the output to your program.
Something like the following in the console:
php -l yourfile.php | pathToYourCProgram
In the above case, you will read the output of PHP from stdin. You can read the entire input, and directly store it to an array.
you can use "popen" function. do man popen to understand the usage of popen. 1st argument of popen is the binary which you want to execute (i.e. "php -l" in your case), and 2nd argument is the mode (read/write). in your case file mode will be read. see the following code to understand how popen works, its fairly easy.
http://www.google.com/notebook/public/17135812868734162318/BDSUiDQoQ-ojrzeck
hope that helps.
If executing the php processor from your C program is not mandatory, you might want to consider the following completely different approach:
Make a small program that parses stdin for error messages and do some post processing. Let's call this program check_errors.
On the command line:
php -l thefile.php | check_errors
This catches the output of php and directs it to check_errors.
It's more Unix-like to build little tools that do one thing, and one thing only, but doing it very well. Using pipes and redirects one may sequence those programs, doing amazing and complex operations.
Related
I'm having some trouble using the PHP exec() function. Whenever the script I'm attempting to run is short, exec() works just fine. But if the script takes any more than a second or so, it fails. Note, I've attempted run the long script manually on the command line, and it works just fine. It seems as though the PHP interpreter is killing my external script if it takes any longer than a second or so to run. Any thoughts or suggestions? Here is my code:
<?php
$fileName = "foobar.docx";
$argVar = $fileName;
exec("python3 /var/www/html/jan8/alexandrina.py /var/www/html/jan8/$argVar");
echo "$output";
?>
And here is my script:
#!/usr/bin/env python3
import docx
import sys
docxFile = "".join(sys.argv[1:])
# The Three Lines Below Create a New Variable "htmlFile"
# "htmlFile" is the same as "docxFile", except ".docx" is cut off
# and replaced with ".html"
myNumber = len(docxFile) - 5
htmlFile = docxFile[0:myNumber]
htmlFile = htmlFile + '.html'
def generateHTML(filename):
doc = docx.Document(filename)
fullText = []
for para in doc.paragraphs:
fullText.append('<p>')
fullText.append(para.text)
fullText.append('</p>')
fullText.append('\n')
return '\n'.join(fullText)
file = open(htmlFile, "w")
file.write(generateHTML(docxFile))
file.close()
print("end python script")
Additional notes: I've increased the max execution time limits in php.ini, but I don't think that should matter, as the "long script" should only take a few seconds to run. Also, when I refer to the "short script", and the "long script", I'm actually referring to the same script. The difference between the "long script" and the "short script" is just the time to execute as it may vary depending on the size of the file I'm asking the script to process. Anyway... any suggestions would really be appreciated!
Ordinarily, php exec function should block until the command you run has completed. I.e., the PHP script will halt, waiting for the command to finish until continuing with the rest of your script. I was half thinking that your server was experiencing a max_execution_time timeout, but you've clearly stated that even just a couple of seconds is too long and even these fairly short scripts are having trouble.
A couple of solutions occur to me. The simplest one is to alter the python command so that a) any output is routed to a file or output stream and b) the process is run in the background. According to the docs on exec:
If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
I also would like you to make use of the two additional optional parameters for the exec function.
$fileName = "foobar.docx";
$argVar = $fileName;
$cmd = "python3 /var/www/html/jan8/alexandrina.py /var/www/html/jan8/$argVar";
// modify your command to toss output, background the process, and output the process id
$cmd_modified = $cmd . " >/dev/null & echo \$!";
$cmd_output = NULL; // this will be an array of output
$cmd_return_value = NULL; // this will be the return value of the script
exec($cmd_modified, $cmd_output, $cmd_return_value);
echo "exec has completed</br>";
echo "output:<br>" . print_r($cmd_output, TRUE) . "<br>";
echo "return value: " . print_r($cmd_return_value, TRUE);
This may help or may not. If it does not, we still might be able to solve the problem using posix commands.
EDIT: according to crispytx, the long scripts are resulting in a $cmd_return_val of 1 which means an error is happening. Try changing this one line:
$cmd_modified = $cmd . " >/dev/null & echo \$!";
to this
$cmd_modified = $cmd . " & echo \$!";
And let us know what the output of $cmd_output is -- it should at the very least have the process id of the newly spawned process.
Thanks for all the help S. Imp. I had a little trouble debugging using your suggestions because I happened to be using AJAX to call the script. However, I wrote simpler script using your suggestions to try and debug the problem and this is what I found:
Array ( [0] => Traceback (most recent call last): [1] => File "/var/www/html/jan8/alexandrina.py", line 28, in [2] => file.write(generateHTML(docxFile)) [3] => UnicodeEncodeError: 'ascii' codec can't encode character '\u2026' in position 25: ordinal not in range(128) )
So it looks like the problem has to do with ascii encoding! Even though the larger file was just a docx file with the same text as the shorter docx file repeated over and over again for 300 pages. It seems that if a docx file exceeds 1 pages, ascii characters are inserted that aren't present in single page docx files. I have no idea if this post will ever end up helping anyone, but who knows!
[SOLVED]
This is my code for executing a command from PHP:
$execQuery = sprintf("/usr/local/bin/binary -mode M \"%s\" %u %s -pathJson \"/home/ec2/fashion/jsonS/\" -pathJson2 \"/home/ec2/fashion/jsonS2/\"", $path, $pieces, $type);
exec($execQuery, $output, $return);
the $return value is always 0 but $output is empty. The $output should be a JSON.
If I execute the same but removing one letter to binary (for example /usr/local/bin/binar ) I get (correctly) a $return = 127.
If I write other parameters (like -mode R which doesn't exit) I got errors from the console (which are correct as well).
If I run the exact $execQuery (which I printf before to be sure about quotation marks) on the console, it executes correctly. It's only the PHP side where I've got the error.
What can be wrong?
Thank you in advance.
Well, a couple of things might be happening...
This binary you're running write to something else that STDOUT (for instance, STDERR)
The env vars available to the PHP user differ from the env vars available to the user running console (and those vars are required)
PHP User does not have permission to access some files involved.
In order to debug, it might be better to use proc_open instead of exec, and check the STDOUT and STDERR. This might give you additional information regarding what's happening.
Suggestion (and shameless advertising)
I wrote a small utility library for PHP that executes external programs in a safer way and provides aditional debug information. It might help you to, at least pinpoint the issue.
I have a great Python program on my webserver, which I want to use from inside my PHP web app.
Here's an example of the python command, and output as you would see it in terminal:
>>> print MBSP.parse('I ate pizza with a fork.')
I/PRP/I-NP/O/NP-SBJ-1/O/i
ate/VBD/I-VP/O/VP-1/A1/eat
pizza/NN/I-NP/O/NP-OBJ-1/O/pizza
with/IN/I-PP/B-PNP/O/P1/with
a/DT/I-NP/I-PNP/O/P1/a
fork/NN/I-NP/I-PNP/O/P1/fork ././O/O/O/O/.
You might recognize this as a typical POS tagger.
In any case, I'm confused about how to use a PHP-based web app to send this program a string like "I ate pizza with a fork", and somehow get the response back in a way that can be further parsed in PHP.
The idea is to use PHP to pass this text to the Python program, and then grab the response to be parsed by PHP by selecting certain types of words.
It seems like in PHP the usual suspects are popen() and proc_open(), but popen() is only for sending, or receiving information - not both? Is popen() able to give me access to this output (above) that I'm getting from the Python program? Or is there a better method? What about curl?
Here are all my options in terms of functions in PHP:
http://us.php.net/manual/en/function.proc-open.php
I'm lost on this, so thanks for your wise words of wisdom!
I use exec() for this purpose.
exec($command, $output);
print_r($output);
If you want to get a little heavier / fancier... give your python script an http (or xmlrpc) front end, and call that with a GET/POST. Might not be worth all that machinery though!
You could use popen(), and pass the input to your Python script as a command line argument, then read the output from the file descriptor popen gives you, or proc_open() if you want to interact bi-directionally with the Python script.
Example 1 in the proc_open manual: http://us.php.net/manual/en/function.proc-open.php gives an example of this.
If your Python needs it as stdin, you could try popening a command line:
echo "I ate pizza!"|my_python_progam.py
and just read the output. As usual, do proper input validation before sending it to the command-line.
Something like this would work
$command = '/usr/bin/python2.7 /home/a4337/Desktop/script.py'
$pid = popen('$command',r)
........
........
.........
pclose($pid)
I need to turn HTML into equivalent Markdown-structured text.
OBS.: Quick and clear way of doing this with PHP & Python.
As I am programming in PHP, some people indicates Markdownify to do the job, but unfortunately, the code is not being updated and in fact it is not working. At sourceforge.net/projects/markdownify there is a "NOTE: unsupported - do you want to maintain this project? contact me! Markdownify is a HTML to Markdown converter written in PHP. See it as the successor to html2text.php since it has better design, better performance and less corner cases."
From what I could discover, I have only two good choices:
Python: Aaron Swartz's html2text.py
Ruby: Singpolyma's html2markdown.rb, based on Nokogiri
So, from PHP, I need to pass the HTML code, call the Ruby/Python Script and receive the output back.
(By the way, a folk made a similar question here ("how to call ruby script from php?") but with no practical information to my case).
Following the Tin Man`s tip (bellow), I got to this:
PHP code:
$t='<p><b>Hello</b><i>world!</i></p>';
$scaped=preg_quote($t,"/");
$program='python html2md.py';
//exec($program.' '.$scaped,$n); print_r($n); exit; //Works!!!
$input=$t;
$descriptorspec=array(
array('pipe','r'),//stdin is a pipe that the child will read from
array('pipe','w'),//stdout is a pipe that the child will write to
array('file','./error-output.txt','a')//stderr is a file to write to
);
$process=proc_open($program,$descriptorspec,$pipes);
if(is_resource($process)){
fwrite($pipes[0],$input);
fclose($pipes[0]);
$r=stream_get_contents($pipes[1]);
fclose($pipes[1]);
$return_value=proc_close($process);
echo "command returned $return_value\n";
print_r($pipes);
print_r($r);
}
Python code:
#! /usr/bin/env python
import html2text
import sys
print html2text.html2text(sys.argv[1])
#print "Hi!" #works!!!
With the above I am geting this:
command returned 1
Array
(
[0] => Resource id #17
1 => Resource id #18
)
And the "error-output.txt" file says:
Traceback (most recent call last):
File "html2md.py", line 5, in
print html2text.html2text(sys.argv1)
IndexError: list index out of range
Any ideas???
Ruby code (still beeing analysed)
#!/usr/bin/env ruby
require_relative 'html2markdown'
puts HTML2Markdown.new("<h1>#{ ARGF.read }</h1>").to_s
Just for the records, I tryed before to use PHP's most simple "exec()" but I got some problemas with some special characters very common to HTML language.
PHP code:
echo exec('./hi.rb');
echo exec('./hi.py');
Ruby code:
#!/usr/bin/ruby
puts "Hello World!"
Python code:
#!usr/bin/python
import sys
print sys.argv[1]
Both working fine. But when the string is a bit more complicated:
$h='<p><b>Hello</b><i>world!</i></p>';
echo exec("python hi.py $h");
It did not work at all.
That's because the html string needed to have its special characters scaped. I got it using this:
$t='<p><b>Hello</b><i>world!</i></p>';
$scaped=preg_quote($t,"/");
Now it works like I said here.
I am runnig:
Fedora 14
ruby 1.8.7
Python 2.7
perl 5.12.2
PHP 5.3.4
nginx 0.8.53
Have PHP open the Ruby or Python script via proc_open, piping the HTML into STDIN in the script. The Ruby/Python script reads and processes the data and returns it via STDOUT back to the PHP script, then exits. This is a common way of doing things via popen-like functionality in Perl, Ruby or Python and is nice because it gives you access to STDERR in case something blows chunks and doesn't require temp files, but it's a bit more complex.
Alternate ways of doing it could be writing the data from PHP to a temporary file, then using system, exec, or something similar to call the Ruby/Python script to open and process it, and print the output using their STDOUT.
EDIT:
See #Jonke's answer for "Best practices with STDIN in Ruby?" for examples of how simple it is to read STDIN and write to STDOUT with Ruby. "How do you read from stdin in python" has some good samples for that language.
This is a simple example showing how to call a Ruby script, passing a string to it via PHP's STDIN pipe, and reading the Ruby script's STDOUT:
Save this as "test.php":
<?php
$descriptorspec = array(
0 => array("pipe", "r"), // stdin is a pipe that the child will read from
1 => array("pipe", "w"), // stdout is a pipe that the child will write to
2 => array("file", "./error-output.txt", "a") // stderr is a file to write to
);
$process = proc_open('ruby ./test.rb', $descriptorspec, $pipes);
if (is_resource($process)) {
// $pipes now looks like this:
// 0 => writeable handle connected to child stdin
// 1 => readable handle connected to child stdout
// Any error output will be appended to /tmp/error-output.txt
fwrite($pipes[0], 'hello world');
fclose($pipes[0]);
echo stream_get_contents($pipes[1]);
fclose($pipes[1]);
// It is important that you close any pipes before calling
// proc_close in order to avoid a deadlock
$return_value = proc_close($process);
echo "command returned $return_value\n";
}
?>
Save this as "test.rb":
#!/usr/bin/env ruby
puts "<b>#{ ARGF.read }</b>"
Running the PHP script gives:
Greg:Desktop greg$ php test.php
<b>hello world</b>
command returned 0
The PHP script is opening the Ruby interpreter which opens the Ruby script. PHP then sends "hello world" to it. Ruby wraps the received text in bold tags, and outputs it, which is captured by PHP, and then output. There are no temp files, nothing passed on the command-line, you could pass a LOT of data if need-be, and it would be pretty fast. Python or Perl could easily be used instead of Ruby.
EDIT:
If you have:
HTML2Markdown.new('<h1>HTMLcode</h1>').to_s
as sample code, then you could begin developing a Ruby solution with:
#!/usr/bin/env ruby
require_relative 'html2markdown'
puts HTML2Markdown.new("<h1>#{ ARGF.read }</h1>").to_s
assuming you've already downloaded the HTML2Markdown code and have it in the current directory and are running Ruby 1.9.2.
In Python, have PHP pass the var as a command line argument, get it from sys.argv (the list of command line arguments passed to Python), and then have Python print the output, which PHP then echoes. Example:
#!usr/bin/python
import sys
print "Hello ", sys.argv[1] # 2nd element, since the first is the script name
PHP:
<?php
echo exec('python script.py Rafe');
?>
The procedure should be basically the same in Ruby.
Use a variable in the Ruby code, and pass it in as an argument to the Ruby script from the PHP code. Then, have the Ruby script return the processed code into stdout which PHP can read.
I think your question is wrong. Your problem is how to convert from HTML to Markdown. Am I right?
Try this http://milianw.de/projects/markdownify/ I think it could help you =)
Another very weird approach will be like the one i used.
Php file -> output.txt
ruby file -> read from output.txt
Ruby file-> result.txt
Php file -> read from result.txt
simple add exec(rubyfile.rb);
Not recommended but this will work for sure.
I have a PHP script which executes a shell command:
$handle = popen('python last', 'r');
$read = fread($handle, 4096);
print_r($read);
pclose($handle);
I echo the output of the shell output. When I run this in the command I get something like this:
[root#localhost tester]# python last
[last] ZVZX-W3vo9I: Downloading video webpage
[last] ZVZX-W3vo9I: Extracting video information
[last] ZVZX-W3vo9I: URL: x
[download] Destination: here.flv
[download] 0.0% of 10.09M at ---b/s ETA --:--
[download] 0.0% of 10.09M at 22.24k/s ETA 07:44
[download] 0.0% of 10.09M at 66.52k/s ETA 02:35
[download] 0.1% of 10.09M at 154.49k/s ETA 01:06
[download] 0.1% of 10.09M at 162.45k/s ETA 01:03
However, when I run that same command from PHP I get this output:
[last] ZVZX-W3vo9I: Downloading video webpage
[last] ZVZX-W3vo9I: Extracting video information
[last] ZVZX-W3vo9I: URL: x
[download] Destination: here.flv
As you can see the bottom bit is missing which is the bit I need!! The problem before was that the percentages were being updated on the same line but now I have changed my Python script so that it creates a new line. But this made difference! :(
This question is related to this one.
Thank you for any help.
Update
Needed to redirect output "2>&1". Arul got lucky :P since I missed the deadline to pick the one true answer which belonged to Pax!
You read only the first 4,096 bytes from the pipe, you'll need to place the fread/print_r in a loop and check for the end-of-file using the feof function.
$handle = popen('python last', 'r');
while(!feof($handle))
{
print_r(fread($handle, 4096));
}
pclose($handle);
The first step is to see where the output is going. The first thing I would do is choose a slightly smaller file so that you're not waiting around for seven minutes for each test.
Step 1/ See where things are being written in the shell. Execute the command python last >/tmp/stdout 2>/tmp/stderr then look at those two files. Ideally, everything will be written to stdout but that may not be the case. This gives you the baseline behavior of the script.
Step 2/ Do the same thing when run from PHP by using $handle = popen('python last >/tmp/stdout 2>/tmp/stderr', 'r');. Your PHP script probably won't get anything returned in this case but the files should still be populated. This will catch any changed behavior when running in a non-terminal environment.
If some of the output goes to stderr, then the solution should be as simple as $handle = popen('python last 2>&1', 'r');
Additionally, the doco for PHP states (my bolding):
Returns a file pointer identical to that returned by fopen(), except that it is unidirectional (may only be used for reading or writing) and must be closed with pclose(). This pointer may be used with fgets(), fgetss(), and fwrite().
So I'm not sure you should even be using fread(), although it's shown in one of the examples. Still, I think line-based input maps more to what you're trying to achieve.
Irrespective of all this, you should read the output in a loop to ensure you can get the output when it's more than 4K, something like:
$handle = popen ('python last 2>&1', 'r');
if ($handle) {
while(! feof ($handle)) {
$read = fgets ($handle);
echo $read;
}
pclose ($handle);
}
Another thing to look out for, if you're output is going to a browser and it takes too long, the browser itself may time out since it thinks the server-side connection has disappeared. If you find a small file working and your 10M/1minute file not working, this may be the case. You can try flush() but not all browsers will honor this.
It is much much easier to do this:
$output = `python last`;
var_dump($output);
The 'ticks' (`) will execute the line and capture the output. Here is a test example:
File test.php:
<?php
echo "PHP Output Test 1\n";
echo "PHP Output Test 2\n";
echo "PHP Output Test 3\n";
echo "PHP Output Test 4\n";
echo "PHP Output Test 5\n";
?>
File capture.php:
<?php
$output = `php test.php`;
var_dump($output);
?>
Output from php capture.php:
string(80) "Test PHP Script
Test PHP Script
Test PHP Script
Test PHP Script
Test PHP Script
"
You can then split the output into an array based on line breaks:
$outputArray = explode('\n', $output);
OR use proc_open(), which gives you much more control than popen as you can specify where you want stdin, stdout and stderr to be handled.
OR you can fopen STDIN or php://stdin and then pipe to php:
? python last | php script.php
I would go with option 1 and using the backticks, easiest way.
I would not be surprised to find that the progress report is omitted when the output is not going to a tty. That is, the PHP is capturing everything that is sent, but the progress report is not being sent.
There is ample precedent for commands behaving differently depending on where the output goes - starting with the good old ls command.
Of course, if you wrote the Python script that you're running, this is much less likely to be the cause of the trouble.
How can you verify whether this hypothesis is valid? Well, you could start by running the script at the command line with the standard output going to one file and the standard error going to another. If you see the progress information in one of those files, you know a whole lot more about what is going on. If you see the progress information on the screen still, then the script is probably echoing the progress information to /dev/tty, but when PHP runs it, there is no /dev/tty for the process. If you don't see the progress information at all (on screen or in a file), then my original hypothesis is probably verified.
Try running stream_get_contents() on $handle. It's a better way to work with resources where you don't know the exact size of what you're trying to retrieve.
Could you read in a while loop instead of using fread()?
while( !feof($handle) )
echo fgets($handle);
You may have to flush() also.
What do you see with
python last | less
?
Maybe the output you want is emitted on STDERR. Then you have to start it this way:
$handle = popen('python last 2>&1', 'r');
The 2>&1 directs STDERR into STDOUT, which you are capturing with popen.
If you're just trying to show the progress of the python command inside a terminal window, then I would recommend the following instead:
<?php
system("python last > `tty`");
You won't need to capture the output then, and the user can even Ctrl+C the download without aborting the PHP script.
You are missing a flush call. (In your python app, and possibly your php app aswell)
That is because when you use standard stream stdin/stdout interactively (from cmdline) they are in a line-buffered mode (in short system flushes on each new line), but when you call it from within your program streams are in a fully buffered mode(doesn't output till system buffer is full).
More info on this here buffering in standard streams