Can anyone give me some pointers with regard PHP command execution and best practice?
Im currently trying to parse some netbackup data, but i am running into issues related to the massive amount of data the system call is returning. In order to cut down the amount of data im retreiving I'm doing something like this:
$awk_command = "awk -F, '{print $1\",\"$2\",\"$3\",\"$4\",\"$5\",\"$6\",\"$7\",\"$9\",\"$11\",\"$26\",\"$32\",\"$33\",\"$34\",\"$35\",\"$36\",\"$37\",\"$38\",\"$39\",\"$40}'";
exec("sudo /usr/openv/netbackup/bin/admincmd/bpdbjobs -report -M $master_name -all_columns | $awk_command", $get_backups, $null);
foreach ($get_backups as $backup_detail)
{
process_the_data();
write_data_to_db();
}
Im using awk to limit the amount of data be received. Without it i end up receiving nearly ~150mb of data, and with it, i get a much more manageable ~800k of data.
You don't need to tell me that the awk shit is nasty - i know that already... But in the interests of bettering myself (and my code) can anyone suggest an alternative?
I was thinking of something like proc_open but really not sure if that is going to provide any benefits.
Use exec to write the data to a file instead of reading it whole into your script.
exec("sudo /usr/openv/netbackup/bin/admincmd/bpdbjobs -report -M $master_name -all_columns | $awk_command > /tmp/output.data");
Then use any memory efficient method to read the file in parts.
Have a look here:
Least memory intensive way to read a file in PHP
Related
I've read here and cannot really understand how to speed up my simple exec() which basically looks like this:
zcat access_log.201312011745.gz | grep 'id=6' | grep 'id2=10' | head -n10
I've added ini_set('memory_limit', 256); to the top of the PHP document, but the script still takes about 1 minute to run (contrasted with about near instant completion in Penguinet). What can I do to improve it?
I would try some of the following:
Change your exec to just run somethig simple, like
echo Hello
and see if it still takes so long - if it does, the problem is in the process creation and exec()ing area.
If that runs quickly, try changing the exec to something like:
zcat access_log.201312011745.gz > /dev/null
to see if it is the "zcat" slowing you down
Think about replacing the greps with a "sed" that quits (using "q") as soon as it finds what you are looking for rather than continuing all the way to end of file - since it seems (by your "head") you are only interested in the first few, not all occurrences of your strings. For example, you seem to be looking for lines that contain "id=6" and also "id2=10", so if you used "sed" like below, it may be faster because "sed" will print it and stop immediately the moment it finds a line with "id=6" followed by "id2=10"
zcat access_log.201312011745.gz | sed -n '/id=2.*id2=10/p;q'
The "-n" says "don't print, in general" and then it looks for "id=2" followed by any characters then "id2=10". If it finds that, it prints the line and the "q" makes it quit immediately without looking through to end of file. Note that I am assuming "id=2" comes before "id2=10" on the line. If that is not true, the "sed" will need additional work.
I have a file that I'm using to log IP addresses for a client. They want to keep the last 500 lines of the file. It is on a Linux system with PHP4 (oh no!).
I was going to add to the file one line at a time with new IP addresses. We don't have access to cron so I would probably need to make this function do the line-limit cleanup as well.
I was thinking either using like exec('tail [some params]') or maybe reading the file in with PHP, exploding it on newlines into an array, getting the last 1000 elements, and writing it back. Seems kind of memory intensive though.
What's a better way to do this?
Update:
Per #meagar's comment below, if I wanted to use the zip functionality, how would I do that within my PHP script? (no access to cron)
if(rand(0,10) == 10){
shell_exec("find . logfile.txt [where size > 1mb] -exec zip {} \;")
}
Will zip enumerate the files automatically if there is an existing file or do I need to do that manually?
The fastest way is probably, as you suggested, to use tail:
passthru("tail -n 500 $filename");
(passthru does the same as exec only it outputs the entire program output to stdout. You can capture the output using an output buffer)
[edit]
I agree with a previous comment that a log rotate would be infinitely better... but you did state that you don't have access to cron so I'm assuming you can't do logrotate either.
logrotate
This would be the "proper" answer, and it's not difficult to set this up either.
You may get the number of lines using count(explode("\n", file_get_contents("log.txt"))) and if it is equal to 1000, get the substring starting from the first \n to the end, add the new IP address and write the whole file again.
It's almost the same as writing the new IP by opening the file in a+ mode.
I have a great Python program on my webserver, which I want to use from inside my PHP web app.
Here's an example of the python command, and output as you would see it in terminal:
>>> print MBSP.parse('I ate pizza with a fork.')
I/PRP/I-NP/O/NP-SBJ-1/O/i
ate/VBD/I-VP/O/VP-1/A1/eat
pizza/NN/I-NP/O/NP-OBJ-1/O/pizza
with/IN/I-PP/B-PNP/O/P1/with
a/DT/I-NP/I-PNP/O/P1/a
fork/NN/I-NP/I-PNP/O/P1/fork ././O/O/O/O/.
You might recognize this as a typical POS tagger.
In any case, I'm confused about how to use a PHP-based web app to send this program a string like "I ate pizza with a fork", and somehow get the response back in a way that can be further parsed in PHP.
The idea is to use PHP to pass this text to the Python program, and then grab the response to be parsed by PHP by selecting certain types of words.
It seems like in PHP the usual suspects are popen() and proc_open(), but popen() is only for sending, or receiving information - not both? Is popen() able to give me access to this output (above) that I'm getting from the Python program? Or is there a better method? What about curl?
Here are all my options in terms of functions in PHP:
http://us.php.net/manual/en/function.proc-open.php
I'm lost on this, so thanks for your wise words of wisdom!
I use exec() for this purpose.
exec($command, $output);
print_r($output);
If you want to get a little heavier / fancier... give your python script an http (or xmlrpc) front end, and call that with a GET/POST. Might not be worth all that machinery though!
You could use popen(), and pass the input to your Python script as a command line argument, then read the output from the file descriptor popen gives you, or proc_open() if you want to interact bi-directionally with the Python script.
Example 1 in the proc_open manual: http://us.php.net/manual/en/function.proc-open.php gives an example of this.
If your Python needs it as stdin, you could try popening a command line:
echo "I ate pizza!"|my_python_progam.py
and just read the output. As usual, do proper input validation before sending it to the command-line.
Something like this would work
$command = '/usr/bin/python2.7 /home/a4337/Desktop/script.py'
$pid = popen('$command',r)
........
........
.........
pclose($pid)
I'm trying to test a php file from a C program(...)
Basically I have a filename that I want to check against php -l and store the output for further processing.
A simple solution in that case would be to redirect the output to a file. And then read the file into an array. You then can have your further processing with the array.
Something like this(in C):
system("php -l yourfile.php > myfile");
FILE *f = fopen("myfile", "rb");
fseek(f, 0, SEEK_END);
long pos = ftell(f);
fseek(f, 0, SEEK_SET);
char *array = malloc(pos);
fread(array, pos, 1, f);
fclose(f);
//your processing part here..
free(array); // free allocated memory
Solution #2: Invoke the PHP interpreter, and pipe the output to your program.
Something like the following in the console:
php -l yourfile.php | pathToYourCProgram
In the above case, you will read the output of PHP from stdin. You can read the entire input, and directly store it to an array.
you can use "popen" function. do man popen to understand the usage of popen. 1st argument of popen is the binary which you want to execute (i.e. "php -l" in your case), and 2nd argument is the mode (read/write). in your case file mode will be read. see the following code to understand how popen works, its fairly easy.
http://www.google.com/notebook/public/17135812868734162318/BDSUiDQoQ-ojrzeck
hope that helps.
If executing the php processor from your C program is not mandatory, you might want to consider the following completely different approach:
Make a small program that parses stdin for error messages and do some post processing. Let's call this program check_errors.
On the command line:
php -l thefile.php | check_errors
This catches the output of php and directs it to check_errors.
It's more Unix-like to build little tools that do one thing, and one thing only, but doing it very well. Using pipes and redirects one may sequence those programs, doing amazing and complex operations.
I have a program on my linux server that asks the same series of questions each time it executes and then provides several lines of output. My goal is to automate the input and output with a php script.
The program is not designed to accept input on the command line. Instead, the program asks question 1 and waits for an answer from the keyboard, then the program asks question 2 and waits for an answer from the keyboard, etc.
I know how to capture the output in an array by writing:
$out = array();
exec("my/path/program",$out);
But how do I handle the input?
Assume the program asks 3 questions and valid answers are: left 120 n
What is the easiest way using php to pass that input to the program?
Can I do it somehow on the exec line?
I’m not a php noob but simply have never needed to do this before.
Alas, my googling is going in circles.
First up, just to let you know that you're trying to reinvent the wheel. What you're really looking for is expect(1), which is a command-line utility intended to do exactly what you want without involving PHP.
However, if you really want to write your own PHP code you need to use proc_open. Here are some good code examples on reading from STDOUT and writing to STDIN of the child process using proc_open:
http://www.php.net/manual/en/function.proc-open.php#79665
How to pass variables as stdin into command line from PHP
http://camposer-techie.blogspot.com/2010/08/ejecutando-comandos-sobre-un-programa.html (this one is in Spanish, sorry, but the code is good)
Finally, there is also an Expect PECL module for PHP.
Hope this helps.
Just add the arguments to the exec line.
exec("/path/to/programname $arg1 $arg2 $arg3");
... but don't forget to apply escapeshellarg() on every argument! Otherwise, you're vulnerable to injected malicious code.
$out = array();
//add elements/parameters/input to array
string $execpath = "my/path/program ";
foreach($out as $parameter) {
$execpath += $parameter;
//$execpath += "-"+$execpath; use this if you need to add a '-' in front of your parameters.
}
exec($execpath);