I am extracting information from proftpd logs. I have to call this one-liner from a PHP script but it does not work anymore from there.
This is the original line, which works:
(gunzip -c xferlog*.gz; cat xferlog?(*)!(.gz)) | grep 'host [0-9]\+ file a _ o r ftpuser' | sort -k 5n,5 -k 2M,2 -k 3n,3 -k 4,4 | tail -1 | cut -c 1-24
This is the error I got when executed in PHP:
$cmd = "(gunzip -c $logFile*.gz; cat $logFile?(*)!(.gz)) | grep '$host [0-9]\+ $file a _ o r $ftpUser' | sort -k 5n,5 -k 2M,2 -k 3n,3 -k 4,4 | tail -1 | cut -c 1-24";
exec($cmd);
sh: Syntax error: "(" unexpected (expecting ")")
I tried several bash scripts that would be called by PHP, but it has not been successful. I had errors like:
bash: command substitution: line 9: syntax error near unexpected token `('
bash: command substitution: line 9: `cat ${LOGS}?(*)!(.gz)'
or
bash: ./extract_date_in_xferlog.sh: line 8: syntax error near unexpected token `('
bash: ./extract_date_in_xferlog.sh: line 8: `(gunzip -c ${LOGS}*.gz; cat ${LOGS}?(*)!(.gz)) | grep "$HOST [0-9]\+ $FILE a _ o r $USER" | sort -k 5n,5 -k 2M,2 -k 3n,3 -k 4,4 | tail -1 | cut -c 1-24'
I am a bit confused, thank you for your help!
The weird wildcard uses extended globbing. You need to enable extglob either as part of your script (probably better) or in your Bash setup (probably where it was before, and then it broke when somebody changed it for unrelated reasons).
You're probably not escaping the quotes correctly in the script.
I suggest handling the shell command as a single quoted string assuming you don't want to embed PHP variables in the shell command, and then making sure that all the single quotes in the command are escaped with \' to avoid prematurely terminating the PHP string.
Alternatively you could use a HEREDOC or NOWDOC style string to avoid escaping issues.
Related
I wrote a very convoluted, very hackjob PHP-cli script that receives and parses JSON of changeable structure, depth and content. For that reason at that time I found it easiest to do the parsing using PHP's shell_exec() and cat | jq | grep.
Sometimes, rarely, on certain input it gives me the message Error: writing output failed: Broken pipe, which is the last message I see in cli output before the script dies. However, even when it does do that, the data is still parsed out correctly, for all the little good it does me.
I isolated the problematic piece of code to:
$jq1='cat '.$randfile.' | jq \'.\' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}"';
$jq1=trim(shell_exec($jq1));
And tried to debug it by seeing what it executes. The first line is the shell_exec argument, echoed before execution, the second line is the result of shell_exec.
Command: cat 5ca15f21.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: standalone
Command: cat 5ca59379.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: season
Error: writing output failed: Broken pipe
Command: cat 5ca7d271.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: extended
Command: cat 5ca7d7a8.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: season
(I have seen the error of my lazy ways and will be rewriting that whole section, but back then I was young and inexperienced and impatient. I'd still like to understand what's going wrong where why.)
What would make it do that sometimes? The input is always jq's pretty-printed JSON, of varying structure.
Even if it does get the broken pipe message, the necessary value is still parsed out and stored in the variable. What causes it to die then? I would like to know for the future if there's a way to make PHP disregard the [non-fatal] error and go on executing.
Why does the shell command that produces the broken pipe error message in shell_exec behave no differently when invoked manually in bash? Where is the broken pipe and what makes it so broken?
Edit:
The cat command could be eliminated if this is a bash command to be executed. I would write it like this.
$jq1=grep -Po --color=none "\w{3,15}"|"\[" -m 1 file.txt | jq \'.\';
$jq1=trim(shell_exec($jq1));
I combined your redundant grep commands and added file to the grep command and only piped the jq command.
I have a PHP script that executes a shell command to find the common items between two files given. This is the beginning of my PHP script:
$E7Bonded_File = "/opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Bonded_cust_stats.csv";
$E7Single_File = "/opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Single_cust_stats.csv";
$E7Common_File = "/opt/IBM/custom/NAC_Dslam/junk/Common_tn_SingleBonded_E7_cust_stats.csv";
//only do this once, with old single/bonded filenames. This will be a list to add to the existing Common file.
exec ("comm -12 <(cut -d ',' -f2 $E7Single_File| sort) <(cut -d ',' -f2 $E7Bonded_File| sort)", $outputCommon);
I see this error message when I run the script:
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `comm -12 <(cut -d ',' -f2 /opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Single_cust_stats.csv| sort) <(cut -d ',' -f2 /opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Bonded_cust_stats.csv| sort)'
I checked, and the parentheses look ok for my exec() line.
When I run the shell command at the command line it returns a listing of numbers like I expect:
comm -12 <(cut -d ',' -f2 junk/PortParameter_E7_Single_cust_stats.csv| sort) <(cut -d ',' -f2 junk/PortParameter_E7_Bonded_cust_stats.csv| sort)
I looked online and I seem to be using exec() correctly. I want the numbers returned to be stored as an array, $outputCommon.
Any ideas about this error message?
*********Update on answer***************
My solution wound up being a combination of both mario and miken32/my co-worker
Adding #!/bin/bash at the top of my php script, and
Adding /bin/bash -c as follows:
exec("/bin/bash -c /opt/IBM/custom/NAC_Dslam/Common_list.sh", $outputShell);
After I moved the comm part to a shell script:
Common_list.sh:
#!/bin/bash
comm -12 <(cut -d ',' -f2 /opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Single_cust_stats.csv| sort) <(cut -d ',' -f2 /opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Bonded_cust_stats.csv| sort)
This error typically comes up for non-bash shells, which don't support <() expression pipes.
On Ubuntu/Debian servers the default /bin/sh is typically dash.
Check for symlinked binaries:
me#snip:~$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Jul 16 2017 /bin/sh -> dash
Or as #theotherguy mentioned, bash runs as restricted_shell when started as sh.
See $_ENV[SHELL] on what Apache/PHP use as default. Change environment vars.
Either adapt that, or wrap the shell_exec cmdline with /bin/bash -c '…'.
Probably the easiest solution would be to make this into an executable script on the server:
#!/bin/bash
if [[ ! -r "$1" ]] || [[ ! -r "$2" ]]; then
printf "File not found\n" >&2
exit 1
fi
comm -12 <(cut -d ',' -f2 "$1"| sort) <(cut -d ',' -f2 "$2"| sort)
And then call that from PHP:
$E7Bonded_File = escapeshellarg("/opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Bonded_cust_stats.csv");
$E7Single_File = escapeshellarg("/opt/IBM/custom/NAC_Dslam/junk/PortParameter_E7_Single_cust_stats.csv");
//only do this once, with old single/bonded filenames. This will be a list to add to the existing Common file.
exec ("/usr/local/bin/your_script.sh $E7Single_File $E7Bonded_File", $outputCommon);
Always escape your shell arguments with escapeshellarg() even if you think they're safe.
when checking the script with a linter, it complains that:
comm -12 <(cut -d ',' -f2 junk/PortParameter_E7_Single_cust_stats.csv| sort) <(cut -d ',' -f2 junk/PortParameter_E7_Bonded_cust_stats.csv| sort)
^-- SC2039: In POSIX sh, process substitution is undefined.
this happens when I define shebang #!/bin/sh, while #!/bin/bash does not complain... therefore the answer might be, to run the script with /usr/bin/bash. escapeshellarg() is indeed useful.
...
try shell_exec(), simply because exec() should not invoke any shell. the one returns the output as string and the other can return an array.
alternatively, you can explicitly invoke bash with exec():
exec('/bin/bash -c " command "', $stdOut);
So in case anyone else still needs to get substitution working in command execution in PHP, there is super dumb simple thing to do:
$a = 'Hello substitution';
var_dump(shell_exec("bash -c 'cat <(echo $a)'"));
Results in:
string(19) "Hello substitution
"
Which is expected/desired.
This entry in my Makefile happily crawls my PHP files and runs PHP's built-in lint on them:
lint:
#find . -name "*.php" | grep -v "^./lib" | grep -v "^./vendor" | xargs -I{} php -l '{}' | grep -v "No syntax errors detected"
The grep -v suppresses all the "No syntax errors detected" messages that would otherwise be produced while still failure messages, if any.
The problem is that make dies when there are no syntax errors and continues when there are errors. This is because of the exit code from grep -v. It thinks it has succeeded when it finds something (an error message) and failed when it finds nothing (all files passed lint).
I looked at negating the exit code of the final call to grep with !:
lint:
#find . -name "*.php" | grep -v "^./lib" | grep -v "^./vendor" | xargs -I{} php -l '{}' | ! grep -v "No syntax errors detected"
but that gives me:
/bin/sh: -c: line 0: syntax error near unexpected token `!'
I can use ! at the commandline fine but in this context it doesn't work for some reason.
I'm curious how I negate an exit code within the context of a pipeline/xargs/grep/make. But mostly I want to solve my problem - open to any suggestions that result in a working lint target in my Makefile that does the right thing.
Return value of pipe is one which is returned by its last command. So you need just revert status of full command line:
lint:
#! find ... | grep -v "No syntax errors detected"
All I want is to backup the database using cronjob. I keep getting this error.
Error
/usr/local/cpanel/bin/jailshell: -c: line 0: unexpected EOF while looking for matching `"'
/usr/local/cpanel/bin/jailshell: -c: line 1: syntax error: unexpected end of file
Cronjob
root mysqldump -e --user=t***b --password=1*** --all-databases | gzip | uuencode `date-database.sql.gz | mail -s "`dateweb1_iepe-wp.sql.gz mysqldump backup" ***#gmail.com
I think jailshell has a problem with shell substitution. I worked around this by creating the command in php.
echo "<?php date_default_timezone_set('America/New_York'); passthru( '/usr/bin/mysqldump -uXXX -pYYY db_name > /home/mysql_backups/backup-'.date(DATE_ATOM));" | php -q
This worked for me inside a cPanel cron job, which uses jailshell
I want to have a grep that has one or more from each of the following groups.
"include,include_once,exec"
AND
"$_GET, $_POST"
So the file must have one or more from BOTH groups. (checking a whole directory of php files)
I'm trying to scan some PHP files for vulnerabilities but I really suck at grep :(
I suggest starting simple with commands like:
grep --color=auto -ER '\$_POST|\$_GET' .
or
grep --color=auto -ER 'include(_once)+|exec\(|require(_once)+' .
The above commands will color the found patterns in the output. Notice the regular expressions (enabled via -E) and the recursive search (via -R). Special characters of the regex are escaped via a backspace character (that is, the $ char and the opening bracket of exec)
To see the context of found occurrences use the -C num argument of grep to see num lines around the found line:
grep --color=auto -C 2 -ER '\$_POST|\$_GET' .
You can also use -A(fter) and -B(efore) if you want to output a different number of lines before or after the line with the found occurrence of your pattern. You may also use the -n argument to get the line number of a found occurrence. If you want to get more fancy you can combine the find command with xargs and grep or use multiple grep commands connected via pipes, but that's better explained in tutorials on the web.
grep -l will just list the files that match; you can then pipe the output of that into another grep:
grep -l -e "$_POST" -e "$_GET" * | grep -e "include" -e "include_once" -e "exec"