I am modifying a set of bash scripts that process video files and reports the processing steps to a mysql database (here is the original code in question).
The function that does the database reporting is called from the main processing script and looks like this in the original:
_report_to_db(){
if [ "${REPORT_TO_DB}" = "Y" ] ; then
echo "INSERT IGNORE INTO tableA (objectIdentifierValue,object_LastTouched) VALUES ('${MEDIA_ID}',NOW()) ON DUPLICATE KEY UPDATE object_LastTouched = NOW()" | mysql --login-path="${LOGIN_PROFILE}" "${DB_NAME}" 2> /dev/null
_db_error_check
fi
}
Since the scripts are meant to be run directly from the command line, when you run them that way it works fine. But I'm running them via php from a web interface and there's some shenanigans going on with the quoting/escaping of whitespace and/or variables.
For instance, the script breaks on the whitespace after ...| mysql and it thinks I'm trying to run mysql as root without a password and totally ignores the --login-path and the other stuff I'm piping to it.
When I call mysql from a variable like so:
_report_to_db(){
if [ "${REPORT_TO_DB}" = "Y" ] ; then
SQL_ARRAY=(INSERT IGNORE INTO tableA (columnA,lastTouched) VALUES ("${SOME_PASSED_VALUE}",NOW()) ON DUPLICATE KEY UPDATE object_LastTouched = NOW();)
MYSQL_COMMAND_ARRAY=(mysql --login-path="${LOGIN_PROFILE}" -e "${SQL_ARRAY[#]}" "${DB_NAME}")
echo "$(${MYSQL_COMMAND_ARRAY[#]})"
_db_error_check
fi
}
... I am able to log into mysql correctly but the SQL query is ignored (when it echos the result you get the standard MySQL --help output.
So far I have tried all kinds of variations on quoting, escaping, referencing the query as a separate string variable, as an array (as you see here).
What is also not helpful is that the original _db_error_check() function only checks the value of the pipe exit status. So if the pipe is ok, but there's a problem further down the path, it fails silently.
_db_error_check(){
if [ "$?" != "0" ] ; then
# reports an error if the pipe exit value ≠ 0
else
# everything is ok! even if there was a mysql error
fi
}
This is not a file or database permissions issue (already triple checked that). Are there quotes or some other stupid thing that I am missing?? Thanks! Oh, I am running OSX El Capitan.
UPDATE
Lol, I was going to post the PHP that calls the script and then I remembered that the PHP is actually calling a Pyhton script that does some other processing too, and that is what calls the bash script. Here it all is:
PHP
$command = escapeshellcmd("/usr/local/bin/python3 /Users/user/path/to/ingest.py " . $user . " 2>&1");
while (# ob_end_flush());
$proc = popen($command, 'r');
echo '<pre>';
while (!feof($proc))
{
echo fread($proc, 4096);
# flush();
}
echo '</pre>';
PYTHON
for item in os.listdir(ingestDir):
if not item.startswith("."):
filePath = os.path.abspath(ingestDir+"/"+item)
fileNameForMediaID = os.path.splitext(item)[0]
try:
ingest = subprocess.Popen(['/usr/local/bin/ingestfile','-e','-u',user,'-I',filePath,'-m',fileNameForMediaID])
ingest.wait()
os.remove(filePath)
except IOError as err:
print("OS error: {0}".format(err))
UPDATE 2
I think this might actually be a weird quirk of my installation (go figure). Using mysql --login-path=myDbUser [etc...] from a shell on my host machine I keep getting the error ERROR 1045 (28000): Access denied for user 'ADMIN'#'localhost' (using password: NO) where the client user is ADMIN and I am trying to login as myDbUser.
I actually uninstalled and reinstalled mysql (via Homebrew) and still have the same results. Using a different machine (running Sierra, but the same mysql version) I can run the above shell command successfully and log into mysql as the target user.
Also on the host machine, I can sudo -u _www zsh and run the command as the Apache user (which is the user running the whole show) without a problem. SO WHY IS IT NOT RUNNING CORRECTLY EITHER IN THE SCRIPT OR EVEN RUN FROM SHELL AS MY MAIN CLIENT USER???
Any ideas? $PATH is identical in all cases mentioned above. Same ~/.mylogin.cnf setups. Is there anything else stupid obvious I missed?
You need to use indirect expansion here:
echo "$(${MYSQL_COMMAND_ARRAY[#]})"
the man says:
If the first character of parameter is an exclamation point (!), and
parameter is not a nameref, it introduces a level of variable
indirection. Bash uses the value of the variable formed from the rest
of parameter as the name of the variable; this variable is then
expanded and that value is used in the rest of the substitution,
rather than the value of parameter itself. This is known as indirect
expansion. If parameter is a nameref, this expands to the name of the
variable referenced by parameter instead of performing the complete
indirect expansion. The exceptions to this are the expansions of
${!prefix*} and ${!name[#]} described below. The exclamation point
must immediately follow the left brace in order to introduce
indirection.
${!name[#]}
${!name[*]}
If name is an array variable, expands to the list of array indices
(keys) assigned in name. If name is not an array, expands to 0 if name
is set and null otherwise. When ‘#’ is used and the expansion appears
within double quotes, each key expands to a separate word.
PS: If I may put forward a piece of my personal opinion, having a chain of php -> python -> bash is the worst coding style one can ever met, you may want to rewrite it into single langue so it will be easier to track down further issues at least.
Related
I just cannot fathom how to get the PHP exec() or shell_exec() functions to treat a '*' character as a wildcard. Is there some way to properly encode / escape this character so it makes it through to the shell?
This is on windows (via CLI shell script if that matters, Terminal or a git-bash yields the same results).
Take the following scenario:
C:\temp\ contains a bunch of png images.
echo exec('ls C:\temp\*');
// output: ls: cannot access 'C:\temp\*': No such file or directory
Permissions is not the problem:
echo exec('ls C:\temp\exmaple.png');
// output: C:\temp\example.png
Therefore the * character is the problem and is being treated as a literal filename rather than a wildcard. The file named * does not exist, so from that point of view, it's not wrong...
It also does not matter if I use double quotes to encase the command:
echo exec("ls C:\temp\*");
// output: ls: cannot access 'C:\temp\*': No such file or directory
I have also tried other things like:
exec(escapeshellcmd('ls C:\temp\*'));
exec('ls C:\temp\\\*');
exec('ls "C:\temp\*"');
exec('ls "C:\temp\"*');
And nothing works...
I'm pretty confused that I cannot find any other posts discussing this but maybe I'm just missing it. At this point I have already worked around the issue by manually programming a glob loop and using the internal copy() function on each file individually, but it's really bugging me that I do not understand how to make the wildcard work via shell command.
EDIT:
Thanks to #0stone0 - The answer provided did not particularly answer my initial question but I had not tried using forward slashes in the path and when I do:
exec('ls C:/temp/*')
It works correctly, and as 0stone0 said, it only returns the last line of the output, which is fine since this was just for proof of concept as I was not actually attempting to parse the output.
Also, on a side note, since posting this question my system had been updated to Win11 22H2 and now for some reason the original test code (with the backslashes) no longer returns the "Cannot access / no file" error message. Instead it just returns an empty string and has no output set to the &$output parameter either. That being said, I'm not sure if the forward slashes would have worked on my system prior to the 22H2 update.
exec() only returns the last output line by default.
The wildcard probably works, but the output is just truncated.
Pass an variable by ref to exec() and log that:
<?php
$output = [];
exec('ls -lta /tmp/*', $output);
var_dump($output);
Without any additional changes, this returns the same as when I run ls -lta /tmp/* in my Bash terminal
That said, glob() is still the preferred way of getting data like this especcially since
You shouldn't parse the output of ls
I have a Make target called test. It looks like this:
test:
/var/www//vendor/bin/phpunit
I want to set up a git hook that runs all tests before commits. In order to do this without getting stalled on deprecation warnings, I want to have my hook run a different Make target. It looks like this:
test-automated:
export SYMFONY_DEPRECATIONS_HELPER = disabled
/var/www//vendor/bin/phpunit
However, when I try to run this target, I get this error message:
export SYMFONY_DEPRECATIONS_HELPER = "disabled" /bin/sh: export: line
1: : bad variable name make: *** [Makefile:14: test-automated] Error 2
What can I do to make Make and PHPUnit run without deprecation warnings?
Addendum: I also tried export SYMFONY_DEPRECATIONS_HELPER=disabled and export SYMFONY_DEPRECATIONS_HELPER="disabled". In both of those cases, the Make target ran, but deprecation warnings were displayed, and Make exited with code 1, which is not the desired outcome.
Your answer is the right way to do it.
A makefile recipe is a shell script, and the content of the recipe must be valid shell syntax; it can't be makefile syntax. In the shell, variable assignment must not have whitespace around the equal sign; if you type this into your shell prompt:
$ export foo = bar
that is not assigning a variable foo to the value bar and exporting it; that is trying to export the 3 variables named foo, =, and bar, and = is not a valid shell variable name so you get this error.
Your change to make this export SYMFONY_DEPRECATIONS_HELPER=disabled is right.
The reason it doesn't work is that each logical line in a recipe is run in a separate shell. So assigning the variable on one logical line then running the program in another logical line is not going to work: the assignment on the first line is lost as soon as that shell exits.
You can use either one of:
test-automated:
SYMFONY_DEPRECATIONS_HELPER=disabled /var/www//vendor/bin/phpunit
as you did or (less good IMO but will work):
test-automated:
export SYMFONY_DEPRECATIONS_HELPER=disabled; \
/var/www//vendor/bin/phpunit
by adding the ; \ at the end you've combined these two physical lines into one logical line.
This isn't really a direct answer to the question I asked, but I found out that this works:
test-automated:
SYMFONY_DEPRECATIONS_HELPER=disabled /var/www//vendor/bin/phpunit
I'll leave the question open in case someone has a better answer.
I have come across an interesting situation with the shell_exec() function in PHP. This command works:
$res = shell_exec("mdb-export -D \"%d/%m/%y\" ".$this->mdbFileName." ".$tableName." &> " . $outputFile);
However, this command does not work:
$res = shell_exec("mdb-export -D \"%d/%m/%y\" ".$this->mdbFileName." \"".$tableName."\" &> " . $outputFile);
When entering the command in a shell (either way - with or without quotes around the table name) it works fine. I would really like to know why the first set of escaped quotes does not affect shell_exec():
\"%d/%m/%y\"
But the second set:
\"".$tableName."\"
Does not allow shell_exec to execute. FYI: all file name variables in the script are full file paths. They were entered in the exact same way when testing in the shell. Any thoughts?
EDIT: Upon failure, shell_exec returns NULL and the command is not executed.
After some more testing, it appears that the $tableName variable contained an extra space. Another part of the program was pulling the table name from the MDB file and left a trailing space.
When echoing out the $tableName variable everything appeared fine because the space was invisible. When running shell_exec without quotes surrounding the table name, the command executed because no errors occur if there are extra spaces between parameters. However, when surrounding the variable with quotes, the mdb-export program started looking for:
"tableName "
instead of
"tableName"
causing a silent, yet complete failure.
This is my code for executing a command from PHP:
$execQuery = sprintf("/usr/local/bin/binary -mode M \"%s\" %u %s -pathJson \"/home/ec2/fashion/jsonS/\" -pathJson2 \"/home/ec2/fashion/jsonS2/\"", $path, $pieces, $type);
exec($execQuery, $output, $return);
the $return value is always 0 but $output is empty. The $output should be a JSON.
If I execute the same but removing one letter to binary (for example /usr/local/bin/binar ) I get (correctly) a $return = 127.
If I write other parameters (like -mode R which doesn't exit) I got errors from the console (which are correct as well).
If I run the exact $execQuery (which I printf before to be sure about quotation marks) on the console, it executes correctly. It's only the PHP side where I've got the error.
What can be wrong?
Thank you in advance.
Well, a couple of things might be happening...
This binary you're running write to something else that STDOUT (for instance, STDERR)
The env vars available to the PHP user differ from the env vars available to the user running console (and those vars are required)
PHP User does not have permission to access some files involved.
In order to debug, it might be better to use proc_open instead of exec, and check the STDOUT and STDERR. This might give you additional information regarding what's happening.
Suggestion (and shameless advertising)
I wrote a small utility library for PHP that executes external programs in a safer way and provides aditional debug information. It might help you to, at least pinpoint the issue.
I have a CakePHP database.php config file that I'd like to leverage to open a command line mysql client instance.
The goal would be to be able to run a shell script like ./db-login.sh from the project root and have it pull the database details (host, port, database name, username, password) from the PHP config file and pass them to the mysql command line as arguments. This is to avoid having to enter the details every time.
I understand it's possible to create a shell alias that has the values hard-coded: I would like a portable script that could be included with any of my Cake projects. I also would like to keep the task of getting the DB credentials into bash variables separate from launching the mysql client. This would open up the ability to re-use the DB credentials in other shell scripts easily (such as a mysqldump backup script.)
Here's what I have so far:
database.php
Consider this file immutable for the purposes of this question. It must exist exactly as you see it.
<?php
class DATABASE_CONFIG {
public $default = array(
'host' => 'localhost',
'login' => 'cakephpuser',
'password' => 'c4k3 roxx!',
'database' => 'my_cakephp_project',
);
}
db-cred.sh
Acts as middleware to convert the PHP variables into bash (friendly) variables.
#!/usr/bin/env php
<?php
include 'database.php';
$db = new DATABASE_CONFIG();
// Selectively wrap quotes. Has no real effect(?)
$p = (!empty($db->default['password']) ? "\"$db->default['password']\"" : '');
echo <<<EOD
DB_HOST="{$db->default['host']}"
DB_NAME="{$db->default['database']}"
DB_USER="{$db->default['login']}"
DB_PASS={$p}
EOD;
db-login.sh
#!/usr/bin/env bash
# Uses Cake's database.php file to log into mysql on the
# command line with the correct arguments.
# Holy yuck, but nothing else works!
eval $( db-cred.sh )
# Try to set an appropriate password clause.
PATTERN=" |'"
if [ -n "$DB_PASS" ]; then
if [[ $DB_PASS =~ $PATTERN ]]; then
PASS_CLAUSE=" -p'${DB_PASS}'"
else
PASS_CLAUSE=" -p${DB_PASS}"
fi
else
PASS_CLAUSE=""
fi
# Get a rough look at what we're about to run.
echo mysql --host=${DB_HOST} --database=${DB_NAME} --user=${DB_USER}${PASS_CLAUSE}
# Call MySQL.
mysql --host=${DB_HOST} --database=${DB_NAME} --user=${DB_USER}${PASS_CLAUSE}
Things that need help:
The db-login.sh script uses eval to suck in the variables. Yuck.
The MySQL password is leaked on the command line. This is not a dealbreaker, but if there's a clean/portable way to avoid it, I'm open to it.
Spaces and quotes in the mysql password don't get passed to bash properly.
A password like my Pass will cause DB_PASS to be set to my in db-login.sh.
Quoting the string in db-cred.sh produces "my instead.
Likewise, my attempts at quoting PASS_CLAUSE are not effective: Only passwords without spaces/quotes will successfully log into the mysql client.
The db-login.sh script uses eval to suck in the variables. Yuck.
Instead of eval, you can use the source command:
source db-cred.sh
The MySQL password is leaked on the command line. This is not a dealbreaker, but if there's a clean/portable way to avoid it, I'm open to it.
You can just unset the MySQL password after it is used:
unset DB_PASS
However, you will always have this problem if you are supplied the password to MySQL via the command line. You could remove the -pPassword clause and force the user to enter the password when MySQL prompts for it, but from your use-case this does not seem feasible.
Spaces and quotes in the mysql password don't get passed to bash properly.
Likewise, my attempts at quoting PASS_CLAUSE are not effective: Only passwords without spaces/quotes will successfully log into the mysql client.
For 3 and 4, I believe all you need to do is properly quote your variable values.
In PHP:
echo <<<EOD
DB_HOST="{$db->default['host']}"
DB_NAME="{$db->default['database']}"
DB_USER="{$db->default['login']}"
DB_PASS="{$p}"
EOD;
Note the added quotes in DB_PASS="{$p}".
As a reminder, you can also escape the quotes in PHP with addslashes().
Okay, so #nickb's suggestions led me down the right track with enough modifications.
Explanation
Issues arise if you try to pass the MySQL password through multiple bash variables. Here's a stripped down example:
#!/bin/bash
set -x
DB_PASS="pass with space and 'quote"
PASS_CLAUSE=" -p'$DB_PASS'"
# Doesn't work.
mysql $PASS_CLAUSE
# Works.
mysql -p"$DB_PASS"
Note that with the set -x we're able to see the commands that are actually being run. This is what helped me identify the double-escaping. The string in $DB_PASS gets re-escaped when it gets saved into $PASS_CLAUSE. Since it gets passed to MySQL with one level of escaping still in place the first connection fails, but using the original variable succeeds.
Final Scripts
db-cred.sh
#!/usr/bin/php
<?php
include 'database.php';
$db = new DATABASE_CONFIG();
echo <<<EOD
DB_HOST="{$db->default['host']}"
DB_NAME="{$db->default['database']}"
DB_USER="{$db->default['login']}"
DB_PASS="{$db->default['password']}"
EOD;
db-login.sh
#!/bin/bash
# Holy yuck, but nothing else works!
eval $( db-cred.sh )
CMD="mysql --host=${DB_HOST} --database=${DB_NAME} --user=${DB_USER}"
PATTERN=" |'"
if [[ ${DB_PASS} =~ ${PATTERN} ]]; then
${CMD} -p"${DB_PASS}"
elif [ -n "${DB_PASS}" ]; then
${CMD} -p${DB_PASS}
else
${CMD}
fi
Final Thoughts
I haven't specifically tested for it, but there may still be issues with passwords that contain double-quotes.
This could still be improved by somehow removing the need for eval. #Dave's idea about using CSV as a go between has potential, but in my tests it further complicated things by adding another level of escaping/unescaping necessary to get the variables into bash cleanly.
The MySQL password is still used on the command line. Writing a mysql config file, using it with --defaults-extra-file=filename and afterwards deleting it might be a way around that.
db-login.sh now represents a pattern for executing command lines with the database credentials and can be easily modified/extended for other tools like mysqldump.