I've got a script that imports data to a MySQL table and when inserting, VARCHAR and TEXT fields have double quotes on the beginning and end. I am using addslashes because some of the fields are supposed to have single quotes, double quotes, commas, and semi-colons. Here's my code:
$csvfile = fopen($csv_file, 'r');
$theData = fgets($csvfile);
$i = 0;
$imports = array();
while (!feof($csvfile))
{
$csv_data[] = fgets($csvfile, 1024);
$data = explode(",", $csv_data[$i]);
$insert_csv = array();
$insert_csv['id'] = md5($EventID.$PerformerID);
$insert_csv['EventID'] = addslashes($data[0]);
$insert_csv['Event'] = addslashes($data[1]);
$insert_csv['PerformerID'] = addslashes($data[2]);
$insert_csv['Performer'] = addslashes($data[3]);
$insert_csv['Venue'] = addslashes($data[4]);
$insert_csv['VenueID'] = addslashes($data[5]);
$insert_csv['VenueStreetAddress'] = addslashes($data[6]);
$insert_csv['DateTime'] = addslashes($data[7]);
$insert_csv['PCatID'] = addslashes($data[8]);
$insert_csv['PCat'] = addslashes($data[9]);
$insert_csv['CCatID'] = addslashes($data[10]);
$insert_csv['CCat'] = addslashes($data[11]);
$insert_csv['GCatID'] = addslashes($data[12]);
$insert_csv['GCat'] = addslashes($data[13]);
$insert_csv['City'] = addslashes($data[14]);
$insert_csv['State'] = addslashes($data[15]);
$insert_csv['StateID'] = addslashes($data[16]);
$insert_csv['Country'] = addslashes($data[17]);
$insert_csv['CountryID'] = addslashes($data[18]);
$insert_csv['Zip'] = addslashes($data[19]);
$insert_csv['TicketsYN'] = addslashes($data[20]);
$insert_csv['IMAGEURL'] = addslashes($data[23]);
$query = "INSERT IGNORE INTO table_name(`id`, `EventID`, `Event`, `PerformerID`, `Performer`, `Venue`, `VenueID`, `VenueStreetAddress`, `DateTime`, `PCatID`, `PCat`, `CCatID`, `CCat`, `GCatID`, `GCat`, `City`, `State`, `StateID`, `Country`, `CountryID`, `Zip`, `TicketsYN`, `IMAGEURL`)
VALUES('{$insert_csv['id']}','{$insert_csv['EventID']}','{$insert_csv['Event']}','{$insert_csv['PerformerID']}','{$insert_csv['Performer']}','{$insert_csv['Venue']}','{$insert_csv['VenueID']}','{$insert_csv['VenueStreetAddress']}','{$insert_csv['DateTime']}','{$insert_csv['PCatID']}','{$insert_csv['PCat']}','{$insert_csv['CCatID']}','{$insert_csv['CCat']}','{$insert_csv['GCatID']}','{$insert_csv['GCat']}','{$insert_csv['City']}','{$insert_csv['State']}','{$insert_csv['StateID']}','{$insert_csv['Country']}','{$insert_csv['CountryID']}','{$insert_csv['Zip']}','{$insert_csv['TicketsYN']}','{$insert_csv['IMAGEURL']}')";
$n = mysql_query($query);
if(!mysql_query($query)){
die("error: ".mysql_error());
}
$i++;
What is causing the double quotes and how can I remove them when inserting the rows? I have also tried stripslashes on the VALUES part of the query but it causes an error due to fields that have single quotes, double quotes, or other delimiters.
Its possible your csv file contains fields that are delimited by double quotes. You can remove the double quotes from the fields by using the trim function. for example:
$insert_csv['EventID'] = trim(addslashes($data[0]), '"');
The above code will remove the double quote from the start and end of the $data[0] string.
TLDR: Instead of using addslashes() use a DB-specific escape function like mysqli_real_escape_string()
What addslashes() does is that it returns a string with backslashes before characters that need to be escaped.
I was going to write the whole explanation, but I think the php.net does a better job of explaining:
Returns a string with backslashes before characters that need to be
escaped. These characters are single quote ('), double quote ("),
backslash () and NUL (the NULL byte).
An example use of addslashes() is when you're entering data into
string that is evaluated by PHP. For example, O'Reilly is stored in
$str, you need to escape $str. (e.g. eval("echo
'".addslashes($str)."';"); )
To escape database parameters, DBMS specific escape function (e.g.
mysqli_real_escape_string() for MySQL or pg_escape_literal(),
pg_escape_string() for PostgreSQL) should be used for security
reasons. DBMSes have differect escape specification for identifiers
(e.g. Table name, field name) than parameters. Some DBMS such as
PostgreSQL provides identifier escape function,
pg_escape_identifier(), but not all DBMS provides identifier escape
API. If this is the case, refer to your database system manual for
proper escaping method.
If your DBMS doesn't have an escape function and the DBMS uses \ to
escape special chars, you might be able to use this function only when
this escape method is adequate for your database. Please note that use
of addslashes() for database parameter escaping can be cause of
security issues on most databases.
Looks like you have a csv file. I recommend using php's in-build fgetcsv() to read the file. This way, you will get an array for every row and then can use that array to insert into the database.
Also, you can directly import csv into mysql if you want it that way:
LOAD DATA INFILE 'D:/myfile.csv'
INTO TABLE my_table
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
Related
I have a dilemma how should I mysql_real_escape_string() my variables without inserting them into the database \n, \r, \x00 when someone uses " ' or <br> on my comment field, I tried with preg_replace instead of mysql_real_escape_string, but seems I don't know exactly how to allow all the chars and signs I want.
mysql_real_escape_string only escapes values so that your queries don't break, it also protects against SQL injection if used correctly.
If you don't want certain characters you will need to use additional functions to strip them before you apply mysql_real_escape_string.
[insert obligatory "use prepared statements" comment]
Ex:
$string = "My name is
John";
$filtered_string = str_replace("\n", " ", $string); // filter
$escaped = mysql_real_escape_string($filtered_string); // sql escape
mysql_query("INSERT INTO `messages` SET `message` = '" . $escaped . "'");
You should be able to use str_replace to help with this:
mysql_real_escape_string(str_replace(array("\n", "\r\n", "\x00", '"', '\''), '', $input));
Having said that, it is a good idea to switch to mysqli or PDO for database read / write. Both of these allow prepared statements, which reduce the risk of SQL injections.
Here's an example of PDO:
$stmt = $PDOConnection->prepare('INSERT INTO example_table (input_field) VALUES (:input_field)');
$stmt->bindParam(':input_field', $input);
$stmt->execute();
How do I escape a dot in insert query?
insert into './$x/.' () lues( );
How to escape the dot before $x and after $x, I tried the above but did not work.
Use forward slash, not back slash.
\
also, you need to escape things by placing the escape character BEFORE the thing it's escaping.
E.g.
"INSERT INTO `\$my_table` VALUES(NULL,$asdf,$jkl)"
Ok, but in your case, try this:
insert into `.$x.` () lues( );
Those are not ' but `
That is the backtick character. Same key as ~ on U.S. keyboards.
Concatenating a variable and a string:
$query = 'INSERT INTO `' . $my_table . '` VALUES(NULL,asdf,' . $jkl . ')";
But inside of double quotes "" you can just put the variable names into the string:
$query = 'INSERT INTO `$my_table` VALUES(NULL,asdf,$jkl )";
(Both of the above result in the exact same string being assigned to variable $query)
Take a look at the PHP page on strings
You can see sections on single quote, double quote, and others that will blow your mind (HEREDOC).
Suppose I have the following string:
insert into table values ('text1;'); insert into table values ('text2')
How do I break those queries (get each individual query) using regular expressions?
I've found a very similar problem: Use regex to find specific string not in html tag ...but it uses a solution that is specific to .NET: behind lookup (in php it complains that is not fixed length).
I would be very grateful if someone could give me some hints on how to deal with this problem.
The trick is to count how many unescaped quote characters you've passed. Assuming that the SQL is syntactically correct, semicolons after an even number of unescaped quote characters will be the ones you want, and semicolons after an odd number of unescaped quote characters will be part of a string literal. (Remember, string literals can contain properly escaped quote characters.)
If you want 100% reliability, you'll need a real SQL parser, like this. (I just Googled "SQL parser in PHP". I don't know if it works or not.)
EDIT:
I don't think it's possible to find pairs of unescaped quote characters using nothing but regex. Maybe a regex guru will prove me wrong, but it just seems too damn difficult to distinguish between escaped and unescaped quote characters in so many possible combinations. I tried look-behind assertions and backrefs with no success.
The following is not a pure-regex solution, but I think it works:
preg_match_all("/(?:([^']*'){2})*[^']*;/U", str_replace("\\'", "\0\1\2", $input), $matches);
$output = array_map(function($str){ return str_replace("\0\1\2", "\\'", $str); }, $matches[0]);
Basically, we temporarily replace escaped quote characters with a string of bytes that is extremely unlikely to occur, in this case \0\1\2. After that, all the quote characters that remain are the unescaped ones. The regex picks out semicolons preceded by an even number of quote characters. Then we restore the escaped quote characters. (I used a closure there, so it's PHP 5.3 only.)
If you don't need to deal with quote characters inside string literals, yes, you can easily do it with pure regex.
Assuming proper SQL syntax it would probably be best to split on the semicolon.
The following regexp will work but only if all quotes come in pairs.
/.+?\'.+?\'.*?;|.+?;/
To avoid escaped single quotes:
/.+?[^\\\\]\'.+?[^\\\\]\'.*?;|.+?;/
To handle Multiple pairs of single quotes.
/.+?(?:[^\\]\'.+?[^\\]\')+.*?;|.+?;/
Tested against the following data set:
insert into table values ('text1;\' ','2'); insert into table values ('text2'); insert into test3 value ('cookie\'','fly');
Returns:
insert into table values ('text1;\' ','2');
insert into table values ('text2');
insert into test3 value ('cookie\'','fly');
I have to admit this is a pretty dirty regexp. It would not handle any sort of SQL syntax errors at all. I enjoyed the challenge of coming up with a pure regex though.
How you want to break?
You can use explode( ' ', $query ) to transform into an array.
Or if you want to get text1 and text2 values with regexp you can use preg_match( '/(\'([\w]+)\')/', $query, $matches ) where $matches[1] is your value.
preg_match_all( '/([\w ]+([\w \';]+))/', $queries, $matches ) will give to you all matches with this pattern of query.
Regex's aren't always good at this type of thing. The following function should work though:
function splitQuery($query) {
$open = false;
$buffer = null;
$parts = array();
for($i = 0, $l = strlen($query); $i < $l; $i++) {
if ($query[$i] == ';' && !$open) {
$parts[] = trim($buffer);
$buffer = null;
continue;
}
if ($query[$i] == "'") {
$open = ($open) ? false: true;
}
$buffer .= $query[$i];
}
if ($buffer) $parts[] = trim($buffer);
return $parts;
}
Usage:
$str = "insert into table values ('text1;'); insert into table values ('text2')";
$str = splitQuery($str);
print_r($str);
Outputs:
Array
(
[0] => insert into table values ('text1;')
[1] => insert into table values ('text2')
)
A problem I recently ran into was that when trying to update a field in my database using this code would not work. I traced it back to having a % sign in the text being updated ($note, then $note_escaped)... Inserting it with sprintf worked fine though.
Should I not be using sprintf for updates, or should it be formed differently?
I did some searching but couldn't come up with anything.
$id = mysql_real_escape_string($id);
$note_escaped = mysql_real_escape_string($note);
$editedby = mysql_real_escape_string($author);
$editdate = mysql_real_escape_string($date);
//insert info from form into database
$query= sprintf("UPDATE notes_$suffix SET note='$note_escaped', editedby='$editedby', editdate='$editdate' WHERE id='$id' LIMIT 1");
You are using sprintf totally wrong. Removing the function call in your code would still do the same thing. It should be:
sprintf("UPDATE notes_%s SET note='%s', editedby='%s', editdate='%s' WHERE id=%d LIMIT 1", $suffix, $note_escaped, $editedby, $editdate, $id);
You should read the manual.
first of all you should be using prepared statements instead of a sprintf-call
but if you absolutely have to do it this way you have to use:
$id = mysql_real_escape_string($id);
$note_escaped = mysql_real_escape_string($note);
$editedby = mysql_real_escape_string($author);
$editdate = mysql_real_escape_string($date);
//insert info from form into database
$query= sprintf("
UPDATE notes_%s /* this is still open for injection, and cannot be properly escaped with mysql_real_escape_string */
SET note='%s',
editedby='%s',
editdate='%s'
WHERE id='%d'
LIMIT 1",
$suffix,
$note_escaped, $editedby, $editdate, $id);
You can escape the % in the source text by replacing it with \% in mysql.
sprintf() is not used much in PHP, unless you need to format data somehow. These two statements work identically in PHP:
$num = 42;
$char = 'q';
$text = sprintf('The number is %d and the character is %s', $num, $char);
$text = "The number is $num and the character is $char";
sprintf's used more in C for "printing" variable data into a string. But PHP can already do that with double-quoted strings, so unless you need to use sprintf's special formatting functions (e.g. %0.2f for a 2-decimal-place float), it's easier to use the regular string method.
From http://php.net/manual/en/function.mysql-real-escape-string.php:
Note: mysql_real_escape_string() does not escape % and _. These are wildcards in MySQL if combined with LIKE, GRANT, or REVOKE.
You need to manually escape the % and _ if any with \% and _. I don't recommend using sprintf, but just improving your escape function.
When I use fputcsv to write out a line to an open file handle, PHP will add an enclosing character to any column that it believes needs it, but will leave other columns without the enclosures.
For example, you might end up with a line like this
11,"Bob ",Jenkins,"200 main st. USA ",etc
Short of appending a bogus space to the end of every field, is there any way to force fputcsv to always enclose columns with the enclosure (defaults to a ") character?
No, fputcsv() only encloses the field under the following conditions
/* enclose a field that contains a delimiter, an enclosure character, or a newline */
if (FPUTCSV_FLD_CHK(delimiter) ||
FPUTCSV_FLD_CHK(enclosure) ||
FPUTCSV_FLD_CHK(escape_char) ||
FPUTCSV_FLD_CHK('\n') ||
FPUTCSV_FLD_CHK('\r') ||
FPUTCSV_FLD_CHK('\t') ||
FPUTCSV_FLD_CHK(' ')
)
There is no "always enclose" option.
Not happy with this solution but it is what I did and worked. The idea is to set an empty char as enclosure character on fputcsv and add some quotes on every element of your array.
function encodeFunc($value) {
return "\"$value\"";
}
fputcsv($handler, array_map(encodeFunc, $array), ',', chr(0));
Building on Martin's answer, if you want to avoid inserting any characters that don't stem from the source array (Chr(127), Chr(0), etc), you can replace the fputcsv() line with the following instead:
fputs($fp, implode(",", array_map("encodeFunc", $row))."\r\n");
Granted, fputs() is slower than fputcsv(), but it's a cleaner output. The complete code is thus:
/***
* #param $value array
* #return string array values enclosed in quotes every time.
*/
function encodeFunc($value) {
///remove any ESCAPED double quotes within string.
$value = str_replace('\\"','"',$value);
//then force escape these same double quotes And Any UNESCAPED Ones.
$value = str_replace('"','\"',$value);
//force wrap value in quotes and return
return '"'.$value.'"';
}
$fp = fopen("filename.csv", 'w');
foreach($table as $row){
fputs($fp, implode(",", array_map("encodeFunc", $row))."\r\n");
}
fclose($fp);
After a lot of scrafffing around and some somewhat tedious character checking, I have a version of the above referenced codes by Diego and Mahn that will correctly strip out encasings and replace with double quotes on all fields in fputcsv. and then output the file to the browser to download.
I also had a secondary issue of not being able to be sure that double quotes were always / never escaped.
Specifically for when outputting directly to browser using the php://input stream as referenced by Diego. Chr(127) is a space character so the CSV file has a few more spaces than otherwise but I believe this sidesteps the issue of chr(0) NULL characters in UTF-8.
/***
* #param $value array
* #return string array values enclosed in quotes every time.
*/
function encodeFunc($value) {
///remove any ESCAPED double quotes within string.
$value = str_replace('\\"','"',$value);
//then force escape these same double quotes And Any UNESCAPED Ones.
$value = str_replace('"','\"',$value);
//force wrap value in quotes and return
return '"'.$value.'"';
}
$result = $array_Set_Of_DataBase_Results;
$fp = fopen('php://output', 'w');
if ($fp && $result) {
header('Content-Type: text/csv');
header('Content-Disposition: attachment; filename="export-'.date("d-m-Y").'.csv"');
foreach($result as $row) {
fputcsv($fp, array_map("encodeFunc", $row), ',', chr(127));
}
unset($result,$row);
die;
}
I hope this is useful for some one.
A "quick and dirty" solution is to add ' ' at the end of all of your fields, if it's acceptable for you:
function addspace($v) {
return $v.' ';
}
fputcsv($handle, array_map('addspace', $fields));
PS: why it's working? see Volkerk answer ;)