I'm having some regex troubles, using Python 2.7 if that matters.
Basically what I'm trying to do is to capture inserted variables in a PHP SQL query string declaration, for example:
$query = "SELECT * FROM `users` WHERE user='$user' AND password='$pass';";
This should return $user when I get the second group from the match.
Here's my regex as it stands right now:
r'.*?\s*=\s*\(\".*?\'(\$[^\']+)\'.*?\"\);'
Example showing that this works and captures $user but not the one above (yes I know it doesn't capture $pass as it ideally should, that's seems to be a limitation with Python's implementation and Regex in general. I do some hacks to get around this in my actual program)
The above works for the example I used. However, when I introduce another case where the inserted variable uses the syntax '{$foo['bar']}', my other regex below doesn't work which accounts for the fact that it contains an apostrophe which doesn't close the variable:
r'.*?\s*=\s*[\(]?\".*?(?:(?:\'(\$[^\']+)\')|(?:\'(\$\{[^\}]+\})\'))?.*?\"[\)]?;'
So basically I want to capture either the '$user' syntax or the one with { }, for example '{$foo['bar']}'. Note that these are not exclusive, it's just that an inserted variable may be of either kind and I want to account for both.
Here's a link to test this out, showing that it doesn't work. Using the second regex also breaks capturing the simple $user, not sure why.
I am not sure what do you mean by limitation in python because following works as it should:
>>> import re
>>> query = "SELECT * FROM `users` WHERE user='$user' AND password='$pass';";
>>> re.findall(r"='(\$\w+)'", query)
['$user', '$pass']
For matching the other query have a look at this regex demo:
='(\{?\$.+?)(?:'(?:\s|;))
And, code example:
>>> query1 = "(\"SELECT table_schema, table_name, create_time FROM information_schema.tables WHERE table_schema='{$_DVWA['db_database']}' AND table_name='users' LIMIT 1\");"
>>> re.findall(r"='(\{?\$.+?)(?:'(?:\s|;))", query1)
["{$_DVWA['db_database']}"]
# it works on the other query as well
>>> re.findall(r"='(\{?\$.+?)(?:'(?:\s|;))", query)
['$user', '$pass']
Related
I am running a very simple SELECT query in MySQL and it's not working.
SELECT string_name FROM table_name;
This is giving me required output. Like
This is string one.
This is string two.
This is string three.
and so on...
But if I am running a query like this
SELECT * FROM table_name WHERE string_name='This is string one'
It's not giving any output. I even tried TRIM function.
SELECT * FROM table_name WHERE TRIM(string_name)=TRIM('This is string one')
But it's still not giving any output.
Please suggest what I am missing here. Is it because of some formatting or am I doing any silly mistake. By the way, Strings are saved as VARCHAR in the database.
To reiterate from comments; sometimes "non-printing" control characters (like newlines) can make their way into data they were never intended to be a part of. You can test for this by checking CHAR_LENGTH of field values versus what you actually see. Obviously, on large amounts of data this can be difficult; but if you know of one problematic value already, you can use this method to confirm this is the problem on that row before attempting to identify the offending character.
Once this problem is confirmed, you can use queries with MySql's ASC() and substring functions to identify character codes until you find the character; it can be best to start from the end of the string and work back, as often the offending characters are at the end.
The character or characters identified in known problem rows are often the cause of other problem rows as well, so identifying the issue in one known row can actually help resolve all such problems.
Once the character code(s) are identified, queries like WHERE string_name LIKE CONCAT('%', CHAR(13), CHAR(10)) should work (in this case for traditional Windows newlines) to identify other similar problem rows. Obviously, adjust character codes and wildcards according to your circumstances.
If no row should ever have those characters anywhere, you should be able to clean up the data with an update like this:
UPDATE theTable SET theString = REPLACE(REPLACE(theString, CHAR(10), ''), CHAR(13), '') to remove the offending characters. Again, use the codes you've actually observed causing the problem; and you can convert them to spaces instead if circumstances are better handled that way, such as a newline between two words.
Have you tried using LIKE for debugging purposes?
SELECT * FROM table_name WHERE string_name LIKE 'This is string one'
/!\ Don't just switch from = to LIKE, read about why here
TLDR:
= is apparently 30x faster.
Use = wherever you can and LIKE wherever you must.
First of all, I must acknowledge the points made by #Uueerdo were actually the the main cause of this issue. Even I was somewhat sure that there are some hidden characters in the string causing all the issue but I was not sure how to find and fix that offending character.
Also, the approach suggested by #Uueerdo to check and replace the offending character using the ASCII code seems quite legit but as he himself mentioned that this process will take lot's of time and one have to manually check every string for that one offending character and then replace it.
Luckily after spending couple of hours on it, I came up with a much faster approach to fix the issue. For that, first of all I would like to share my use case.
My first query was for selecting all the strings from a database and printing the result on page.
$result = mysqli_query($conn, "SELECT * from table_name");
while($row = mysqli_fetch_array($result)){
$string_var = $row["string_name"];
echo $string_var;
echo "<br>";
}
The above code was working as expected and printing all the string_name from the table. Now, if I wanted to use the variable $string_var for another SELECT query in the same table, it was giving me 0 results.
$result = mysqli_query($conn, "SELECT * FROM table_name");
while($row = mysqli_fetch_array($result)){
$string_var = $row["string_name"];
echo "String Name : ".$string_var."";
$sec_result = ($conn, "SELECT * FROM table_name WHERE string_var='$string_name'");
if(mysqli_num_rows($sec_result) > 0){
echo "Has Results";
} else {
echo "No Results";
}
}
In this snippet, my second query $sec_result was always giving me No Results as output.
What I simply did to fix this issue.
$result = mysqli_query($conn, "SELECT * FROM table_name";
while ($row = mysqli_fetch_array($result)){
$string_var = $row["string_name"];
$row_id = $row["id"];
$update_row = mysqli_query($conn, "UPDATE table_name SET string_name='$string_var' WHERE id=$row_id");
}
This step updated all the strings from the table without any hidden/problem causing character.
I am not generalising this approach and I am not sure if this will work in every use case but it helped me fix my issue in less than a minute.
I request #Uueerdo and others with better understanding on this to post a more generic approach so that it can help others because I think many people who can't find a right approach in such conditions, end up using LIKE in place of = but that completely changes the core idea of the query.
i am using mysqlclient,
in one of my query, as shown below
sprintf (query, "select user from pcloud_session where id = '%s'", sid);
here some time this sid is with % sign in it like the example
2Cq%yo4i-ZrizGGQGQ71eJQ0
but when there is this % this query always fail, i think i have to escape this %, but how ?
i tried with \ and %% , but both of this not working, please help me here
UPDATE:
When using session.hash_bits_per_character = 6, in php session ,the default charset contains a character (comma) that will always be urlencoded(here it is %2C). This results in cookie values having this %2C in it, but session db having a comma instead of it. any idea about fixing this problem ?.. sorry for the confusion
Thanks
There's no need to escape a literal '%' in MySQL query text.
When you say the query "always fail", is it the call to the mysql_query function that is returning an error? Does it return a SQL Exception code, or is it just not returning the resultset (row) you expect?
For debugging, I suggest you echo out the contents of the query string, after the call to sprintf. We'd expect the contents of the string to be:
select user from pcloud_session where id = '2Cq%yo4i-ZrizGGQGQ71eJQ0'
And I don't see anything wrong with that SQL construct (assuming the id column exists in pcloud_session and is of character datatype. Even if id was defined as an integer type, that statement wouldn't normally throw an exception, the string literal would just be interpreted as integer value of 2.)
There should be no problem including a '%' literal into the target format of an sprintf. And there should be no problem including a '%' literal within MySQL query text.
(I'm assuming, of course, that sid is populated by a call to mysql_real_escape_string function.)
Again, I suggest you echo out the contents of query, following the call to sprintf. I also suggest you ensure that no other code is mucking with the contents of that string, and that is the actual string being passed as an argument to mysql_query function. (If you are using the mysql_real_query function, then make sure you are passing the correct length.)
UPDATE
Oxi said: "It does not return a SQL Exception code, it just does not return the result[set] I expect. I did print the query, it prints with % in it."
#Oxi
Here's a whole bunch of questions that might help you track down the problem.
Have you run a test of that query text from the mysql command line client, and does that return the row(s) you expect?
Is that id column defined as VARCHAR (or CHAR) with a length of (at least) 24 characters? Is the collation on the column set as case insensitive, or is it case sensitive?
show create table pcloud_session ;
(I don't see any characters in there that would cause a problem with characterset translation, although that could be a source of a problem, if your application is not matching the database charactarset encoding.)
Have you tested queries using a LIKE predicate against that id column?
SELECT id, user FROM pcloud_session WHERE id LIKE '2Cq\%yo4i-%' ESCAPE '\\'
ORDER BY id LIMIT 10 ;
SELECT id, user FROM pcloud_session WHERE id LIKE '2Cq%'
ORDER BY id LIMIT 10 ;
Are you getting no rows returned when you expect one row? Are you getting too many rows returned, or are you getting a different row than the one you expect?
That is an oddball value for an id column. At first, it looks almost as if the value is represented in a base-64 encoding, but it's not any standard encoding, since it includes the '%' and the '-' characters.
If you're going to do this in C without an interface library, you must use mysql_real_escape_string to do proper SQL escaping.
There shouldn't be anything intrinsically wrong with using '%inside of a string, though, as the only context in which it has meaning is either directly inprintftype functions or as an argument toLIKE` inside of MySQL.
This proves to be really annoying, but it's absolutely necessary. It's going to make your code a lot more complicated which is why using low-level MySQL in C is usually a bad idea. The C++ wrapper will give you a lot more support.
You really shouldn't escape the string yourself. The safest option is to let the MySQL API handle it for you.
For a string of maximum length n, start by allocating a string of length 2*n+1:
int sidLength = strlen(sid);
// worst-case, we need to escape every character, plus a byte for the ASCIIZ
int maxSafeSidLength = sidLength * 2 + 1;
char *safeSid = malloc(maxSafeSidLength);
// copy "sid" to "safeSid", escaping as appropriate
mysql_real_escape_string(mysql, safeSid, sid, sidLength);
// build the query
// ...
free(safeSid);
There's a longer example at the mysql_real_escape_string page on dev.mysql.com, in which they build the entire query string, but the above approach should work for supplying safeSid to sprintf.
I have a PHP script that is generating a MySQL select statement:
select * from words where word = 'Classic'
There is exactly one word in the words table with the variable word equal to Classic.
When my PHP page executes, I get no results from the query. If I echo the string that is being used to execute the query, cut and paste that into the SQL window in PHPMyAdmin in the database, I also get no results. However, if I re-type that EXACT string into the SQL window in PHPMyAdmin (with the same quote characters), I get the proper result of one row.
The word Classic from the select statement is gotten from a PHP GET (see code below). I can echo the $word variable, and get the correct result of 'Classic'. What am I doing wrong?
Here is my code:
<?php
require ('dbconnect.php');
$word = $_GET["word"];
$selectStr = "SELECT * FROM words WHERE word = '" . $word . "'";
if ($results = MySQL($dbName, $selectStr))
{
$rowCount = MySQL_NUMROWS($results);
}
$resultRow = MYSQL_FETCH_ROW($results);
$wordID = $resultRow[0];
?>
Please, please, please sanitize that word. mysql_real_escape_string() should do the trick.
$selectStr = "SELECT * FROM words WHERE word LIKE '" . $sanitized_word_i_promise . "'"; should work :)
Just to explain: "=" should work for exact matches. This includes uppercase / lowercase, spaces etc. You should probably trim that result first too, before using it in the query.
If you have foo stored in the database (note the space at the end) - it won't match foo, without a space. You'll want to use LIKE 'foo%' - probably.
Either way, Sourabh is right, although performance wise, this isn't a big hit when trying to match exact strings, you should look for the problem in other places first (such as, is the item in the database an exact match?).
First off you should not take any user input and directly input it into a query without sanitizing it, or using a prepared statement.
Now that we've gotten that out of the way: have you tried doing a strcmp() with the variable and your string written in? Such as
echo strcmp($_GET['word'], "Classic")
If you get a result other than 0 it means they are not the same, most likely there will be a whitespace of some sort in the $_GET variable. use trim() on it to take out whitespace. Also could be a case sensitivity issue as well.
I just discovered this, I use MySQL as my database.
When i do something like:
$query = $this->db->where('id', '6rubbish')->get('posts', 1);
it generates and executes this SQL:
SELECT *
FROM (`posts`)
WHERE `id` = '6rubbish'
LIMIT 1
The surprising thing is that it actually fetches the post with the ID 6.
I find this very vulnerable in same cases because i'm trying to exactly match the ID, not to do a LIKE query.
Any ideas?
Yes.
Read Type Conversion in Expression Evaluation
Use intval() PHP function to extract the integer part of the variable
Or use is_int to exclude any variable that is not a pure integer
But the origin of the problem is that your query generator library doesn't understand the variable types, PHP being a dynamic typed language doesn't help too.
I don't know what library you're using, maybe there is an option to tell that you're passing an int? It should protect you from SQL injection, I hope, try with:
$query = $this->db->where('id', "don't")->get('posts', 1);
and see if the generated SQL has the single quoted escaped (doubled or preceded by backslash).
That is because SELECT 6 = '6rubbish' will give you true, and your id is number type.
Your id field is, no doubt, a numeric data type.
When MySQL evaluates expressions, it converts operands to compatible types (see docs).
'6rubbish' (a string) gets converted to 6 (a number) and, hence, you get a match.
I have a function below which works perfectly, but now the client came back and asked that the number only be taken to do the search because most of his clients won't type in the suffix "h" or whatever it may be as per my example below:
38039 or 38039h
However he also said he only has one group of product codes which begin with "T" so they could be typing in "T760" in which case we would need the prefix.
My code below does a search on the exact product currently, can anyone help me work in these examples?
<?php
//Find Stock Value
function checkstock($prodCode) {
$prodCode = strtoupper($prodCode);
require '../../../../config.php';
$dbh = new PDO(DB_DSN, DB_USER, DB_PASS);
$sql = "SELECT * FROM isproducts WHERE prodCode = '".
$prodCode."' AND AllowSalesOrder = '1'";
$stmt = $dbh->query($sql);
$obj = $stmt->fetch(PDO::FETCH_OBJ);
$count = $stmt->rowCount();
echo ($count == 1 ?
ROUND($obj->FreeStockQuantity, 0) : 'Invalid product code '.$prodCode.'');
}
//Call Stock Function
checkstock($_POST['productcode']);
?>
Change the query to like below ?
SELECT * FROM isproducts
WHERE
(
prodCode='{$prodCode}' // for product with prefix or suffix
OR prodCode LIKE '{$prodCode}%' // without suffix
OR prodCode='T{$prodCode}' // without prefix
)
AND AllowSalesOrder = ''";
Wild-card by single character
OR prodCode LIKE '{$prodCode}_' // single character wild-card
It seems that you may have products with the same number but not the same suffix? like 8512n and 8512h ?
You could use LIKE '%$code%'
$sql = "SELECT * FROM isproducts WHERE prodCode LIKE '%".$prodCode."%' AND AllowSalesOrder = ''";
and I think its the more secure way in your case, so that all products containing the number will appear, regardless of suffix or prefix.
The above may return more than one product, so the user still has to choose which one it is he is actually looking for.
You can use % wild card for this kind of problem.
check this out....
http://www.w3schools.com/SQL/sql_wildcards.asp
it might be useful... :-)
In some of my code I use the following strategy:
# psuedo-code ... NOT intended for real use:
SELECT COUNT(*) FROM someTable WHERE someColumn = "{XXX}"
# If that returns exactly one than use the corresponding query
SELECT COUNT(*) FROM someTable WHERE someColumn LIKE "{XXX}"
# If that returns exactly one then use it
SELECT COUNT(*) FROM someTable WHERE someColumn LIKE "{XXX}%"
# If that returns exactly one then use it
SELECT COUNT(*) FROM someTable WHERE someColumn LIKE "%{XXX}%"
# If that returns exactly one then use it
... (where {XXX} is the placeholder for the user supplied search term).
The idea here is that I first try a precise match, then I try it under the assumption that the term already may contain SQL wildcards, then I try suffixing the % wild card and finally I try wrapping it with % wild cards.
At any point if I've found an unambiguous match then I use it. If I find more than one match at any point (not shown in the psuedo-code here) then I might throw an exception or I might return them or a subset of them based on the specifics of what I'm doing.
(In reality I'm using the parameter interpolation features of Python or Perl or sanitizing my inputs to allow wild cards while preventing SQL injections; so the code doesn't look like what I'm showing here. This is just to convey the general idea).
My goal is to allow my scripts to be called with the minimum unambiguous arguments supplied which sounds roughly similar to what your clients are requesting here.
From a usability perspective most users will get the first characters of any input right. So exact match following by suffixed wildcard match is most likely to succeed most of the time. In my case my users are likely to be familiar with SQL wildcards and may prefer to use them to construct their own unambiguous match; and logically that attempt has to be inserted before I start suffixing or wrapping it with my own wildcards.
This is why I use this specific sequence of matching attempts.