I have an application where I need to be able to let the users put anything into an input field and am having issues with null bytes. I am passing the data via AJAX to PHP 5.5 and can see it's being passed from the AJAX request correctly, but when I immediately var_dump the $_POST on the PHP side, a string that contained '%00' comes through as ''. As an aside, I'm protecting my database from injection by using query bindings. Also, the user base is exclusively internal to my company. So, I'm not really concerned with the security aspect of it. How can I get PHP to let these null bytes through?
Echoing a null-byte in PHP would indeed result in an empty-looking string, so this makes perfect sense.
echo chr(0); // outputs nothing
Related
I'm stuck with a php issue.
I've got to send a sql query with a POST form with an ajax request.
This query is like :
SELECT * FROM table WHERE field LIKE '%512%'
The problem is when i take back this query from POST var in php, it shows :
SELECT * FROM table WHERE field LIKE 'Q2%'
and it obviously fail...
I tried changing to utf-8 it didn't change anything.
Javascript seems to send the correct text, but php change it when reading POST.
Is ther a way I can prevent php from reading %51 as ascii code ?
Just in case, the website is written with Code Igniter.
Thanks
Don't send entire SQL queries from a Javascript client to a server. Just don't. That's worse than an SQL injection vulnerability, it's simply carte blanche for query execution by anyone for anything.
Data in HTTP requests is typically encoded using percent encoding, and guess what: %51 happens to stand for "Q" in this encoding. You need to properly encode your data when sending it in an HTTP request, for example using encodeURIComponent(). If you want to send "%51", you need to actually be sending %2551. The specifics will depend on how you're sending that data exactly.
Also consider reading The Great Escapism (Or: What You Need To Know To Work With Text Within Text).
Though you can use %25512%, but I agree with deceze.
Using %25 will be interpreted to % sign
so use %25Your number
POST Data : URL encoding replaces unsafe ASCII characters with a "%" followed by two hexadecimal digits.
First : encode data using javascript function.
var res = encodeURI("SELECT * FROM table WHERE field LIKE '%512%'");
then send the data using ajax.
In PHP, Decode data using function :
$data=urldecode(string $_POST['yourdata']);
For further information you can go through these links:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURI
http://php.net/manual/en/function.urldecode.php
http://ascii.cl/htmlcodes.htm
I have been investigating a problem with data sent to my PHP webservice with a POST request and who were sometimes truncated somewhere in the middle. I have found that it is due to an unescaped ampersand (&) which cut the data in the middle. For example if in POST the data is:
data=foobar&morethings
then I will only have "data" => "foobar" in my $_POST array and the "morethings" part is lost.
The obvious solution would be to fix the software which sends the POST request to my webservice so that it escapes ampersands, but this is not practically possible right now (we can not make our users update the software so easily). Therefore I have to find a temporary workaround.
Is there a possibility, from PHP to retrieve the raw data as it was sent to the webservice before it was parsed by whatever is cutting the POST data in pieces?
file_get_contents('php://input');
http://php.net/manual/en/wrappers.php.php
Yes, a couple options.
Use $xml = file_get_contents('php://input'); to read the raw post data.
Use $HTTP_RAW_POST_DATA. This is less reliable though as that variable isn't always populated depending on PHP ini settings.
I have a PHP script that fetches a relatively large amount of data, and formats it as HTML unordered lists for use in an Ajax application.
Considering the data is in the order of tens to possibly more than a hundred KB, and that I want to be able to differentiate between the different lists with Javascript, what would be the best way to go about doing this?
I thought about json_encode, but that results in [null] when more than a certain amount of rows are requested (maybe PHP memory limit?).
Thanks a lot,
Fela
Certain illegal characters in the string could be breaking the json_encode() function in PHP which you will need to sanitize before this will work correctly. You could do this using regular expressions if this becomes a problem.
However, if you are sending requests with that amount of data it may be unwise to send this using AJAX as your application will seem very unresponsive. It may be better to get this data directly from the database as this would be a far faster method although you will have to obviously compromise.
I cannot click up, but I agree with Daniel West.
Ensure your strings are UTF-8 encoded or use mysql_set_charset('utf8') when you connect. The default charset for mysql is unfortunately Latin/Windows. Null is the result of a failed encoding because of this, if it was out of memory, the script itself would fail.
I would pass the data around via JSON, and do it in small batches that you can present incrementally. This will make it appear faster and may get around the memory issues you are having. If the data above the scroll ( first 40 lines or so ) loads quick, it is ok if the rest of the page takes several seconds to load. If you want to get tricky, you can even load the first page and then wait for a scroll event to load the rest, so you don't have to hit the server too much if the user never scrolls to look at the data below the scrollbar. If php is returning null from the json_encode it is because of invalid characters. If you cant control the data, you could just send HTML from the server to the client and avoid all the encoding/decoding, but this means more data to transfer.
With jquery you can transform your unordered list into a javascript array in your ajax application.
$.map( $('li'), function (element) { return $(element).text() });
Also, underscorejs as some very neat functions for javascript arrays and collections.
http://documentcloud.github.com/underscore/
I would prefer JSON, and I thought the 'null' you get from PHP encode is resulted from jSON's double escaping mechanism with javaScript. I have explained it in another post.
You need to double escape special character(One suspicious 'null' cause is '\n' in your case)
json parse error with double quotes
Original Question
mysql-server-6.0.10
I have this problem, I'm using the COMPRESS function to update a field with a BASE64 encoded value. The "updated" base64 string is only slightly different than the value that is currently in the database but it IS different. The only part is different is the last few bytes of the string. The problem is that mysql is seeing these values as the SAME and therefore skipping the update. It returns without error letting my program thing it did what it was told but then you go back and the data is unchagned.
For example, the first one is base64 string that was previously INSERTED to the db, the second one is what I'm trying to overwrite with:
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO2k6MjM0MjM0MjM0MzI0MDAwO319
YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=
I'm running this query:
UPDATE TABLE SET fieldname=COMPRESS('YToxOntpOjA7YToxOntzOjE1OiJ0cmFja2luZ19udW1iZXIiO3M6MTU6IjIzNDIzNDIzNDMyNDExMSI7fX0=') WHERE id = 'SOMEID';
Mysql comes back with 0 rows affected.
I can reproduce this in an SQL Query window via PHPMyAdmin so I know its not a problem with code.
How can I force this to just update regardless if the strings BEGIN the same way, since it obviously inst checking the entire string? Right now my workaround is to clear the present data with one query, then run another query to update against nothing. This is inefficient though and I'm not happy with it.
First question update
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
See the bug report I filed:
https://bugs.php.net/bug.php?id=60111
Second question update and solution
So it really turns out I was way ahead of my self on this. I was using JSON_NUMERIC_CHECK incorrectly. I was using it as a blanket solution to ensure all numeric values would be, well, numeric. The flaw in this logic is that JSON_NUMERIC_CHECK's purpose in life is to attempt to convert all numeric strings to integers, in 32-bit OS environment there is a limit to how many digits an int val can be and thus using the exponential expression as it should!
It comes down to the fact that in my system, there is a possibility of a [tracking number] could actually be an extremely long string of just numbers. JSON_NUMERIC_CHECK was doing exactly what it was suppose to do by converting this to an exponentially expressed int val.
The actual fix to this was to not use JSON_NUMERIC_CHECK and to make sure that my var types were correct where I actually needed int vals before creating the object that is later passed to the front end via JSON.
Phew, a mouthfull, but i figured if anyone else runs into this problem it might be helpful. Chao!
I'm sorry guys, nevermind, the problem is that value i'm supplying is being treated as INT instead of STRING and its over the max size of an INT so its resetting to the MAX INT and therefore being the same string.
Sometimes just thinking through it enough to post a problem on here is enough to help me figure it out :)
UPDATE:
I actually found out this is a result of a PHP bug with JSON_NUMERIC_CHECK!
The value was getting saved in the database properly (couldnt tell because of the compression) but when my script to go back was loading the data out of the database via JSON_ENCODE the values were getting mangled by JSON_NUMERIC_CHECK.
UPDATE:
I'm actually just way ahead of my self. using JSON_NUMERIC_CHECK incorrectly. My bad!
Quick question, is it a better idea to call htmlentities() (or htmlspecialchars()) before or after inserting data into the database?
Before: The new longer string will cause me to have to change the database to hold longer values in the field. (maxlength="800" could change to a 804 char string)
After: This will require a lot more server processing, and hundreds of calls to htmlspecialchars() could be made on every page load or AJAX load.
SOOO. Will converting when results are retrieved slow my code significantly? Should I change the DB?
I'd recommend storing the most raw form of the data in the database. That gives you the most flexibility when choosing how and where to output that data.
If you find that performance is a problem, you could cache the HTML-formatted version of this data somehow. Remember that premature optimization is a bad thing.
I have no experience of php but generally I always convert or escape nearest to output. You don't know when your output requirements will change, for example you may want to spit out data as XML, or JSON arrays and so escaping for HTML and then storing means you're limited to using the data as HTML alone.
In a php/MySQL web app, data flows in two ways
Database -> scripting language (php) -> HTML output -> browser ->screen
and
Keyboard-> browser-> $_POST -> php -> SQL statement -> database.
Data is defined as everything provided by the user.
ALWAYS ALWAYS ALWAYS....
A) process data through mysql_real_escape_string as you move it into an SQL statement, and
B) process data through htmlspecialchars as you move it into the HTML output.
This will protect you from sql injection attacks, and enable html characters and entities to display properly (unless you manage to forget one place, and then you have opened up a security hole).
Did I mention that this has to be done for every single piece of data any user could ever have touched, altered or provided via a script?
p.s. For performance reasons, use UTF-8 encoding everywhere.
It's best to store text as raw and encode it as needed, to be honest, you always need to htmlencode your data anyways when you're outputting it to the wbe page to prevent XSS hacking.
You shouldn't encode your data before you put it in the database. The main reason are:
If such data is near the column size limit, say 32 chars, if the title was "Steve & Fred blah blah" then you might go over that column limit because a 1 char & becomes a 5 char & amp;
You are assuming the data will always be displayed in a web page, in the future you never know where you'll be looking at the data and you might not want it encoded, now you have to decode it and it's possible you might not have access to PHP's decode function
It is the way of the craftsman to "measure twice, optimize once".
If you don't need high performance for your website, store it as raw data and when you output it do what you want.
If you need performance then consider storing it twice: raw data to do what you want with it and another field with the filtered data. It could be seen as redundant, but CPU is expensive, while data storage is really cheap.
The easiest way is store the data "as is" and then convert to htmlentities wherever it is needed.
The safest solution is to filter the data before it goes in into the Database as this prevents possible attacks on your server and database from the lack of security implementation, and then convert it however you need when needed. Also if you are using PDO this will happen automatically for you using prepared statements.
http://php.net/PDO
We had this debate at work recently. We decided to store the escaped values in the database, because before (when we were storing it unescaped) there were corner cases where data was being displayed without being escaped. This can lead to XSS. So we decided to store it escaped to be safe, and if you want it unescaped you have to do the work yourself.
Edit: So to everyone who disagrees, let me add some backstory for my case. Let's say you're working in a team of 50+ people... and data from the database is not guaranteed to be HTML-Encoded on the way out - there's no built-in mechanism for it so the developer has to write the code to do it. And this data is shown all over the place so it's not going through 1 developer's code it's going through 30's - most of whom have no clue about this data (or that it could even contain angle brackets which is rare) and merely want to get it shown on the page, move on, and forget about it.
Do you still think it's better to put the data, in HTML, into the database and rely on random people who are not-you to do things properly? Because frankly, while it certainly may not seem warm-fuzzy-best-practicey, I prefer to fail closed (meaning when the data comes through in a Word Doc it looks like Value<Stock rather than Value<Stock) rather than open (so the Word Doc looks right with no work, but some corner of the platform may/likely-is vulnerable to XSS). You can't have both.