I have a table for things properties.
Such as id, amount, owner, attributes.
The attributes is stores as blob, and could be any small information about the thing itself,.
Like, the thing may have text on it, or it may has something about it that makes it more unique.
When you export a table/row in PhpMyAdmin, I get these really nice values
INSERT INTO `user_things` (`owner_id`, `id`, `thingId`, `count`, `attributes`) VALUES
(1, 6, 101, 1, 0x10c8431200);
And what I'm trying to do, is being able get information from the blob in PHP that displays the blob the same way PhpMyAdmin displayed it, so I can with PHP build a string of query there the blob value stays at a 0x00000 value.
I've tried to use this code: (mysqlQuery is a simplified function I created for mysqli queries, and that one works perfect so don't worry about that part)
$thing = mysqlQuery("SELECT * FROM `user_things` LIMIT 1;");
$blob = $thing["attributes"];
but it doesn't work very well, and I've tried base64_encode and that doesn't give me the 0x000 either.
I would highly appreciate any help on this, how I can get the blob to the same type of value that PhpMyAdmin displays it as.
0x is just a convention for expressing hexadecimal values.
The 0x10c8431200 hexadecimal value is the same like 72079315456 decimal value.
You can try to convert binary data to hex with the bin2hex function.
Example:
$thing = mysqlQuery("SELECT * FROM `user_things` LIMIT 1;");
$blob = $thing["attributes"];
echo bin2hex($blog);
Related
I am trying to save a string in binary to save space in my MySQL DB. However, I am unable to retrieve the binary data correctly, so what am I doing wrong?
I have a database like:
CREATE TABLE `Test` (
ID INT NOT,
Value BLOB
);
I insert the data with statements like:
INSERT INTO `Test` (ID, Value) VALUES(1, b'10000001');
However, I can't retrieve the binary string when I select the value from the inserted row with the underneath query:
SELECT * FROM `Test`; //NOT WORKING. Returns Value=0
SELECT ID, BIN(Value) FROM `Test`; //NOT WORKING. Returns Value=0
What should I use? I have already tried using BIN(Value) in my query as shown above, but without any success. Thanks in advance.
Starting with a literal string of ones and zeroes is incredibly odd, but:
$ones_and_zeroes = '10000001';
// If you have a literal string of ones and zeroes that you need to convert
// to an actual number there's base_convert()
$hex_string = base_convert($ones_and_zeroes, 2, 16);
// $hex_string is now "81"
// you can convert this to an *actual* binary string with hex2bin()
$bin = hex2bin($hex_string);
// $bin now contains the byte 0x81
// now if you want to do it right:
$dbh = new PDO( ... );
$stmt = $dbh->prepare('INSERT INTO `Test` (ID, Value) VALUES(1, :bin)');
$stmt->execute(['bin'=>$bin]);
// or whatever the gross mysqli equivalent is.
I'm not going to show you how to do it wrong in PHP, but of you wanted to hand-write this query it would be:
INSERT INTO `Test` (ID, Value) VALUES(1, 0x81);
Hex-encoded data prefixed with '0x' being the only safe way [that I am aware of] to embed binary data in a text query, though it is EMINENTLY preferable to use prepared statements whenever possible as hex-encoding data like this doubles the length.
The best advice is to check if the insert query ran successfully i.e. with expected data inserted into DB using any MySQL admin tool.
Only then can you isolate whether the problem is with the retrieval or the insets itself
I recently switched from MySQL to PostgreSQL. I have one problem left however.
Previously, I would store small images in the BLOB format in MySQL.
PostgreSQL doesn't know such thing as a BLOB.
I tried using BYTEA field type instead. This actually inserts an large (hexadecimal?) string I guess, but now I'm stuck trying to get this string back to displaying an actual image in PHP..
Any ideas? Thanks in advance.
Here is a piece of code I use to save the image in the database:
$data = bin2hex(file_get_contents('php://input'));
if (!empty($data)) {
$sql = "UPDATE asset SET data = X'%s' WHERE uuid = '%s'";
$args = array($data, $asset_uuid);
}
psql (9.1.3) and php 5.3.6 are used
Bytea is a byte array. It's not a bit pattern. See section 4.2.1.5 of PostgreSQL Lexical Structure.
The correct way to enter bytea is '\x...' with hex values. So what you want is SET data = '\x%s'.
You might also want to look into prepared statements with pg_prepare.
Edit: I was able to insert a (text) file into a bytea with this:
$source = file_get_contents( 'hello.php' );
$insert = pg_prepare( $conn, '', 'insert into t (name, data) values($1,$2)' );
pg_execute( $conn, '', array( 'hello.php', $source ) );
3rd Edit: This works fine to insert the file into the database. However, the pgsql driver in PHP is quite impolite. The only way to retrieve the actual data back is using the old bytea escape mechanism, as detailed here: pg_unescape_bytea.
pg_query('SET bytea_output = "escape";');
$result = pg_query( 'select data from t' );
while ( $row = pg_fetch_row( $result ) ) {
echo pg_unescape_bytea( $row[0] );
}
I'm sorry about how annoying this is. The PostgreSQL interface in PHP can do with some major overhaul for binary values.
To insert bytea contents with the pg_* API, the binary value should always be run through the pg_escape_bytea() function, even if it's passed to the pg_execute or pg_query_params functions.
This is because the pg_* layer doesn't "know" that a particular parameter has binary contents, and it does not implement any real support for parameter types anyway. So the text representation must be used. It can either be in the escape form or the hex form, it doesn't matter to the PG server, and it's independant of the value of bytea_output, which is meaningful only for values read from the server.
Example:
$esc=pg_escape_bytea("\000\001\002");
pg_query_params('INSERT INTO some_table(some_col) VALUES($1)', array($esc));
To read bytea contents with the pg_* API, the value must be run through pg_unescape_bytea() after the fetch. Assuming the client library is not older than 9.0 (libq.so.5.3 or higher), it can decode the contents whether it's in hex form or escape form and it will autodetect it. Only with an older library would it be necessary to force bytea_output to escape for it to decode properly, either dynamically with SET or statically for the whole database (ALTER DATABASE SET bytea_output=escape) or in postgresql.conf for the whole instance.
Example:
$p=pg_query("SELECT some_col FROM some_table WHERE...");
$r=pg_fetch_array($p);
$contents = pg_unescape_bytea($r[0]);
Both answers posted here gave me some thoughts, but none were 100% of the answer.
So, I will explain in this answer what I did to get it to work.
When displaying the image, I used this:
header('Content-Type: image/jpeg');
$data = pack("H*", pg_unescape_bytea($data));
echo $data;
I'm running PHP 5.3.8, in PHP 5.4.0 it turns out you can use hex2bin instead of pack.
When adding the image to the database, I used this:
$data = pg_escape_bytea($data); // Escape input for PostgreSQL
$sql = "UPDATE asset SET data = '%s'WHERE uuid = '%s'";
I'm glad it is working now. Thank you both Daniel and Johann!
I am trying to match a md5 has (generated through php) to its original value in a SQLExpress database.
I am using the following function in my SQL query
master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', 'ID'), 1, 0)
Where 'ID' is the field in the database.
However they both seem to return different values for the md5 hash. I have been using '12290' as a static value to test this.
php md5() returns: 0bd81786a8ec6ae9b22cbb3cb4d88179
The following SQL Statement returns the same output:
DECLARE #password VARCHAR(255)
SET #password = master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', '12290'), 1, 0)
SELECT #password
Yet when I run the following statement from the table:
SELECT ID, master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', CONVERT(NVARCHAR(255), ID)), 1, 0) AS temp
FROM Clients
ORDER BY ID ASC
The 'temp' value matching to the 'ID' value of 12290 returns: 1867dce5f1ee1ddb46ff0ccd1fc58e03
Any help on the matter would be much appreciated!
Thanks
Python helped me to help you.
>>> from hashlib import md5
>>> md5('1\x002\x002\x009\x000\x00').digest().encode('hex')
'1867dce5f1ee1ddb46ff0ccd1fc58e03'
NVARCHAR is Unicode type and it seems from the above experiment that '12990' is stored as UTF-16LE in your database: '1\02\09\09\00\0'.
Assuming that the data encoding in the PHP is UTF-8 data and you don't want to change the existing data in the database, this is how you can fix your PHP script:
<?php
$password = '12290';
$hash = md5(mb_convert_encoding($password, 'UTF-16LE', 'UTF-8')) . "\n";
echo $hash;
?>
Output:
susam#swift:~$ php utf16le-hash.php
1867dce5f1ee1ddb46ff0ccd1fc58e03
In case the data in PHP is in some other encoding such as ASCII, ISO-8859-1, etc. you can change the third argument to mb_convert_encoding accordingly. The list of all supported encodings is available at: http://www.php.net/manual/en/mbstring.supported-encodings.php
Also, see http://www.php.net/manual/en/function.mb-convert-encoding.php
I don't have SQL server to test this on, but the CONVERT command might be creating the NVARCHAR with 240-odd trailing blanks (as you have specified NVARCHAR(255))
Try setting the NVARCHAR to the length of the ID to test:
ARE #password VARCHAR(255)
SET #password = master.sys.fn_varbintohexsubstring(0, HASHBYTES('MD5', CONVERT(NVARCHAR(5), '12290')), 1, 0)
SELECT #password
Try with different lengths in the CONVERT - is there any difference?
One of two things is most likely the problem:
Either the ID column in that row has a value not exactly equal to '12290' (e.g. extra whitespace)
Or the CONVERT function produces such a value
In any case, a standard debugging approach would be to use an SQL query to SELECT the string lengths of that ID field and the return value of CONVERT; if either is not equal to 5, you found the error.
Alternatively you can perform a dump of the table in question including data, and look at the generated INSERT statement to see what the database says the value in that column is.
I'm having some problems when trying to unserialize an array in PHP.
I have two methods in a class, one to handle storing a serialized array in a mysql table and one to retrieve it.
I get the following error when executing the second method (when retrieving / unserializing):
Notice: unserialize(): Error at offset 50623 of 50647 bytes in /Users/benwaine/NetBeansProjects/SentimentEngineMk2/lib/TSE/Mapper/Filter.php on line 39
The string is stored ina blob field in MySQL. The array is multi-dimentional and composed of a token with an array of floats. The problem is intermittent perhaps suggesting the issue is with the tokens?
// Structure
keywordA - 0.234
- 0.234
KeywordB - 0.23
- 0.47
// Methods
public function insertFilterForKeyword($id, array $filterProbs)
{
$sql = "INSERT INTO filter (keyword_id, filter_probs, sample_size_pos, sample_size_neg)
VALUE (:id, :filterProbs, :pos, :neg)";
$stmt = $this->pdo->prepare($sql);
$meta = array_shift($filterProbs);
$stmt->bindParam(':id', $id);
$stmt->bindParam(':filterProbs',serialize($filterProbs));
$stmt->bindParam(':pos', $meta['sample_size_pos']);
$stmt->bindParam(':neg', $meta['sample_size_neg']);
$result = $stmt->execute();
return $result;
}
public function getFilterByKeyword($keyword)
{
$sql = 'SELECT f.id, f.keyword_id, f.filter_probs, f.sample_size_pos, f.sample_size_neg
FROM filter f
JOIN keyword k ON k.id = f.keyword_id
WHERE k.keyword = :keyword';
$stmt = $this->pdo->prepare($sql);
$stmt->bindParam(':keyword', $keyword);
$result = $stmt->execute();
$data = $stmt->fetch(PDO::FETCH_ASSOC);
$probArray = unserialize($data['filter_probs']);
$filter = new TSE_Filter($probArray);
return $filter;
}
I'm guessing the error has to do with characters in the data that are causing deserialization issues.
After googling I tried methods the deserialization line in the following ways, none of which worked:
1) base64_encode / decode the serialized string.
//to safely serialize
$safe_string_to_store = base64_encode(serialize($multidimensional_array));
//to unserialize...
$array_restored_from_db = unserialize(base64_decode($encoded_serialized_string));
2) Use a preg match to alter potentially difficult characters:
unserialize(preg_replace('/;n;/', ';N;', strtolower($data['filter_probs'])));
Any help would be much appreciated!
Thanks.
I'd've guessed length of the strin in the db being the issue. What happens if you echo out the serialised string once you've pulled it out of the database? Does it look like a finished string or does it break? I think blob is around 65k characters, you could try changing it to a longblob and then resave the data?
Also, if you have one that you know doesn't work, skip the database save and try to unserialise the string straight away. Then atleast you know if the issue is with unserialising, or the data in your db?
In your sample code, you're using urlencode on the serialized string. You're not decoding the urlencoding when you unserialize, which would cause the unserialization to fail.
Perhaps that's the problem?
(You should be able to store the serialized data without urlencoding)
My guess is that it has something to do with column, connection or client charsets.
From http://dev.mysql.com/doc/refman/5.1/en/charset-conversion.html:
If the column has a binary data type (BINARY, VARBINARY, BLOB), all the values that it contains must be encoded using a single character set (the character set you're converting the column to). If you use a binary column to store information in multiple character sets, MySQL has no way to know which values use which character set and cannot convert the data properly.
So I'd try setting the column to CHARACTER SET BINARY explicitly first. Remember that serialized strings contain null bytes and other weird stuff if you serialize objects (I realize you aren't doing that, but who knows what your array contains as keys and values).
Also, you're not flagging the serialized argument as a blob when preparing it, so PDO is probably applying some character conversion as well. For the insert, try using this:
$stmt->bindParam(':filterProbs', serialize($filterProbs), PDO::PARAM_LOB);
And maybe use the same approach when fetching as well, just with PDOStatement::bindColumn() (and remember fetching with PDOStatement::fetch(PDO::FETCH_BOUND) then).
Check database scheme definition for field filter.filter_probs if it is long enough to save serialized string.
Also you can validate the array before serialize and after unserialize containing same value.
Does the comment by getmequick http://php.net/manual/en/function.unserialize.php help? Search for 'error at offset'...
So I have a BINARY(255) field in MySQL, and I want to store a Base64 string in it and retrieve it as needed using PHP.
I'm storing the data using
'UPDATE table SET bindata=0x'.bin2hex(base64_decode($b64));
Which seems to be working fine, at any rate, there is a value going into the database. But I can't figure out how to get my original base64 string back. I've tried using base64_encode on the SELECT results, using HEX(bindata) and then running hex2bin and base64_encode. Neither seems to work.
If the base64 string, once converted into binary, has a bunch of zeros at the end, that's fine.
I would store the value this way:
$sql = "UPDATE table SET bindata=?";
$stmt = $pdo->prepare($sql);
Pass as your query parameter the result of base64_decode($b64). It's okay to pass a binary string this way.
$stmt->execute([base64_decode($b64)]);
When you fetch the binary string from the database, query it like this:
$sql = "SELECT bindata FROM table ...";
$stmt = $pdo->prepare($sql);
$stmt->execute();
It's okay to fetch a binary string. After you fetch that into a result row, you can re-encode it.
while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
echo base64_encode($row['bindata']);
}