Related
I don't understand why
SELECT UUID();
Returns something like:
3f06af63-a93c-11e4-9797-00505690773f
But if I insert it into a binary(16) field (the UUID() function) with for instance a BEFORE INSERT trigger and run a select, it returns something like:
0782ef48-a439-11
Note that these two UUIDs are not the same data.
I realize binary and an UUID string doesn't look identical, but shouldn't the selected data at least be just as long? Otherwise how can it possibly be equally likely to be unique?
Is it better to store it as char(36)? I just need it to be unique to prevent duplicate inserts. It is never selected or used for joins.
EDIT:
before trigger would be like:
BEGIN
if NEW.UUID IS NULL THEN
NEW.UUID = UUID();
END IF
END
So, as a response to comments. The correct way to store a 36-char UUID as binary(16) is to perform the insert in a manner like:
INSERT INTO sometable (UUID) VALUES
(UNHEX(REPLACE("3f06af63-a93c-11e4-9797-00505690773f", "-","")))
UNHEX because an UUID is already a hexed value. We trim (REPLACE) the dashes in the statement to bring the length down to 32 characters (our 16 bytes represented as HEX). You can do this at any point before storing it, obviously, so it doesn't have to be handled by the database.
You may retrieve the UUID like this:
SELECT HEX(UUID) FROM sometable;
Just in case someone comes across this thread and is unsure how this works.
And remember: If you're selecting a row using the UUID, use UNHEX() on the condition:
SELECT * FROM sometable WHERE UUID = UNHEX('3f06af63a93c11e4979700505690773f');
or literal notation (as mentioned by Alexis Wilke):
SELECT * FROM sometable WHERE UUID = 0x3f06af63a93c11e4979700505690773f;
And NOT HEX()on the column:
SELECT * FROM sometable WHERE HEX(UUID) = '3f06af63a93c11e4979700505690773f';
The last solution, while it works, requires that MySQL HEXes all UUIDs before it can determine which rows match. It's very inefficient.
Edit: If you're using MySQL 8 you should have a look at the UUID functions as mentioned in SlyDave's answer. This answer is still correct, but it doesn't optimise the UUID indexes which can be done natively using those functions. If you're on < MySQL 8 you can implement Devon's polyfill, which provides identical functionality on previous versions of MySQL.
As of MySQL 8 you can use two new UUID functions:
BIN_TO_UUID
SELECT BIN_TO_UUID(uuid, true) AS uuid FROM foo;
-- 3f06af63-a93c-11e4-9797-00505690773f
UUID_TO_BIN
INSERT INTO foo (uuid) VALUES (UUID_TO_BIN('3f06af63-a93c-11e4-9797-00505690773f', true));
This method also supports rearranging the time component of the uuid to enhance indexing performance (by ordering it chronologically), simply set the second argument to true - this only works for UUID1.
If you are using the true on UUID_TO_BIN flag for indexing performance (recommended), you must also set it on BIN_TO_UUID otherwise it won't convert back properly.
See the documentation for further details.
https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_uuid-to-bin
http://mysqlserverteam.com/mysql-8-0-uuid-support/
Polyfill for BIN_TO_UUID and UUID_TO_BIN for MySQL 5 with the swap_flag parameter.
DELIMITER $$
CREATE FUNCTION BIN_TO_UUID(b BINARY(16), f BOOLEAN)
RETURNS CHAR(36)
DETERMINISTIC
BEGIN
DECLARE hexStr CHAR(32);
SET hexStr = HEX(b);
RETURN LOWER(CONCAT(
IF(f,SUBSTR(hexStr, 9, 8),SUBSTR(hexStr, 1, 8)), '-',
IF(f,SUBSTR(hexStr, 5, 4),SUBSTR(hexStr, 9, 4)), '-',
IF(f,SUBSTR(hexStr, 1, 4),SUBSTR(hexStr, 13, 4)), '-',
SUBSTR(hexStr, 17, 4), '-',
SUBSTR(hexStr, 21)
));
END$$
CREATE FUNCTION UUID_TO_BIN(uuid CHAR(36), f BOOLEAN)
RETURNS BINARY(16)
DETERMINISTIC
BEGIN
RETURN UNHEX(CONCAT(
IF(f,SUBSTRING(uuid, 15, 4),SUBSTRING(uuid, 1, 8)),
SUBSTRING(uuid, 10, 4),
IF(f,SUBSTRING(uuid, 1, 8),SUBSTRING(uuid, 15, 4)),
SUBSTRING(uuid, 20, 4),
SUBSTRING(uuid, 25))
);
END$$
DELIMITER ;
--
-- Tests to demonstrate that it works correctly. These are the values taken from
-- https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_uuid-to-bin
--
-- If you run these SELECTs using the above functions, the
-- output of the two columns should be exactly identical in all four cases.
SET #uuid = '6ccd780c-baba-1026-9564-5b8c656024db';
SELECT HEX(UUID_TO_BIN(#uuid, 0)), '6CCD780CBABA102695645B8C656024DB';
SELECT HEX(UUID_TO_BIN(#uuid, 1)), '1026BABA6CCD780C95645B8C656024DB';
SELECT BIN_TO_UUID(UUID_TO_BIN(#uuid,0),0), '6ccd780c-baba-1026-9564-5b8c656024db';
SELECT BIN_TO_UUID(UUID_TO_BIN(#uuid,1),1), '6ccd780c-baba-1026-9564-5b8c656024db';
Included are the SELECT samples from https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_uuid-to-bin that demonstrate that the above code returns the exact same results as the 8.0 function. These functions are considered DETERMINISTIC as they always produce the same output for a given input. See https://dev.mysql.com/doc/refman/8.0/en/create-procedure.html
I am using MariaDB so BIN_TO_UUID functions family do not exist. I managed to get the corresponding values anyway.
bin -> hex
Here, uuid is the binary(16) value of an uuid; you'll use the value below to SELECT a readable version of it.
LOWER(CONCAT(
SUBSTR(HEX(uuid), 1, 8), '-',
SUBSTR(HEX(uuid), 9, 4), '-',
SUBSTR(HEX(uuid), 13, 4), '-',
SUBSTR(HEX(uuid), 17, 4), '-',
SUBSTR(HEX(uuid), 21)
))
hex -> bin
Here, cc6e6d97-5501-11e7-b2cb-ceedca613421 is a readable version of an UUID, and you'll use the value below in a WHERE clause to seek for it.
UNHEX(REPLACE('cc6e6d97-5501-11e7-b2cb-ceedca613421', '-', ''))
Cheers
The other answers are correct. The UUID() function returns a 36 character string and that needs to be converted using the shown functions (UNHEX() or, on newer platforms, UUID_TO_BIN()).
However, if you use your own software to create your UUIDs, then you can use the Hexadecimal Literal notation instead.
So I would use the following with the MySQL UUID() function:
INSERT INTO sometable (id) VALUES (UNHEX(REPLACE(UUID(), '-', ''))); -- all versions
INSERT INTO sometable (id) VALUES (UUID_TO_BIN(UUID()); -- since v8.0
But use this in case I generate my own UUIDs;
INSERT INTO sometable (id) VALUES 0x3f06af63a93c11e4979700505690773f;
Similarly, you can use Hexadecimal Literals in your WHERE clauses:
SELECT * FROM sometable WHERE id = 0x3f06af63a93c11e4979700505690773f;
This will be faster if you do not have to convert your data to a UUID string each time.
Note: the 'x' in '0xaBc is case sensitive. The hexadecimal digits are not, however.
In MySQL 4.0 and above you can change the size of UUID like using the MID
SELECT MID(UUID(),1,32); # 32 characters long UUID
SELECT MID(UUID(),1,11); # 11 characters long UUID
As #nickdnk pointed out you shouldn't do this. The total length of UUID makes them unique. Striping a part of them can lead to non unique values.
I'm using PHP 7 with Phalcon PHP and I'm trying to create a method to generate a booking number. Here is my current method :
public function generateNumber($company_code) {
// Build the prefix : COMPANY20190820
$prefix = $company_code . date('Ymd');
// It's like SELECT count(*) FROM bookings WHERE number LIKE 'COMPANY20190820%'
$counter = Bookings::count(array(
"number LIKE :number:",
"bind" => array('number' => $prefix.'%')
));
// Concat prefix with bookings counter with str_pad
// COMPANY20190820 + 005 (if 4 bookings in DB)
$booking_number = $prefix . str_pad($counter + 1, 3, 0, STR_PAD_LEFT);
// Return COMPANY20190820005
return $booking_number;
}
So I have a problem because sometime I have to delete 1 or multiple bookings so I can get :
COMPANY20190820001
COMPANY20190820002
COMPANY20190820005
COMPANY20190820006
COMPANY20190820007
And I need to add after the last in my DB so here 007, because I can get duplicated booking number if I count like that.
So how can I do to take the last and increment according the last booking number of the current day ?
You need to rethink what you want to do here as it will never work that way.
As I see it you have at least two options:
Use an auto-increment id and use that in combination with the prefix
Use a random fairly unique string (e.g. UUID4)
You should never manually try to get the current maximum id as that may and most likely will at some point result in race conditions and brittle code as a result of that.
So I found a solution, maybe there is a better way to do that but my function works now:
public function generateNumber($company_code) {
// Build the prefix : COMPANY20190820
$prefix = $company_code . date('Ymd');
// Get the last booking with the today prefix
// e.g : COMPANY20190820005
$last_booking = Bookings::maximum(array(
"column" => "number",
"number LIKE :number:",
"bind" => array('number' => $prefix.'%')
));
// Get the last number by removing the prefix (e.g 005)
$last_number = str_replace($prefix, "", $last_booking);
// trim left 0 if exist to get only the current number
// cast to in to increment my counter (e.g 5 + 1 = 6)
$counter = intval(ltrim($last_number, "0")) + 1;
// Concat prefix + counter with pad 006
$booking_number = $prefix . str_pad($counter, 3, 0, STR_PAD_LEFT);
// Return COMPANY20190820006
return $booking_number;
}
I reckon that the use case you describe does not justify the hassle of writing a custom sequence generator in PHP. Additionally, in a scenario where booking deletion is expected to happen, ID reusing feels more a bug than a feature, so your system should store a permanent counter to avoid reusing, making it less simple. Don't take me wrong, it can be done and it isn't rocket science, but it's time and energy you don't need to spend.
Your database engine surely has a native tool to generate autoincremented primary keys, with varying names and implementations (SQL Server has identity, Oracle has sequences and identity, MySQL has auto_increment...). Use that instead.
Keep internal data and user display separated. More specifically, don't use the latter to regenerate the former. Your COMPANY20190820007 example is trivial to compose from individual fields, either in PHP:
$booking_number = sprintf('%s%s%03d',
$company_code,
$booking_date->format('Ymd'),
$booking_id
);
... or in SQL:
-- This is MySQL dialect, other engines use their own variations
SELECT CONCAT(company_code, DATE_FORMAT(booking_date, '%Y%m%d'), LPAD(booking_id, 3, '0')) AS booking_number
FROM ...
You can (and probably should) save the resulting booking_number, but you cannot use it as source for further calculations. It's exactly the same case as dates: don't need to store dates in plain English in order to eventually display them to the end-user and you definitively don't want to parse English dates back to actual dates in order to do anything else beyond printing.
You also mention the possibility of generating long pure-digit identifiers, as Bookings.com does. There're many ways to do it and we can't know which one they use, but you may want to considering generating a numeric hash out of your auto-incremented PK via integer obfuscation.
you could split your database field in two parts, so you hold the prefix and the counter separately.
then, you simply select the highest counter for your desired prefix and increment that one.
if you can't change the table structure, you could alternatively order by the id descendingly and select the first. then you can extract its counter manually. keep in mind you should pad the numbers then, or you get #9 even if #10 exists.
if padding is not an option, you can direct the database to replace your prefix. that way, you can cast the remaining string to a number and let the database sort - this will cost some performance, though, so keep the amount of records low.
I don't know if it is a PHP issue, or PDO issue, or SQLITE issue, but when I search for values in columns which have NUMBER data type, or simply retrieve a SUM of them, php get them as string istead of number (integer, float...)
<?php
//...
$res = $pdo->query("SELECT SUM(id) as 'foo' FROM anytable");
$bar = $res->fetch(); //this value should be (int)
$bar['test'] = 12.50; //this value is (float)
$bar['test'] = 666; //this value is (int)
echo json_encode($bar);
?>
I get this object:
{ foo: "580", test: 12.50, test2: 666 }
This is a sample code, my real query is really more complex and I don't want to parse every field (roughly 90) as integer or float, since they are mixed data type
EDIT: As noted by VolKerk, the implementation in PDO for sqlite is fairly poor. I tested the feature proposed $pdostmt->getColumnMeta( $intColId ) which give us some hint. Since they don't match syntactically the sqlite driver types, I'll test something more and leave here result by time, if anyone is interested.
$hints = $pdostmt->getColumnMeta( $intColId );
$coltype = $hints['native_type'];
results:
sqlite data type | native_type
---------------------------------
DECIMAL | 'double'
Though there are sqlite functions like sqlite3_column_double() or sqlite3_column_int() which would allow to return a range of "native" types, the function behind PDOStatement::fetch() when using the sqlite driver is currently limited to NULL, sqlite3_column_blob() and as the default for any other type: sqlite3_column_text(). So, regardless of whether the storage type is INTEGER,REAL or TEXT, you will get ...text.
But at least PDOStatement::getColumnMeta can tell you a bit more about the native type. The sqlite driver distinguishes between SQLITE_NULL, SQLITE_FLOAT, SQLITE_BLOB, SQLITE_TEXT and SQLITE_INTEGER.
I need to store updated_at timestamp with high precision on a laravel application, using the format "m-d-Y H:i:s.u" (including milisseconds)
According to laravel documentation, I can customize the date format by setting the $dateFormat property on a class, but...
The main problem is that Laravel's schema builder adds a column of type timestamp in the database when I use $table->nullableTimestamps() And according to mysql documentation, columns of type TIMESTAMP only allow the precision up to seconds..
Any ideas on how I could achieve that?
You can't because the PHP PDO driver doesn't support fractional seconds in timestamps. A work around is to select the timestamp as a string instead so the PDO driver doesn't know its really a timestamp, simply by doing $query->selectRaw(DB::raw("CONCAT(my_date_column) as my_date_column")) however this means you can't use the default select for all fields so querying becomes a real pain. Also you need to override getDateFormat on the model.
// override to include micro seconds when dates are put into mysql.
protected function getDateFormat()
{
return 'Y-m-d H:i:s.u';
}
Finally in your migration rather than nullableTimestamps, outside of the Schema callback do:
DB::statement("ALTER TABLE `$tableName` ADD COLUMN created_at TIMESTAMP(3) NULL");
Note this example was for 3 decimal places however you can have up to 6 if you like, by changing the 3 to a 6 in two places, in the alter table and in the sprintf and also adjusting the multiplier * 1000 to 1000000 for 6.
Hopefully some day PHP PDO will be updated to fix this, but its been over 5 years and nothings changed so I don't have my hopes up. In case you are interested in the details, see this bug report:
http://grokbase.com/t/php/php-bugs/11524dvh68/php-bug-bug-54648-new-pdo-forces-format-of-datetime-fields
I found that link in this other answer which might help you more understand the issue:
https://stackoverflow.com/a/22990991/259521
PHP is really showing its age lately, and I would consider this issue one of my reasons for considering moving to the more modern Node.js.
Based on malhal's answer, I was able to get fractional timestamp reading to work here. Pasting the answer here for convenience:
class BaseModel extends Model
{
protected $dateFormat = 'Y-m-d\TH:i:s.u';
protected function asDateTime($value)
{
try {
return parent::asDateTime($value);
} catch (\InvalidArgumentException $e) {
return parent::asDateTime(new \DateTimeImmutable($value));
}
}
public function newQuery()
{
$query = parent::newQuery();
if($this->usesTimestamps()) {
$table = $this->getTable();
$column = $this->getDeletedAtColumn();
$query->addSelect(DB::raw("concat($table.$column) as $column"));
}
return $query;
}
}
There is a lot going on here because it gets the query with scopes applied and then adds a select for the updated_at column to the end, which overwrites any previously loaded updated_at column later while Laravel hydrates the Model from the query. For being an ugly hack, it worked surprisingly well the very first time.
Timestamps are stored internally as Carbon in Laravel:
dd(MyModel->first()->updated_at->format('Y-m-d H:i:s.u'));
Output:
2017-04-14 22:37:47.426131
Also be sure to run a migration to convert your columns to fractional timestamps. Microsecond precision raises the size of timestamps from 4 bytes to 7 bytes but this is 2017, don't let saving a byte or two by choosing millisecond precision cost you a fortune later when you find yourself serving stale cache entries:
\DB::statement("ALTER TABLE my_table MODIFY updated_at TIMESTAMP(6) NULL DEFAULT NULL");
Sadly I haven't found a way to modify the migration schema timestamps() function to do this.
Ahoy Stack Overflow! This be mai first post...
I'm attempting to identify users with a salted unique public key.
Algorithm - Should I use uniqid(), sha256, sha512, something else? All hashes will be salted. NIST recommended SHA256, but I prefer to hear what others might suggest.
Generation - Does hash(SALT + AUTO_INCREMENT_PK + CREATED_TIMESTAMP) suffice? More entropy?
I'd use email, as it is unique for each user, however the user can modify their email address. I was also considering storing signup_email so that hashes would not have to be re-calculated.
MySQL Storage - Currently, our ID's are INT(255) auto_increment primary key's. As stated earlier, potentially hundreds of millions of keys. Depending on the crypto algo, I should have a fixed-size ID. Can I keep INT(255) or should I use CHAR(n)?
---------------------- Thanks for reading :) -------------------------------
One thing: If you don't trust the users with their IDs, sending them over GET or POST will not work; those are all visible to motivated users.
I would use SHA256 using a salt.counter.time string, and use the output to generate GUIDs for the actual id. This would minimize the possibility for collisions.
You will have to use CHAR for MySQL to store GUIDs.
See the comments at http://us2.php.net/manual/en/function.uniqid.php for more in-depth info. AFAIK GUID is not part of the PHP core so you have to fake it a bit.
If you are using user id as the way to allow a user to do anything with your service, if one user "guesses" the user id of another one, he'll be able to do whatever he wants with that one's account ?
You do not have any kind of other password or anything to go along that ?
Well, in that case, you need something quite unique, don't you ;-)
(Hoping I understood the question well -- but that might not be the case -- sorry, if it isn't)
What do you think of using Globally Unique Identifier (like, for instance, 61350955-9755-4AF3-8C19-6DBC42CA69E2) for your users ?
For an example of how they look like, take a look at http://createguid.com/
As a sidenote, that GUID is quite long ; which means lots of bytes in your DB, if you have millions users... So, it probably shouldn't be used as any kind of primary/foreign key.
What about using the smallest possible integer (that fits the number of users you'll have) as primary/foreign key, as that one will be duplicated in many places of the application ; and only have the "long user id" stored only once, in your user table ?
I wrote this class that gives you an unique id of 24 chars, compatible with the id field of MongoDB (and using the same logic to construct it). Might be useful in the future.
<?php
/**
* Generator for Mongo-like ObjectIds in pure PHP
* Author: Mauricio Piacentini
*
* Inspired by https://github.com/justaprogrammer/ObjectId.js
*
*/
class ObjectIdFactory
{
private $_datetime = null;
private $_machine = null;
private $_pid = null;
private $_increment = null;
public function __construct()
{
$this->_machine = str_pad(dechex(rand(0, 16777215)), 6, "0", STR_PAD_LEFT);
$this->_pid = str_pad(dechex(rand(0, 32767)), 4, "0", STR_PAD_LEFT);
$this->_increment = rand(0, 16777215);
//We need a DateTime object to get timestamps, cache it
$this->_datetime = new DateTime();
}
public function getNewId($forcedincrement = null)
{
if (is_null($forcedincrement)) {
$this->_increment++;
if ($this->_increment > 0xffffff) {
$this->_increment = 0;
}
} else {
$this->_increment = $forcedincrement;
}
$timestamp = $this->_datetime->getTimestamp();
$timestamp_final = str_pad(dechex($timestamp), 8, "0", STR_PAD_LEFT);
$increment_final = str_pad(dechex($this->_increment), 6, "0", STR_PAD_LEFT);
return $timestamp_final . $this->_machine . $this->_pid . $increment_final;
}
}
https://github.com/piacentini/ObjectId.php
Have you looked into using a UUID?
A quick google search yields some good resources/links.
Personally I use md5(uniqid(mt_rand(), true)) which will create 32 character identifier (a 128 bit hex number) that is extremely difficult to predict.