Related
I have function which checks if value exists, in this case it is API key. What I am trying to achieve is, before creating new api key for each account registration, I want to loop my function to generate new key if existing already in database. Key is simple string generated using:
$apiKey = bin2hex(random_bytes(16));
My function:
function apiCheckKey($apiKey) {
global $conn;
$sql = "SELECT * FROM `api` WHERE `key` = '".$apiKey."'";
$result = mysqli_query($conn, $sql);
if (mysqli_num_rows($result)) {
return true;
} else {
return false;
}
}
My check:
if(!apiCheckKey($apiKey)) {
// loop
}
How can I run loop efficiently to generate new key and eliminate duplicates? Please keep in mind, database will contain 100 000+ records...
There's a few things to keep in mind when doing this:
Ensure you have a UNIQUE constraint on this column so it's impossible to add duplicate values. You can't rely on a SELECT COUNT(*) FROM x WHERE key=? test before inserting as that's vulnerable to race conditions.
Generate an API key that's sufficiently random that collisions are unlikely. A >=20 character random string using all letters, both upper and lower case, plus numbers will have 704,423,425,546,998,022,968,330,264,616,370,176 possible forms so a collision is astronomically unlikely. If you have shorter keys collisions become a lot more probable due to effects like the pigeonhole principle and the birthday paradox.
In the unlikely event a collision does occur, make your code generate a new key and retry the insert. A UNIQUE constraint violation is a very specific MySQL error code you can handle. Check your error value if/when the INSERT fails and dispatch accordingly.
Test your code by generating a few million keys to be sure it's operating properly.
I've the following code:
$salt=uniqid(mt_rand(), false);
#Add data to tables
mysql_query("INSERT INTO accounts VALUES('$user', '".hash('sha512',$pass+$salt)."', '$salt', '$cookie_value')");
mysql_query("INSERT INTO passwordreset VALUES('$user', NULL, NULL)");
#cookie creation
#.....
#cookie update
mysql_query("UPDATE accounts SET cookie='$cookie_value' WHERE user='$user'");
I sanitize data from form using these functions:
$var = htmlentities($var, ENT_QUOTES, "UTF-8");
return mysql_real_escape_string($var);
Today I logged into phpMyAdmin and I saw that passwords and salts for all users are the same.
Don't remind me about deprecated mysql_* I know it, that's just quick draft.
String concatenation in PHP uses . not +. Thus:
hash('sha512',$pass+$salt)
Should be
hash('sha512',$pass.$salt) // or
hash('sha512',"${pass}${salt}")
This is PHP, $pass+$salt should be $pass . $salt
There's a few things I would comment on your current code:
Using the + operator (as opposed to .) on two strings results in the sum of both values cast to integer (if a string is not numeric it's cast to int(0)); when it's passed to hash() it gets cast to a string again, so your passwords will typically all be sha512("0"). I'm not sure why your salts all have the same value though, unless the column data type is INT in your database.
You can use uniqid(mt_rand(), true) to collect more entropy, resulting in a better salt.
You should hash passwords with a dedicated password hash, such as the BlowFish option in crypt() (make sure your column width is big enough); this way you can get rid of the salt column and you can choose how much work is required to verify the hash in a backward compatible manner.
The cookie column is for an auto-login feature I assume? It's better to create a separate table for this containing a randomized string as the primary key and a foreign key to your users table. This way you can support auto-login from multiple browsers.
error here
$pass+$salt
should be
$pass.$salt
. is used for string catenation in php
How would I go about generating this... I want to keep my primary key sequential and have a 12 digit unique pin generated for each new object added to the database.
The reason it cant just be autoincrement is i don't want the sequential numbers to be easily guessable.
It needs to be integer numbers, because I'm going to have verification codes that need to be dialed on a phone pad.
Use a concatenation of a unique incremented number and a randomly generated number.
The unique incremented number ensures that the result is unique, and the randomly generated number makes it hardly guessable.
This is simple and guaranteed to have no collision (1). The result is incremental, partly random, and non-predictable (provided that the random number part is generated with a good PRNG).
(1): You have to either pad id and random with zeros, or to separate them with some non-digit character.
With a MySQL db, this translates to:
CREATE TABLE foo (
id int not null auto_increment,
random int not null,
...
primary key (id)
);
Maybe you can use UUID_SHORT(). Not 12 digits long, but still could be a viable option:
mysql> select uuid_short();
+-------------------+
| uuid_short() |
+-------------------+
| 22048742962102272 |
+-------------------+
So:
INSERT INTO `table` (`id`, `text`) VALUES (UUID_SHORT(), 'hello world!');
Note: If you really want to have exactly 12 digits, then don't even try to substring the result, if would not ensure the uniqueness of the identifier and may cause collisions.
<?php
$allowed_characters = array(1,2,3,4,5,6,7,8,9,0);
for($i = 1;$i <= 12; $i++){
$pass .= $allowed_characters[rand(0, count($allowed_characters) - 1)];
}
echo $pass;
?>
demo: http://sandbox.phpcode.eu/g/c0190/4
Generally, I will prefer to do something a little bit more low tech. I obscure the values in PHP and leave them as auto-incrementing in JS.
$seeds = array( /*series 100 of very large >= 10-digit numbers*/ );
$seedID = rand( count( $seeds ) ); // randomly choose one of those.
// a string combination which represents the ID + some hash.
$id = bcadd( $seeds[ $seedID ], /* id retrieved from database */ );
// make sure we've not accidentally passed the 10^12 point
$id = bcmod( $id, 1000000000000 );
// make sure to pad
$id = str_pad('' . $id, 3, "0", STR_PAD_LEFT);
$outID = substr( $id, 0, 5 ) . $seedID . substr( $id, 6 );
Then, when receiving the ID from the user:
$seedID = substr( $outID, 6, 2 );
$tmpID = substr( $outID, 0, 5 ) . substr( $outID, 8 );
$id = bcsub( $tmpID, $seeds[ $seedID ] );
// we passed the modulus se we need to add this back in.
if( $id < 0 ) $id = bcmod( bcadd( $id, 1000000000000 ), 1000000000000 );
This will basically mean that you're simply obscuring whatever number you want -- you can use auto_increment with impunity!
One method would be to take your primary key value, salt it with a few other random-ish bits of data (username, current time, process ID, fixed string, etc...) and hash it with md5 or sha1. You then take the hash string and convert it into digits via basic string operations. That'll give you a relatively unique numeric code.
of course, with only 12 digits, you're far more likely to end up with a collision than by using the raw string hash - but since you're requiring this to be dialed on a keypad, it's an acceptable tradeoff.
If the pins are invalidated/deleted after usage, then the collision chances will be much reduced.
You want two things
Uniqueness
Incremental
If you want both the things from same sequence you will run out of luck (literally)
Uniqueness is guaranteed by having large sample space + random + check-unique. Which means, the actual number could be anywhere in between the sample space.
But if you want unique + incremental property, you are dividing sample space by 2. In 64 tries you would have reduced a 64 bit int sample space to 1 bit sample space.
Good luck !
All solutions so far lack one thing essential to your application: Security!
You said you will be using these numbers as a (product) verification code - so you really, really want this to be unpredictable, otherwise it will get exploited.
Neither MySQL's built-in RANDOM function nor any of the random functions PHP provides today are secure random functions. They behave pseudo-randomly, alright, but they all are predictable!
Your only chance is to whip up something of your own using /dev/urandom on a *nix machine or leveraging the Crypto API on Windows. OpenSSL does provide secure random numbers based on these mechanisms - you could reuse this either in a C extension for PHP or by reading the output from a command line script called from PHP. See also this answer.
About your requirement for the numbers to be sequential - is this really so important? It does complicate things enormously. Otherwise you would be good to go with a simple secure 6 byte random number encoded to a string using hex encoding (yielding a 12 character string). Although I would recommend making it 10 bytes and 20 characters to be safer.
But if you want to be sequential, which I interpret as monotonously increasing (because a simple +1 would be trivially predictable), this makes things just so much more complicated. And you don't gain anything from this complexity, the only thing that might happen is that you break the security by inventing some obscure scheme that is easily exploitable.
My suggestion: Add another column that acts as a plain old auto-incremented ID and add the code as a random number constructed as above as a separate column. As far as I see, there's no need to require the product activation code to be the ID at the same time.
I am developing a mysql database.
I "need" a unique id for each user but it must not auto increment! It is vital it is not auto increment.
So I was thinking of inserting a random number something like mt_rand(5000, 1000000) into my mysql table when a user signs up for my web site to be. This is where I am stuck?!
The id is a unique key on my mysql table specific to each user, as I can not 100% guarantee that inserting mt_rand(5000, 1000000) for the user id will not incoherently clash with another user's id.
Is there a way in which I can use mt_rand(5000, 1000000) and scan the mysql database, and if it returns true that it is unique, then insert it as the user's new ID, upon returning false (somebody already has that id) generate a new id until it becomes unique and then insert it into the mysql database.
I know this is possible I have seen it many times, I have tried with while loops and all sorts, so this place is my last resort.
Thanks
You're better off using this: http://dev.mysql.com/doc/refman/5.0/en/miscellaneous-functions.html#function_uuid
Or using this: http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
But if you actually want to do what you are saying, you can just do something like:
$x;
do {
$x = random_number();
"SELECT count(*) FROM table WHERE id = $x"
} while (count != 0);
// $x is now a value that's not in the db
You could use a guid. That's what I've seen done when you can't use an auto number.
http://php.net/manual/en/function.com-create-guid.php
Doesn't this function do what you want (without verification): http://www.php.net/manual/en/function.uniqid.php?
I think you need to approach the problem from a different direction, specifically why a sequence of incrementing numbers is not desired.
If it needs to be an 'opaque' identifier, you can do something like start with a simple incrementing number and then add something around it to make it look like it's not, such as three random numbers on the end. You could go further than that and put some generated letters in front (either random or based on some other algorithm, such as the day of the month they first registered, or which server they hit), then do a simple checksuming algorithm to make another letter for the end. Now someone can't easily guess an ID and you have a way of rejecting one sort of ID before it hits the database. You will need to store the additional data around the ID somewhere, too.
If it needs to be a number that is random and unique, then you need to check the database with the generated ID before you tell the new user. This is where you will run into problems of scale as too small a number space and you will get too many collisions before the check lucks upon an unallocated one. If that is likely, then you will need to divide your ID generation into two parts: the first part is going to be used to find all IDs with that prefix, then you can generate a new one that doesn't exist in the set you got from the DB.
Random string generation... letters, numbers, there are 218 340 105 584 896 combinations for 8 chars.
function randr($j = 8){
$string = "";
for($i=0;$i < $j;$i++){
srand((double)microtime()*1234567);
$x = mt_rand(0,2);
switch($x){
case 0:$string.= chr(mt_rand(97,122));break;
case 1:$string.= chr(mt_rand(65,90));break;
case 2:$string.= chr(mt_rand(48,57));break;
}
}
return $string;
}
Loop...
do{
$id = randr();
$sql = mysql_query("SELECT COUNT(0) FROM table WHERE id = '$id'");
$sql = mysql_fetch_array($sql);
$count = $sql[0];
}while($count != 0);
For starters I always prefer to do all the randomization in php.
function gencode(){
$tempid=mt_rand(5000, 1000000);
$check=mysql_fetch_assoc(mysql_query("SELECT FROM users WHERE id =$tempid",$link));
if($check)gencode();
$reg=mysql_query("INSERT INTO users id VALUES ('$tempid')",$link);
//of course u can check for if $reg then insert successfull
I'm programming a script using PHP and MySQL and I want to get a
unique id (consisting of a string: capitals and small
letters with numbers) like: gHYtUUi5b.
I found many functions in PHP that can generate such numbers but I'm afraid about how to ensure the id is unique!
UPDATE: uuid is long, I mean such id like: (P5Dc) an 11 alphanumeric char.
EDIT: This answer has been flagged for being dangerous in the context of destroying a database. Do NOT use this code to generate unique ids in databases!
I use UUID() to create a unique value.
example:
insert into Companies (CompanyID, CompanyName) Values(UUID(), "TestUUID");
You may like the way that we do it. I wanted a reversible unique code that looked "random" -a fairly common problem.
We take an input number such as 1,942.
Left pad it into a string: "0000001942"
Put the last two digits onto the front: "4200000019"
Convert that into a number: 4,200,000,019
We now have a number that varies wildly between calls and is guaranteed to be less than 10,000,000,000. Not a bad start.
Convert that number to a Base 34 string: "2oevc0b"
Replace any zeros with 'y' and any ones with 'z': "2oevcyb"
Upshift: "2OEVCYB"
The reason for choosing base 34 is so that we don't worry about 0/O and 1/l collisions. Now you have a short random-looking key that you can use to look up a LONG database identifier.
A programmatic way can be to:
add a UNIQUE INDEX to the field
generate a random string in PHP
loop in PHP ( while( ! DO_THE_INSERT ) )
generate another string
Note:
This can be dirty, but has the advantage to be DBMS-agnostic
Even if you choose to use a DBMS specific unique ID generator function (UUID, etc)
it is a best practice to assure the field HAS to be UNIQUE, using the index
the loop is statistically not executed at all, it is entered only on insert failure
If you use MySQL with version higher than 5.7.4, you can use the newly added RANDOM_BYTES function:
SELECT TO_BASE64(RANDOM_BYTES(16));
This will result in a random string such as GgwEvafNLWQ3+ockEST00A==.
How you generate the unique_ids is a useful question - but you seem to be making a counter productive assumption about when you generate them!
My point is that you do not need to generate these unique id's at the time of creating your rows, because they are essentially independent of the data being inserted.
What I do is pre-generate unique id's for future use, that way I can take my own sweet time and absolutely guarantee they are unique, and there's no processing to be done at the time of the insert.
For example I have an orders table with order_id in it. This id is generated on the fly when the user enters the order, incrementally 1,2,3 etc forever. The user does not need to see this internal id.
Then I have another table - unique_ids with (order_id, unique_id). I have a routine that runs every night which pre-loads this table with enough unique_id rows to more than cover the orders that might be inserted in the next 24 hours. (If I ever get 10000 orders in one day I'll have a problem - but that would be a good problem to have!)
This approach guarantees uniqueness and takes any processing load away from the insert transaction and into the batch routine, where it does not affect the user.
Use UUID function.
I don't know the source of your procedures in PHP that generates unique values. If it is library function they should guarantee that your value is really unique. Check in documentation. You should, hovewer, use this function all the time. If you, for example, use PHP function to generate unique value, and then you decide to use MySQL function, you can generate value that already exist. In this case putting UNIQUE INDEX on the column is also a good idea.
DELIMITER $$
USE `temp` $$
DROP PROCEDURE IF EXISTS `GenerateUniqueValue`$$
CREATE PROCEDURE `GenerateUniqueValue`(IN tableName VARCHAR(255),IN columnName VARCHAR(255))
BEGIN
DECLARE uniqueValue VARCHAR(8) DEFAULT "";
DECLARE newUniqueValue VARCHAR(8) DEFAULT "";
WHILE LENGTH(uniqueValue) = 0 DO
SELECT CONCAT(SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1)
) INTO #newUniqueValue;
SET #rcount = -1;
SET #query=CONCAT('SELECT COUNT(*) INTO #rcount FROM ',tableName,' WHERE ',columnName,' like ''',newUniqueValue,'''');
PREPARE stmt FROM #query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
IF #rcount = 0 THEN
SET uniqueValue = #newUniqueValue ;
END IF ;
END WHILE ;
SELECT uniqueValue;
END$$
DELIMITER ;
And call the stored procedure as GenerateUniqueValue('tableName','columnName'). This will give you a 8 digit unique character everytime.
To get unique and random looking tokens you could just encrypt your primary key i.e.:
SELECT HEX(AES_ENCRYPT(your_pk,'your_password')) AS 'token' FROM your_table;
This is good enough plus its reversable so you'd not have to store that token in your table but to generate it instead.
Another advantage is once you decode your PK from that token you do not have to do heavy full text searches over your table but simple and quick PK search.
Theres one small problem though. MySql supports different block encryption modes which if changed will completely change your token space making old tokens useless...
To overcome this one could set that variable before token generated i.e.:
SET block_encryption_mode = 'aes-256-cbc';
However that a bit waste... The solution for this is to attach an encryption mode used marker to the token:
SELECT CONCAT(CONV(CRC32(##GLOBAL.block_encryption_mode),10,35),'Z',HEX(AES_ENCRYPT(your_pk,'your_password'))) AS 'token' FROM your_table;
Another problem may come up if you wish to persist that token in your table on INSERT because to generate it you need to know primary_key for the record which was not inserted yet... Ofcourse you might just INSERT and then UPDATE with LAST_INSERT_ID() but again - theres a better solution:
INSERT INTO your_table ( token )
SELECT CONCAT(CONV(CRC32(##GLOBAL.block_encryption_mode),10,35),'Z',HEX(AES_ENCRYPT(your_pk,'your_password'))) AS 'token'
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = "your_table";
One last but not least advantage of this solution is you can easily replicate it in php, python, js or any other language you might use.
Below is just for reference of numeric unique random id...
it may help you...
$query=mysql_query("select * from collectors_repair");
$row=mysql_num_rows($query);
$ind=0;
if($row>0)
{
while($rowids=mysql_fetch_array($query))
{
$already_exists[$ind]=$rowids['collector_repair_reportid'];
}
}
else
{
$already_exists[0]="nothing";
}
$break='false';
while($break=='false'){
$rand=mt_rand(10000,999999);
if(array_search($rand,$alredy_exists)===false){
$break='stop';
}else{
}
}
echo "random number is : ".$echo;
and you can add char with the code like -> $rand=mt_rand(10000,999999) .$randomchar; // assume $radomchar contains char;
For uniqueness what I do is I take the Unix timestamp and append a random string to it and use that.
<?php
$hostname_conn = "localhost";
$database_conn = "user_id";
$username_conn = "root";
$password_conn = "";
$conn = mysql_pconnect($hostname_conn, $username_conn, $password_conn) or trigger_error(mysql_error(),E_USER_ERROR);
mysql_select_db($database_conn,$conn);
// run an endless loop
while(1) {
$randomNumber = rand(1, 999999);// generate unique random number
$query = "SELECT * FROM tbl_rand WHERE the_number='".mysql_real_escape_string ($randomNumber)."'"; // check if it exists in database
$res =mysql_query($query,$conn);
$rowCount = mysql_num_rows($res);
// if not found in the db (it is unique), then insert the unique number into data_base and break out of the loop
if($rowCount < 1) {
$con = mysql_connect ("localhost","root");
mysql_select_db("user_id", $con);
$sql = "insert into tbl_rand(the_number) values('".$randomNumber."')";
mysql_query ($sql,$con);
mysql_close ($con);
break;
}
}
echo "inserted unique number into Data_base. use it as ID";
?>
crypt() as suggested and store salt in some configuration file, Start salt from 1 and if you find duplicate move to next value 2. You can use 2 chars, but that will give you enough combination for salt.
You can generate string from openssl_random_pseudo_bytes(8). So this should give random and short string (11 char) when run with crypt().
Remove salt from result and there will be only 11 chars that should be enough random for 100+ millions if you change salt on every fail of random.
You might also consider using crypt()* to generate a [nearly-guaranteed] unique ID inside your contraints.
USE IT
$info = random_bytes(16);
$info[6] = chr(ord($info[6]) & 0x0f | 0x40);
$info[8] = chr(ord($info[8]) & 0x3f | 0x80);
$result =vsprintf('%s%s-%s-%s-%s-%s%s%s', str_split(bin2hex($info), 4));
return $result;
This generates random ids:
CREATE TABLE Persons (
ID Integer PRIMARY KEY AUTOINCREMENT,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int
);
You could use Twitter's snowflake.
In short, it generates a unique id based on time, server id and a sequence. It generates a 64-bit value so it is pretty small and it fits in an INT64. It also allows for sorting values correctly.
https://developer.twitter.com/en/docs/basics/twitter-ids
In sum, it allows multiple servers, highly concurrency, sorting value and all of them in 64 bits.
Here it is the implementation for MySQL
https://github.com/EFTEC/snowflake-mysql
It consists of a function and a table.