Related
I have a database table with various fields involving jobs done on ships including a field named created which uses DATE format. The result i want to achieve is to have a unique reference number for each job. The format i want to use for this reference number is:
example : Lets say the date of the job is 23/11/2013 like today. Then the number would be 1311/1 the next job 1311/2 and goes on. If the month changes and the date of the next job is for example 15/12/2013 the refence number i would like to have if its the first job of the month is 1312/1.
So the two first digits of my reference number would show the year,the next two the month and the number after the slash i would like it to be an auto_increment number that will reset each month.My code so far is :
$job_num = 1;
foreach($random as $rand) {
$vak = $rand->created;
$gas = $rand->id;
$vak1 = substr($vak, 2, 2);
$vak2 = substr($vak, 5, -3);
$vak3 = substr($vak, 8, 10);
if(date(j) > 1) {
echo $vak1.$vak2.'/'.$job_num.'<br>';
$job_num++;
} else {
$job_num = 1;
echo $vak1.$vak2.'/'.$job_num.'<br>';
$job_num++;
}
}
So as u can see i want to achieve all this inside a foreach statement. And although the above code kinda works,the porblem i have is that at the 1st of any month in other words when date(j) = 1 if i insert more than one job in my database the $job_num variable resets as many times as the jobs i have inserted resulting in identical refence numbers.
I am really new in programming and php so if anyone could help me solve this, i would really appreciate it.
Thanks in advance:)
You can't do this with the auto-increment mechanism if you use InnoDB, which is MySQL's default storage engine.
You can do it with the MyISAM storage engine, but you really shouldn't use MyISAM, for many reasons.
So you'll have to assign the repeating numbers yourself. This means you have to lock the table while you check what is the current maximum number for the given month, then insert a new row with the next higher number.
If that seems like it would impair concurrent access to the table, you're right. Keep in mind that MyISAM does a table-lock during insert/update/delete of any row.
If you can use the MyISAM engine, you can get this behavior without procedural code.
create table demo (
yr_mo integer not null,
id integer auto_increment,
other_columns char(1) default 'x',
primary key (yr_mo, id)
) engine=MyISAM;
insert into demo (yr_mo) values (1311);
insert into demo (yr_mo) values (1311);
insert into demo (yr_mo) values (1311);
insert into demo (yr_mo) values (1311);
insert into demo (yr_mo) values (1312);
The last INSERT statement starts a new month.
Now if you look at the autoincrement values the MyISAM engine assigned . . .
select * from demo;
YR_MO ID OTHER_COLUMNS
--
1311 1 x
1311 2 x
1311 3 x
1311 4 x
1312 1 x
This is MyISAM's documented behavior; look for "MyISAM Notes".
If you want the form yymm/n for presentation, use something like this.
select concat(yr_mo, '/', id) as cat_key
from demo;
function generateRandomData(){
# $db = new mysqli('localhost','XXX','XXX','scores');
if(mysqli_connect_errno()) {
echo 'Failed to connect to database. Please try again later.';
exit;
}
$query = "insert into scoretable values(?,?,?)";
for($a = 0; $a < 1000000; $a++)
{
$stmt = $db->prepare($query);
$id = rand(1,75000);
$score = rand(1,100000);
$time = rand(1367038800 ,1369630800);
$stmt->bind_param("iii",$id,$score,$time);
$stmt->execute();
}
}
I am trying to populate a data table in mysql with a million rows of data. However, this process is extremely slow. Is there anything obvious I'm doing wrong that I could fix in order to make it run faster?
As hinted in the comments, you need to reduce the number of queries by catenating as many inserts as possible together. In PHP, it is easy to achieve that:
$query = "insert into scoretable values";
for($a = 0; $a < 1000000; $a++) {
$id = rand(1,75000);
$score = rand(1,100000);
$time = rand(1367038800 ,1369630800);
$query .= "($id, $score, $time),";
}
$query[strlen($query)-1]= ' ';
There is a limit on the maximum size of queries you can execute, which is directly related to the max_allowed_packet server setting (This page of the mysql documentation describes how to tune that setting to your advantage).
Therfore, you will have to reduce the loop count above to reach an appropriate query size, and repeat the process to reach the total number you want to insert, by wrapping that code with another loop.
Another practice is to disable check constraints on the table you wish to do bulk insert:
ALTER TABLE yourtablename DISABLE KEYS;
SET FOREIGN_KEY_CHECKS=0;
-- bulk insert comes here
SET FOREIGN_KEY_CHECKS=1;
ALTER TABLE yourtablename ENABLE KEYS;
This practice however must be done carefully, especially in your case since you generate the values randomly. If you have any unique key within the columns you generate, you cannot use that technique with your query as it is, as it may generate a duplicate key insert. You probably want to add a IGNORE clause to it:
$query = "insert INGORE into scoretable values";
This will cause the server to silently ignore duplicate entries on unique keys. To reach the total number of requiered inserts, just loop as many time as needed to fill up the remaining missing lines.
I suppose that the only place where you could have a unique key constraint is on the id column. In that case, you will never be able to reach the number of lines you wish to have, since it is way above the range of random values you generate for that field. Consider raising that limit, or better yet, generate your ids differently (perhaps simply by using a counter, which will make sure every record is using a different key).
You are doing several things wrong. First thing you have to take into account is what MySQL engine you're using.
The default one is InnoDB, previously the default engine is MyISAM.
I'll write this answer under assumption you're using InnoDB, which you should be using for plethora of reasons.
InnoDB operates in something called autocommit mode. That means that every query you make is wrapped in a transaction.
To translate that to a language that us mere mortals can understand - every query you do without specifying BEGIN WORK; block is a transaction - ergo, MySQL will wait until hard drive confirms data has been written.
Knowing that hard drives are slow (mechanical ones are still the ones most widely used), that means your inserts will be as fast as the hard drive is. Usually, mechanical hard drives can perform about 300 input output operations per second, ergo assuming you can do 300 inserts a second - yes, you'll wait quite a bit to insert 1 million records.
So, knowing how things work - you can use them to your advantage.
The amount of data that the HDD will write per transaction will be generally very small (4KB or even less), and knowing today's HDDs can write over 100MB/sec - that indicates that we should wrap several queries into a single transaction.
That way MySQL will send quite a bit of data and wait for the HDD to confirm it wrote everything and that the whole world is fine and dandy.
So, assuming you have 1M rows you want to populate - you'll execute 1M queries. If your transactions commit 1000 queries at a time, you should perform only about 1000 write operations.
That way, your code becomes something like this:
(I am not familiar with mysqli interface so function names might be wrong, and seeing I'm typing without actually running the code - the example might not work so use it at your own risk)
function generateRandomData()
{
$db = new mysqli('localhost','XXX','XXX','scores');
if(mysqli_connect_errno()) {
echo 'Failed to connect to database. Please try again later.';
exit;
}
$query = "insert into scoretable values(?,?,?)";
// We prepare ONCE, that's the point of prepared statements
$stmt = $db->prepare($query);
$start = 0;
$top = 1000000;
for($a = $start; $a < $top; $a++)
{
// If this is the very first iteration, start the transaction
if($a == 0)
{
$db->begin_transaction();
}
$id = rand(1,75000);
$score = rand(1,100000);
$time = rand(1367038800 ,1369630800);
$stmt->bind_param("iii",$id,$score,$time);
$stmt->execute();
// Commit on every thousandth query
if( ($a % 1000) == 0 && $a != ($top - 1) )
{
$db->commit();
$db->begin_transaction();
}
// If this is the very last query, then we just need to commit and end
if($a == ($top - 1) )
{
$db->commit();
}
}
}
DB querying involves many interrelated tasks. As a result it is an 'expensive' process. It is even more 'expensive' when it comes to insertion/update.
Running query once is the best way to enhance performance.
You can prepare the statements in the loop and run it once.
eg.
$query = "insert into scoretable values ";
for($a = 0; $a < 1000000; $a++)
{
$values = " ('".$?."','".$?."','".$?."'), ";
$query.=$values;
...
}
...
//remove the last comma
...
$stmt = $db->prepare($query);
...
$stmt->execute();
Have a look at this gist I've created. It takes about 5 minutes to insert a million rows on my laptop.
How can I automatically generate a unique, random 6 digit number to insert into a column of a mysql table? The randomly generated number must not already exist in the column.
I am accessing mysql via php.
The table format is like so, with the random number going in the reqnumber column:
id,status,reqnumber
function gen(){
$num = rand(100000,999999);
if($num == ifnumberinyourdatabase){
gen();
}
return $num;
}
You can also use recursive function here.
which check's if number is your database if it is generate new one if not return the unique number
function gen(){
$num = rand(100000,999999);
$query_idgetrs = "SELECT * FROM servicetbl where reqnumber = $num";
$idgetrs = mysql_query($query_idgetrs, $dbconnection) or die(mysql_error());
$row = mysql_num_rows($idgetrs);
if($row >= 1){
gen();
}
return $num;
}
Just generate a random number and then use str_pad():
$myRandom = str_pad(rand(1,999999), 6, '0', STR_PAD_LEFT);
The problem that you're going to run into is that since you require this to be random, there's no way to know if it exists in the table until it's generated. You'd have to make a loop and keep checking in DB.
Put unique constraint on reqnumber field and put error handling code in PHP
Although random is ok, please note that 6 digits only offers 1 million combinations. I'm not sure how long it would be before you started getting duplicate primary key errors.
A much better solution would be to use a unique value. This is very different to a random value as the unique value guarantees to by different every time. MySql has the auto_increment datatype to help you with this. Unfortunately, you are still limited to 1 million entries when using 6 digits.
If you want a totally random, long identifier, check out MySql's UUID function. It will generate a unique string that is guaranteed to never repeat. However it is much longer than 6 characters because that's what it can take to achieve uniqueness.
A part of your table structure must be:
`id` mediumint(6) AUTO_INCREMENT NOT NULL,
PRIMARY KEY (`id`)
If you really need 6 digits always:
ALTER TABLE tbl AUTO_INCREMENT = 100000;
or use
sprintf()
In MySQL I have to check whether select query has returned any records, if not I insert a record. I am afraid though that the whole if-else operation in PHP scripts is NOT as atomic as I would like, i.e. will break in some scenarios, for example if another instance of the script is called where the same record needs to be worked with:
if(select returns at least one record)
{
update record;
}
else
{
insert record;
}
I did not use transactions here, and autocommit is on. I am using MySQL 5.1 with PHP 5.3. The table is InnoDB. I would like to know if the code above is suboptimal and indeed will break. I mean the same script is re-entered by two instances and the following query sequence occurs:
instance 1 attempts to select the record, finds none, enters the block for insert query
instance 2 attempts to select the record, finds none, enters the block for insert query
instance 1 attempts to insert the record, succeeds
instance 2 attempts to insert the record, fails, aborts the script automatically
Meaning that instance 2 will abort and return an error, skipping anything following the insert query statement. I could make the error not fatal, but I don't like ignoring errors, I would much rather know if my fears are real here.
Update: What I ended up doing (is this ok for SO?)
The table in question assists in a throttling (allow/deny, really) amount of messages the application sends to each recipient. The system should not send more than X messages to a recipient Y within a period Z. The table is [conceptually] as follows:
create table throttle
(
recipient_id integer unsigned unique not null,
send_count integer unsigned not null default 1,
period_ts timestamp default current_timestamp,
primary key (recipient_id)
) engine=InnoDB;
And the block of [somewhat simplified/conceptual] PHP code that is supposed to do an atomic transaction that maintains the right data in the table, and allows/denies sending message depending on the throttle state:
function send_message_throttled($recipient_id) /// The 'Y' variable
{
query('begin');
query("select send_count, unix_timestamp(period_ts) from throttle where recipient_id = $recipient_id for update");
$r = query_result_row();
if($r)
{
if(time() >= $r[1] + 60 * 60 * 24) /// The numeric offset is the length of the period, the 'Z' variable
{/// new period
query("update throttle set send_count = 1, period_ts = current_timestamp where recipient_id = $recipient_id");
}
else
{
if($r[0] < 5) /// Amount of messages allowed per period, the 'X' variable
{
query("update throttle set send_count = send_count + 1 where recipient_id = $recipient_id");
}
else
{
trigger_error('Will not send message, throttled down.', E_USER_WARNING);
query('rollback');
return 1;
}
}
}
else
{
query("insert into throttle(recipient_id) values($recipient_id)");
}
if(failed(send_message($recipient_id)))
{
query('rollback');
return 2;
}
query('commit');
}
Well, disregarding the fact that InnoDB deadlocks occur, this is pretty good no? I am not pounding my chest or anything, but this is simply the best mix of performance/stability I can do, short of going with MyISAM and locking entire table, which I don't want to do because of more frequent updates/inserts vs selects.
It seems like you already know the answer to the question, and how to solve your problem. It is a real problem, and you can use one of the following to solve it:
SELECT ... FOR UPDATE
INSERT ... ON DUPLICATE KEY UPDATE
transactions (don't use MyIsam)
table locks
This can and might happen depending on how often this page is executed.
The safe bet would be to use transactions. The same thing you wrote would still happen, except that you could safely check for the error inside the transaction (In case the insertion involves several queries, and only the last insert breaks) allowing you to rollback the one that became invalid.
So (pseudo):
#start transaction
if (select returns at least one record)
{
update record;
}
else
{
insert record;
}
if (no constraint errors)
{
commit; //ends transaction
}
else
{
rollback; //ends transaction
}
You could lock the table as well but depending on the work you're doing, you'd have to get an exclusive lock on the entire table (you cannot SELECT ... FOR UPDATE non-existing rows, sorry) but that would also block reads from your table until you are finished.
I'm programming a script using PHP and MySQL and I want to get a
unique id (consisting of a string: capitals and small
letters with numbers) like: gHYtUUi5b.
I found many functions in PHP that can generate such numbers but I'm afraid about how to ensure the id is unique!
UPDATE: uuid is long, I mean such id like: (P5Dc) an 11 alphanumeric char.
EDIT: This answer has been flagged for being dangerous in the context of destroying a database. Do NOT use this code to generate unique ids in databases!
I use UUID() to create a unique value.
example:
insert into Companies (CompanyID, CompanyName) Values(UUID(), "TestUUID");
You may like the way that we do it. I wanted a reversible unique code that looked "random" -a fairly common problem.
We take an input number such as 1,942.
Left pad it into a string: "0000001942"
Put the last two digits onto the front: "4200000019"
Convert that into a number: 4,200,000,019
We now have a number that varies wildly between calls and is guaranteed to be less than 10,000,000,000. Not a bad start.
Convert that number to a Base 34 string: "2oevc0b"
Replace any zeros with 'y' and any ones with 'z': "2oevcyb"
Upshift: "2OEVCYB"
The reason for choosing base 34 is so that we don't worry about 0/O and 1/l collisions. Now you have a short random-looking key that you can use to look up a LONG database identifier.
A programmatic way can be to:
add a UNIQUE INDEX to the field
generate a random string in PHP
loop in PHP ( while( ! DO_THE_INSERT ) )
generate another string
Note:
This can be dirty, but has the advantage to be DBMS-agnostic
Even if you choose to use a DBMS specific unique ID generator function (UUID, etc)
it is a best practice to assure the field HAS to be UNIQUE, using the index
the loop is statistically not executed at all, it is entered only on insert failure
If you use MySQL with version higher than 5.7.4, you can use the newly added RANDOM_BYTES function:
SELECT TO_BASE64(RANDOM_BYTES(16));
This will result in a random string such as GgwEvafNLWQ3+ockEST00A==.
How you generate the unique_ids is a useful question - but you seem to be making a counter productive assumption about when you generate them!
My point is that you do not need to generate these unique id's at the time of creating your rows, because they are essentially independent of the data being inserted.
What I do is pre-generate unique id's for future use, that way I can take my own sweet time and absolutely guarantee they are unique, and there's no processing to be done at the time of the insert.
For example I have an orders table with order_id in it. This id is generated on the fly when the user enters the order, incrementally 1,2,3 etc forever. The user does not need to see this internal id.
Then I have another table - unique_ids with (order_id, unique_id). I have a routine that runs every night which pre-loads this table with enough unique_id rows to more than cover the orders that might be inserted in the next 24 hours. (If I ever get 10000 orders in one day I'll have a problem - but that would be a good problem to have!)
This approach guarantees uniqueness and takes any processing load away from the insert transaction and into the batch routine, where it does not affect the user.
Use UUID function.
I don't know the source of your procedures in PHP that generates unique values. If it is library function they should guarantee that your value is really unique. Check in documentation. You should, hovewer, use this function all the time. If you, for example, use PHP function to generate unique value, and then you decide to use MySQL function, you can generate value that already exist. In this case putting UNIQUE INDEX on the column is also a good idea.
DELIMITER $$
USE `temp` $$
DROP PROCEDURE IF EXISTS `GenerateUniqueValue`$$
CREATE PROCEDURE `GenerateUniqueValue`(IN tableName VARCHAR(255),IN columnName VARCHAR(255))
BEGIN
DECLARE uniqueValue VARCHAR(8) DEFAULT "";
DECLARE newUniqueValue VARCHAR(8) DEFAULT "";
WHILE LENGTH(uniqueValue) = 0 DO
SELECT CONCAT(SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1),
SUBSTRING('ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789', RAND()*34+1, 1)
) INTO #newUniqueValue;
SET #rcount = -1;
SET #query=CONCAT('SELECT COUNT(*) INTO #rcount FROM ',tableName,' WHERE ',columnName,' like ''',newUniqueValue,'''');
PREPARE stmt FROM #query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
IF #rcount = 0 THEN
SET uniqueValue = #newUniqueValue ;
END IF ;
END WHILE ;
SELECT uniqueValue;
END$$
DELIMITER ;
And call the stored procedure as GenerateUniqueValue('tableName','columnName'). This will give you a 8 digit unique character everytime.
To get unique and random looking tokens you could just encrypt your primary key i.e.:
SELECT HEX(AES_ENCRYPT(your_pk,'your_password')) AS 'token' FROM your_table;
This is good enough plus its reversable so you'd not have to store that token in your table but to generate it instead.
Another advantage is once you decode your PK from that token you do not have to do heavy full text searches over your table but simple and quick PK search.
Theres one small problem though. MySql supports different block encryption modes which if changed will completely change your token space making old tokens useless...
To overcome this one could set that variable before token generated i.e.:
SET block_encryption_mode = 'aes-256-cbc';
However that a bit waste... The solution for this is to attach an encryption mode used marker to the token:
SELECT CONCAT(CONV(CRC32(##GLOBAL.block_encryption_mode),10,35),'Z',HEX(AES_ENCRYPT(your_pk,'your_password'))) AS 'token' FROM your_table;
Another problem may come up if you wish to persist that token in your table on INSERT because to generate it you need to know primary_key for the record which was not inserted yet... Ofcourse you might just INSERT and then UPDATE with LAST_INSERT_ID() but again - theres a better solution:
INSERT INTO your_table ( token )
SELECT CONCAT(CONV(CRC32(##GLOBAL.block_encryption_mode),10,35),'Z',HEX(AES_ENCRYPT(your_pk,'your_password'))) AS 'token'
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = "your_table";
One last but not least advantage of this solution is you can easily replicate it in php, python, js or any other language you might use.
Below is just for reference of numeric unique random id...
it may help you...
$query=mysql_query("select * from collectors_repair");
$row=mysql_num_rows($query);
$ind=0;
if($row>0)
{
while($rowids=mysql_fetch_array($query))
{
$already_exists[$ind]=$rowids['collector_repair_reportid'];
}
}
else
{
$already_exists[0]="nothing";
}
$break='false';
while($break=='false'){
$rand=mt_rand(10000,999999);
if(array_search($rand,$alredy_exists)===false){
$break='stop';
}else{
}
}
echo "random number is : ".$echo;
and you can add char with the code like -> $rand=mt_rand(10000,999999) .$randomchar; // assume $radomchar contains char;
For uniqueness what I do is I take the Unix timestamp and append a random string to it and use that.
<?php
$hostname_conn = "localhost";
$database_conn = "user_id";
$username_conn = "root";
$password_conn = "";
$conn = mysql_pconnect($hostname_conn, $username_conn, $password_conn) or trigger_error(mysql_error(),E_USER_ERROR);
mysql_select_db($database_conn,$conn);
// run an endless loop
while(1) {
$randomNumber = rand(1, 999999);// generate unique random number
$query = "SELECT * FROM tbl_rand WHERE the_number='".mysql_real_escape_string ($randomNumber)."'"; // check if it exists in database
$res =mysql_query($query,$conn);
$rowCount = mysql_num_rows($res);
// if not found in the db (it is unique), then insert the unique number into data_base and break out of the loop
if($rowCount < 1) {
$con = mysql_connect ("localhost","root");
mysql_select_db("user_id", $con);
$sql = "insert into tbl_rand(the_number) values('".$randomNumber."')";
mysql_query ($sql,$con);
mysql_close ($con);
break;
}
}
echo "inserted unique number into Data_base. use it as ID";
?>
crypt() as suggested and store salt in some configuration file, Start salt from 1 and if you find duplicate move to next value 2. You can use 2 chars, but that will give you enough combination for salt.
You can generate string from openssl_random_pseudo_bytes(8). So this should give random and short string (11 char) when run with crypt().
Remove salt from result and there will be only 11 chars that should be enough random for 100+ millions if you change salt on every fail of random.
You might also consider using crypt()* to generate a [nearly-guaranteed] unique ID inside your contraints.
USE IT
$info = random_bytes(16);
$info[6] = chr(ord($info[6]) & 0x0f | 0x40);
$info[8] = chr(ord($info[8]) & 0x3f | 0x80);
$result =vsprintf('%s%s-%s-%s-%s-%s%s%s', str_split(bin2hex($info), 4));
return $result;
This generates random ids:
CREATE TABLE Persons (
ID Integer PRIMARY KEY AUTOINCREMENT,
LastName varchar(255) NOT NULL,
FirstName varchar(255),
Age int
);
You could use Twitter's snowflake.
In short, it generates a unique id based on time, server id and a sequence. It generates a 64-bit value so it is pretty small and it fits in an INT64. It also allows for sorting values correctly.
https://developer.twitter.com/en/docs/basics/twitter-ids
In sum, it allows multiple servers, highly concurrency, sorting value and all of them in 64 bits.
Here it is the implementation for MySQL
https://github.com/EFTEC/snowflake-mysql
It consists of a function and a table.