In my database (MySQL) I have a table (MyISAM) containing a field called number. Each value of this field is either 0 or a positive number. The non zero values must be unique. And the last thing is that the value of the field is being generated in my php code according to value of another field (called isNew) in this table. The code folows.
$maxNumber = $db->selectField('select max(number)+1 m from confirmed where isNew = ?', array($isNew), 'm');
$db->query('update confirmed set number = ? where dataid = ?', array($maxNumber, $id));
The first line of code select the maximum value of the number field and increments it. The second line updates the record by setting it freshly generated number.
This code is being used concurrently by hundreds of clients so I noticed that sometimes duplicates of the number field occur. As I understand this is happening when two clients read value of the number field almost simultaneously and this fact leads to the duplicate.
I have read about the SELECT ... FOR UPDATE statement but I'm not quite sure it is applicable in my case.
So the question is should I just append FOR UPDATE to my SELECT statement? Or create a stored procedure to do the job? Or maybe completely change the way the numbers are being generated?
This is definitely possible to do. MyISAM doesn't offer transaction locking so forget about stuff like FOR UPDATE. There's definitely room for a race condition between the two statements in your example code. The way you've implemented it, this one is like the talking burro. It's amazing it works at all, not that it works badly! :-)
I don't understand what you're doing with this SQL:
select max(number)+1 m from confirmed where isNew = ?
Are the values of number unique throughout the table, or only within sets where isNew has a certain value? Would it work if the values of number were unique throughout the table? That would be easier to create, debug, and maintain.
You need a multi-connection-safe way of getting a number.
You could try this SQL. It will do the setting of the max number in one statement.
UPDATE confirmed
SET number = (SELECT 1+ MAX(number) FROM confirmed WHERE isNew = ?)
WHERE dataid = ?
This will perform badly. Without a compound index on (isNew, number), and without both those columns declared NOT NULL it will perform very very badly.
If you can use numbers that are unique throughout the table I suggest you create for yourself a sequence setup, which will return a unique number each time you use it. You need to use a series of consecutive SQL statements to do that. Here's how it goes.
First, when you create your tables create yourself a table to use called sequence (or whatever name you like). This is a one-column table.
CREATE TABLE sequence (
sequence_id INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`sequence_id`)
) AUTO_INCREMENT = 990000
This will make the sequence table start issuing numbers at 990,000.
Second, when you need a unique number in your application, do the following things.
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
UPDATE confirmed
SET number = LAST_INSERT_ID()
WHERE dataid = ?
What's going on here? The MySQL function LAST_INSERT_ID() returns the value of the most recent autoincrement-generated ID number. Because you inserted a row into that sequence table, it gives you back that generated ID number. The DELETE FROM command keeps that table from snarfing up disk space; we don't care about old ID numbers.
LAST_INSERT_ID() is connection-safe. If software on different connections to your database uses it, they all get their own values.
If you need to know the last inserted ID number, you can issue this SQL:
SELECT LAST_INSERT_ID() AS sequence_id
and you'll get it returned.
If you were using Oracle or PostgreSQL, instead of MySQL, you'd find they provide SEQUENCE objects that basically do this.
Here's the answer to another similar question.
Fastest way to generate 11,000,000 unique ids
Related
I have a table called INVOICES that receives entries from a PHP script. It has many columns, but the two most relevant are INVOICE_ID and INVOICE_TYPE. Basically the INVOICE_TYPE is a number from 0 to 3, which designates different types of invoices.
Up to this point, everything ran smoothly until two users submitted invoices while the server had a hiccup and wrote both in as the same INVOICE_ID. The reason for this is the PHP script reads the MAX INVOICE_ID of the INVOICE_TYPE, then adds 1, then inserts the new row with that INVOICE_ID. In essence, it is programmatically a primary key. 99.9% of the time it worked, but that one time it was a problem.
I have tried finding SQL solutions but do not have sufficient knowledge of it. I have tried doing it myself in an SQL query to read the MAX, increment, and the insert but just throws an exception that you cannot select and insert from the same table at once.
What I'm wondering is if there is an auto-increment that could be conditional to the INVOICE_TYPE, to only increment if the type is matched. Any suggestions would help at this point.
An unique index over the two columns (INVOICE_ID, INVOICE_TYPE) will make one of such hiccupy queries fail.
CREATE UNIQUE INDEX id_type_unique ON INVOICES (INVOICE_ID, INVOICE_TYPE);
INSERT INTO INVOICES (INVOICE_ID, INVOICE_TYPE) VALUES (1, 5); -- okay
INSERT INTO INVOICES (INVOICE_ID, INVOICE_TYPE) VALUES (1, 5); -- error
If you insert only one row to one table at once simplest solution is to apply unique index on both columns.
CREATE UNIQUE INDEX invoice_id_type_unique
ON INVOICES(INVOICE_ID,INVOICE_TYPE);
But if you execute more queries based on the same data you need to use transactions to prevent modifying/inserting only part of data.
START TRANSACTION;
SELECT #invoice_id:=MAX(INVOICE_ID) FROM INVOICES WHERE INVOICE_TYPE=1;
INSERT INTO INVOICES (...,#invoice_id,...);
...
... #OTHER QUERIES UPDATING DATA
COMMIT;
make (INVOICE_ID) as UNIQUE, this will solved your problem, sql will not allowed duplicate value in same column.
The way i like it is to create a table to handle all the sequences and a stored procedure that i can call with the name of the sequence that i like to know the next value, something similar to:
START TRANSACTION;
SELECT value INTO result FROM Sequences WHERE name like paramSeqName FOR UPDATE;
UPDATE Sequences SET value = value + 1 WHERE name like paramSeqName;
COMMIT;
There is a good example here:
http://www.microshell.com/database/mysql/emulating-nextval-function-to-get-sequence-in-mysql/
Together with my team, I am working on a functionality to generate invoice numbers. The requirements says that:
there should be no gaps between invoice numbers
the numbers should start from 0 every year (the together with the year we will have a unique key)
the invoice numbers should grow accordinlgy to the time of the creation of the invoices
We are using php and postgres. We tought to implement this in the following way:
each time a new invoice is persisted on the database we use a BEFORE INSERT trigger
the trigger executes a function that retrieves a new value from a postgres sequence and writes it on the invoice as its number
Considering that multiple invoices could be created during the same transaction, my question is: is this a sufficiently safe approach? What are its flaws? How would you suggest to improve it?
Introduction
I believe the most crucial point here is:
there should be no gaps between invoice numbers
In this case you cannot use a squence and an auto-increment field (as others propose in the comments). Auto-increment field use sequence under the hood and nextval(regclass) function increments sequence's counter no matter if transaction succeeded or failed (you point that out by yourself).
Update:
What I mean is you shouldn't use sequences at all, especially solution proposed by you doesn't eliminates gap possibility. Your trigger gets new sequence value but INSERT could still failed.
Sequences works this way because they mainly meant to be used for PRIMARY KEYs and OIDs values generation where uniqueness and non-blocking mechanism is ultimate goal and gaps between values are really no big deal.
In your case however the priorities may be different, but there are couple things to consider.
Simple solution
First possible solution to your problem could be returning new number as maximum value of currently existing ones. It can be done in your trigger:
NEW.invoice_number =
(SELECT foo.invoice_number
FROM invoices foo
WHERE foo._year = NEW._year
ORDER BY foo.invoice_number DESC NULLS LAST LIMIT 1
); /*query 1*/
This query could use your composite UNIQUE INDEX if it was created with "proper" syntax and columns order which would be the "year" column in the first place ex.:
CREATE UNIQUE INDEX invoice_number_unique
ON invoices (_year, invoice_number DESC NULLS LAST);
In PostgreSQL UNIQUE CONSTRAINTs are implemented simply as UNIQUE INDEXes so most of the times there no difference which command you will use. However using that particular syntax presented above, makes possible to define order on that index. It's really nice trick which makes /*query 1*/ quicker than simple SELECT max(invoice_number) FROM invoices WHERE _year = NEW.year if the invoice table gets bigger.
This is simple solution but has one big drawback. There is possibility of race condition when two transactions try to insert invoice at the same time. Both could acquire the same max value and the UNIQUE CONSTRAINT will prevent the second one from committing. Despite that it could be sufficient in some small system with special insert policy.
Better solution
You may create table
CREATE TABLE invoice_numbers(
_year INTEGER NOT NULL PRIMARY KEY,
next_number_within_year INTEGER
);
to store next possible number for certain year. Then, in AFTER INSERT trigger you could:
Lock invoice_numbers that no other transaction could even read the number LOCK TABLE invoice_numbers IN ACCESS EXCLUSIVE;
Get new invoice number new_invoice_number = (SELECT foo.next_number_within_year FROM invoice_numbers foo where foo._year = NEW.year);
Update number value of new added invoice row
Increment UPDATE invoice_numbers SET next_number_within_year = next_number_within_year + 1 WHERE _year = NEW._year;
Because table lock is hold by the transaction to its commit, this probably should be the last trigger fired (read more about trigger execution order here)
Update:
Instead of locking whole table with LOCK command check link provided by Craig Ringer
The drawback in this case is INSERT operation performance drop down --- only one transaction at the time can perform insert.
Just looking for some tips and pointers for a small project I am doing. I have some ideas but I am not sure if they are the best practice. I am using mysql and php.
I have a table called nomsing in the database.
It has a primary key called row id which is an integer.
Then I have about 8 other tables referencing this table.
That are called nomplu, accsing,accplu, datsing, datplu for instance.
Each has a column that references the primary key of nomsing.
Withing my php code I have all the information to insert into the tables except one thing , the row id primary key of the nomsing table. So that php generates a series of inserts like the following.
INSERT INTO nomsing(word,postress,gender) VALUES (''велосипед","8","mask").
INSERT INTO nomplu(word,postress,NOMSING?REFERENCE) VALUES (''велосипеды","2",#the reference to the id of the first insert#).
There are more inserts but this one gets the point across. The second insert should reference the auto generated id for the first insert. I was this to work as a transaction so all inserts should complete or none.
One idea I have is to not auto generate the id and generate it myself in php. That way would know the id given before the transaction but then I would have to check if the id was already in the db.
Another idea I have is to do the first insert and then query for the row id of that insert in php and then make the second insert. I mean both should work but they don't seem like an optimal solution. I am not too familiar with the database transactional features but what would be the best approach to do in this case. I don't like the idea of inserting then querying for the id and then running the rest of the queries. Just seems very inefficient or perhaps I am wrong.
Just insert a row in the master table. Then you can fetch the insert id ( lastInserId when on PDO) and use that to populate your other queries.
You could use the php version as given by JvdBerg , or Mysql's LAST_INSERT_ID. I usually use the former option.
See a similar SO question here.
You could add a new column to the nomsing table, called 'insert_order' (or similar) with a default value of 0, then instead of generating one SQL statement per insert create a bulk insert statement e.g.
INSERT INTO nomsing(word,postress,gender, insert_order)
VALUES (''велосипед","8","mask",1), (''abcd'',"9","hat",2).....
you generate the insert_order number with a counter in your loop starting at one. Then you can perform one SELECT on the table to get the ids e.g.
SELECT row_id
FROM nomsing
WHERE insert_order > 0;
now you have all the IDs you can now do a bulk insert for your following queries. At the end of your script just do an update to reset the insert_order column back to 0
UPDATE nomsing SET insert_order = 0 WHERE insert_order > 0;
It may seem messy to add an extra column to do this but it will add a significant speed increase over performing one query at a time.
Okay, so let's say I have a mysql database table with two columns, one is for id and the other is for password. If I have three rows of data and the id values go from 1 to 3 and I delete row 3 and then create another row of data, I will see id=4 instead of id=3 on the newly created row. I know this has to do with the auto increment value but I was wondering if I can add some code in a php file that will automatically reset all the id numbers such that you start at id=1 and go up to the last id number in increments of 1 after a row has been deleted?
My goal is to create a form where the user enters a password and the system will match the password with a password value in the database. If there is a match, the row with the matched password will be deleted and the column of id numbers will be reordered such that no id numbers are skipped.
Update: I'm making a rotating banner ad system by setting a random number from 1 to 4 to a variable so that the php file will retrieve a random ad from id=1 to id=4 by using the random number variable. If the random number happens to be 3 and id=3 does not exist, there will be a gap in the row of banner ads. If there is a way to work around big gaps in this situation, please tell me. thanks in advance
Just execute the following SQL query:
ALTER TABLE `tbl_name` AUTO_INCREMENT = 1;
…but it sounds like a terrible idea, so don't do it. Why is the value of your primary key so important? Uniqueness is far more important, and reseting it undermines that.
You can only use
ALTER TABLE 'tbl' AUTO_INCREMENT=#
to reset to a number above the highest value number. If you have 1, 2, 3, and you delete 2, you cannot use this to fill 2. If you delete 3, you could use this to re-use 3 (assuming you haven't put anything higher). That is the best you can do.
ALTER TABLE 'table' AUTO_INCREMENT = 1;
However running this code is not the best idea. There is something wrong with your application if you depend on the column having no gaps. Are you trying to count the number of users? if so use COUNT(id)? Are you trying to deal with other tables? If so use a foreign key.
If you are dead set on doing this the Wrong Way you could try to look for the lowest free number and do the incrementing on your own. Keep in mind the race conditions involves however.
Also, keep in mind that if you change the actual numbers in the database you will need to change all references to it in other tables and in your code.
Well, you can actually just specify the id number you'd like a record to have as part of your insert statement, for example:
INSERT INTO person VALUES(1,'John','Smith','jsmith#devnull.fake','+19995559999');
And if there's not a primary key collision (no record in the database with id=1), then MySQL will happily execute it.
The ALTER TABLE 'tbl' AUTO_INCREMENT=# thing also works, and means you don't have to keep track of the counter.
While you're thinking about this, though, you might want to read some of the discussion on natural vs surrogate keys. The idea of having your id # be specifically important is a bit unusual and might be a sign of a troubled design.
You could do that by:
Inventing a mechanism that provides the next available id when you want to insert (e.g. a transaction involving reading and incrementing an integer column somewhere -- pay special attention to the transaction isolation level!)
Using UPDATE to decrement all ids greater than the one you just deleted (again, with a transaction -- don't forget that foreign keys must be ON UPDATE CASCADE!)
But it begs the question: why do you want this? is it going to be worth the trouble?
It's almost certain that you can achieve whatever your goal is without such witchery.
Update (to address comment):
To select a random number of rows, you can do e.g. in MySQL
SELECT id FROM banners ORDER BY RAND() LIMIT 5
to select 5 random, guaranteed existing banner ids.
A word of caution: there are quite a few people who view ORDER BY RAND() as a bad performance hog. However, it is IMHO not quite right to put every case in the same basket. If the number of rows in the table is manageable (I would consider anything below 10K to be not that many) then ORDER BY RAND() provides a very nice and succint solution. Also, the documentation itself suggests this approach:
However, you can retrieve rows in
random order like this:
mysql> SELECT * FROM tbl_name ORDER BY RAND();
ORDER BY RAND() combined with
LIMIT is useful for selecting a random
sample from a set of rows:
mysql> SELECT * FROM table1, table2 WHERE a=b AND c ORDER BY RAND() LIMIT 1000;
RAND() is not meant to be
a perfect random generator. It is a
fast way to generate random numbers on
demand that is portable between
platforms for the same MySQL version.
I have to insert data into two tables, Items and Class_Items. (A third table, Classes is related here, but is not being inserted into).
The primary key of Items is Item_ID, and it's an auto-incrementing integer. Aside from this primary key, there are no unique fields in Items. I need to know what the Item_ID is to match it to Classes in Class_Items.
This is all being done through a PHP interface. I'm wondering what the best way is to insert Items, and then match their Item_ID's into Class_Items. Here are the two main options I see:
INSERT each Item, then use mysql_insert_id() to get its Item_ID for the Class_Items INSERT query. This means one query for every Item (thousands of queries in total).
Get the next Autoincrement ID, then LOCK the Class_Items table so that I can just keep adding to an $item_id variable. This would mean just two queries (one for the Items, one for the Class_Items)
Which way is best and why? Also, if you have an unlisted alternative I'm open to whatever is most efficient.
The most efficient is probably going to be to use parameterized queries. That would require using the mysqli functions, but if you're to the point of needing to optimize this kind of query you should think about being there anyway.
No matter how you cut it, you've got two inserts to make. Doing the first, grabbing the new ID value as you've described (which imposes insignificant overhead, because the value is on hand to mysql already,) and using it in the second insert is pretty minimal.
I would investigate using stored procedures and/or transactions to make sure nothing bad happens.
I'm working on a project with mysql and what I did is the following (without using autoincrement fields):
1- I created a table called SEQUENCE with one field of type BIGINT called VALUE with an initial value of 1. This table will store the id value that will be incremented each time you insert a new record.
2- Create a store procedure and handle the id increment inside it within a transaction.
Here is an example.
CREATE PROCEDURE `SP_registerUser`(
IN _username VARCHAR(40),
IN _password VARCHAR(40),
)
BEGIN
DECLARE seq_user BIGINT;
START TRANSACTION;
#Validate that user does not exist etc..........
#Register the user
SELECT value FROM SEQUENCE INTO seq_user;
UPDATE SECUENCE SET value = value + 1;
INSERT INTO users VALUES(seq_user, _username, SHA1(_password));
INSERT INTO user_info VALUES(seq_user, UTC_TIMESTAMP());
COMMIT;
END //
In my case I want to store the user id in two different tables (users and user_info)