This is for a file sharing website. In order to make sure a "passcode", which is unique to each file, is truely unique, I'm trying this:
$genpasscode = mysql_real_escape_string(sha1($row['name'].time())); //Make passcode out of time + filename.
$i = 0;
while ($i < 1) //Create new passcode in loop until $i = 1;
{
$query = "SELECT * FROM files WHERE passcode='".$genpasscode."'";
$res = mysql_query($query);
if (mysql_num_rows($res) == 0) // Passcode doesn't exist yet? Stop making a new one!
{
$i = 1;
}
else // Passcode exists? Make a new one!
{
$genpasscode = mysql_real_escape_string(sha1($row['name'].time()));
}
}
This really only prevents a double passcode if two users upload a file with the same name at the exact same time, but hey better safe than sorry right? My question is; does this work the way I intend it to? I have no way to reliably (read: easily) test it because even one second off would generate a unique passcode anyway.
UPDATE:
Lee suggest I do it like this:
do {
$query = "INSERT IGNORE INTO files
(filename, passcode) values ('whatever', SHA1(NOW()))";
$res = mysql_query($query);
} while( $res && (0 == mysql_affected_rows()) )
[Edit: I updated above example to include two crucial fixes. See my answer below for details. -#Lee]
But I'm afraid it will update someone else's row. Which wouldn't be a problem if filename and passcode were the only fields in the database. But in addition to that there's also checks for mime type etc. so I was thinking of this:
//Add file
$sql = "INSERT INTO files (name) VALUES ('".$str."')";
mysql_query($sql) or die(mysql_error());
//Add passcode to last inserted file
$lastid = mysql_insert_id();
$genpasscode = mysql_real_escape_string(sha1($str.$lastid.time())); //Make passcode out of time + id + filename.
$sql = "UPDATE files SET passcode='".$genpasscode."' WHERE id=$lastid";
mysql_query($sql) or die(mysql_error());
Would that be the best solution? The last-inserted-id field is always unique so the passcode should be too. Any thoughts?
UPDATE2: Apperenatly IGNORE does not replace a row if it already exists. This was a misunderstanding on my part, so that's probably the best way to go!
Strictly speaking, your test for uniqueness won't guarantee uniqueness under a concurrent load. The problem is that you check for uniqueness prior to (and separately from) the place where you insert a row to "claim" your newly generated passcode. Another process could be doing the same thing, at the same time. Here's how that goes...
Two processes generate the exact same passcode. They each begin by checking for uniqueness. Since neither process has (yet) inserted a row to the table, both processes will find no matching passcode in database, and so both processes will assume that the code is unique. Now as the processes each continue their work, eventually they will both insert a row to the files table using the generated code -- and thus you get a duplicate.
To get around this, you must perform the check, and do the insert in a single "atomic" operation. Following is an explanation of this approach:
If you want passcode to be unique, you should define the column in your database as UNIQUE. This will ensure uniqueness (even if your php code does not) by refusing to insert a row that would cause a duplicate passcode.
CREATE TABLE files (
id int(10) unsigned NOT NULL auto_increment PRIMARY KEY,
filename varchar(255) NOT NULL,
passcode varchar(64) NOT NULL UNIQUE,
)
Now, use mysql's SHA1() and NOW() to generate your passcode as part of the insert statement. Combine this with INSERT IGNORE ... (docs), and loop until a row is successfully inserted:
do {
$query = "INSERT IGNORE INTO files
(filename, passcode) values ('whatever', SHA1(NOW()))";
$res = mysql_query($query);
} while( $res && (0 == mysql_affected_rows()) )
if( !$res ) {
// an error occurred (eg. lost connection, insufficient permissions on table, etc)
// no passcode was generated. handle the error, and either abort or retry.
} else {
// success, unique code was generated and inserted into db.
// you can now do a select to retrieve the generated code (described below)
// or you can proceed with the rest of your program logic.
}
Note: The above example was edited to account for the excellent observations posted by #martinstoeckli in the comments section. The following changes were made:
changed mysql_num_rows() (docs) to mysql_affected_rows() (docs) -- num_rows doesn't apply to inserts. Also removed the argument to mysql_affected_rows(), as this function operates on the connection level, not the result level (and in any case, the result of an insert is boolean, not a resource number).
added error checking in the loop condition, and added a test for error/success after loop exits. The error handling is important, as without it, database errors (like lost connections, or permissions problems), will cause the loop to spin forever. The approach shown above (using IGNORE, and mysql_affected_rows(), and testing $res separately for errors) allows us to distinguish these "real database errors" from the unique constraint violation (which is a completely valid non-error condition in this section of logic).
If you need to get the passcode after it has been generated, just select the record again:
$res = mysql_query("SELECT * FROM files WHERE id=LAST_INSERT_ID()");
$row = mysql_fetch_assoc($res);
$passcode = $row['passcode'];
Edit: changed above example to use the mysql function LAST_INSERT_ID(), rather than PHP's function. This is a more efficient way to accomplish the same thing, and the resulting code is cleaner, clearer, and less cluttered.
I'd personally would have write it on a different way but I'll provide you a much easier solution: sessions.
I guess you're familiar with sessions? Sessions are server-side remembered variables that timeout at some point, depending on the server configuration (the default value is 10 minutes or longer). The session is linked to a client using a session id, a random generated string.
If you start a session at the upload page, an id will be generated which is guaranteed to be unique as long the session is not destroyed, which should take about 10 minutes. That means that when you're combining the session id and the current time you'll never have the same passcode. A session id + the current time (in microseconds, milliseconds or seconds) are NEVER the same.
In your upload page:
session_start();
In the page where you handle the upload:
$genpasscode = mysql_real_escape_string(sha1($row['name'].time().session_id()));
// No need for the slow, whacky while loop, insert immediately
// Optionally you can destroy the session id
If you do destroy the session id, that would mean there's a very slim chance that another client can generate the same session id so I wouldn't advice that. I'd just allow the session to expire.
Your question is:
does this work the way I intend it to?
Well, I'd say... yes, it does work, but it could be optimized.
Database
To make sure to not have the same value in the field passcode on the database layer, add a unique key to this:
/* SQL */
ALTER TABLE `yourtable` ADD UNIQUE `passcode` (`passcode`);
(duplicate key handling has to be taken care of than ofc)
Code
To wait a second until a new Hash is created, is ok, but if you're talking heavy load, then a single second might be a tiny eternity. Therefore I'd rather add another component to the sha1-part of your code, maybe a file id from the same database record, userid or whatever which makes this really unique.
If you don't have a unique id at hand, you still can fall back to a random number rand-function in php.
I don't think mysql_real_escape_string is needed in this context. The sha1 returns a 40-character hexadecimal number anyway, even if there are some bad characters in your rows.
$genpasscode = sha1(rand().$row['name'].time());
...should suffice.
Style
Two times the passcode generation code is used in your code sample. Start cleaning this up in moving this into a function.
$genpasscode = gen_pc(row['name']);
...
function gen_pc($x)
{
return sha1($row[rand().$x.time());
}
If I'd do it, I'd do it differently, I'd use the session_id() to avoid duplicates as good as possible. This way you wouldn't need to loop and communicate with your database in that loop possibly several times.
You can add unique constraint to your table.
ALTER TABLE files ADD UNIQUE (passcode);
PS: You can use microtime or uniqid to make the passcode more unique.
Edit:
You make your best to generate a unique value in php, and unique constraint is used to guarantee that at database side. If your unique value is very unique, but in very rare case it failed to be unique, just feel free to give the message like The system is busy now. Please try again:).
Related
I have found two different ways to, first, get the next invoice number and, then, save the invoice in a multi-tenant database where, of course, each tenant will have his own invoices with different incremental numbers.
My first (and actual) approach is this (works fine):
Add a new record to the invoices tables. No matter the invoice number yet (for example, 0, or empty)
I get the unique ID of THAT created record after insert
Now I do a "SELECT table where ID = $lastcreatedID **FOR UPDATE**"
Here I get the latest saved invoice number with "SELECT #A:=MAX(NUMBER)+1 FROM TABLE WHERE......"
Finally I update the previously saved record with that invoice number with an "UPDATE table SET NUMBER = $mynumber WHERE ID = $lastcreatedID"
This works fine, but I don't know if the "for update" is really needed or if this is the correct way to do this in a multi-tenant DB, due to performance, etc.
The second (and simpler) approach is this (and works too, but I don't know if it is a secure approach):
INSERT INTO table (NUMBER,TENANT) SELECT COALESCE(MAX(NUMBER),0)+1,$tenant FROM table WHERE....
That's it
Both methods are working, but I would like to know the differences between them regarding speed, performance, if it may create duplicates, etc.
Or... is there any better way to do this?
I'm using MySQL and PHP. The application is an invoice/sales cloud software that will be used by a lot of customers (tenants).
Thanks
Regardless of if you're using these values as database IDs or not, re-using IDs is virtually guaranteed to cause problems at some point. Even if you're not re-using IDs you're going to run into the case where two invoice creation requests run at the same time and get the same MAX()+1 result.
To get around all this you need to reimplement a simple sequence generator that locks its storage while a value is being issued. Eg:
CREATE TABLE client_invoice_serial (
-- note: also FK this back to the client record
client_id INTEGER UNSIGNED NOT NULL PRIMARY KEY,
serial INTEGER UNSIGNED NOT NULL DEFAULT 0
);
$dbh = new PDO('mysql:...');
/* this defaults to 'on', making every query an implicit transaction. it needs to
be off for this. you may or may not want to set this globally, or just turn it off
before this, and back on at the end. */
$dbh->setAttribute(PDO::ATTR_AUTOCOMMIT,0);
// simple best practice, ensures that SQL errors MUST be dealt with. is assumed to be enabled for the below try/catch.
$dbh->setAttribute(PDO::ATTR_ERRMODE_EXCEPTION,1);
$dbh->beginTransaction();
try {
// the below will lock the selected row
$select = $dbh->prepare("SELECT * FROM client_invoice_serial WHERE client_id = ? FOR UPDATE;");
$select->execute([$client_id]);
if( $select->rowCount() === 0 ) {
$insert = $dbh->prepare("INSERT INTO client_invoice_serial (client_id, serial) VALUES (?, 1);");
$insert->execute([$client_id]);
$invoice_id = 1;
} else {
$invoice_id = $select->fetch(PDO::FETCH_ASSOC)['serial'] + 1;
$update = $dbh->prepare("UPDATE client_invoice_serial SET serial = serial + 1 WHERE client_id = ?");
$update->execute([$client_id])
}
$dbh->commit();
} catch(\PDOException $e) {
// make sure that the transaction is cleaned up ASAP, then let the exception bubble up into your general error handling.
$dbh->rollback();
throw $e; // or throw a more pertinent error/exception of your choosing.
}
// both committing and rolling back will release the lock
At a very basic level this is what MySQL is doing in the background for AUTOINCREMENT columns.
Do not use MAX(id)+1. It will, someday, bite you. There will be two invoices with the same number, and it will take us a few paragraphs to explain why it happened.
Instead, use AUTO_INCREMENT the way it is intended.
INSERT INTO Invoices (id, ...) VALUES (NULL, ...);
SELECT LAST_INSERT_ID(); -- specific to the conne ction
That is safe even when multiple connections are doing the same thing. No FOR UPDATE, no BEGIN, etc is necessary. (You may want such for other purposes.)
And, never delete rows. Instead, use the standard business practice of invalidating bad invoices. Imagine being audited.
All that said, there is still a potential problem. After a ROLLBACK or system crash, an id may be "burned". Also things like INSERT IGNORE allocate the id before checking to see whether it will be needed.
If you can live with the caveats, use AUTO_INCREMENT.
If not, then create a 1-row, 2-column table to simulate a sequence number generator: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#sequence
Or use MariaDB's SEQUENCE
Both the approaches do work, but each with its own demerits in high traffic situations.
The first approach runs 3 queries for every invoice you create, putting extra load on your server.
The second approach can lead to duplicates in events where two invoices are generated with very little time difference (such that the SELECT query return same max number for both invoices).
Both the approaches may lead to problems in high traffic conditions.
Two solutions to the problems are listed below:
Use generated columns: Mysql supports generated columns, which are basically derived using other column values for each row. Refer this
Calculate invoice number on the fly: Since you're using the primary key as part of the invoice, let the DB handle generating unique primary keys, and then generate invoice numbers on the fly in your business logic using the id for each invoice.
I have two tables:
people:
peopleID (PK)
name
peopleaddress:
addressID (PK)
peopleID (fK)
address
addresstype
...other fields...
A person can have more addresses.
I have a form to add a new person (and addresses) and another form to edit info. When I load the edit form, it takes the info from the DB about that person and put them in the fields value="".
Right now, when I submit the edit form, I use 2 sql commands:
$stmt = $conn->prepare("DELETE from peopleaddress WHERE peopleID=?");
and
$stmt = $conn->prepare("INSERT INTO peopleaddress (peopleID, addresstype, active, street.....) VALUES (?, ?, ?, ....)");
It works well, only downside is that addressID changes every update and it tends to grow fast.
Am I doing it the right way or there is a way in php or better sql to say:
if new address exists » update
if doensn't exist » insert
if all fields of existing address empty » delete
Thanks for your help!
Here's what you're looking for: "INSERT ... ON DUPLICATE KEY UPDATE"
INSERT INTO peopleaddress (peopleID, addresstype, active, ...) VALUES (1,2,3, ...)
ON DUPLICATE KEY UPDATE addresstype=2, active=3, ...;
It will find the address if it is already in the database and update it, if it can. Otherwise, it does the insert.
You may have to re-work your '?' substitutions.
If someone adds a new address to an existing account it's okay to have a new row (e.g. id), keeping old addresses isn't necessarily a good or bad thing and having that as a separate table is good (some people may have a home and a business address in example).
If all the fields are missing then your code's logic (PHP?) shouldn't even be getting to the point of any SQL queries.
function admin_members_address_update()
{
// 1. Create $required array.
// 2. Check foreach $_POST[array_here].
// 3. If any of the required $_POST variables aren't set HTTP 403 and end the function.
//if something wrong, HTTP 403
else
{
///Normal code here.
}
}
You should always check for variables you expect to be set just like you always escape (e.g. not trust) client data and have your error reporting set to maximum. A hacker will omit a form field easily with any dev tool and try to find vulnerabilities based on error messages generated by the server.
Always check for failure before you presume success.
//if () {}
//else if () {}
//else if () {}
//else {}
SQL should be handled in a similar fashion:
$query1 = 'SELECT * from table;';
$result1 = mysqli_query($db,$query1);
if ($result1) {}
else {/*SQL error reporting, send __magic_function__ as param with well named functions*/}
That is the very rough of it (don't have access to my actual code) though it should give you something to go on. Comment if you'd like me to update my answer when I have access to my code later.
A MySQL unsigned INT (max 4294967295) will support an average of 13 inserts per second, 24x7, for 10 years. That's already probably as many addresses as there are on earth. If you're pushing the limit somehow, a BIGINT will be massively larger and probably never run out in a million years.
Basically, don't worry about using up IDs with auto increment, the decision of whether to DELETE -> INSERT or UPDATE/DELETE/INSERT should be based on whether you need to maintain persistent IDs for individual addresses. Deleting then inserting assigns a new ID, even if it's really the same address, which is undesirable if you want to create a foreign key that references address IDs. If you don't need that you don't need to worry about it, though as a personal preference I would probably incorporate UPDATE.
I have the following function which seems to run forever. Its creating a random string then checking if its in the database. If it is it should run again and again until it has one that is new. It should then return the value to me
public function checkPromo(){
$continue = true;
while ($continue){
$promo = $this->getRandString(6);
$query = sprintf("SELECT * FROM table WHERE field=%s",
$this->db->cleanCode($promo, "text"));
$result = $this->db->query($query);
if($this->db->num_rows($result) >= 1){
$continue = false;
}
}
return $promo;
}
Your specific problem is that your check is backwards. You're looping until you find a duplicate, which will likely take quite a while or forever.
Even if you fixed it, it's a bad algorithm. It will take exponentially longer to generate a new unique code the more codes you have in your database, since the chance of duplicates increases. Especially if your codes are only 6 characters long the chance for duplication increases a lot.
Note that you're also prone to race conditions with your code (think it through from that perspective).
There are two approaches to generating unique ids:
use an incrementing counter, which means you're using a central generator which keeps track of existing ids
use a decentralised approach where you're not keeping track of your ids, but you're using an algorithm that is random enough and has a large enough space that collisions are so unlikely as to be irrelevant in practice
You're combining the worst of both worlds: you're using a central system to keep track of your ids, but you're using random id generation with it. Use one or the other, not both.
If you're going to use a database anyway, just use a standard auto_increment id. If you want it to look a bit random, hash it with MD5 or such.
Alternatively, simple pre-generate all possible codes (just 6 characters isn't a lot), and pick one at random using a method which is not prone to race conditions. Something along the lines of this:
UPDATE codes
SET claimed_user_id = %d
WHERE claimed_user_id IS NULL
ORDER BY RAND()
LIMIT 1
Otherwise, if you want decentralised random ids, use an appropriate algorithm, which is pretty much UUID.
while (true) is a dangerous thing. As you noted, its defaulted to run again and again creating an infinite loop.
You'd be better offer utilizing an auto-incremented value(link assumes MySQL) or generate a pseudo unique id with PHP (such as uniqid()).
You could expand on these values further by prefixing or suffixing a common word, say promo, or padding or hashing them to create a standard length.
Assuming you simply want to generate a promocode that was not recorded in the database and taking into account the context, I advise you change the attitude a little bit:
public function generatePromo(){
$promo = $this->getRandString(6);
$query = sprintf("SELECT * FROM table WHERE field=%s", $this->db->cleanCode($promo, "text"));
$result = $this->db->query($query);
if($this->db->num_rows($result) > 0){ # in case a record with this "text" already exists, run this method again
$this->generatePromo();
} else {
return $this->promo = $promo; # otherwise return the value/store it in the object
}}
If you expect your table to contain only unique values, then I suggest you make it UNIQUE. That will significantly reduce query time.
I want to add 1 to the value of the previous row for hit_count, but I'm afraid doing it may not be safe if multiple queries are being run quickly (i.e. the page is being loaded several times a second - this is for a web-app I'm making, and so I want to make sure any amount of page loads is supported well).
Here's what I had in mind:
$result = mysql_query("SELECT * FROM rotation");
$fetch = mysql_fetch_assoc($result);
$update_hit = $fetch['hit_counter']+1;
$query = "INSERT INTO rotation (hit_counter, rotation_name) VALUES ('$update_hit', '$rotation_name')";
$result = mysql_query($query);
I thought about setting the hit_counter column to a UNIQUE KEY, but I don't know what else I'd do after that.
I would use AUTO_INCREMENT but the problem is, I need the actual hit_counter value within the rest of the script.
Any ideas, comments, advice would be greatly appreciated!
Edit: I used hit_count and hit_counter, was a typo. Updated to avoid any confusion.
You can use the DUPLICATE KEY functionality when you make name + counter a unique value:
INSERT INTO rotation SET hit_counter = 1, rotation_name= 'name'
ON DUPLICATE KEY UPDATE hit_counter = hit_counter + 1
Performance wise (and if your requirements allow it) I advice pushing updates in bulk (per 100 hits) using a caching mechanism like f.e. memcached.
You could use AUTO_INCREMENT, if you need the inserted id within the rest script, you can use mysql_insert_id to get it.
How do i go about looking into a table and searching to see if a row exist. the back gorund behind it is the table is called enemies. Every row has a unique id and is set to auto_increment. Each row also has a unique value called monsterid. the monster id isn't auto_increment.
when a monster dies the row is deleted and replaced by a new row. so the id is always changing. as well the monsterid is changed too.
I am using in php the $_GET method and the monsterid is passing through it,
basically i am trying to do this
$monsterID = 334322 //this is the id passed through the $_GET
checkMonsterId = "check to see if the monster id exist within the enemies table"
if monsterid exist then
{RUN PHP}
else
{RUN PHP}
If you need anymore clarity please ask. and thanks for the help in advance.
Use count! If it returns > 0, it exists, else, it doesn't.
select count(*) from enemies where monsterid = 334322
You would use it in PHP thusly (after connecting to the database):
$monsterID = mysql_real_escape_string($monsterID);
$res = mysql_query('select count(*) from enemies where monsterid = ' . $monsterid) or die();
$row = mysql_fetch_row($res);
if ($row[0] > 0)
{
//Monster exists
}
else
{
//It doesn't
}
Use count, like
select count(*) from enemies where monsterid = 334322
However be sure to make certain you've added an index on monsterid to the table. Reason being that if you don't, and this isn't the primary key, then the rdbms will be forced to issue a full table scan - read every row - to give you the value back. On small datasets this doesn't matter as the table will probably sit in core anyway, but once the number of rows becomes significant and you're hitting the disk to do the scan the speed difference can easily be two orders of magnitude or more.
If the number of rows is very small then not indexing is rational as using an non-primary key index requires additional overhead when inserting data, however this should be a definite decision (I regularly impress clients who've used a programmer who doesn't understand databases by adding indexes to tables which were fine when the coder created them but subsequently slow to a crawl when loaded with real volumes of data - quite amazing how one line of sql to add an index will buy you guru status in your clients eyes cause you made his system usable again).
If you're doing more complex queries against the database using subselect, something like finding all locations where there is no monster, then look up the use of the sql EXISTS clause. This is often overlooked by programmers (the temptation is to return a count of actual values) and using it is generally faster than the alternatives.
Simpler :
select 1 from enemies where monsterid = 334322
If it returns a row, you have a row, if not, you don't.
The mysql_real_escape_string is important to prevent SQL injection.
$monsterid = mysql_real_escape_string($_GET['monsterid']);
$query = intval(mysql_query("SELECT count(*) FROM enemies WHERE monsterid = '$monsterid'));
if (mysql_result > 0) {
// monster exists
} else {
// monster doesn't exist
}