I have I have static HTML / JS page without any database (I don't need it). However I start fight with some issue. I need to generate random ID which have to be unique(used once and never ever again).
Stardard MySQL way:
On standard application I will do it with storing all used IDs in database
and when needed new one I will simply
// I generate ID here, let's say it's 123
SELECT COUNT(id) FROM table WHERE id = 123
My solution which may be not the best one:
I am thinking alternative may be some storing in file
// Let's generate ID here, and make it 123 again
$handle = fopen("listOfIDs.txt", "r");
if ($handle) {
$used = false;
while (($line = fgets($handle)) !== false) {
if($line = 123) {
$used = true;
break;
}
}
fclose($handle);
return $used;
} else {
// error opening the file.
}
Dispite the fact I can imagine my option may work it could be super slow when file become bigger.
Question:
What's the best way to keep simple unique IDs without using database.
Edit:
I forgot to mentioned that unique ID have to be numbers only.
You could use uniqid(), str_rand(something), timestamp, some hash with random data that you probably never get twice.
Or, doing your way, you could have a single line with information in this file, just the last used id:
$fp = fopen("lastID.txt", "r+");
if (flock($fp, LOCK_EX)) { // lock the file
$lastId = fgets($fp); // get last id
$newId = $lastId +1; //update the id
ftruncate($fp, 0); // truncate file
fwrite($fp, $newId); //write the new Id
fflush($fp); // flush output
flock($fp, LOCK_UN); // release the lock
} else {
throw new Exception("Could not get the lock");
}
fclose($fp);
return $newId;
Maybe you can use a mix of the current timestamp, the request ip, and a randomnumber:
$unique = microtime().$_SERVER['REMOTE_HOST'].rand(0,9999);
Sure theorically you can get a request that executes this operation in the same microtime, with the same remote host, and matching the random number, but it's almost impossible, I think.
You can use PHP uniqid with $more_entropy = true, which increases the likelihood that the result will be unique.
$id = uniqid("", true);
You can also work some random number out using current date and time, etc. You should consider how fast the users may generate these numbers, if it's super fast, there might be a possibility of two different users generating the same $id because it happened at the same time.
Related
I explain, I have a Symfony2 project and I need to import users via csv file in my database. I have to do some work on the datas before importing it in MySQL. I created a service for this and everything is working fine but it takes too much time to execute and slow my server if I give it my entire file. My files have usually between 500 and 1500 rows and I have to split my file in ~200 rows files and import one by one.
I need to handle related users that can be both in the file and/or in database already. Related users are usually a parent of a child.
Here is my simplified code :
$validator = $this->validator;
$members = array();
$children = array();
$mails = array();
$handle = fopen($filePath, 'r');
$datas = fgetcsv($handle, 0, ";");
while (($datas = fgetcsv($handle, 0, ";")) !== false) {
$user = new User();
//If there is a related user
if($datas[18] != ""){
$user->setRelatedMemberEmail($datas[18]);
$relation = array_search(ucfirst(strtolower($datas[19])), UserComiti::$RELATIONSHIPS);
if($relation !== false)
$user->setParentRelationship($relation);
}
else {
$user->setRelatedMemberEmail($datas[0]);
$user->addRole ( "ROLE_MEMBER" );
}
$user->setEmail($mail);
$user->setLastName($lastName);
$user->setFirstName($firstName);
$user->setGender($gender);
$user->setBirthdate($birthdate);
$user->setCity($city);
$user->setPostalCode($zipCode);
$user->setAddressLine1($adressLine1);
$user->setAddressLine2($adressLine2);
$user->setCountry($country);
$user->setNationality($nationality);
$user->setPhoneNumber($phone);
//Entity Validation
$listErrors = $validator->validate($user);
//In case of errors
if(count($listErrors) > 0) {
foreach($listErrors as $error){
$nbError++;
$errors .= "Line " . $line . " : " . $error->getMessage() . "\n";
}
}
else {
if($mailParent != null)
$children[] = $user;
else{
$members[] = $user;
$nbAdded++;
}
}
foreach($members as $user){
$this->em->persist($user);
$this->em->flush();
}
foreach($children as $child){
//If the related user is already in DB
$parent = $this->userRepo->findOneBy(array('username' => $child->getRelatedMemberEmail(), 'club' => $this->club));
if ($parent !== false){
//Check if someone related to related user already has the same last name and first name. If it is the case we can guess that this user is already created
$testName = $this->userRepo->findByParentAndName($child->getFirstName(), $child->getLastName(), $parent, $this->club);
if(!$testName){
$child->setParent($parent);
$this->em->persist($child);
$nbAdded++;
}
else
$nbSkipped++;
}
//Else in case the related user is neither file nor in database we create a fake one that will be able to update his profile later.
else{
$newParent = clone $child;
$newParent->setUsername($child->getRelatedMemberEmail());
$newParent->setEmail($child->getRelatedMemberEmail());
$newParent->setFirstName('Unknown');
$this->em->persist($newParent);
$child->setParent($newParent);
$this->em->persist($child);
$nbAdded += 2;
$this->em->flush();
}
}
}
It's not my whole service because I don't think the remaining would help here but if you need more information ask me.
While I do not heave the means to quantitatively determine the bottlenecks in your program, I can suggest a couple of guidelines that will likely significantly increase its performance.
Minimize the number of database commits you are making. A lot happens when you write to the database. Is it possible to commit only once at the end?
Minimize the number of database reads you are making. Similar to the previous point, a lot happens when you read from the database.
If after considering the above points you still have issues, determine what SQL the ORM is actually generating and executing. ORMs work great until efficiency becomes a problem and more care needs to go into ensuring optimal queries are being generated. At this point, becoming more familiar with the ORM and SQL would be beneficial.
You don't seem to be working with too much data, but if you were, MySQL alone supports reading CSV files.
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed.
https://dev.mysql.com/doc/refman/5.7/en/load-data.html
You may be able to access this MySQL specific feature through your ORM, but if not, you would need to write some plain SQL to utilize it. Since you need to modify the data you are reading from the CSV, you would likely be able to do this very, very quickly by following these steps:
Use LOAD DATA INFILE to read the CSV into a temporary table.
Manipulate the data in the temporary table and other tables as required.
SELECT the data from the temporary table into your destination table.
I know that it is very old topic, but some time ago I created a bundle, which can help import entities from csv to database. So maybe if someone will see this topic, it will be helpful for him.
https://github.com/jgrygierek/BatchEntityImportBundle
https://github.com/jgrygierek/SonataBatchEntityImportBundle
I have this function, and this deletes textfiles after a certain age from my database automatically.
$r = new textfiles;
$db = new DB;
$currTime_ori = $db->queryOneRow("SELECT NOW() as now");
...
if($this->site->textfilesretentiondays != 0)
{
echo "PostPrc : Deleting textfiles older than ".$this->site->textfilesretentiondays." days\n";
$result = $db->query(sprintf("select ID from textfiles where postdate < %s - interval %d day", $db->escapeString($currTime_ori["now"]), $this->site->textfilesretentiondays));
foreach ($result as $row)
$r->delete($row["ID"]);
}
Now I would edit this function so that at first all textfiles are automatically downloaded in a root directory /www/backup and then the script should delete the textfiles with the string $r->delete($row["ID"]);
At the moment I have no idea how I could implement this.
For me it's seems to be impossible to give you an completely answer to your question because leak of informations.
Do you store the whole file-content in database or only the path and filename?
It would help us to see whats the content of "$row" which represents one row from database.
If you just store the filename (and optionally the path) you could use the "copy" (http://php.net/manual/de/function.copy.php) function from php to copy the file to your backup-directory. Please note, you have to ensure that the user who's executing the script or running the web-server have the privileges to write into the directory.
You could add this functionality to class textfiles as as method like makeBackup.
There are few information, but I'll give it a try. If you want to backup the rows before deleting them, you can store them in .txt file in json_encoded form using this piece of code inserted in the FOREACH loop, before delete command:
$myfile = fopen("/www/backup/".$row["ID"].".txt", "w") or die("Unable to open file!");
$txt = json_encode($row);
fwrite($myfile, $txt);
fclose($myfile);
By your approach ..
function delete ($id){
$result = $db->query(sprintf("select * from textfiles where id=$id);
//if you have filepath use copy as SebTM suggested
$path = $row['path']; //assuming path is the column name in ur db
$filename = basename($path); //to get filename
$backup_location = '/www/backup/'.$filename;
copy($path, $backup_location);
//if you have data in db
$content = $row['data'] //assuming data to be backed up to file is in a field 'data'
$backup_location = '/www/backup/file.txt';
file_put_contents($backup_location, $content);
}
But this is not the most optimal approach , you could shift even the initial query into delete function above , and call delete function only once, instead of calling it in a loop ..
So I have a script for accepting and processing request from another scripts and/or applications. However, one of the task has to be done with my script is assigning each request a unique, sequential "ID" to each of them.
For example, let's says that application A is giving a 1000 request to my script, and in the same time, application B is giving 500 request to my script. I have to give them 1500 unique, sequential number, like 2001~3500 to each of them.
The order between them however, does not matter, so I can give them numbers like this :
#2001 for 1st request from A (henceforth, A1)
#2002 for A2
#2003 for B1
#2004 for A3
#2005 for B2
...and so on...
I've tried creating a file that stores that number and a separated lock file with a function like this :
private function get_last_id()
{
// Check if lock file exists...
while (file_exists("LAST_ID_LOCKED")) {
// Wait a little bit before checking again
usleep(1000);
}
// Create the lock file
touch("LAST_ID_LOCKED");
// Create the ID file for the first time if required
if (!file_exists("LAST_ID_INDICATOR")) {
file_put_contents("LAST_ID_INDICATOR", 0);
}
// Get the last ID
$last_id = file_get_contents("LAST_ID_INDICATOR");
// Update the last ID
file_put_contents("LAST_ID_INDICATOR", $last_id + 1);
// Delete the lock file
unlink("LAST_ID_LOCKED");
return $last_id;
}
This code, however, would create a race condition, where if I send them those 1500 request, the last ID will have quite a number missings, (e.g. only reach 3211 instead of 3500).
I've also tried using flock like this, but to no avail :
private function get_last_id()
{
$f = fopen("LAST_ID_INDICATOR", "rw");
while (true) {
if (flock($f, LOCK_SH)) {
$last_id = fread($f, 8192);
flock($f, LOCK_UN);
fclose($f);
break;
}
usleep($this->config["waiting_time"]);
}
$f = fopen("LAST_ID_INDICATOR", "rw");
while (true) {
if (flock($f, LOCK_SH)) {
$last_id = fread($f, 8192);
$last_id++;
ftruncate($f, 0);
fwrite($f, $last_id);
flock($f, LOCK_UN);
fclose($f);
break;
}
usleep($this->config["waiting_time"]);
}
return $last_id;
}
So, what else can I do to look for a solution for this situation?
Notes: Due to server limitation, I'm limited to PHP 5.2 without something like semaphores and such.
Since no-one seems to be giving an answer, I'll give you a possible solution.
Use the Lamport's Bakery Algorithm as part of your solution.
Edit: The filter lock would work even better if you don't need the order preserved.
Obviously this will have its own challenges implementing but it's worth a try and if you get it right, it might just do the trick for what you want to do.
Since you mentioned semaphores, I assume you know enough knowledge to understand the concept.
This can be found in chapter 2 of "The art of multiprocessor programming".
If you have access to a database whith locking capabilities you can use that. E.g. for MySQL with skeleton PHP code:
Create a table with one row and one column (if you do not want "dual-use" of an already existing table):
$sql = 'CREATE TABLE TABLENAME (COLUMNNAME INTEGER) ENGINE=MyISAM';
excecuteSql($sql) ...
Create a PHP function to (re)set the counter/Id value:
$sql = 'UPDATE TABLENAME SET COLUMNNAME=0';
executeSql($sql); ...
Create a PHP function to get a unique, successive id:
$sql = "SELECT GET_LOCK('numberLock',10)";
executeSql($sql); ...
$sql = 'SELECT * FROM TABLENAME';
if ($result = mysqli_query($link, $sql)) {
$row = mysqli_fetch_row($result);
$wantedId = $row[0];
// do something with the id ...
mysqli_free_result($result);
}
$sql = 'UPDATE TABLENAME SET COLUMNNAME=COLUMNNAME+1';
executeSql($sql); ...
$sql = "SELECT RELEASE_LOCK('numberLock')";
executeSql($sql); ...
I am in need to create a CSV file getting the data from a mySQL DB.
The fact is that I want the CSV tp be corrected labeled and not just writing the data like this:
id,name,url
1,thisismyname,thisismyurl
I need the CSV file to look well ordered and each data inserted in the relative column.
Also with the function I am going to add below I can only grab the data from the DB and write it to the CSV file as it is. But I need to work with the data and have the CSV labeled in this way:
Campaign Name:
Name of the campaign
Campaign Url:
Url of the campaign
Tot visits:
Tot of visits
Tot unique visits:
Tot of unique visits
id name url
1 thisname this url
2 thisname this url
3 thisname this url
4 thisname this url
5 thisname this url
This is the PHP code I have so far..I need to understand how to achieve a correct structure of the CSV with PHP and adding the lines in it the exact way I want..
Thanks for your help!
function genCSV($filename, $attachment = true, $headers = true) {
// send response headers to the browser
header('Content-Type: text/csv');
header('Content-Disposition: attachment;filename=' . $filename);
$fp = fopen('php://output', 'w');
$query = "SELECT * FROM campaigns";
$result = mysql_query($query) or die(mysql_error());
if ($headers) {
// output header row (if at least one row exists)
$row = mysql_fetch_assoc($result);
if ($row) {
fputcsv($fp, array_keys($row));
// reset pointer back to beginning
mysql_data_seek($result, 0);
}
}
while ($row = mysql_fetch_assoc($result)) {
fputcsv($fp, $row);
}
fclose($fp);
}
Here is a much less elegant solution than the one proposed by #Tom Regner.
I needed to backup certain database tables (all those with a given prefix) but not others. This method, though somewhat slow, allows you to select exactly which tables and which columns from those tables are copied. It was originally written to allow each piece of data to be AES encrypted before being entered into the file but there are other uses for it. As written here, the result is a CSV file with the first line containing the list of columns for the table and the rest containing the data in CSV. It will stand adaptation to output the result of any sql into CSV, if you like.
Obviously: mysqlidb = mysqli databse resource, backups/ = directory to put finished files in.
FWIIW, here is the code:
$sql="SHOW TABLES LIKE 'yourtable%'";
$result = $mysqlidb->query($sql);
$tableresult=$mysqlidb->query($sql);
while($tables=$tableresult->fetch_assoc())
{
$keys=array_keys($tables);
$tablename=$tables[$keys[0]];
echo "Writing $tablename <BR>";
$file=fopen("backups/$tablename.enc","w");
$cols=array();
$sql="SHOW COLUMNS FROM $tablename";
$result=$mysqlidb->query($sql);
while($row=$result->fetch_assoc())
{
$cols[]=$row['Field'];
}
fputcsv($file,$cols);
$sql="SELECT * FROM $tablename";
$result=$mysqlidb->query($sql);
while($row=$result->fetch_assoc())
{
fputcsv($file,$row);
}
fclose($file);
}
i'm trying to generate a unique id using this code:
$RCode = md5(uniqid(rand(), true));
then i want it to check in my database if the RCode is unique.
if it isnt i want it to use that part of the code again and check in my database again if it is unique,
if it is unique it should write to my database.
i have all the code i need for checking and writing into the database, i just have no idea how to make it loop back to the start.
Help is appreciated!
Thanks a lot in advance!
Don't bother checking at first. Instead put a unique constraint on the column, that way the insert will fail if the RCode isn't unique. Then you can handle that error/exception and try another hash. The probability of a collision is low so in this case you probably aren't going to be hammering the database.
Typical example for do-while loop.
Some PHP-pseudocode:
do {
$rcode = md5(uniqid(rand(), true));
$res = mysql_result(mysql_query("SELECT COUNT(*) FROM records WHERE rcode='$rcode'"));
} while ($res[0] > 0);
mysql_query("INSERT INTO records (rcode) VALUES ('$rcode')");
$found = false;
while (! $found) {
//try..
if (...unique...) {
$found = true;
}
}
Going to the start is as easy as implementing a while loop. Heck, you could even use goto (kidding!).
But I don't understand why you don't want to use auto_increment.
You can use the "loop forever then break out on success" method:
while (true) {
$RCode = ...;
if ($RCode does not exist in db) {
break;
}
}
write to the db
Edit: Or better, make sure the field has a unique constraint on it, then test for uniqueness by checking for failure of an insert:
while (true) {
$RCode = ...
try to insert RCode
if (no failure) {
break;
}
}
This will be more resilient to concurrent hits.
This answer is a little different to what you asked but it solves the problem of a unique ID in a different way that may be better to use depending on your application.
To be honest I like to use variables such as date and time combined with another variable such as an IP address for this type of thing, you can then be very certain that your ID will be unique because the date and time will not reoccur and in the event there are 2 requests in the same second the IP address of the user is completely unique at this time also. Plus no having to check with the database. An example would be
$idstring = date('Ymdhis');
$ipstring = $_SERVER['REMOTE_HOST'];
$hashme = $idstring.$ipstring;
$idhash = md5($hashme);
I hope this is helpful.
As you hash uniqid() returned value using md5() (the same goes for any other hashing algorithm), possibility of getting not unique string is extremely low.
In my opinion, its so low that checking for that string to be unique would be overkill.
All you need to do is insert value in the database. It will be unique!
Edit:
You should use uniqid(null, true) to get 23 chars long, unique string. If that is what you need - unique string.
you can just run MD5 on the timestamp...
$uniqueid = md5(time());
First, don't md5 uniqueid. uniqueid is good enough and it does not have the overhead of md5 if used by itself.
Second, you are far better off obscuring an automatically incrementing number than using a UUID or some equivalent.
But, if you must:
do {
$rcode = uniqid(rand(), true);
$res = mysql_query("SELECT COUNT(*) FROM records WHERE rcode='$rcode'");
} while ($res && mysql_num_rows($res));
mysql_query("INSERT INTO records (rcode) VALUES ('$rcode')");
// OR!
// assuming unique index on rcode
do {
$rcode = uniqid(rand(), true);
} while (mysql_query("INSERT INTO records (rcode) VALUES ('$rcode')"););