I am developing an application in PHP for which I need to implement a big file handler.
Reading and writing the file is not a problem, but checking the content of the file is a problem.
I built a recursive function which checks whether or not a variable is already used in the same document.
private function val_id($id){
if(!isset($this->id)){
$this->id = array();
}
if(in_array($id, $this->id)){
return $this->val_id($id+1);
}else{
$this->id[] = $id;
return $id;
}
}
When in_array($id,$this->id) returns FALSE, the $id will be added to $this->id (array which contains all used ids) and returns a valid id.
When this returns TRUE, it returns the same function with parameter $id+1
Since we are talking about over 300000 records a time, PHP won't not to be able to store such big arrays. It seems to quit writing lines in the documents I generate when this array gets too big. But I don't receive any error messages like that.
Since the generated documents are SQL files with multiple rows INSERT another solution could be to check if the id already exists in the database. Can MySQL catch these exceptions and try these entries again with adding 1 to id? How?
How do you think I need to solve this problem?
Kind regards,
Wouter
make error messages to appear.
increase memory_limit
instead of values store the parameter in the key - so you'll be able to use isset($array[$this->id]) instead of in_array()
Use INSERT IGNORE to disable duplicate key check in mysql and remove your key check in php. Your statement could look like this.
INSERT IGNORE INTO tbl_name SET key1 = 1, col1 = 'value1'
If you want to add 1 to the id always you could use ON DUPLICATE KEY to increment your key by one:
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
Why should 30.000 records be a problem? Each record in a standard PHP array takes 144 bytes, for 30.000 that would mean 4218,75 kByte. No big deal.
Otherwise, Your Common Sense's idea with the array-key is worth a thought, because it's faster.
Related
$connection->createCommand()->batchInsert('user', ['name', 'age'], [
[$names, $ages],
])->execute();
I know that i can do a batch insert using the code above in Yii 2. But how can i prevent duplicate entry using batchInsert? For example, if I have a duplicate name, i dont want to insert it in the db
Two possible options:
1) Use unique constraint for name column in your database table.
That way you just try to execute query and catch exception:
try {
// Your query goes here
} catch (\yii\db\Exception) {
// Handle error
}
\yii\db\Exception is more common database operations related exception, you can use more specific \yii\db\IntegrityException.
2) Exclude duplicates from array in PHP before feeding data and executing query. Depending on how array is constructed you can do it:
during building this array in foreach for example by checking if item with same name already exists in formed array, if yes - append element to it, otherwise - skip.
or afterwards using for example array_filter function.
I recommend first approach, because even you decide to handle duplicates in PHP, according data structure and additional protection in database won't be superfluos.
I have a strange situation.
Suppose I have a very simple function in php (I used Yii but the problem is general) which is called inside a transaction statement:
public function checkAndInsert($someKey)
{
$data = MyModel::model()->find(array('someKey'=>$someKey)); // search a record in the DB.If it does not exist, insert
if ( $data == null)
{
$data->someCol = 'newOne';
$data->save();
}
else
{
$data->someCol = 'test';
$data->save();
}
}
...
// $db is the instance variable used for operation on the DB
$db->transaction();
$this->checkAdnInsert();
$db->commit();
That said, if I run the script containing this function by staring many processes, I will have duplicate values in the DB. For example, if I have $someKey='pippo', and I run the script by starting 2 processes, I will have two (or more) records with column "someCol" = "newOne". This happens randomly, not always.
Is the code wrong? Should I put some constraint in DB in form of KEYs?
I also read this post about adding UNIQUE indexes to TokuDB which says that UNIQUE KEY "kills" write performance...
The approach you have is wrong. It's wrong because you delegate the authority for integrity/uniqueness check to PHP, but it's the database that's responsible for that.
In other words, you don't have to check whether something exists and then insert. That's bad because there's always some slight ping involved between PHP and MySQL and as you already saw - you can get false results for your checks.
If you need unique values for certain column or combination of columns, you add a UNIQUE constraint. After that you simply insert. If the record exists, insert fails and you can deal with it via Exception. Not only is it faster, it's also easier for you because your code can become a one-liner which is much easier to maintain or understand.
Here is a part of my php code:
foreach ($value->ahkam as $k => $v){
echo $v->id."\n";
//Save into db one hokm
$addHokm = "INSERT INTO qm_hokm (hokm_id, type, tooltip, line, x1, y1, x2, y2, radius, XOrigin, YOrigin, page_id)
VALUES ($v->id,$v->type,'tooltip',0,$v->x1,$v->y1,$v->x2,$v->y2,$v->r,$v->XOrigin,$v->YOrigin,$pageNumber)";
if(!mysqli_query($con, $addHokm))
echo "Failed to insert into db...".$v->id."\n";
}
In fact, I am fetching a json structure sent by an ajax request from a client.
I have many values in $value->ahkam but the problem is that only the first query is run and the others give me the error msg. Any help plz
UPDATE:
the result of echo is:
0
1
Failed to insert into db...1
2
Failed to insert into db...2
As you see, the hokm number 0 is added but not the others, I need to mention also that $pageNumer is a foeign key
The problem is in your foreign key, it must not be unique. Like that, you can add multiple entries for one page_id. I hope it is the correct answer:)
Based on your comments, it appears that your query is inserting a duplicate value for the page_id value, which appears to be set as a field that cannot have duplicate values. According to your query, you're using $pageNumber for that field, but I don't see it changing in your loop. You either need to get rid of the constraint preventing you from using the same value or make sure that $pageNumber has a value that isn't being used already.
$blanknumber = $_POST["blankstartnumber"];
while ($blanknumber <= ($_POST["blankendnumber"] ))
{
echo "$blanknumber";
$blankid = $blanknumber;
$query = "INSERT INTO blank (Blank_ID) VALUES ('$blankid')";
mysql_query($query,$con);
$blanknumber++;
}
So the values are added into the database. Lets say if I have the starting number at 1 and ending at 5. It will all the those values, but it's still trying to add more into the database. I also tried adding an IF statement aswell. if ($blanknumber != $_POST["blankendnumber"])
12345 Error: Duplicate entry '5' for
key 'PRIMARY'
Make sure your $POST value is an integer; by default, I believe it will be cast as a string.
$_POST['varName'] = (int) $_POST['varName'];
edit:
$blanknumber = $_POST["blankstartnumber"];
while ($blanknumber <= ($_POST["blankendnumber"] ))
This should only execute once, since you're setting both comparison variables equal. Definitely 2x check your code.
The database error indicates that Blank_ID is your primary key for that table, and you'd already inserted a 5 into the row. A primary key's values can exist only once in the entire table - duplicates are forbidden (if they were allowed, it wouldn't be a primary key anymore).
If your while loop isn't ending, I'd suggest dumping out both the blankendnumber and blankstartnumber before the loop starts, making sure you've got the right values in there.
It looks like it's actually functioning properly, but you might not have tidy'ed up your db table prior to running. If your output was:
123455 Error: Duplicate entry '5'...
Then, you'd have a programming error, as 5 is getting run twice. Instead, I think you already have data in the blank table that causes a conflict.
Edit: to automatically have MySQL handle the duplicate key error gracefully, you can use the ON DUPLICATE KEY clause to update the row.
INSERT INTO blank (Blank_ID) VALUES (5) ON DUPLICATE KEY UPDATE mod_date = NOW();
I want to generate a random string of about 5 characters long. I can create it ok, but I'm having trouble checking if it exists in an array (or database in real situation) and creating a new one if it does.
I use a function like this to generate the string:
function rand_string(){
return substr(md5(microtime()), 0, 5);
}
But them I'm lost.
I need to check if it exists already.
If it does, make a new one
And repeat
Try this:
function rand_string(){
$str = substr(md5(microtime()), 0, 5);
if(exists_in_db($str)) $str = rand_string();
return $str;
}
Just a warning that if you are using this to generate a unique string by adding it to the database once you've determined it's not been used then this is not safe in a concurrent environment.
In the interval between you checking it's not in the database, and adding a record containing it later on another thread could do the same...
If you are using it this way, probably the safest approach is to ensure that the field containing the string has a unique constraint on it and try to add it. If you suceeded in adding it then you know it was unique, if you didn't then it wasn't. And this is safe to do in a multithreaded environment.
If you are simply checking against a static list of strings and do not intend to add the generated string to the database then ignore this post :P
To check if it's in the DB run a query after.
$unique=FALSE;
while(!$unique)
{
$str = substr(md5(microtime()), 0, 5);
//Insert SQL Code to check if used here
if($row['ID']=='')
$unique=TRUE;
}
on a database you could..:
select substr(newid(),1,5) as name
for a new string and to check..:
select count(*) as cnt from yourtable where yourcolumn = __GENERATED_string__
build a while around it and you'Re done
With the mysql_query(), use a select statement to see if you return any results (i.e., "Select * from table where col1 = 'String'". Then test to see if any rows were returned. Loop through these calls until you have a truly random, unused value.