Fast inserts results in data loss - php

I have a strange bug here related to mysql and php. I'm wondering if it could be a performance problem on our server's behalf too.
I got a class used to manage rebate promotional codes. The code is great, works fine and doesn exactly what it is supposed to do. The saveChanges() operation sends an INSERT or UPDATE depending on the state of the object and in the current context will only insert cause i'm trying to generate coupon codes.
The classe's saveChanges goes like this: (I know, i shouldn't be using old mysql, but i've got no choice due to architectural limitations of the software, so don't complain about that part please)
public function saveChanges($asNew = false){
//Get the connection
global $conn_panier;
//Check if the rebate still exists
if($this->isNew() || $asNew){
//Check unicity if new
if(reset(mysql_fetch_assoc(mysql_query('SELECT COUNT(*) FROM panier_rabais_codes WHERE code_coupon = "'.mysql_real_escape_string($this->getCouponCode(), $conn_panier).'"', $conn_panier))) > 0){
throw new Activis_Catalog_Models_Exceptions_ValidationException('Coupon code "'.$this->getCouponCode().'" already exists in the system', $this, __METHOD__, $this->getCouponCode());
}
//Update the existing rebate
mysql_query($q = 'INSERT INTO panier_rabais_codes
(
`no_rabais`,
`code_coupon`,
`utilisation`,
`date_verrou`
)VALUES(
'.$this->getRebate()->getId().',
"'.mysql_real_escape_string(stripslashes($this->getCouponCode()), $conn_panier).'",
'.$this->getCodeUsage().',
"'.($this->getInvalidityDate() === NULL ? '0000-00-00 00:00:00' : date('Y-m-d G:i:s', strtotime($this->getInvalidityDate()))).'"
)', $conn_panier);
return (mysql_affected_rows($conn_panier) >= 1);
}else{
//Update the existing rebate
mysql_query('UPDATE panier_rabais_codes
SET
`utilisation` = '.$this->getCodeUsage().',
`date_verrou` = "'.($this->getInvalidityDate() === NULL ? '0000-00-00 00:00:00' : date('Y-m-d G:i:s', strtotime($this->getInvalidityDate()))).'"
WHERE
no_rabais = '.$this->getRebate()->getId().' AND code_coupon = "'.mysql_real_escape_string($this->getCouponCode(), $conn_panier).'"', $conn_panier);
return (mysql_affected_rows($conn_panier) >= 0);
}
}
So as you can see, the code itself is pretty simple and clean and returns true if the insert succeeded, false if not.
The other portion of the code generates the codes using a random algorithm at goes like this:
while($codes_to_generate > 0){
//Sleep to delay mysql choking on the input
usleep(100);
//Generate a random code
$code = strtoupper('RC'.$rebate->getId().rand(254852, 975124));
$code .= strtoupper(substr(md5($code), 0, 1));
$rebateCode = new Activis_Catalog_Models_RebateCode($rebate);
$rebateCode->setCouponCode($code);
$rebateCode->setCodeUsage($_REQUEST['utilisation_generer']);
try{
if($rebateCode->saveChanges()){
$codes_to_generate--;
$generated_codes[] = $code;
}
}catch(Exception $ex){
}
}
As you can see here, two things to note. The number of codes to generate and the array of generated codes only get filled if i get a return true from the saveChanges, so mysql HAS to report that the information was inserted for this part to happen.
Another tidbit is the first line of the while:
//Sleep to delay mysql choking on the input
usleep(100);
Wtf? Well this post is all about this. My code works flawlessly with small amounts of codes to generate. But if i ask mysql to save more than a few codes at once, i have to throttle myself using usleep or mysql drops some of these lines. It will report that there are affected rows but is not saving them.
Under 100 lines, i don't need throttling and then i need to usleep depending on the amount of lines to insert. It must be something simple but i don't know what. Here is a sum of the lines i tried to insert and the minimum usleep throttle i had to implement:
< 100 lines: none
< 300 lines: 2 ms
< 1000 lines: 5 ms
< 2000 lines: 10 ms
< 5000 lines: 20 ms
< 10000 lines: 100 ms
Thank you for your time

Are you sure that your codes are all inserted and not updated, because, update a non existing line does nothing.

Related

why I cannot get cached data in transaction of Redis?

I am writing a lottery program in PHP, because there would be large concurrent request for this program, I have limited number of each prize, 10 in this example. I don't want to see any prize exceed the stock. so I put the entire logic in transaction of Redis(I use predis(https://github.com/nrk/predis) as my PHP redis client), but it doesn't work, after more than 10 times requests to this program, I found more than 10 records in database that I could not understand. does anyone who knows the reason? very appreciate to your explanation, thank you!
here is my PHP code:
$this->load->model('Lottery_model');
$money = $this->_get_lottery();//which prize do you get
if($money > 0){
$key = $this->_get_sum_key($money);
$dbmodel = $this->Lottery_model;
// Executes a transaction inside the given callable block:
$responses = $redis->transaction(function ($tx) use ($key, $money, $dbmodel){
$qty = intval($tx->get($key));
if($qty < 10){
//not exceed the stock limit
$dbmodel->add($customer, $money); // insert record into db
$tx->incr($key);
}else{
log_message('debug', $money . ' dollar exceed the limit');
}
});
}else{
log_message('debug', 'you are fail');
}
after reading documentation about transaction of Redis, I know the usage of above code is totally wrong. then I modified it to below version, using optimistic lock and check-and-set.
$options = array(
'cas' => true, // Initialize with support for CAS operations
'watch' => $key, // Key that needs to be WATCHed to detect changes
'retry' => 3,
);
try{
$responses = $redis->transaction($options, function ($tx) use ($key, $money, $username, $dbmodel, &$qty){
$qty = intval($tx->get($key));
if($qty < 10){
$tx->multi();
$tx->incr($key);
$dbmodel->add($username, $money);// insert into mysql db
}else{
log_message('debug', $money . ' dollar exceed the limit');
}
});
}catch(PredisException $e){
log_message('debug', 'redis transaction failed');
}
But the problem is that the number of record in database exceeds the limitation of the prize, the total number saved in Redis won't be. what is the common solution to solve this kind of problem? do I have to lock INNodb table in this case?
You need to understand how Redis transactions work - in a nutshell, all commands making a transaction are buffered by the client (predis in your case) and then fired to the server all at once. Your code attempts to use the result of a read request (get) before the transaction had been executed. Please refer to the documentation for more details: https://redis.io/topics/transactions
Either read the qty outside the transaction, and use WATCH to protect against competing updates, or move this logic in its entirety to a Lua script.

SQL insert Or update times out if large date in loop

i have some data with userid and date.
Sometimes there is large datas i need to loop through and update the sql database but the database times out.
is there any better way i can do this please, sample code below.
foreach($time[$re->userid][$today] as $t){
if(($re->time >= $t->in_from) && ($re->time < $t->in_to)
&& md5($t->WorkDay."_in".$re->date) != $in){//in
$tble = tools::sd("{$t->WorkDay} in");
}
if(($re->time >= $t->out_from) && ($re->time < $t->out_to)
&& md5($t->WorkDay."_out".$re->date) != $out){//out
$tble = tools::sd("{$t->WorkDay} out");
if($tble =='nout'){
$re->date2 = tools::ndate($re->date . "- 1");
}
}
if(!empty($tble)){
$q = array(
"id" => $re->userid
, "dt" => $re->date2
, "{$tble}" => $re->time
);
dump($q); // insert into sql
}
}
dump function:::
function dump($d ='')
{
if(!empty($d)){
end($d);
$tble = key($d);
$d['ld'] = "{$d['dt']} {$d[$tble]}";
$r = $GLOBALS['mssqldb']->get_results("
IF NOT EXISTS (select id,ld,dt,{$tble} from clockL
WHERE id = '{$d['id']}'
AND dt ='{$d['dt']}')
INSERT INTO clockL (id,ld,dt,{$tble})
VALUES ('{$d['id']}','{$d['ld']}','{$d['dt']}'
,'{$d[$tble]}')
ELSE IF EXISTS (select id,{$tble} from clockL
WHERE id = '{$d['id']}'
AND dt ='{$d['dt']}'
AND {$tble} = 'NOC'
)
update clockL SET {$tble} ='{$d[$tble]}', ld ='{$d['ld']}' WHERE id = '{$d['id']}'
AND dt ='{$d['dt']}' AND {$tble} ='NOC'
");
//print_r($GLOBALS['mssqldb']);
}
}
Thank You.
Do the insert/update outside of the loop. Enclose it in a transaction so that you don't get an inconsistent database state if the script dies prematurely. Using one big query is usually faster than making lots of small queries. You might also set higher values for time and memory limits, but be aware of the consequencies.
Are you aware of a PHP function called set_time_limit()? You can find the detailed documentation here.
This can manipulate the execution time, which is 30 seconds default. If you set it to 0, eg set_time_limit(0), there will be no execution time limit.
May be looping is the reason for time out.
Because when your performing the insert/update operations in side the loop, the connection to the database will be in open state until the loop is terminated which may cause the time out problem.
Try doing the insert/update operation outside of the loop.

Why str_shuffle always generate similar patterns?

I am trying to generate 1500 authentication code using the following code:
<?php
include "../include/top.php";
set_time_limit(0);
ini_set("memory_limit", "-1");
$end=0;
while ($end<1500)
{
//generate an authentication code
$string="ABCDEFGHJKLMNPQRSTUVWXYZ123456789";
$string= substr(str_shuffle($string),5,8) ;
//check whether generated code already exist
$query = "select count(*) from auth where code = '$string' ";
$stmt = prepare ($query);
execute($stmt);
$bind = mysqli_stmt_bind_result($stmt, $count);
check_bind_result($bind);
mysqli_stmt_fetch($stmt);
mysqli_stmt_free_result($stmt);
//If generated code does not already exist, insert it to Database table
if ($count == 0)
{
echo $string."<br>";
$query = "insert into auth (Code) values ('$string')";
$stmt = prepare ($query);
execute($stmt);
$end++;
}
}
?>
It generated and inserted 1024 codes in database and printed 667 codes in browser within 15 seconds and the browser continue loading without inserting/printing further codes, until I close the browser window after half an hour.
After that when opening any web page in browser from the WAMP, It shows like the browser is loading and does not show the content. That is, I need to restart the WAMP after running this script before opening any web pages.
I have tried this many times.
Why the script does not generate 1500 codes and always stop when it reach the count 667/1024?
UPDATE
As an experiment, I have added an ELSE clause to the IF condition and wrote the code to print "Code Already Exist" in ELSE clause. And ran the script with an empty(truncated) copy of the same table , then it print and inserted 1024 codes and after that, it print "Code Already Exist" continuously (Around 700 000+ entries within 5 minutes and continuing). And again when running the script with table having only 1024 rows, It doesn't print or insert even a single code. Instead it infinitely continuing print "Code Already Exist".
Another thing I observed that the very first 1024 iteration of the WHILE loop passes the IF condition(if the table is empty). And all the subsequent iteration failed the IF condition.
I dont think that randomiser in str_shuffle is up to this.
If I run it once I get 1024 unique codes and then it just generates duplicates. If I then restart it, it will generate another 976 unique codes giving a total of 2000 codes on the database.
I therefore assume that the randomiser used by str_shuffle() needs a reset to accomplish the generation of the required 1500 unique codes.
Try this minor modification, it will at least stop the execution after 15000 failed attempts at generating a unique code.
Basically I think you have to come up with a much better randomisation mehanism.
<?php
include "../include/top.php";
set_time_limit(0);
ini_set("memory_limit", "-1");
$end=0;
$dups=0;
while ($end<1500 && $dups < 15000)
{
//generate an authentication code
$string="ABCDEFGHJKLMNPQRSTUVWXYZ123456789";
$string= substr(str_shuffle($string),5,8) ;
//check whether generated code already exist
$query = "select count(*) from auth where code = '$string' ";
$stmt = prepare ($query);
execute($stmt);
$bind = mysqli_stmt_bind_result($stmt, $count);
check_bind_result($bind);
mysqli_stmt_fetch($stmt);
mysqli_stmt_free_result($stmt);
//If generated code does not already exist, insert it to Database table
if ($count == 0) {
echo $string."<br>";
$query = "insert into auth (Code) values ('$string')";
$stmt = prepare ($query);
execute($stmt);
$end++;
} else {
$dups++
echo "DUPLICATE for $string, Dup Count = $dups<br>";
}
}
?>
Why you need to restart = you set php timeout so it never times out.
I don't see any specific coding errors. The use of the str_shuffle for creating a authentication code is peculiar because it will prevent duplicated letters, making a much smaller possible range of values. So it may just be repeating the patterns.
Try something like this instead, so that you are shuffling the shuffled string:
$origstring= str_shuffle("ABCDEFGHJKLMNPQRSTUVWXYZ123456789");
while ($end<1500 && $dups < 15000)
{
$origstring = str_shuffle( $origstring);
$string = substr( $newstring, 5, 8);
Or, use something like this to generate the string so that you can have duplicates, creating a much larger range of possible values:
$characters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890';
for ($i = 0; $i < 8; $i++)
{
$code .= $characters[mt_rand(0, 35)];
}
You have to fine-tune some variables in your php.ini configuration file, find those:
; Maximum execution time of each script, in seconds
; http://php.net/max-execution-time
; Note: This directive is hardcoded to 0 for the CLI SAPI
max_execution_time = 600
And, not mandatory, you can change those also:
suhosin.post.max_vars = 5000
suhosin.request.max_vars = 5000
After modification, restart your web server.

MySQL INSERTing rows into a database with datetime sanity checks

I am making a meeting room booking system in which there should be no times within the start and end dates so in theory the validation should check for no dates/times within one start and end date time frame.
I have two tables, I can insert into it fine with both start and end dates so the only columns i am interested in at the moment are these
meetingrooms
|------------------------------------||- bookingtime -|-bookingend-|
I understand the principle behind the sanity check and the check i can do in psudocode. Here is the code i have got so far -
>
p4a_db::singleton()->query("INSERT INTO meetingrooms(location_id, bookingtime, bookingend, merono_id)
WHERE bookingtime < " . $date . " AND bookingend > " . $date . "
OR
bookingdate < " . $date . " AND bookingend > " . $dateend . "
VALUES(?,?,?,?)",
array($location, $date, $dateend, $merono));
I don't want to insert data directly into the statement but until i understand how to do this i am stuck, so the question,
How do i perform a sanity check before the data is inserted so that i don't get dates within booked times.
any help would be greatly appreciated.
Edit:
I've been overthinking my answer and I realized that the old solution will not work in your case since you need the time span, comparing the start and end date is useless.
My way of processing this would be:
Save the dates as int, use 24h system (7:40am is 740, 9:50pm is 2150)
Check for stored dates where: (Start<NewStart<End)
Check for stored dates where: (Start<NewEnd<End)
When processing several rooms, just store room number + time as int. That way you can still use the method from 2 and 3.
2 and 3 can be done in a sql query, check out this link.
Old answer (checking for duplicates)
This is an example of how to check for duplicates (in this case email) before inserting the text:
$emailexist = $mysqli->prepare("select email from users where email = ?");
$emailexist->bind_param('s', $email);
$emailexist->execute();
$emailexist->store_result();
if ($emailexist->num_rows > 0) {
$emailexist->close();
$mysqli->close();
return true;
}
else {
$emailexist->close();
$mysqli->close();
return false;
}
It checks if there are rows which contain the string. If so (if number of rows higher than 0) it returns true (which means, the date already exists).
You can just adapt this to you code.
However, you could also just set the columns to UNIQUE. Then you get an error when trying to insert it. It is easier and you won't have problems with concurrent connections.
after a long and intensive search, I have now got a working example of this method, along with a method of protecting against sql injection, here's the code;
if ($this->BookingValue == 1)
{
$sql = "SELECT COUNT(*) as num FROM meeting_room_bookings
WHERE
(
(? < start_at AND ? > start_at)
OR
(? > start_at AND ? < end_at)
)
AND
meeting_room_id = ?";
$result = p4a_db::singleton()->fetchRow($sql, array($date, $date, $date, $dateend, $merono));
if ( 0 == $result["num"] )
{
p4a_db::singleton()->query("INSERT INTO meeting_room_bookings (start_at, end_at, meeting_room_id)
VALUES
(?,?,?)", array($date, $dateend, $merono));
return true;
}
else
{
return false;
There isn't much to explain about this code, but in term of differences, (excluding the change in column names with the table) the query is now prepared before the value is set, then it is possible to use it in an if statement, thus allowing the validation to take place to filter results between different dates.
along with this i have added validation to stop dates from other meeting rooms being included within the statement via the AND statement where the meeting room id is limeted to a single value.
Although now, which will lead on to a separate question is another thrown error that comes from this statement, i know the insert is sound but something from this prepared statement causes the error:
SQLSTATE[HY093]: Invalid parameter number: number of bound variables does not match number of tokens
File: Pdo.php, Line: 234
Although now i am looking into a error that is thrown from the prepared statement and will update this answer when there is a fix, thanks for the help.

comparing timestamps

HI, My php is very rusty and I can't quite remember how to do this.
I have a script that i only want to call every 15 minutes. I've created a table called last_updated. What I want to do is have some code at the top of my script that queries this last_updated table and if now() minus the last updated is greater than 15 minutes then run the script and also update last_updated to now...if it isn't don't run the script. Does this make sense?
Now I know when I'm updating last_updated I need to use now() To put a new timestamp in but I;m not sure how to do the comparing of now with the db value to see if it's greater then 15 mins.
Any ideas
<?php
$pdo = new PDO('mysql:host=your_host;dbname=your_database', $user, $password, array(PDO::ATTR_ERRMODE=>PDO::ERRMODE_EXCEPTION));
// query the database. change
$stmt = $pdo->query('SELECT UNIX_TIMESTAMP(last_updated_date) FROM last_updated ORDER BY last_updated_date DESC LIMIT 1');
$lastUpdatedTimestamp = $stmt->fetch(PDO::FETCH_COLUMN);
if ((time() - $lastUpdatedTimestamp) > (60 * 15)) {
touch($file);
// do stuff
}
time() gives you the current time in seconds. You should probably unroll 60 * 15 to 900, I just provided it with both numbers to illustrate what was going on.
Also, a file might be better for this than a database table. Have a look at the touch()
function. It changes the modification time of a file, or creates an empty file with the current time as the mod time if it doesn't exist. You can check the file mod time with filemtime()
<?php
$lastUpdated = null;
$file = '/path/to/writable/file/with/nothing/in/it';
if (file_exists($file)) {
$lastUpdated = filemtime($lastUpdated);
}
if (!$lastUpdated || (time() - $lastUpdated) > 900) {
touch($file);
// do stuff
}
You seem to use MySQL as the DBMS. In that case and if you want you can let MySQL do most of the work:
SELECT
pit < Now()-Interval 15 Minute as mustUpdate
FROM
last_updated
WHERE
siteId=?
pit is your DateTime field and siteId is some condition you may have if you store more than one record in the table (which sounds like a good idea to me).
The result (if there is such a record with siteId=?) contains a field mustUpdate which either contains 0 or 1, 1 indicating that the value of pit is more than 15 minutes in the past.

Categories