I have a rounds mysql table (innodb) that is used to track all games that given user plays. It has the following columns: id (int), userId (int), gameId (int), status (int).
Id is table's primary key, userId and gameId represent foreign keys to other tables and in status the int values represent 3 different states if the game round - in progress, finished, error.
One user is not allowed to proceed with new game round until his previous round is still in progress.
My code does the following:
SELECT id FROM rounds WHERE userId = <user> AND gameId = <game> AND status = <in progress> LIMIT 1
If the select returns result, an error is thrown. Otherwise I'm creating new round:
INSERT INTO rounds (userId, gameId, status) VALUES (<user>, <game>, <in progress>);
There is no other code between the select and insert and most of the time everything seems to be working correctly and creating new game round, while the previous is still in progress, results in error.
But when I test the code for concurrency and performance under big load, there are some rounds that manage to get inserted simultaneously. (I'm using simple node script that send 200 requests asynchronously.)
Do you have any ideas how should I alter the code so that in all cases I have maximum 1 active round?
I seem to be stuck on this problem and I know there must be simple solution buy I just can't see it. :(
Any help is greatly appreciated!
PS: I've tried using INSERT ... INTO rounds SELECT ... FROM ROUNDS WHERE NOT EXISTS (SELECT id FROM rounds WHERE userId=<user> AND gameId=<game> AND status=<in progress>) but this results in deadlocks..
EDIT:
The code looks something like this:
In game:
public function play() {
$gamesInProgress = $this->repo->getGamesInProgress($this->user->getId(), $this->id, self::STATUS_IN_PROGRESS);
if($gamesInProgress) throw \Exception('Only one active game is allowed.');
$this->createGame();
// some other code
}
In Repo:
public function checkForGamesInProgress($userId, $gameId, $status) {
$stmt = $this->dbh->prepare('SELECT `id` FROM `rounds` WHERE `userId`=:userId AND `gameId`=:gameId AND `status`=:status LIMIT 1');
$stmt->prepare([
'userId' => $userId,
'gameId' => $gameId,
'status' => $status
]);
return $stmt->fetchColumn();
}
It would be a good practice to use "composite key". This way the database ensured that no duplicated rows are inserted. But then you will have to handle the database error when (accidentally) trying to insert the duplicated entry.
I know that you have tried this, but what happens when you run this?
INSERT INTO rounds (userId, gameId, status)
SELECT <user>, <game>, '<in progress>' FROM rounds WHERE NOT EXISTS(
SELECT id FROM rounds WHERE userId = <user> AND gameId = <game> AND status = '<in progress>' LIMIT 1)
Related
I'm trying to make a tracking system on my website that's very basic, with just the amount of people present recorded.
Unfortunately, the code below doesn't work. I checked the error logs in my server and basically here's the issue : After the first execute there are no entities found, even though there is already an entry with that value, and so the code goes straight to the "else" and then crashes because there is already an entry with that primary key. Can someone help me find why it doesn't find the entity on the first execute?
Here is the code :
$q = "SELECT date, amount FROM tracking WHERE date = ?";
$req = $bdd->prepare($q);
$req->execute(date("Y-m-d"));
$results = $req->fetchAll();
if (count($results) != 0){
$results["amount"] = $results["amount"] + 1;
$track = $bdd->prepare("UPDATE tracking SET amount = ? WHERE DATE(date) = ?");
$track->execute(array($results["amount"], date("Y-m-d")));
exit;
}
else{
$q = 'INSERT INTO tracking (date, amount) VALUES (:val1, :val2)';
$req = $bdd->prepare($q);
$req->execute(
[
"val1" => date(Y-m-d),
"val2" => 1,
]
);
}
Thanks
It looks like your tracking table must only have one row per date. There's a way to handle that directly in MySQL's query language.
First, make your date column the primary key of your table, or create a unique index on it. You create the unique index like this.
CREATE UNIQUE INDEX trackdate ON tracking(date);
Then use this single query to do your insertion / update.
INSERT INTO tracking (date, amount) VALUES (CURDATE(), 1)
ON DUPLICATE KEY UPDATE amount = amount + 1;
Each time you run this query it will either insert the necessary row, or increment the amount column. And it does it "atomically," meaning that if two different php program instances try to do it concurrently, it won't get confused.
I currently have a script that runs every 5 minutes and selects the data from a table on server 1 and an identical table on server2. This is a workaround for replication, essentially, since we don't have that option currently.
The script is successful but I've realized that it misses records sometimes, for whatever reason. The current script selects all records from the destination table, stores the max primary key, selects all data from the source table and then inserts anything with a greater Primary key into the dest. table.
I'd like to modify the script slightly and instead of using max id, just say "if a row has an primary key that doesn't exist in the destination table, insert that row there."
Again these are cloned tables so the structure is the same and they both use AI Primary Keys.
Here's the current working script:
$latest_result = $conn2->query("SELECT MAX(`SESSIONID`) FROM
`ambition`.`session`");
$latest_row = $latest_result->fetch_row();
$latest_session_id = $latest_row[0];
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "SELECT * FROM
`cdrdb`.`session` WHERE `SESSIONID` > $latest_session_id");
// Loop on the results
while($source = $source_data->fetch_assoc()) {
// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM
ambition.session WHERE SESSIONID = '".$source['SESSIONID']."' ") or
die(mysqli_error($conn2));
//if query returns false, rows don't exist with that new ID.
if ($row_exists->num_rows == 0){
//Insert new rows into ambition.session
$stmt = $conn2->prepare("INSERT INTO ambition.session (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE //etc. There are a lot of columns so I
ommitted the others
Is there a way I can slightly modify this to just insert what doesn't exist rather than relying on the MAX ID?
Or is there something here that would be a culprit as to why it's missing records?
You could use INSERT INTO SELECT and check if value is already in target:
INSERT INTO trg_table (cols)
SELECT cols
FROM src_table s
WHERE NOT EXISTS (SELECT 1 FROM trg_table t WHERE t.id = s.id);
Following is my database in mysql:
Id Username Password
1 admin admin
2 jay jay1
3 suman xyza
4 chintan abcde
This is my code in php:
$fetchid = mysql_query(" SELECT MAX(Id) As max From user;");
$row = mysql_fetch_array($fetchid);
$largest = $row['max'];
$largest++;
$user= $_POST['username'];
$pass= $_POST['password'];
$result = mysql_query(" INSERT INTO `proshell`.`user` (
`Id` ,
`Username` ,
`Password`
)"."
VALUES (
'".$largest."', '".$user."', '".$pass."'
);");
Problem:
Now if I delete row with Id=1 and then re-enter the data then it should use ID=1 then Again I reinsert the data it use ID=5
It works like this:
if I delete row with Id=1 and then re-enter the data the Id it gets is 5 but then 1 is free so,
What should I write to perform that task.
First, if you set your Id column to AUTO_INCREMENT you don't need the following part in your code at all:
$fetchid = mysql_query(" SELECT MAX(Id) As max From user;");
$row = mysql_fetch_array($fetchid);
$largest = $row['max'];
$largest++;
Because AUTO_INCREMENT will automatic add new value to your ID colume.
But if you don't set it to AUTO_INCREMENT, the above code will grab the MAXIMUM ID value (in this case, 4).
When you re-enter your data again after you delete the row 1, the MAXIMUM ID still 4, so your new ID value will be 5 (from $largest++;).
.....
If you really need to use consecutive ids as you PK, you need to re-write you code but I suggest you to use UUID for you ID column instead.
You can easily generate UUID by using uuid().
How about the UUID performance? Refer to Dancrumb's answer about this:
A UUID is a Universally Unique ID. It's the universally part that you should be considering here.
Do you really need the IDs to be universally unique? If so, then UUIDs
may be your only choice.
I would strongly suggest that if you do use UUIDs, you store them as a
number and not as a string. If you have 50M+ records, then the saving
in storage space will improve your performance (although I couldn't
say by how much).
If your IDs do not need to be universally unique, then I don't think
that you can do much better then just using auto_increment, which
guarantees that IDs will be unique within a table (since the value
will increment each time)
see. UUID performance in MySQL?
EDIT: I don't suggest you run query on the whole table just to find the MAX ID value before inserting new value everytime, because it will give you a performance penalty (Imagine that if you have million rows and must query on them everytime just to insert a new row, how much workload causes to your server).
It is better to do the INSERT just as INSERT, no more than that.
EDIT2:
If you really want to use consecutive ids, then how about this solution?
Create new TABLE just for store the ids for insert (new ids and the ids that you deleted).
For example:
CREATE TABLE cons_ids (
ids INT PRIMARY KEY,
is_marker TINYINT DEFAULT 0
);
then initial ids with values from 1-100 and set marker to be '1' on some position, e.g. 80th of whole table. This 'marker' uses to fill your ids when it's nearly to empty.
When you need to INSERT new Id to your first table, use:
$result = mysql_query("SELECT ids, marker FROM cons_ids ORDER BY ids ASC LIMIT 1;");
$row = mysql_fetch_row($result);
and use $row[0] for the following code:
INSERT INTO yourtable (Id, Username, Password)
VALUES ($row[0], $username, $password);
DELETE FROM cons_ids
WHERE ids = $row[0];
This code will automatically insert the lowest number in cons_ids as your Id and remove it from the cons_ids table. (so next time you do insert, it will be the next lowest number)
Then following with this code:
if ($row[1] == 1) {
//add new 100 ids start from the highest ids number in cons_ids table
//and set new marker to 80th position again
}
Now each time you delete a row from your first table, you just add the Id from the row that you deleted to cons_ids, and when you do INSERT again, it will use the Id number that you just deleted.
For example: your current ids in cons_ids is 46-150 and you delete row with Id = 14 from first table, this 14 will add to your cons_ids and the value will become 14, and 46-150. So next time you do INSERT to your first table, your Id will be 14!!.
Hope my little trick will help you solve your problem :)
P.S. This is just an example, you can modify it to improve its performance.
First of all, as I understand, you are selecting highest column ID which should be always the last one (since you set auto-increment on ID column).
But what are you trying to do is actually filling up holes after delete query, right?
If you are really looking for such approach, try to bypass delete operation by making new boolean column where you flag record if it is active or not (true/false).
SQL table change:
Id Username Password Active
1 admin admin false
2 jay jay1 true
3 suman xyza false
4 chintan abcde true
PHP request:
$fetchid = mysql_query(" SELECT MIN(Id) As min FROM user WHERE active = false;");
$result = mysql_query(" INSERT INTO `proshell`.`user` (
`Id` ,
`Username` ,
`Password`
`Active`
)"."
VALUES (
'".$largest."', '".$user."', '".$pass."', 'true'
);");
I have the following two tables
Table player:
player_id (int)(primary)
player_name (varchar)
player_report_count (int)
Table report:
report_id (int)(primary)
player_id
report_description
report_location
Firstly I ask the user for the player_name and insert it into the player database. From here the player is given an id.
Then I tried to grab the value of the players report count and increment the current value by one (which isn't working).
This is followed by grabbing the playerId from the player table and then inserting into the corresponding column from the report table (also does not work).
When I insert some values into the database, the names, description and report are added to the database however the playerID remains at 0 for all entries and the player_report_count remains at a consistent 0.
What is the correct way to make these two features function? And also is there a more efficient way of doing this?
<?php
$records = array();
if(!empty($_POST)){
if(isset($_POST['player_name'],
$_POST['report_description'],
$_POST['report_location'])){
$player_name = trim($_POST['player_name']);
$report_description = trim($_POST['report_description']);
$report_location = trim($_POST['report_location']);
if(!empty($player_name) && !empty($report_description) && !empty($report_location)){
$insertPlayer = $db->prepare("
INSERT INTO player (player_name)
VALUES (?)
");
$insertPlayer->bind_param('s', $player_name);
$reportCount = $db->query("
UPDATE player
SET player_report_count = player_report_count + 1
WHERE
player_name = $player_name
");
$getPlayerId = $db->query("
SELECT player_id
FROM player
WHERE player_name = $player_name
");
$insertReport = $db->prepare("
INSERT INTO report (player_id, report_description, report_location)
VALUES (?, ?, ?)
");
$insertReport->bind_param('iss', $getPlayerId, $report_description, $report_location);
if($insertPlayer->execute()
&& $insertReport->execute()
){
header('Location: insert.php');
die();
}
}
}
Main issue here is you are getting player details before inserting it. $getPlayerId will return empty result always.
Please follow the order as follows.
Insert player details in to player table and get payerid with mysql_insert_id. After binding you need to execute to insert details to the table.
Then bind and execute insert report .
Then update the player table by incrementing report count with playerid which you got in step 1.
Note : use transactions when inserting multiple table. This will help you to rollback if any insert fails.
MySQL Query will return result object. Refer it from here https://stackoverflow.com/a/13791544/3045153
I hope it will help you
If you need to catch the ID of the last insterted player, This is the function you need if you're using PDO or if it's a custom Mysql Class, you need the return value of mysql_insert_id() (or mysqli_insert_id()) and then directly use it in the next INSERT INTO statement
Background:
I am parsing a 330 meg xml file into a DB (netflix catalog) using PHP script from the console.
I can successfully add about 1,500 titles every 3 seconds until i addd the logic to add actors, genre and formats. These are separate tables linked by an associative table.
right now I have to run many, many queries for each title, in this order ( i truncate all tables first, to eliminate old titles, genres, etc)
add new title to 'titles' and capture insert id
check actor table for exising actor
if present, get id, if not insert
actor and get insert id
insert title id and actor id into
associative table
(steps 2-4 are repeated for genres too)
This drops my speed don to about 10 per 3 seconds. which would take eternitty to add the ~250,00 titles.
so how would I combine the 4 queries into a single query, without adding duplicate actors or genres
My goal is to just write all queries into a data file, and do a bulk insert.
I started by writing all associative queries into a data file, but it didn't do much for performance.
I start by inserting th etitle, and saving ID
function insertTitle($nfid, $title, $year){
$query="INSERT INTO ".$this->titles_table." (nf_id, title, year ) VALUES ('$nfid','$title','$year')";
mysql_query($query);
$this->updatedTitleCount++;
return mysql_insert_id();
}
that is then used in conjunction with each actor's name to create the association
function linkActor($value, $title_id){
//check if we already know value
$query="SELECT * FROM ".$this->persons_table." WHERE person = '$value' LIMIT 0,1";
//echo "<br>".$query."<br>";
$result=mysql_query($query);
if($result && mysql_num_rows($result) != 0){
while ($row = mysql_fetch_assoc($result)) {
$value_id=$row['id'];
}
}else{
//no value known, add to persons table
$query="INSERT INTO ".$this->persons_table." (person) VALUES ('$value')";
mysql_query($query);
$value_id=mysql_insert_id();
}
//echo "linking title:".$title_id." with rel:".$value_id;
$query = " INSERT INTO ".$this->title_persons_table." (title_id,person_id) VALUE ('$title_id','$value_id');";
//mysql_query($query);
//write query to data file to be read in bulk style
fwrite($this->fh, $query);
}
This is a perfect opportunity for using prepared statements.
Also take a look at the tips at http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html, e.g.
To speed up INSERT operations that are performed with multiple statements for nontransactional tables, lock your tables
You can also decrease the number of queries. E.g. you can eliminate the SELECT...FROM persons_table to obtain the id by using INSERT...ON DUPLICATE KEY UPDATE and LAST_INSERT_ID(expr).
( sorry, running out of time for a lengthy description, but I wrote an example before noticing the time ;-) If this answer isn't downvoted too much I can hand it in later. )
class Foo {
protected $persons_table='personsTemp';
protected $pdo;
protected $stmts = array();
public function __construct($pdo) {
$this->pdo = $pdo;
$this->stmts['InsertPersons'] = $pdo->prepare('
INSERT INTO
'.$this->persons_table.'
(person)
VALUES
(:person)
ON DUPLICATE KEY UPDATE
id=LAST_INSERT_ID(id)
');
}
public function getActorId($name) {
$this->stmts['InsertPersons']->execute(array(':person'=>$name));
return $this->pdo->lastInsertId('id');
}
}
$pdo = new PDO("mysql:host=localhost;dbname=test", 'localonly', 'localonly');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// create a temporary/test table
$pdo->exec('CREATE TEMPORARY TABLE personsTemp (id int auto_increment, person varchar(32), primary key(id), unique key idxPerson(person))');
// and fill in some data
foreach(range('A', 'D') as $p) {
$pdo->exec("INSERT INTO personsTemp (person) VALUES ('Person $p')");
}
$foo = new Foo($pdo);
foreach( array('Person A', 'Person C', 'Person Z', 'Person B', 'Person Y', 'Person A', 'Person Z', 'Person A') as $name) {
echo $name, ' -> ', $foo->getActorId($name), "\n";
}
prints
Person A -> 1
Person C -> 3
Person Z -> 5
Person B -> 2
Person Y -> 6
Person A -> 1
Person Z -> 5
Person A -> 1
(someone might want to start a discussion whether a getXYZ() function should perform an INSERT or not ...but not me, not now....)
Your performance is glacially slow; something is very Wrong. I assume the following
You run your dedicated, otherwise-idle database server on respectable hardware
You have tuned it to some extent (i.e. at least configure it to use a few gigs of ram properly) - engine-specific optimisations will be required
You may be being stung by doing lots of tiny operations with autocommit on; this is a mistake as it generates an unreasonable number of disc IO operations. You should do a large amount of work (100, 1000 records etc) in a single transaction then commit it.
The lookups may be slowing things down because of the simple overhead of doing the queries (the queries themselves will be really easy as you'll have an index on actor name).
I also question your method of assuming that no two actors have the same name - surely your original database contains a unique actor ID, so you don't get them mixed up?
Can you use a language other than PHP? If not, are you running this as a PHP stand-alone script or through a webserver? The webserver is probably adding a lot of overhead you don't need.
I do something very similar at work, using Python, and can insert a couple thousand rows (with associative table lookups) per second on your standard 3.4 GHz, 3GB RAM, machine. MySQL database isn't hosted locally but within the LAN.