We have a database that has more than 70 tables with more than 10k of users. One of the tables in the database is to keep recording of user's latest actions/changes in order to keep them pending until the gameweek finishes (Football matches), let's call this table 'Substitute'.
However, every week when we advance the system (apply changes) at the end of gameweek, we check to see if there's any duplicated data stored regarding user's changes (we have noticed that a while ago), and remove them manually from the database.
The rows have primary keys (which is not duplicated) only the data is duplicated. It's like the insert query is being fired twice. (not sure about this or why)
Example: Let's say the user has a team of football players (11 on the field and 4 substitutes)
when the user chooses player and clicks substitute, that player (on the field) will be exchanged with the one who's in on the bench (Substitute). This process will happen immediately without the need of user to save (JavaScript). But, it not be saved in the team details, instead, it will be saved in the Substitute table that keep track of changes.
User wants to remove player 1 and enter player 12 (1->12). For the first step the system will record
id Team from_player to_player
0 x 1 12
And if the user do another substitution
id Team from_player to_player
0 x 1 12
1 x 2 13
And when the user substitute back the player (12->1) that record will be deleted(1->12), since the user substituted back the player and the later subs cancels the first.
id Team from_player to_player
1 x 2 13
But SOMETIMES it record that row more than once
id Team from_player to_player
0 x 1 12
1 x 1 12
2 x 1 12
3 x 2 13
Or the duplicate could be in between
id Team from_player to_player
0 x 1 2
1 x 2 13
2 x 1 2
3 x 1 2
N.B This happens only to around 10-100 users every week, even though there are more than 10k users registered in our database.
So what do you think is causing the problem?
I don't want to use INSERT IGNORE, I want to find the root of the problem and stop it.
So basically, my questions are:
1- Is it likely to be a server or client side problem?
2- Could ajax code be called/fired twice under certain circumstances?
3- Could be there an error in the sql server where it executes the same query twice?
Really appreciate your help.
UPDATE:
4- If the problem is with the client side, how i inspect it? Any suggested way?
For those who ask, what does happen when the user subs back the same player.
Let's say this is part of the original players stetting.
| Original Status | After First Subs|After Second Subs|
| | | |
On the field | p1 | P12 | P1 |
-----------------------|-----------------|-----------------|-----------------|
| | | |
Subs(at the bench) | p12 | P1 | P12 |
| | | |
Original Status p1 on the field, p12 on the bench (substitute)
Record of actions:
s1- P1->P12
Status= P12 on the field. P1 on the bench
s2- P12->P1
Status= P1 back on the field. P12 back to the bench.
Note that s2 wont be recorded, but, s1 will be deleted. it's like A * -1, and then -A * -1 again. The will cancel each other.
ajax mite be called more than once, tried inspecting using your firebug. you might need to disable your substitute method until the success flag is up.
Maybe the user fired the query more than once through the UI. Since the id is the PK, the firing the same query twice.
Another question.
When given the situation:
id Team from_player to_player
0 x 1 12
1 x 1 12
2 x 1 12
3 x 2 13
when user substitutes back 12->1, does your function delete "ALL" previous records or just 1?
Related
I have registration form for users. They can registrate to the tour and I need to generate unique UID for each of their registration. I use ID as primary key and its autoincrement, but its only for system purpose only. I need to generate another UID which is used for pairing payments from bank account.
The format of UID I need to generate is {TOUR_NUMBER}{REGISTRATION_NUMBER}, so the numerical series can be like 59001, 59002, 59003 and so on (59 is number of the tour).
Currently I use this scenario.
1) count how many users registered to certain tour already
2) increment that number
3) with while() check if that incremented number is not already used
4a) if not, save the registration
4b) if it is, increment number again and continue with 3)
Problem is, that if there is many users at the same time, system generate the same UID for more users. I've tried to start transaction before point 1) and commit them after 4a) but it did not help.
--- EDIT ---
Maybe I did not explain my problem sufficiently. Let me try it again. Here are simplified tables I have. Tour_id and registration_id columns are auto-incremented.
tour_id
name
1
England
2
Germany
registration_id
name
tour_id
payment_id
1
George
1
1001
2
John
1
1002
3
Jane
1
1003
4
Luke
2
2001
payment_id is generated from tour_id and incremented number of total users registrated to the certain tour. The payment_id format can not be changed, it must be the way it is now, unfortunatelly. The value is used primarly for the payment pairing so it must be connected to certain user from the beginning and saved within user's tour registration.
I just need to ensure, that the payment_id will not be the same, which happend now from time to time, because delays in executing queries in my scenario above I guess. If it happends, the table looks like the table below, which I need to avoid - George and John have the same payments_id because they did registration at the same time. And 1002 is missing.
registration_id
name
tour_id
payment_id
1
George
1
1001
2
John
1
1001
3
Jane
1
1003
4
Luke
2
2001
Big problem...
I'm implementing an online ticket sale system in PHP and MySQL. I have a table called "block_of_tickets", or something like that...
This table looks like:
+-----------+------------+--------------+--------------+--------------+
| idblock | block_name | total_tickets| block_gender | idblock_pair |
+-----------+------------+--------------+--------------+--------------+
| 1 | Block 1- M | 100 | MALE | 2 |
+-----------+------------+--------------+--------------+--------------+
| 2 | Block 1- F | 100 | FEMALE | 1 |
+-----------+------------+--------------+--------------+--------------+
Where:
idblock: The id (primary key) of the block of tickets.
block_name: The name of the block. In the example I have a "Block 1- M" and "Block 1- F" to represente the "Block 1 - Male" and "Block 1 - Female", respectively.
total_tickets: the total of available tickets
block_gender: the gender of the block of tickets
idblock_pair: the block wich is pair of the current block.
Note: There are also other columns, like "price", etc.
Here is the (big) problem:
When there is a "idblock_pair", it means that both block of tickets will share the same total_tickets (available tickets), so both cells must have exactly the same value in this case. As you can see in the example above, block 1 points to block 2 and vice-versa.
Lots of people buy lots of tickets in (almost) the same time, wich means that each sold ticket must decrement 1 in the "total_tickets" field, for both cells.
Database Normalization can solve this. However, it would lose a lot in performance.
I'm almost sure that I should use "SELECT... FOR UPDATE"... but I don't know how, since it's the same table, and a "deadlock" can occur...
How to solve this problem? Do I have to use Triggers? Proccedures? Do I have to use the PHP processing (and transactions) to solve this?
In the example below, one ticket were sold, and now I'm decrementing the total_tickets by 1:
START TRANSACTION;
SELECT *
FROM block_of_tickets
WHERE idblock in (1,2) FOR UPDATE;
UPDATE block_of_tickets
SET total_tickets = (total_tickets - 1)
WHERE idblock in (1,2);
COMMIT;
Is this a nice solution?
I am attempting to upgrade an Old Custom eCommerce system. I am trying to make a log of quantity changes made to an item based on rows from a table (call it "old_table") that the old custom system creates. I can't seem to be able to crack this problem.
As someone starts to make a change to an Item in the system, it creates a record in the "old_table" with the values:
Time | User | Current Qty | Status | Item
-----------------------------------------------------------
2015-07-09 05:30:47 | Admin | 10 | Start | ABC 1
When the person finishes changing the quantity for an Item in the system, it creates a record in the "old_table" with the values:
Time | User | Current Qty | Status | Item
-----------------------------------------------------------
2015-07-09 05:50:47 | Admin | 09 | End | ABC 1
--- Note: Sadly, if the quantity is set to "00" there will not be an "End" entry. I have guessed that i would set a 24 hour limit then would consider that user to have set it to "00" ---
My Goal
In a new Table, I am trying to create a row stating that the User "Admin" subtracted/added a total of "X" of the Item "ABC 1" and if within 24 hours there is no "End" row within the "old_table" for the given item to make a row stating that the User "Admin" set the value to "00".
Anyone have Ideas, suggestions or a solution? I have tried for weeks to solve this. Is it not possible?
You tagged this with mysql AND sql-server. Which system/version is the one in use?
With SQL-Server (>=2008) you could use window function (OVER clause: https://msdn.microsoft.com/en-us/library/ms189461.aspx) which gives you grouped count, sum, avg and other aggregate function.
I have a PHP script pulling a JSON file that is static and updates every 10 seconds. It has details about some events that happen and it just adds to the top of the JSON file. I then insert them into a MySQL database.
Because I have to pull every event every time I pull the file, I will only be inserting new events. The easy way would be to search for the event in the database (primary keys are not the same), but I am talking about ~4000 events every day, and I do not want that many queries just to see if it exists.
I am aware of INSERT IGNORE, but it looks like it only uses PRIMARY_KEY to do this.
What can I do (preferably easily) to prevent duplicates on two keys?
Example:
I have a table events with the following columns:
ID (irrelevant, really)
event_id (that I need to store from the source)
action_id (many action_ids belong to one event_id)
timestamp
whatever...
And my data is my JSON comes out on the first pull like this:
event_id|action_id|...
1 | 1
1 | 2
1 | 3
2 | 1
2 | 2
2 | 3
Then the next pull is this:
event_id|action_id|...
1 | 1
1 | 2
1 | 3
1** | 4**
1** | 5**
2 | 1
2 | 2
2 | 3
2** | 4**
I only want the rows marked with asterisks to be inserted, and the others to be ignored. Remember, primary_key column id is completely in this table, and I just use it for ubiquity.
What command can I use to "INSERT" every event I pull, but ONLY adding those that aren't duplicated by way of the two columns event_id and action_id.
Thanks.
Create a unique index of both columns.
CREATE
UNIQUE INDEX event_action
ON tablename (event_id, action_id)
I do have a DB table, which is kind of spool for performing tasks:
| id | status | owner | param1 |
+----+--------+-------+--------+
| 1 | used | user1 | AAA1 |
| 2 | free | user2 | AAA2 |
| 3 | free | user1 | AAA3 |
| 4 | free | user1 | AAA4 |
| 5 | free | user3 | AAA2 |
This table is being access by many parallel processes, what would be the best way to assure, that each row from the table would be "used" just by single process but also at the same time given out in the same order as they are in table (sorted by id column value)?
My first idea was to simply mark always next row in queue with simple update:
UPDATE table
SET status = "used"
WHERE owner = "userX"
AND status <> "used"
ORDER BY id
LIMIT 1
and then fetch the marked row.
This was not performing at all - with some data (e.g. 3.000.000 rows) and bigger loads process list was full UPDATE statements and mysql crashed with "Out of sort memory" error...
So my next idea is doing following steps/queries:
step1
get the first unused row:
SELECT id
FROM table
WHERE owner = "userX"
AND status = "free"
ORDER BY id
LIMIT 1
step2
try to mark it as used if it is still free:
UPDATE table
SET status = "used"
WHERE id = <id from SELECT above>
AND status = "free"
step3
go to step1 if row was NOT updated (because some other process already used it) or go to step4 if row was updated
step4
do the required work with successfully found row
The disadvantage is that on many concurrent processes there will be always a lot of jumping between steps 1. and 2. till each process finds its "own" row. So to be sure that system works stable - I would need to limit the number of tries each process does and risk that processes may reach the limit and find nothing while there are still entries in the table.
Maybe there is some better way to solve this problem?
P.S. everything is done at the moment with PHP+MySQL
Just a suggestion, instead of sorting and limiting to 1 maybe just grab min(id):
SELECT MIN(id)
FROM table
WHERE owner = "userX"
AND status = "free"
I am also using a MySQL database to choose rows that need to be enqueued for lengthy processing, preferring to do them in the order of the primary index ID column, also using optimistic concurrency control as shown above (no transactions needed). Thank you to #sleeperson for the answer using min(id), that is far superior to order by / limit 1.
I am posting one additional suggestion that allows for graceful restart. I implemented the following that's done only at startup time:
step0
get lingering rows:
SELECT id
FROM table
WHERE owner = "userX"
AND status = "used"
call step4
Etc. After a crash or other unwelcome (yet oh so common event) this will distribute out for processing the rows that should have been done previously, instead of leaving them marked 'used' in the database to trip me up later.