I have a call center web application that I wrote in php. I need a better way of managing access to the client database. I may have a list of 1000 people who need to be called, and I may have 10 people querying that database at the same time in order to pull a person to call. What's the best way I can keep the same record from NOT coming up between the people who are calling.
Currently, I grab a record, then write to a field on it to indicate that it's locked. So when the next person queries the DB, it checks to make sure it's not pulling anything that was marked as locked. This works fine if it's a slow night. When you have a lot of people going at once, it's just not a fast enough way of fixing it.
Any ideas?
When you query for the person, you could try doing something loike this instead.
Select yourPerson from yourTable WITH (rowlock, xlock) WHERE yourPersonid = 1;
This will give you an exclusive row lock on the person to prevent other people from using the same person concurrently.
I guess that perhaps you are doing something like:
declare #id int, #phone varchar(20), #name varchar(100)
select top (1) #id = id, #phone = phone, #name = name
from People
where status = 'awaiting'
update People
set status = 'in_process'
where id = #id --previously grabbed
--- etc.
It is not safe, because of some other process could also select the same record before your update statement
In this scenario you can use update statement (or delete, depending on your logic) with the output clause
declare #person table (id int, phone varchar(20), name varchar(100))
update top (1) p
set status = 'in_process'
output inserted.id, inserted.phone, inserted.name into #person
from People p
where status = 'awaiting'
Related
I'm setting up to gather long time statistics. It will be recorded in little blocks that I'm planning to stick all into one TEXT field, latest first.. sorta like this
[date:03.01.2016,data][date:02.01.2016,data][date:01.01.2016,data]...
it will be more frequent than that (just a sample) but should remain small enough to keep recording for decades, yet big enough to make me want to optimize it.
I'm looking for 2 things
Can you append to the front of a field in mysql?
Can you read the field partially, just the first 100 characters for example?
The blocks will be fixed length so I can accurately estimate how many characters I need to download to display statistics for X time period.
The answer to your two questions is "yes":
update t
set field = concat($newval, field)
where id = $id;
And:
select left(field, 100)
from t
where id = $id;
(These assume that you have multiple rows in the table.)
That said, you method of storing the data is absolutely not the right thing to do in a relational database.
Presumably, you want a table that looks something like this:
create table t (
tId int auto_increment primary key,
creationDate date,
data <something>
);
(This may be more complicated if data should be multiple columns.)
Then you insert into the table:
insert into t(createDate, data)
select $date, $data;
And you can fetch the most recent row:
select t.*
from t
order by tId desc
limit 1;
All of these are just examples, because your question doesn't give a complete picture of the data.
I was tasked to create this organization registration system. I decided to use MySQL and PHP to do it. Each organization in table orgs has a max_members column and has a unique id org_id. Each student in table students has an org column. Every time a student joins an organization, his org column is equated to the org_id of that organization.
When someone clicks join on an organization page, a PHP file executes.
In the PHP file, a query retrieves the total number of students whose org is equal to the org_id of the organization being joined.
$query = "SELECT COUNT(student_id) FROM students WHERE org = '$org_id'";
The maximum members is also retrieved from the orgs table.
$query = "SELECT max_members FROM orgs WHERE org_id = '$org_id'";
So I have variables $total_members and $max_members. A basic if statement checks if $total_members < $max_members, then updates the student's org equal to the org_id. If not, then it does nothing and notifies the student that the organization is full.
What my main concern is what if this situation happened:
Org A only has one slot left. 29/30 members.
Student A clicks join on Org A (and at the same time)
Student B clicks join on Org A
Student A retrieves data: There is one slot left
Student B retrieves data: There is one slot left
Student A's org = Org A's org_id
Student B's org = Org A's org_id
After the scripts have executed, Org A will show up with 31/30 members
Can this happen? If yes, how can I avoid it?
I've thought about using MySQL variables like this:
SET #org_id = 'what-ever-org';
SELECT #total_members := COUNT(student_id) FROM students WHERE org_main = #org_id;
SELECT #max_members := max_members FROM orgs WHERE org_id = #org_id;
UPDATE students SET org_main = IF(#total_members < #max_members, #org_id, '') WHERE student_id = 99999;
But I don't know if it would make a difference.
Row locking does not apply in my case. I think. I'd love to be proven wrong though.
The code I've written above is a simplified version of the original code. The original code included checking registration dates, org days, etc, however, these things are not related to the question.
What you're describing is usually called a race-condition. It occurs, because you perform two non-atomic operations on your database. To avoid this you need to use transactions, which ensure that the database server prevents this kind of interference. Another approach would be to use a "before update trigger".
Transaction
As you're using MySQL you have to make sure that the DB engine your tables are running on is InnoDB, because MyISAM just doesn't have transactions. Before you do your SELECT you need to start a transaction. Either send START TRANSACTION manually to the database or use a proper PHP implementation for it, e.g. PDO (PDO::beginTransaction()).
In a running transaction you can then use the suffix FOR UPDATE in your SELECT statement, which will lock the rows that have been selected by the query:
SELECT COUNT(student_id) FROM students WHERE org = :orgId FOR UPDATE
After your UPDATE statement you can commit the transaction, which will write the changes permanently to the database and remove the locks.
If you expect a lot of these simultaneous requests to happen, be aware that locking can cause some delay in the response, because the database might wait for a lock to be released.
Trigger
Instead of using transactions you can also create a trigger on the database, which will run before an update is executed on the database. In this trigger you could test if the maximum number of students has been exceeded and throw an error if that's the case. However, this can be a very challenging approach, especially if the value to be checked depends on something in the UPDATE statement. Also it is debatable if it's a good idea to implement this kind of logic on the database level.
There are two ways, use synchronized function in PHP which performs this operation. But if you want to implement all the logic in MySQL (I'd like this method), please use Stored Procedure.
Create a stored procedure as (not exactly):
CREATE PROCEDURE join_org(stu_id int, org_id, int, OUT success int)
DECLARE total_members int;
DECLARE max_members_Allowed;
SELECT COUNT(student_id) INTO total_members FROM students WHERE org = 'org_id';
SELECT max_members INTO max_members_Allowed FROM orgs WHERE org_id = 'org_id';
IF(max_members_Allowed > total_members) Then
UPDATE student SET orgid='org_id';
SET success = 1;
ELSE
SET success = 0;
END IF;
Then register this out variable named 'success' int your PHP code to indicate success or failure. Call this procedure when user clicks join.
I'm rebuilding an add-buddy module and I have a question about the database table design.
In the current version i have a design like this:
id | user1_id | user2_id | status
Each time an user invites another, two rows are created, in one row the current users id is placed in the field user1_id, in the next row the users id is placed in user2_id, which enables me to use a rather simple MySQL query in the list mode of the module:
WHERE a.id=b.user2_id AND b.status = 1 AND b.user1_id = '".$get_user."'
Now, i try to find another way to build this, as i would prefer to have only one row for each friendship, instead of the currently used two.
I have come up with several approaches for table-design and building the logged in users buddylist and i would like to see your opinion about this.
using only one field "ids" for both users ids, comma-seperated. in the select-query i check if the field ids contains the logged in users id, probably using LIKE for that. Than in the while loop i would remove the current users id from the field with str_replace($current_user_id, '', $row['id']);
one row, with fields id1 and id2, using a simple OR in the select-query, in the while loop for building the list i would place an if-statement:
if ($row['id1'] == $current_user_id) use $row['id2'] else use $row['id1']
using two select-queries with UNION like this:
(select * from users where id1 = user_id)
UNION
(select * from users where id2 = user_id)
What do you think of this approaches would be the best for this kind of thing, or do you have another idea?
Edit: thought i would have found an easy solution for table and a matching query, but didnt work, so i deleted it.
Like #Nonym suggested, you could use the status column. Set -1 when the friendship is awaiting confirmation, 0 when the friendship has been denied, and 1 if the friendship is accepted.
Getting a list of all friends for a certain user ID is as simple as calling:
SELECT user2_id FROM users WHERE status = 1 AND user1_id = <your user id> .
Since your database is going to be filled with denied invitations, you could have a cronjob running every 24 hours or so, which will delete all denied invitations, reducing space. A query like
DELETE FROM users WHERE status = 0
would be of use. Actually, you can move even further and add another field called date, which will indicate the date when the request was sent and in the same cronjob include deleting records with status = -1, which have been in the table for too long.
EDIT
Just like #jayden said, the user_id's may be mixed, so the best way is to keep these two consistent. Like user1_id being always the current user and user2_id being the receiver. And to know, who invited who, add another field like addressee or any other (probably) more suitable name, which will hold the id of the user who initiated the request.
For starters, and while waiting for a response to the comment, I'll have to say this as soon as possible:
Approach # 1: Try to avoid doing this as much as possible. Data should be what it is; data, so refrain (when you can) from putting logic into the data. Put your logic perhaps in a view, function or stored procedure :)
[edit]
Now I'm confused. Let's take a step back, shall we?
First, you have a table that appears this way (and I will just make a rough 'sketch' of it) :
TABLE USERS
id (auto increment?)
user1_id (the one initiating the friendship invite)
user2_id (the one user1_id is inviting to form a friendship)
status (the status of the friendship; like
A record is added to the table USERS when:
A user, whose id goes into user1_id invites someone whose id then goes into user2_id , and a status is set to mean something like 2 if an invite had just been sent, and 1 if it was accepted or perhaps 0 if it was rejected, etc
What part/s in the above statements is/are incorrect?
[edit] part ii:
Everything aside, if it comes down to:
Get all records where the viewing user is either an initiator (user1_id) or is a recipient of a buddy request (user2_id)
Then you don't really need a union:
SELECT
CASE
WHEN user1_id = <Id_Of_Viewing_User> THEN user2_id
WHEN user2_id = <Id_Of_Viewing_User> THEN user1_id
END AS user_id
FROM users
WHERE
user1_id = <Id_Of_Viewing_User>
OR
user2_id = <Id_Of_Viewing_User>
Can you try the SQL above? That way, you won't need to worry about the programming part. Whatever comes out in user_id is the id of someone who has either requested to be the viewing user's friend of someone the viewing user requested to be a buddy with.
Is this... anywhere.. anywhere at all.. near what you want?
I need retrieve data from 2 tables at the same time, the tables are not linked by foreigns keys or such.
$query1 = "select idemployee from employee where address like 'Park Avenue, 23421'";
$query2 "select idcompany from company where bossName like 'Peter'";
How can I do this with a kinda thread in PHP?. I've heard that threads are no safe in PHP.
UPDATED:
I got an input field that needs to looks data in both tables, is like search on both tables and show the posible results based on the employee address or boss's name, so you can type an address or just the boss's name. It's just a representation on what I need
Either use a single query, or look into something like Gearman to have workers performing jobs asynchronously (I assume the current code is only an example: if the queries you have there are performing so badly you want to perform them async. then you most likely have a database problem). Having some deamon processes ready to go to perform tasks is relatively simple.
.
Um...
$query1 = "select idemployee from employee where address like ?";
$query2 = "select idcompany from company where bossName like ?";
$stmt1 = $pdo->prepare($query1);
$stmt1->execute(array('Park Avenue, 23421'));
$employee = $stmt1->fetch();
$stmt2 = $pdo->prepare($query2);
$stmt2->execute(array('Peter'));
$company = $stmt2->fetch();
What am I missing?
You could use MYSQLI_ASYNC and http://docs.php.net/mysqli.poll (both only available with php 5.3+ and mysqlnd).
But then you'll need a separate connection to the MySQL server for each query.
Depends on what do you want to do with those queries. If, for example, you are using an AJAX form and can make two requests, you should create separate scripts, where each returns the results for each query. That is effectively running them in separate processes, so they execute simultaneously.
There is no such thing as threading per se in PHP, you can see a hack around it here (using full fledged processes.)
Counter answer to my previous:
Create a new table
CREATE TABLE EmployeeBossXref (
id INT auto_increment,
employee_id INT,
boss_id INT,
company_id INT,
FOREIGN KEY (employee_id) REFERENCES Employee(id),
FOREIGN KEY (boss_id) REFERENCES Employee(id),
FOREIGN KEY (company_id) REFERENCES Company(id)
) ENGINE=InnoDB;
Then change SQL to:
select Employee.name, Boss.name, Company.name FROM Employee
JOIN EmployeeBossXref ebx ON ebx.employee_id=Employee.id
JOIN Employee Boss ON Boss.id=ebx.boss_id
JOIN Company ON Company.id=ebx.company_id
WHERE Employee.address LIKE 'Park Avenue, 23421'
AND Boss.name LIKE 'Peter';
With this system, all bosses are employees (which they logically are!), employees can have more than one, or no boss.
You dont. Do you have an engineering reason you need to do this?
Making two queries simultaneously is still going to hit the same database, and the database is going to do the same amount of work. Its not going to make anything faster, and, you'll have the overhead of the additional threads/processes being created.
If you really need better concurrency, consider a 2nd (or 3rd or 4th) real-time replicated database for SELECT queries, to offload some of the work from the main database.
I have a website that has user ranking as a central part, but the user count has grown to over 50,000 and it is putting a strain on the server to loop through all of those to update the rank every 5 minutes. Is there a better method that can be used to easily update the ranks at least every 5 minutes? It doesn't have to be with php, it could be something that is run like a perl script or something if something like that would be able to do the job better (though I'm not sure why that would be, just leaving my options open here).
This is what I currently do to update ranks:
$get_users = mysql_query("SELECT id FROM users WHERE status = '1' ORDER BY month_score DESC");
$i=0;
while ($a = mysql_fetch_array($get_users)) {
$i++;
mysql_query("UPDATE users SET month_rank = '$i' WHERE id = '$a[id]'");
}
UPDATE (solution):
Here is the solution code, which takes less than 1/2 of a second to execute and update all 50,000 rows (make rank the primary key as suggested by Tom Haigh).
mysql_query("TRUNCATE TABLE userRanks");
mysql_query("INSERT INTO userRanks (userid) SELECT id FROM users WHERE status = '1' ORDER BY month_score DESC");
mysql_query("UPDATE users, userRanks SET users.month_rank = userRanks.rank WHERE users.id = userRanks.id");
Make userRanks.rank an autoincrementing primary key. If you then insert userids into userRanks in descending rank order it will increment the rank column on every row. This should be extremely fast.
TRUNCATE TABLE userRanks;
INSERT INTO userRanks (userid) SELECT id FROM users WHERE status = '1' ORDER BY month_score DESC;
UPDATE users, userRanks SET users.month_rank = userRanks.rank WHERE users.id = userRanks.id;
My first question would be: why are you doing this polling-type operation every five minutes?
Surely rank changes will be in response to some event and you can localize the changes to a few rows in the database at the time when that event occurs. I'm pretty certain the entire user base of 50,000 doesn't change rankings every five minutes.
I'm assuming the "status = '1'" indicates that a user's rank has changed so, rather than setting this when the user triggers a rank change, why don't you calculate the rank at that time?
That would seem to be a better solution as the cost of re-ranking would be amortized over all the operations.
Now I may have misunderstood what you meant by ranking in which case feel free to set me straight.
A simple alternative for bulk update might be something like:
set #rnk = 0;
update users
set month_rank = (#rnk := #rnk + 1)
order by month_score DESC
This code uses a local variable (#rnk) that is incremented on each update. Because the update is done over the ordered list of rows, the month_rank column will be set to the incremented value for each row.
Updating the users table row by row will be a time consuming task. It would be better if you could re-organise your query so that row by row updates are not required.
I'm not 100% sure of the syntax (as I've never used MySQL before) but here's a sample of the syntax used in MS SQL Server 2000
DECLARE #tmp TABLE
(
[MonthRank] [INT] NOT NULL,
[UserId] [INT] NOT NULL,
)
INSERT INTO #tmp ([UserId])
SELECT [id]
FROM [users]
WHERE [status] = '1'
ORDER BY [month_score] DESC
UPDATE users
SET month_rank = [tmp].[MonthRank]
FROM #tmp AS [tmp], [users]
WHERE [users].[Id] = [tmp].[UserId]
In MS SQL Server 2005/2008 you would probably use a CTE.
Any time you have a loop of any significant size that executes queries inside, you've got a very likely antipattern. We could look at the schema and processing requirement with more info, and see if we can do the whole job without a loop.
How much time does it spend calculating the scores, compared with assigning the rankings?
Your problem can be handled in a number of ways. Honestly more details from your server may point you in a totally different direction. But doing it that way you are causing 50,000 little locks on a heavily read table. You might get better performance with a staging table and then some sort of transition. Inserts into a table no one is reading from are probably going to be better.
Consider
mysql_query("delete from month_rank_staging;");
while(bla){
mysql_query("insert into month_rank_staging values ('$id', '$i');");
}
mysql_query("update month_rank_staging src, users set users.month_rank=src.month_rank where src.id=users.id;");
That'll cause one (bigger) lock on the table, but might improve your situation. But again, that may be way off base depending on the true source of your performance problem. You should probably look deeper at your logs, mysql config, database connections, etc.
Possibly you could use shards by time or other category. But read this carefully before...
You can split up the rank processing and the updating execution. So, run through all the data and process the query. Add each update statement to a cache. When the processing is complete, run the updates. You should have the WHERE portion of the UPDATE reference a primary key set to auto_increment, as mentioned in other posts. This will prevent the updates from interfering with the performance of the processing. It will also prevent users later in the processing queue from wrongfully taking advantage of the values from the users who were processed before them (if one user's rank affects that of another). It also prevents the database from clearing out its table caches from the SELECTS your processing code does.