The following is a part of a bigger PHP script which adds or disables users from a MySQL database using a CSV file and a client id field as input.
There are two tables involved, users and users_clients. The later keeps the relationship between users and clients as an user can belong to multiple clients.
This is the structure of the tables
users structure (it has more fields)
id | int(11) (primary key)
user | varchar(100)
pass | varchar(100)
category | int(11)
date | timestamp
name | varchar(100)
email | varchar(255)
users indexes
SEARCH | user | FULLTEXT
SEARCH | name | FULLTEXT
SEARCH | email | FULLTEXT
users_clients structure
id_user | int(11)
id_client | int(11)
status | enum('active','inactive')
This is the basic flow of the script for adding each user from the CSV file:
Check if the user exists for that client.
SELECT
LOWER(user)
FROM
users u
INNER JOIN users_clients uc ON u.id = uc.id_user
WHERE
u.user = '$user'
and uc.id_client = $id_client
If it doesn't exist, add it to the database.
INSERT INTO
users ($fields,id_client)
VALUES
($values,'$id_operation')
Get the id of the inserted user.
I know I could use something like mysql_insert_id here, but what about the race conditions?.
SELECT
u.id as id
FROM
users u
WHERE
u.user = '$user'
and u.id_client = '$id_operation'
Associate the user with the corresponding client.
INSERT INTO
users_clients (id_user, id_client)
VALUES
('$id_user','$id_client')
There are currently 400.000 users in the table. The script takes 10+ minutes to process a CVS with 500 users.
How would you improve this so that it is faster?
Thanks in advance.
PD: If you want to see the complete function, it's available at pastebin.
INSERT INTO table (id,a,b,c) VALUES (5454,1,2,3)
ON DUPLICATE KEY
UPDATE table SET foo WHERE id=xyz;
Set indexes in the DB
use mysqli instead of mysql
collect all the stuff you want to insert and do it with a prepared statement / stored procedure like here How to insert an array into a single MySQL Prepared statement w/ PHP and PDO
don't do 500 SELECTs, simple get the entire database and work through it via a foreach/while loop, checking for the stuff you need
use a construct like above
Important: For the above statement the column id should have an unique index !!!
Wrap INSERTs into transaction and don't worry, mysql_insert_id() is completely safe unless you switch to another database connection.
It is also possible to wrap all your queries to a transaction in result of massive speed improvement.
Related
Let's say I got a website which, when visited, shows what is your lucky word today. The problem is that every word can be lucky for only one person so you need to be fast visiting the website. Below is a sample table with lucky words:
+---------------------+
| lucky_word |
+---------------------+
| cat |
| moon |
| piano |
| yellow |
| money |
+---------------------+
My question is: how can I prevent two (or more) users from accessing that table at one time. I assume that every user reads the first lucky_word from the existing table and the chosen word is deleted immediately so it won't be the next user's lucky word. For instance, I want to avoid cat to be shown to more than one visitor.
Should I solve this using an appropriate MySQL query or some lines in a PHP code or both?
You can use a locking read within a transaction; for example, using PDO:
$pdo = new PDO('mysql:charset=utf8;dbname='.DBNAME, USERNAME, PASSWORD);
$pdo->beginTransaction();
$word = $pdo->query('SELECT lucky_word FROM myTable LIMIT 1 FOR UPDATE')
->fetchColumn();
$pdo->prepare('DELETE FROM myTable WHERE lucky_word = ?')
->execute(array($word));
$pdo->commit();
In MySQL you can lock tables, to prevent other sessions reading and/or writing to the table. In the case of WRITE locks, the first session to request the lock will hold the table until it is released, and then the second session will get it until unlocked, and so forth. That way you can be sure that no two sessions are accessing or manipulating the same data at the same time.
Read all about it in the manual:
https://dev.mysql.com/doc/refman/5.6/en/lock-tables.html
How about adding a datestamp to the table updated when that particular word is used?
You could then use the following pseudo sql...
select word from words where lastdate <> [today];
update words set lastdate = today where word = [word];
A quickly method I used for a similar task:
1) create a table "unique_sequence" with just un field: id -> INT AUTOINCREMENT
CREATE TABLE `erd`.`unique_sequence` (
`id` INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`));
2) when a user arrives on the site:
INSERT INTO unique_sequence VALUES();
SELECT word FROM lucky_word WHERE id = LAST_INSERT_ID();
As The ID that was generated by LAST_INSERT_ID() is maintained in the server on a per-connection basis it should be multi-user safe...
... and so we can be sure that every new user will get a unique ID that match the ones in lucky_word table.
I know it makes little sense... and i'm new to using MySQL...
What i'm trying to do here is, link one tables row to another tables row...
for an example there are two tables..
one table is for user registration and same table is used for login as well...
and the next table is for user posts.. like status updates and all...
here is how i want it...
user_log_info:-
id ( primary )
firstname
lastname
username
email
password
posts:-
id ( primary )
userposts
posted_by
date_post
so as you can see, i want the user_log_info tables username to be automatically copied to posts posted_by row... And i have no idea how i can archive this...
You haven't given nearly enough information to give a full answer, but I'll do my best with what you've given.
Tables
+-----------------+ +-----------------+
| users_log_info | | posts |
+-----------------+ +-----------------+
| int ID (primary)| | int ID (primary)|
+-----------------+ | int posted_by |
+-----------------+
(I left off fields that are irrelevant to what you seem to want to do, I'm just simplifying it)
posted_by is an unofficial foreign key, or referencing the primary key of another table.
To insert, what you can do is along the lines of this:
INSERT INTO posts(...., posted_by) VALUES (...., user.ID)
Where .... is referencing all of your other information to insert
Then, to find information on someone who posted something:
SELECT * FROM users_log_info WHERE ID = Post.posted_by
Or if you want to find all posts by a user:
SELECT * FROM posts WHERE posted_by = user.ID
So, if Bob, who is User ID 3 wants to post "Hi", you might be able to do:
INSERT INTO posts(content, posted_by) VALUES('Hi', bob.ID)
And then when you are outputting the post you might do this:
post = (however you choose the post to put on the page)
userPosted = SELECT * FROM users_log_info WHERE ID = post.posted_by
print post.content + " posted by: " userPosted.Name
Essentially, the field "posted_by" is, to "posts" an arbitrary number, but you know that it links to, or references, a user. It does that by referencing "ID", which is the primary key of users_log_info, so that when you want to get information from users_log_info, is all you need to do is select the entry which has the ID that corresponds to "posted_by". I do recommend naming it something like posterID, however, for easier identification.
I have Db of 100,000 users in MYSQL. In that DB i am having column ID,username,Fname,Lname, etc..
When www.example.com/Jim or www.example.com/123 (Where JIM is username and 123 is ID in the users table)
I am using MYSQL query : select * from users where ID = 123 OR username = Jim
I am executing above query in PHP.
Output of the above query is :
| ID | Username | fname | lname |
+----+----------+--------+---------+
|123 | jim | Jim | Jonson |
My Problem is its taking huge time to select username or ID in the DB.
I have used following query
SELECT * FROMusersUSE INDEX (UsersIndexId) where id=123
Is this right way to call Index ?
EXPLAIN SELECT * FROM `users` WHERE ID =327
OP
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE users Const PRIMARY,UsersIndexId PRIMARY 4 const
1
I sugest you take a look at this: How MySQL Uses Indexes
Quoting from the first paragraph:
Indexes are used to find rows with specific column values quickly.
Without an index, MySQL must begin with the first row and then read
through the entire table to find the relevant rows. The larger the
table, the more this costs. If the table has an index for the columns
in question, MySQL can quickly determine the position to seek to in
the middle of the data file without having to look at all the data. If
a table has 1,000 rows, this is at least 100 times faster than reading
sequentially.
That should help speed up your search.
(Edit: Updated the link to a newer version of the SQL docs)
PS: More specifically, column indexes might be what you want.
You can find more info about adding indexes here: Create Index Syntax
To complete #Kjartan answer, you can try the following :
ALTER TABLE users ADD INDEX id_i (`ID`);
ALTER TABLE users ADD INDEX username_i (`Username`);
Your queries should be faster.
At the moment I have three tables that I am trying to connect and figure out what queries will get the results I need and also follow best practices. MySQL is still pretty new to me, and this is my first stumbling block that I can't figure out.
I am trying to build a simple URL shortener that can "link" multiple long URLs to one short URL. I basically want to have a link www.example.com/google then have google.com, google.co.uk, google.it, etc... and GEO target when the user accesses the link.
My three tables are set up as:
short_id | user_id | short_url //Short URL Table
long_id | user_id | long_url | country_code //Long URL table
user_id | name | password | email | created //User table
I am not sure if foreign keys are the best route. Also, I understand how to add a user, but what queries would I have to run to have a user add a short/long url and have the user_id field in "user" table match the user_id fields in the other tables.
Thanks for the help.
select * from short_url_table
left join long_url_table
using (user_id)
left join user_table
using (user_id) ;
How to connect two mysql tables in which I am inserting data in same time?
I have table customers which is my app table, and I have user which is used by library I use for my framework, this library handles users and authorization.
I wanted to have user_id (which is id from user) in customers, but I am creating those two tables in same time.
Any ideas? Thanks!
the php command mysql_insert_id gives you the id of the last record inserted into a table. So from my undestanding if your inserting a user you could get the id then insert that into another table?
http://php.net/manual/en/function.mysql-insert-id.php
Or have I understood your question wrongly?
It is simply not possible. Nothing can happen at the same time in a program.
What is possible is to:
Start a database transaction
perform your first query
retrieve the table's key; if it's an autoincrement, there are built-in ways to retrieve the last inserted key in every database API
perform your second query using the retrieved key as a parameter
Commit the transaction; But if an error occurred, you need to rollback the whole transaction
This is how it is done, and it behaves exactly like you want it to.
How about a link table?
users_customers
user_id | customer_id
=====================
1 | 648785552
5 | 145778304
4 | 654566055
You can then join the tables together using this table, for example like this:
SELECT users.name, customers.address
FROM users
JOIN users_customers
ON users_customers.user_id = users.id
JOIN customers
ON users_customers.customer_id = customers.id