I have a table in my database :
And I'm using MySQL to insert/update/create info however the time it takes to execute queries is slow example:
First when a user runs my app it verifies they can use it by retrieving all IPs from the db with
SELECT IP FROM user_info
Then using a while loop if the IP is in the db do another query:
"SELECT Valid FROM user_info WHERE IP='".$_SERVER['REMOTE_ADDR']."'""
If Valid is 1 then they can use it otherwise they can't however if their IP is not found in the db it creates a new entry for them using
"INSERT INTO user_info (IP,First) VALUES ('".$_SERVER['REMOTE_ADDR']."','".date("Y/m/d") ."')")"
Now this first script has finished it accesses another - this one was supposed to update the db every minute however i don't think i can accomplish that now ; the update query is this:
"UPDATE user_info SET Name='".$Name."', Status='".$Status."', Last='".$Last."', Address='".$Address."' WHERE IP='".$_SERVER['REMOTE_ADDR']."'")"
All together it takes average 2.2 seconds and there's only 1 row in the table atm
My question is how do i speed up mysql queries? I've read about indexes and how they can help improve performance but i do not fully understand how to use them. Any light shed on this topic would help.
Nubcake
Indexes will become very important as your site grows, but they won't help when you have only one row in your table and it cannot be the reason why your queries take so long. You seem to have some sort of fundamental problem with your database. Maybe it's a latency issue.
Try starting with some simpler queries like SELECT * FROM users LIMIT 1 or even just SELECT 1 and see if you still get bad performance.
Lesser the number of queries, lesser the latency of the system. Try merging queries being made on the same table. For example, your second and third queries can be merged and you can execute
INSERT INTO user_info ...... WHERE Valid=1 AND IP= ...
Check the number of rows affected to know if a new row was added or not.
Also, do not open/close your sql connection at any point in between. The overheads of establishing a new connection could be high.
you could make IP Primary Key and Index
Related
My situation: My website will look at a cookie for a remember me token and a user ID. If the cookie exists it will unhash it and look up the user ID and compare the token. with a "WHERE userid = '' and rememberme = ''".
My question is: Will MySQL optimize this query on the unique userid so that the query does not scan the entire database for this 20+ character token? Or instead should I just select the token from the database and then use a php if comparison to check if the tokens are the same?
In short (tl;dr): Would it be better to check if a token matches in with a MySQL select query, or to grab all the tokens from a databases database and compare the values with a php if conditional?
Thanks!
Simple answer:
YES, the database will definitely optimism your search AS LONG AS THE variable you are searching in the WHERE ... portion is indexed! You definitely should not retrieve all the information via SQL and then do a PHP conditional if you are worried about performance.
So if the id column in your table is not indexed, you should index it. If you have let say... 1 million rows already in your table and run a command like SELECT * FROM user WHERE id = 994321, you would see a definite increase in performance.
Elaborating:
A database (like MySQL) is made to be much faster at executing queries/commands than you would expect that to happen in php for instance. In your specific situation, lets say you are executing this SQL statement:
$sql = "SELECT * FROM users WHERE id = 4";
If you have 1 million users, and the id column is not indexed, MySQL will look through all 1 million users to find all the rows with id = 4. However, if it is indexed, there is something called a b tree that MySQL makes (behind the scenes) which works similarly to how the indexing of a dictionary work.
If you try to find the world slowly in a dictionary, you might open the book in the middle, find words that start with the letter M and then look in the middle again of the pages on your right side hoping to find a letter closer to S. This method of looking for a word is much faster than looking at each single page from the beginning 1 by 1.
For that very reason, MySQL has created indexes to help performance and this feature should definitely be taken advantage of to help increase the speed of your queries.
Comparing it on MySQL-side should be fast. It should find the corresponding row by ID first (fast) and then compare the hash (also fast, since there will be only 1 row to check).
Try analyzing the query with EXPLAIN to find out the actual execution plan.
In my opinion it will be always faster to use WHERE clause no matter what (real) database server will be used. Database engines have strong algorithms for searching data written in language that is compiling to low-level code dedicated to platform, so it cannot be even compared with some loop written in interpreted PHP.
And remember that for PHP loop you will have to send all records from DB to PHP.
If you Data Base its on a separate server than you Apache PHP there is not doubt it would be faster if you write a query in MySQL.
If your PHP and MySQL server is on the same physical server probably PHP would be faster cause the comparison will be made on the RAM But have all the User Id array into RAM would be a waste of RAM so you can use Indexes that would speed up your query
ALTER TABLE table ADD INDEX idx__tableName__fieldName (field)
im currently doing a query that pulls a string from my db. but it has to check for every new row i import. and a file of 100k takes almost 4 hours to import. thats way too long. im assuming that my sql code to check if he exist is the thing slowing it down.
ive heard about indexing but i have no clue what it is or how to use it.
this is the current code im using:
$sql2 = $pdo->prepare('SELECT * FROM prospects WHERE :ssn = ssn');
$sql2->execute(array(':ssn' => $ssn));
if($sql2->fetch(PDO::FETCH_NUM) > 0){
so everytime the phpscript reads a new row, it does this check. problem is, that i cant put it in "on duplicate key" in the sql code. it has to check before going to any sql, because if this is empty, then it should continue doing its thing.
what could i do to make this more efficient regarding time? and also, if index is the way to go, could someone enlighten me how this would be done by either posting examples, linking a guide or php.net page. and how i could read from that index to do what i am in my code
So you have 100k records and you don't have any index? Start then with creating one
CREATE INDEX ssn_index ON prospects (ssn)
Now, each time when you try to select something from prospects table with where condition on ssn column MySQL is going to check where it should look for the records by the index. If this column is strongly selective (there are many different values) the query is going to be performed fast.
You can check your execution plan by querying
EXPLAIN SELECT * FROM prospects WHERE :ssn = ssn
Currently,I am working on one php project. for my project extension,i needed to add more data in mysql database.but,i had to add datas in only one particular table and the datas are added.now,that table size is 610.1 MB and number of rows is 34,91,534.one more thing 22 distinct record is in that table,one distinct record is having 17,00,000 of data and one more is having 8,00,000 of data.
After that i have been trying to run SELECT statement it is taking more time(6.890 sec) to execute.in that table possible number of columns is having index.even though it is taking more time.
I tried two things for fast retrieval process
1.stored procedure with possible table column index.
2.partitions.
Again,both also took more time to execute SELECT query against some distinct record which is having more number of rows.any one can you please suggest me better alternative for my problem or let me know, if i did any mistake earlier which i had tried.
When working with a large amount of rows like you do, you should be careful of heavy complex nested select statements. With each iteration of nested selects it uses more resources to get to the results you want.
If you are using something like:
SELECT DISTINCT column FROM table
WHERE condition
and it is still taking long to execute even if you have indexes and partitions going then it might be physical resources.
Tune your structure and then tune your code.
Hope this helps.
I have two tables, to make it easy, consider the following as an example.
contacts (has name and email)
messages (messages but also has name and email w/c needs to be synced to the contacts table)
now please, for those who are itching to say "use relational method" or foreign key etc. I know, but this situation is different. I need to have a "copy" of the name and email of the messages on the messages table itself and need to sync it from time to time only.
As per the syncing requirement, I need to sync the names on the messages with the latest names on the contacts table.
I basically have the following UPDATE SQL in a loop for all rows in Contacts table
UPDATE messages SET name=(
SELECT name FROM contacts WHERE email = '$cur_email')
WHERE email='$cur_email'
the above loops through all the contacts and is fired as many contacts as I have.
I have several looping ideas to do this as well without using internal SELECT but I just thought the above would be more efficient (is it?), but I was wondering if there's an SQL way that's more efficient? Like:
UPDATE messages SET name=(
SELECT name FROM contacts WHERE email = '$cur_email')
WHERE messages.email=contacts.email
something that looks like a join?
I think it should be more efficient
UPDATE messages m JOIN contacts n on m.email=n.email SET m.name=n.name
Ok. i figured it out now.. using JOINS on update
like:
UPDATE messages JOIN contacts ON messages.email=contacts.email
SET messages.email = contacts.email
WHERE messages.email != contacts.email
it's fairly simple!
BUT... I'm not sure if this is really the ANSWER TO MY POST, since my question is what the "BEST WAY is" in terms of efficiency..
Executing the above query on 2000 records took my system a 4second pause.. where as executing a few select , php loop, and a few update statements felt like it was faster..
hmmmmm
------ UPDATE --------
Well i went ahead and created 2 scripts to test this ..
on my QuadCore i7 Ivybridge machine, surprisingly
a single Update query via SQL JOIN is MUCH SLOWER than doing a rather multi query and loop approach..
on one side i have the above simple query running on 1000 records, where all records need updating...
script execution time was 4.92 seconds! and caused my machine to hicup for a split second.. noticed a 100% spike on one of my cores..
succeeding calls to the script (where no fields where needing update) took the same amount of time! ridiculous..
The other side, involving SELECT JOIN query to all rows needing an update, and a simple UPDATE query looped in a foreach() function in PHP..
took the script
3.45 seconds to do all the updates.. # around 50% single core spike
and
1.04 seconds on succeeding queries (where no fields where needing update)
Case closed...
hope this helps the community!
ps
This is what i meant when debating some logic with programmers who are too much into "coding standards".. where their argument is "do it on the SQL side" if you can as it is faster and more of the standard rather than crude method of evaluating and updating in loops w/c they said was "dirty" code.. sheesh.
MyPHP Application sends a SELECT statement to MySQL with HTTPClient.
It takes about 20 seconds or more.
I thought MySQL can’t get result immediately because MySQL Administrator shows stat for sending data or copying to tmp table while I'd been waiting for result.
But when I send same SELECT statement from another application like phpMyAdmin or jmater it takes 2 seconds or less.10 times faster!!
Dose anyone know why MySQL perform so difference?
Like #symcbean already said, php's mysql driver caches query results. This is also why you can do another mysql_query() while in a while($row=mysql_fetch_array()) loop.
The reason MySql Administrator or phpMyAdmin shows result so fast is they append a LIMIT 10 to your query behind your back.
If you want to get your query results fast, i can offer some tips. They involve selecting only what you need and when you need:
Select only the columns you need, don't throw select * everywhere. This might bite you later when you want another column but forget to add it to select statement, so do this when needed (like tables with 100 columns or a million rows).
Don't throw a 20 by 1000 table in front of your user. She cant find what she's looking for in a giant table anyway. Offer sorting and filtering. As a bonus, find out what she generally looks for and offer a way to show that records with a single click.
With very big tables, select only primary keys of the records you need. Than retrieve additional details in the while() loop. This might look like illogical 'cause you make more queries but when you deal with queries involving around ~10 tables, hundreds of concurrent users, locks and query caches; things don't always make sense at first :)
These are some tips i learned from my boss and my own experince. As always, YMMV.
Dose anyone know why MySQL perform so difference?
Because MySQL caches query results, and the operating system caches disk I/O (see this link for a description of the process in Linux)