I have approx 230K records in MySQL user table and i was updating 14K user information via php with simple foreach from variable and using update query to update the column of 14K user . It took 1 hour, 9 min approx to run the query. First it took 13 min to update approx 1100 records and than it took another 56 min to update the remaining records.
Is this usual or do i need to upgrade my system, where i am running. I have 8GB RAM, i5 processor 2nd gen and using ubuntu 14.04.
If this is usual, what is the best way big companies handle such or bigger number of user records in fastest possible way in mysql ?
Just in case, if some one needs to know the exact code that ran.
foreach($_SESSION['tupple'] as $key => $value) {
$commitUpdate = $mysqli->query("UPDATE user_table set col_name='".$mysqli->real_escape_string($value['col_name'])."' where id='".$mysqli->real_escape_string($value['id'])."'");
}
You can speed things up using a prepared query, so it only has to parse the query once.
$stmt = $mysqli->prepare("UPDATE user_table SET col_name = ? WHERE id = ?");
$stmt->bind_param("si", $col_name, $id);
foreach ($_SESSION['tupple'] as $value) {
$col_name = $value['col_name'];
$id = $value['id'];
$stmt->execute();
}
Also, if you're using InnoDB, start a transaction before the loop, and commit it at the end.
$mysqli->begin_transaction(MYSQLI_TRANS_START_READ_WRITE);
// above loop
$mysqli->commit();
It's also possible to update multiple rows in a single query:
UPDATE user_table
SET col_name = CASE id
WHEN $id1 THEN '$name1'
WHEN $id2 THEN '$name2'
WHEN $id3 THEN '$name3'
...
END
WHERE id IN ($id1, $id2, $id3, ...)
You can use this type of syntax to group multiple entries in $_SESSION['tupple'] into batches.
I want to perform a mysql UPDATE query and then get an array of ids that were effected in the change.
This is my update query
mysql_query("UPDATE table SET deleted='1' WHERE id='$id' OR foo='$foo' OR bar='$bar'");
I know that I can do something like this to get the created id from an INSERT query
mysql_query("INSERT INTO table (id,foo,bar) VALUES ('$id','$foo','$bar')");
$newid = mysql_insert_id();
I don't think MySQL has anything like the OUTPUT or RETURNING clauses that other databases support. You can get the list of ids by running a select before the update:
create table temp_table ids_to_update as
SELECT id
FROM table
WHERE (deleted <> '1' or deleted is null) and *id='$id' OR foo='$foo' OR bar='$bar');
Note that MySQL doesn't do an update when the value doesn't change. Hence the first condition -- which you may or may not find important.
Then, to ensure integrity (in the event of intervening transactions that change the data), you can do:
update table t join
temp_table tt
on t.id = tt.id
set deleted = '1';
You could also wrap the two queries in a single transaction, but I think using a temp table to store the ids is probably easier.
I'm using InnoDB.
I have table A
ID | DATA
1 | Something
2 | something else
table B
user_id | DATA
1 | NULL
my program reads a row from table A and updates table B, then deletes the row from table A after the update statement.
is it possible for two users (2 different concurrent sessions) to read the same row from table A? how can I avoid that?
that's my program
$core = Database::getInstance();
$q = $core->dbh->prepare("SELECT * FROM `tableA` LIMIT 1");
$q->execute();
$result = $q->fetch();
$q = $core->dbh->prepare("UPDATE `tableB` SET `data` = ? where `user_id`= ?");
$q->execute(array($result['data'],$ID));
// how to prevent a second user from reading the same row before the next statement gets executed
$q = $core->dbh->prepare("DELETE FROM `tableA` where `ID`= ?");
$q->execute(array($result['ID']));
Hope this helps and makes clear the view of what you want to achieve.
https://dev.mysql.com/doc/refman/5.0/en/internal-locking.html
You can SELECT ... FOR UPDATE which puts an exclusive lock onto the row (at least if the transaction isolation level is set to somewhat reasonable).
This only works if the storage engine of the table is InnoDB. Also you have to execute all queries inside of the same transaction (so execute BEGIN query at the beginning and COMMIT at the end).
Problem
I have a webpage that does the following (the code is much simplified to show only relevant code.
mysql_query("insert into table1 (field1) values ('value')");
$last_id = mysql_insert_id();
$result = mysql_query("select * from table1 t inner join ... where id = $last_id");
write_a_file_using_result($result);
It happened, that the file was created using a different data set than what I found in the table row.
The only explanation I have is:
call1: page was called 1. time with data set 1.
call1: data set 1 gets inserted for connection 1 but gets not committed to the table.
call2: page was called 2. time with data set 2
call2: data set 2 gets inserted for connection 2 and mysql_insert_id returns the same value
call1: file is generated with date set 1
call2: file cannot be written, because it already exists
Result: The file is generated with data set 1 while the table row contains data row 2.
Config
mysql 5.0.51b
The table:
CREATE TABLE `table1` (
`id` int(11) NOT NULL auto_increment,
(...)
Question
I know that MyISAM does not support transactions. But I really expect that it is impossible to insert two rows and get twice the same id inserted, so that the row can be overwritten.
Is MyISAM unsafe to this point or is there another explanation that I overlook ?
Note
I know the mysql extension for php is outdated, but I did not yet rewrite the application.
Is MyISAM unsafe to this point
No. mysql_insert_id guaranteed to return the right value only.
or is there another explanation that I overlook ?
Most likely. Check your code.
Haven't heard about id issues in MyISAM.
You can try to set link identifier when calling last_insert_id, for example
$link = mysql_connect(...);
mysql_query("insert into table1 (field1) values ('value')",$link);
$last_id = mysql_insert_id($link);
$result = mysql_query("select * from table1 t inner join ... where id = $last_id",$link);
write_a_file_using_result($result);
How do I get the ID of the last updated row in MySQL using PHP?
I've found an answer to this problem :)
SET #update_id := 0;
UPDATE some_table SET column_name = 'value', id = (SELECT #update_id := id)
WHERE some_other_column = 'blah' LIMIT 1;
SELECT #update_id;
EDIT by aefxx
This technique can be further expanded to retrieve the ID of every row affected by an update statement:
SET #uids := null;
UPDATE footable
SET foo = 'bar'
WHERE fooid > 5
AND ( SELECT #uids := CONCAT_WS(',', fooid, #uids) );
SELECT #uids;
This will return a string with all the IDs concatenated by a comma.
Hm, I am surprised that among the answers I do not see the easiest solution.
Suppose, item_id is an integer identity column in items table and you update rows with the following statement:
UPDATE items
SET qwe = 'qwe'
WHERE asd = 'asd';
Then, to know the latest affected row right after the statement, you should slightly update the statement into the following:
UPDATE items
SET qwe = 'qwe',
item_id=LAST_INSERT_ID(item_id)
WHERE asd = 'asd';
SELECT LAST_INSERT_ID();
If you need to update only really changed row, you would need to add a conditional update of the item_id through the LAST_INSERT_ID checking if the data is going to change in the row.
This is officially simple but remarkably counter-intuitive. If you're doing:
update users set status = 'processing' where status = 'pending'
limit 1
Change it to this:
update users set status = 'processing' where status = 'pending'
and last_insert_id(user_id)
limit 1
The addition of last_insert_id(user_id) in the where clause is telling MySQL to set its internal variable to the ID of the found row. When you pass a value to last_insert_id(expr) like this, it ends up returning that value, which in the case of IDs like here is always a positive integer and therefore always evaluates to true, never interfering with the where clause. This only works if some row was actually found, so remember to check affected rows. You can then get the ID in multiple ways.
MySQL last_insert_id()
You can generate sequences without calling LAST_INSERT_ID(), but the
utility of using the function this way is that the ID value is
maintained in the server as the last automatically generated value. It
is multi-user safe because multiple clients can issue the UPDATE
statement and get their own sequence value with the SELECT statement
(or mysql_insert_id()), without affecting or being affected by other
clients that generate their own sequence values.
MySQL mysql_insert_id()
Returns the value generated for an AUTO_INCREMENT column by the
previous INSERT or UPDATE statement. Use this function after you have
performed an INSERT statement into a table that contains an
AUTO_INCREMENT field, or have used INSERT or UPDATE to set a column
value with LAST_INSERT_ID(expr).
The reason for the differences between LAST_INSERT_ID() and
mysql_insert_id() is that LAST_INSERT_ID() is made easy to use in
scripts while mysql_insert_id() tries to provide more exact
information about what happens to the AUTO_INCREMENT column.
PHP mysqli_insert_id()
Performing an INSERT or UPDATE statement using the LAST_INSERT_ID()
function will also modify the value returned by the mysqli_insert_id()
function.
Putting it all together:
$affected_rows = DB::getAffectedRows("
update users set status = 'processing'
where status = 'pending' and last_insert_id(user_id)
limit 1"
);
if ($affected_rows) {
$user_id = DB::getInsertId();
}
(FYI that DB class is here.)
This is the same method as Salman A's answer, but here's the code you actually need to do it.
First, edit your table so that it will automatically keep track of whenever a row is modified. Remove the last line if you only want to know when a row was initially inserted.
ALTER TABLE mytable
ADD lastmodified TIMESTAMP
DEFAULT CURRENT_TIMESTAMP
ON UPDATE CURRENT_TIMESTAMP;
Then, to find out the last updated row, you can use this code.
SELECT id FROM mytable ORDER BY lastmodified DESC LIMIT 1;
This code is all lifted from MySQL vs PostgreSQL: Adding a 'Last Modified Time' Column to a Table and MySQL Manual: Sorting Rows. I just assembled it.
Query :
$sqlQuery = "UPDATE
update_table
SET
set_name = 'value'
WHERE
where_name = 'name'
LIMIT 1;";
PHP function:
function updateAndGetId($sqlQuery)
{
mysql_query(str_replace("SET", "SET id = LAST_INSERT_ID(id),", $sqlQuery));
return mysql_insert_id();
}
It's work for me ;)
SET #uids := "";
UPDATE myf___ingtable
SET id = id
WHERE id < 5
AND ( SELECT #uids := CONCAT_WS(',', CAST(id AS CHAR CHARACTER SET utf8), #uids) );
SELECT #uids;
I had to CAST the id (dunno why)... or I cannot get the #uids content (it was a blob)
Btw many thanks for Pomyk answer!
Hey, I just needed such a trick - I solved it in a different way, maybe it'll work for you. Note this is not a scalable solution and will be very bad for large data sets.
Split your query into two parts -
first, select the ids of the rows you want to update and store them in a temporary table.
secondly, do the original update with the condition in the update statement changed to where id in temp_table.
And to ensure concurrency, you need to lock the table before this two steps and then release the lock at the end.
Again, this works for me, for a query which ends with limit 1, so I don't even use a temp table, but instead simply a variable to store the result of the first select.
I prefer this method since I know I will always update only one row, and the code is straightforward.
ID of the last updated row is the same ID that you use in the 'updateQuery' to found & update that row. So, just save(call) that ID on anyway you want.
last_insert_id() depends of the AUTO_INCREMENT, but the last updated ID not.
My solution is , first decide the "id" ( #uids ) with select command and after update this id with #uids .
SET #uids := (SELECT id FROM table WHERE some = 0 LIMIT 1);
UPDATE table SET col = 1 WHERE id = #uids;SELECT #uids;
it worked on my project.
Further more to the Above Accepted Answer
For those who were wondering about := & =
Significant difference between := and =, and that is that := works as a variable-assignment operator everywhere, while = only works that way in SET statements, and is a comparison operator everywhere else.
So SELECT #var = 1 + 1; will leave #var unchanged and return a boolean (1 or 0 depending on the current value of #var), while SELECT #var := 1 + 1; will change #var to 2, and return 2.
[Source]
If you are only doing insertions, and want one from the same session, do as per peirix's answer. If you are doing modifications, you will need to modify your database schema to store which entry was most recently updated.
If you want the id from the last modification, which may have been from a different session (i.e. not the one that was just done by the PHP code running at present, but one done in response to a different request), you can add a TIMESTAMP column to your table called last_modified (see http://dev.mysql.com/doc/refman/5.1/en/datetime.html for information), and then when you update, set last_modified=CURRENT_TIME.
Having set this, you can then use a query like:
SELECT id FROM table ORDER BY last_modified DESC LIMIT 1;
to get the most recently modified row.
No need for so long Mysql code. In PHP, query should look something like this:
$updateQuery = mysql_query("UPDATE table_name SET row='value' WHERE id='$id'") or die ('Error');
$lastUpdatedId = mysql_insert_id();