I have a MySQL table with over 16 million rows and there is no primary key. Whenever I try to add one, my connection crashes. I have tried adding one as an auto increment in PHPMyAdmin and in command line but the connection is always lost after about 10 minutes.
What I would like to do is loop through the table's rows in PHP so I can limit the number of results and with each returned row add an auto-incremented ID number. Since the number of impacted rows would be reduced by reducing the load on the MySQL query, I won't lose my connection.
I want to do something like
SELECT * FROM MYTABLE LIMIT 1000001, 2000000;
Then, in the loop, update the current row
UPDATE (current row) SET ID='$i++'
How do I do this?
I am open to a MySQL solution as well. I just need a process that will not cause me to lose my MySQL connection
Note: the original data was given to me as a txt file. I don't know if there are duplicates but I cannot eliminate any rows. Also, no rows will be added. This table is going to be used only for querying purposes. When I have added indexes, however, there were no problems.
It is easier
ALTER TABLE table_name MODIFY COLUMN id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
In case you want to remove the primary key after the modification
ALTER TABLE table_name MODIFY COLUMN id INT
I would avoid setting primary keys manually. Much safer to let MySQL do it. You can temporarily increase the timeout in the config file:
wait_timeout = 28800
interactive_timeout = 28800
Related
I have a form from which i am inserting data into mysql works fine.But when i delete some data from mysql, and inserted values into database again the autoincrement value is starting from the previous row value.
ForExample:
If i have 1,2,3,4,5 as id's in mydatabse and if i delete 4 and 5 id's from database
and started inserting next data from PHP. then the id's are coming from 6.... But i need to get id as 4 .can any one give suggestions.Thanks in advance.
I'm afraid MySQL does not allow you to "reset" AUTO_INCREMENT fields like that. If you need that behavior, you have to stop using AUTO_INCREMENT and generate your IDs manually.
Auto increment does not (and cannot) guarantee an unbroken sequence.
You can implement this yourself as "SELECT MAX(ID) + 1 FROM MYTABLE;"
But be warned: You will take a slight but noticeable performance hit.
If you are running updates concurrently you risk deadlocks
(again if you are running updates concurrently) you will risk having two inserts with the same key.
You can also implement this by running your own counter in a separate table. You must have program logic to decrement this correctly on a deletion, and, again you will get a performance hot and risk of deadlock as the "counter" will become an object of contention.
You should not play with AUTO_INCREMENT value in a production environment let MySQL take care of its value for you.
If you need to know how many row you have you can use
SELECT COUNT(id) FROM tbl;
Anyway if you really want to change its value the syntax is :
ALTER TABLE tbl AUTO_INCREMENT=101;
I have a MyISAM table in MySQL with three columns - an auto-incrementing ID, an integer (customer id) and a decimal (account balance).
At this time, whenever I initialize my system, I completely wipe the table using:
truncate table xyz;
alter table xyz auto_increment=1001;
Then I repopulate the table with data from PHP. I usually end up having up to 10,000 entries in that table.
However, due to new requirements to the system, I now need to also be able to update the table while the system is running, so I can no longer wipe the table and have to use UPDATE instead of INSERT and update the balances one by one which will be much slower than inserting 20 new records at a time as I'm doing now.
PHP only sends the customer id and the amount to MySQL - the other id is not actually in use.
So my question is this:
Does it make sense to put an index on the customer id to speed up updating given that the input from PHP is most likely not going to be in order? Or will adding the index slow it down enough to not make it worthwhile?
I also don't know if the index is used at all for the UPDATE command or not ...
Any other suggestions on how to speed this up?
It depends on what your update query is. Presumably it is like:
update xyz
set val = <something>
where id = <something else>
If so, an index on id will definitely help speed things up, because you are looking for one record.
If your query looks like:
update xyz
set val = 1.001 * val;
An index will neither help, nor hurt. The entire table will need to be scanned and the index does not get involved.
If your query is like:
update xyz
set id = id+1;
Then an index will be slower. You have to read and write to every row of the table, plus you then have the overhead of maintaining the index.
Ok I'll make this into an answer. If you are saying:
Update xyz set balance=100 where customer_id = 123;
Then yes an index on customer_id will definitely increase the speed since it will find the row to update much quicker.
But for improvement, if you have columns (id,customer_id,balance) and customer_id is unique and id is just an auto incremented column get rid of the id column and make customer_id the primary key. Primary keys do not have to be auto incremented integer columns.
I've got an application where for the first 100 rows (think "first 100 customers") I'd like the default of a certain column to be, say, 10. Then after that promotion is over, I'd like it to be 2. Is there a way to do that automatically so that I'm not flooded with 1000 rows unexpectedly?
Could I query the number of rows each time a new row is added, then run a SQL statement that would alter the default from 10 to 2 if the mysql_num_rows is over 100?
I try to avoid altering a table schema in code, so how about something like:
Leave the default at the permanent value (e.g. 2).
Lock the table for insert.*
Insert a row.
Check the resulting row ID (or however you have set up row counting).
If the ID is less than or equal to 100, update the column (e.g. 2 -> 10).
Unlock the table.
*: There are alternate ways to do this, such as with transactions, that may be better. It depends on the volume you are expecting.
I have a table with the column name id and the settings of both PRIMARY KEY and AUTO INCREMENT. When a row is deleted mid-way through the table, i.e. row 29 in rows 1 - 70, id 29 will simply disappear. Is it possible to have it so that the rows shift up, but the id remains there, i.e. deleting 70 instead of 29, so that the row that had id 30 will now adopt 29, 31 will adopt 30 etc?
This will make it a lot easier when inserting new data into the table in my case if this is possible.
I'm really looking to have the values used up before a new one is created, so it's either this which would be great, or looking for missing values and updating them.
UPDATE:
I wanted this to say, get the row above a certain row, and if all the id columns were in straight value order (no gaps) then I'd be able to do it relatively easily.
In general, it's a hassle to update primary keys in a database since this often sets of a chain of updates through all the tables which reference that primary key. Many engines allow you to force this update to happen automatically using the CASCADE option but it still results in a lot of otherwise unnecessary database updating.
The normal use of auto-incrementing integer primary keys is as permanent, meaningless, immutable values.
That said, if you really want to update those keys, you could do so with the command
UPDATE YourTable SET id = id - 1 WHERE id > (value you deleted)
Alternatively, you can maintain a second integer column, not the primary key of the table, and update this column after each deletion.
answer to the edited question
Previous ID:
SELECT id FROM $mytable WHERE id < $id ORDER BY ID DESC LIMIT 1;
Next ID:
SELECT id FROM $mytable WHERE id > $id ORDER BY ID ASC LIMIT 1;
This will make it a lot easier when inserting new data into the table
in my case if this is possible.
It shouldn't make any difference in your inserts. The column is an auto increment, so it's an unspecified column in your inserts. How does a gap at 29 make it any more or less difficult to insert your 1000th record?
On top of that, if you're re-assigning values to the column that is your primary key, you lose all referential integrity for any tables that reference that column.
To answer your question directly, you can simply drop the auto increment column and recreate it. It'll re-start the numbering at 1 and remove the gaps. However, I strongly recommend against doing this since it's completely unnecessary.
Auto-increment primary keys need to be unique, but they don't need to be contiguous.
You're trying to solve a problem that does not need to be solved. Gaps in auto-increment primary keys are okay. You don't need them to be contiguous. Trying to make them contiguous causes more problems:
It's costly to run queries to find gaps.
It's costly to update lots of rows to shift primary key values to fill the gap. What if you have 10 million rows, and there's a gap after row 2?
It's likely that you'll get a new INSERT while you're shifting values. Now you have to shift again.
If you shift rows to fill the gap, don't forget to ALTER TABLE to change the next auto-increment value that will be generated. Now you have to run two UPDATEs and a table-locking ALTER after every DELETE, which is terrible for scalability.
If you re-use primary key values, you'll confuse your application. It may have deleted a row for a reason. If an application searches for a value based on the primary key, it should find that the row was deleted. It should not find some other arbitrary row that has been shifted into that place.
If you need primary keys to be contiguous, then you're using them as a de facto COUNT, or else some kind of ranking. This is not what the auto-increment mechanism is designed to do.
I am using the following query on MySQL using PHP
$sql = SELECT MAX(SrNo) FROM cart;
$result = mysql_query($sql);
The structure of table CART is
CART (SrNo int(10));
Now I am using the result to do some kind of processing and inserting the maximum value into this table by incrementing one. My problem is that if user1 has got the maximum value of SrNo and is in-between the processing. During this time user2 also requests the server got the same maximum value of SrNo as user1 got and starts processing.
Now when both are done with the processing + insertion into the table, I will have two duplicates in the table CART. How can I prevent this from happening?
In other words, I want no one else to get the maximum value of SrNo until unless one user is finished doing its processing.
NOt a trivial thing with a web application that creates a new connection on each request.
You'd need to add a lockedBy and lockedTime columns to this table, and put into them an ID of user that requested the lock as well as timestamp of when the lock was requested. You need the timestamp, so that you can ignore locks that are longer than certain amount of time.
wouldn't you be fine with the AUTO_INCREMENT feature for PRIMARY KEY?
create table cart ( SrNo int(10) AUTO_INCREMENT PRIMARY KEY ) ENGINE = InnoDB;
then just simply insert new lines and it will automatically increment the new values. That would probably very easily do the trick you are (maybe?) trying to do.
But if you need to lock the maxmium, you can do this:
start transaction;
select max(SrNo) from cart for update;
/* do some other stuff, insert the max value + 1 etc... */
commit;
Remember: You should use transaction for any operation which is not 1 single query!
if you set SrNo as the primary key on the table, then the secound time you try to add the row going to fail, and if it fails, you can request a new number.
ALTER TABLE cart ADD PRIMARY KEY (SrNo);