I have an extremely large database table - nearly 20 million records.
The records do not have a unique ID number. So, I've inserted the new field.
Now, I would like to populate it with ID numbers, increasing by 1, starting with the first ID number being 10,000,001.
FYI - I am using WAMP on a local machine and I've dialed all my max times upto 5000 seconds and dialed up several other variables in php.ini and mysql.ini in order to do the upload in the first place (which took more than 10 hours!!).
In the past, or with other DB's, I might have exported the data into excel and then whipped up some text to paste back into phpmyadmin to UPDATE the records. This is fine when working with 5K records, or even 100K records, but this seems unmanagable with 20 million records.
Thanks in advance!!
Just run these two queries one after the other in the SQL tab:
ALTER TABLE mytable AUTO_INCREMENT=10000001;
ALTER TABLE mytable ADD `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST;
MySQL will then create the id field and fill it in sequentially starting at 10000001.
This works in SQL Server, maybe you can adapt it to MySQL:
declare #value int
set #value=10000000
update your_table
set #value+=1,id=#value
It will update all your ID rows starting at 10000001 increasing by 1.
I hope, at least, gives you some ideas.
All you need to do is set the column to be AUTO_INCREMENT and mysql will number the rows for you. Let's say you want your new column to be named 'id'.
alter table yourtable auto_increment = 10000001;
alter table yourtable add id int unsigned primary key auto_increment;
You can issue these commands in the sql panel of phpMyAdmin -- just leave off the semicolon at the end.
Related
I have a php script that logs inputs from a form into a mysql database table. I'm looking for a way to insert this data untill 3 rows are created, after which it has to update the existing rows so that the first one updates to the new input, the second one to the former first input and the third one to the former second input.
Table:
CREATE TABLE IF NOT EXISTS inputlog (
id int(11) NOT NULL auto_increment,
userid int(11) NOT NULL default '0',
name text,
value text,
PRIMARY KEY (id)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;`
For the first three rows i use:
insert into inputlog (userid,name,value) values('$userid','$name','$value')
After that is has to become:
update inputlog set value = '$value' where userid = '$userid' and name = '$name'
where it has to update all the successive rows.
How can i accomplish this?
Too long for comments, so...
Looks like you want to have only 3 rows in your table because you want the data to be sorted by the id. So id=1 will be the latest value, then id=2 and finally id=3.
In short, do not do that, the id field can be any value. Do not code for that. The danger is if you use the id in another table as a foreign key, you will loose referential integrity. What I propose is:
Add an timestamp column for each row.
Every time you insert a new value, set the timestamp column to NOW()
When selecting, sort on the timestamp and limit to 3 results
If you MUST have only 3 rows, you can then delete the row except for the 3 most recent timestamps.
But... if you must do that...
perform a SELECT with the first 2 lines
truncate the table (delete all rows)
insert the new line, then the 2 stored lines
You will then ahve your 3 rows in the order you want. But without seeing the entire reasoning for your application, my "spider sense" tells me you will hit a wall later on...
And check the comments for other things to worry about.
I am creating a ticketing system that will keep track of tickets that a customer creates. The ticket's basic information will be stored in a table 'tickets' who's structure is as follows:
Primary Key (int 255)
Ticket_Key (varchar)
Ticket Number (varchar 500)
Label
Date Created
Delete
and so on..
The issue is that there will eventually be a large amount of tickets and we need a more uniform way of identifying tickets. I would like PHP to create a Ticket Number in the ticket number that will contain mixed values. The date (in format 20111107), followed by a auto incremented value 1001. 1002, 1003, ...). So the Ticket Number will be 201111071001 for an example.
The issue is how do I program this in PHP to insert to the MySQL database? Also, how do I prevent the possibility of duplicate values in the Unique Id in PHP? There will be a very large amount of customers using the table to insert records.
What about using an auto-increment and combining this with the date field to generate a sequence number for that date and hence a ticketId.
So your insert process would be something like this:
INSERT INTO table (...ticket info...)
You would then retrieve the auto-increment for this row and run a query like this
UPDATE table SET sequence = (SELECT ($id-MAX(auto_increment)) FROM table WHERE date_created=DATE_SUB(CURDATE(),INTERVAL 1 DAY)) WHERE auto_increment=$id
You could then easily create a ticketId of format YYYMMDDXXXX. Assuming you never retro-add tickets in the past this would only ever require these two queries even under heavy usage.
[EDIT] Actually, after looking into this there is a much better way to do this natively in MySQL. If you define two columns (date and sequence) and make them a primary key (both columns) with the sequence field as an auto-increment then MySQL will update the sequence column as an auto-increment per date (i.e. it will start with value 1 for each date).
[EDIT] A table structure along these lines would do the job for you:
CREATE TABLE IF NOT EXISTS `table` (
`created_date` date NOT NULL,
`ticket_sequence` int(11) NOT NULL auto_increment,
`label` varchar(100) NOT NULL,
[other fields as required]
PRIMARY KEY (`created_date`,`ticket_sequence`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
When retrieving the data you could then do something like
SELECT CONCAT( DATE_FORMAT(created_date,'%Y%m%d'),LPAD(ticket_sequence,4,'0')) AS ticket_number, other fields.... FROM table
as i understand that you want to make one result of two different fields like datefield and ticketnumfield
in mysql you do this through the command:
SELECT concat( datefield, ticketnumfeild ) FROM `tbl_name`
this query return the result like 201111071001
I did something like this before where I wanted to refresh the counter for each new day. Unfortunately I do not speak PHP so you will have to settle for explanation and maybe some pseudo code.
Firstly, create a couple of fields in a config file to keep track of your counter. This should be a date field and a number fields...
LastCount (Number)
LastCountDate (Date)
Then you make sure that your ticket number field in your database table is set to only unique values, so it throws an error if you try to insert a duplicate.
Then in your code, you load your counter values (LastCount and LastCountDate) and you process them like so...
newCount = LastCount;
if LastCountDate == Today
increment newCount (newCount++)
else
reset newCount (newCount = 1)
you can then use newCount to create your ticket number.
Next, when you try to insert a row, if it is successful, then great. If it fails, then you need to increment newCount again, then try the insert again. Repeat this until the insert is successful (put it in a loop)
Once you have successfully inserted the row, you need to update the database with the Count Values you just used to generate the ticket number - so they are ready for use the next time.
Hope that helps in some way.
I had heavy SQL dump of a table. I used bigdump lib to import it in MySql database on my server.
Although it worked fine, but now I have duplicated entries in that table.
same table on local server has 8 * 105 records but on server it has 15 * 105 records.
Can you suggest me a query to delete duplicate entries from this table?
Here is my table structure.
Table name is : techdata_products
P.S. This table does not have any primary key.
SQL is not my strong point but I think you can export the result of this query:
SELECT DISTINCT * FROM table;
And then, create a new table and import your results.
First starters why do you have no primary key? You could have simply made that id field that auto increments a primary key to prevent duplicates. My suggestion would be to create a new table and do a
Select Distinct * from table and put the results into a new table that has a primary key
I'm creating a change log db that's the exact same as my active db except it has a changedate DATE field at the end.
The db is basically one primary key id, and about 50 other columns of various data types. The script I have in php is it tries to insert new ids and if it gets the error message for duplicate primary key, then it should get that row, insert it into my backup db with a curdate() call as the final date value, delete the entry from my first db, then insert the new entry.
I have all the other parts of the script finished except the part where I have to insert everything from the first table + an extra column for curdate(). Or if there's a better solution to my problem of inserting into a backup database when a duplicate primary key comes in when there's a fairly high amount of rows please share that.
You could do an INSERT INTO SELECT:
INSERT INTO `backupTable` SELECT *, NOW() FROM `originalTable` WHERE id = '$id';
You have to specify the ID for the entry you wish to copy to your backup db. You have also to be sure, that the IDis not already in your backup table. You can use REPLACE INTO to workaround this case.
REPLACE INTO `backupTable` SELECT *, NOW() FROM `originalTable` WHERE id = '$id';
basicly, you can create a TIMESTAMP column with CURRENT_TIMESTAMP as default value.
when you insert a row to that table, the current date/time will be automaticly inserted.
is that what you were looking for ?
BTW: i would recommend to kill the problem at it source and make sure a duplicate primary key will not be inserted to the datatable..
to do that, you can use the SELECT LAST_INSERT_ID();
I asked this question a little earlier today but am not sure as to how clear I was.
I have a MySQL column filled with ordered numbers 1-56. These numbers were generated by my PHP script, not by auto_increment.
What I'd like to do is make this column auto_incrementing after the PHP script sets the proper numbers. The PHP script works hand in hand with a jQuery interface that allows me to reorder a list of items using jQuery's UI plugin.
Once I decide what order I'd like the entries in, I'd like for the column to be set to auto increment, such that if i were to insert a new entry, it would recognize the highest number already existing in the column and set its own id number to be one higher than what's already existing.
Does anyone have any suggestions on how to approach this scenario?
I'd suggest creating the table with your auto_increment already in place. You can specify a value for the auto_inc column, and mysql will use it, and still the next insert to specify a NULL or 0 value for the auto_inc column will magically get $highest + 1 assigned to it.
example:
mysql> create table foobar (i int auto_increment primary key);
mysql> insert into foobar values (10),(25);
mysql> insert into foobar values (null);
mysql> select * from foobar;
# returns 10,25,26
You can switch it to MySQL's auto_increment implementation, but it'll take 3 queries to do it:
a) ALTER TABLE to add the auto_increment to the field in question
b) SELECT MAX(id) + 1 to find out what you need to set the ID to
c) ALTER TABLE table AUTO_INCREMENT =result from (b)
MySQL considers altering the AUTO_INCREMENT value a table-level action, so you can't do it in (a), and it doesn't allow you to do MAX(id) in (c), so 3 queries.
You can change that with a query, issued through php, using the mysql console interface or (easiest) using phpmyadmin.
ALTER TABLE table_name CHANGE old_column_name new_column_name column_definition;
ALTER TABLE table_name AUTO_INCREMENT = highest_current_index + 1
column_definiton:
old_column_definition AUTO_INCREMENT
More info:
http://dev.mysql.com/doc/refman/5.1/en/alter-table.html
http://dev.mysql.com/doc/refman/5.1/en/create-table.html
EDIT
Always use mysql_insert_id or the appropiate function of your abstraction layer to get the last created id, as LAST_INSERT_ID may lead to wrong results.
No, stop it. This isn't the point of auto_increment. If you aren't going to make them ordered by the id then don't make them auto_increment, just add a column onto the end of the table for ordering and enjoy the added flexibility it gives you. It seems like you're trying to pack two different sets of information into one column and it's really only going to bite you in the ass despite all the well-meaning people in this thread telling you how to go about shooting yourself in the foot.
In MySQL you can set a custom value for an auto_increment field. MySQL will then use the highest auto_increment column value for new rows, essentially MAX(id)+1. This means you can effectively reserve a range of IDs for custom use. For instance:
CREATE TABLE mytable (
id INT UNSIGNED PRIMARY KEY AUTO_INCREMENT,
col1 VARCHAR(256)
);
ALTER TABLE mytable AUTO_INCREMENT = 5001;
In this schema all ids < 5001 are reserved for use by your system. So, your PHP script can auto-generate values:
for ($i=1; $i<=56; $i++)
mysql_query("INSERT INTO mytable SET id = $i, col1= 'whatevers'");
New entries will use the non-reserved range by not specifying id or setting it to null:
INSERT INTO mytable SET id = NULL, col1 = 'whatevers2';
-- The id of the new row will be 5001
Reserving a range like this is key - in case you need more than 56 special/system rows in the future.
ALTER TABLE <table name> <column name> NOT NULL AUTO_INCREMENT
More info:
AUTO_INCREMENT Handling in InnoDB
Server SQL Modes