I have a scripts that retrieves huge data on a table. I want to create a mysqldump to insert data into another database table with different fields. I want the format of phpMyAdmin where it repeats the INSERT INTO Table VALUES(values1),(values2), ...(values100); if reaches certain amount of value sets depends on what you set.
ex: If I have 550 data sets and i want to be devided the data by 100 so that i will have 6 sets of INSERT INTO query.
INSERT INTO tablename VALUES(value1), (value2), .... (value100);
INSERT INTO tablename VALUES(value101), (value102), .... (value200);
INSERT INTO tablename VALUES(value201), (value202), .... (value300);
INSERT INTO tablename VALUES(value301), (value302), .... (value400);
INSERT INTO tablename VALUES(value401), (value402), .... (value500);
INSERT INTO tablename VALUES(value501), (value502), .... (value550);
If you're using mysqldump and wish to output mutiple rows into a single insert, then you need to use the --extended-insert option:
mysqldump extended insert option
I'm not sure it's possible to specify with mysqldump to specify that a specific number of rows are included in each INSERT statement generated in the dump. Rather you can set the net_buffer_length (although it's not recommended that you do change it), so the actual amount may vary depending on the data in each row.
You could use array_chunk(), something like:
$toInsert = array( '(values1)', '(values2)', '(values3)', '(values4)' ); //etc.
$sqlStart = 'INSERT INTO tablename (field1, field2, field3, etc) VALUES ';
foreach (array_chunk($toInsert, 100) as $insertSet) {
$sql = $sqlStart . implode(', ', $insertSet);
//execute $sql
}
Are you actually doing much with the data though? You might be able to do it all in SQL with INSERT INTO table (field1, field2) SELECT somefield, somefield2 FROM another table
While fetching the rows, increment a counter, and when it hits a certain value, have it create a new insert statement?
Some code that might not be correct (no PHP for a LONG time), but you will probably get the idea
$i = 0;
$insertstatements = array();
$currentinsertstatement;
while ($temp = mysql_fetch_assoc($result)) {
// do something with the data
$insertpart = "(value_xxx)";
if ($i % 100 == 0) {
// first value
if ($i != 0) $insertstatements[count($insertstatements)] = $currentinsertstatement;
$currentinsertstatement = "INSERT INTO tablename VALUES " . $insertpart;
} else {
$currentinsertstatement .= ", " . $insertpart;
// somewhere in the middle of the insert statement
}
$i++;
}
if ($i % 100 != 0) {
$insertstatements[count($insertstatements] = $currentinsertstatement;
}
You definitely should use transactions for huge inserts, if your storage engine supports them (like innoDB):
BEGIN;
INSERT INTO tablename VALUES...
INSERT INTO tablename VALUES...
COMMIT;
If something goes wrong, you can safely ROLLBACK the last operation, restart you script, etc.
Related
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
Is the database query faster if I insert multiple rows at once:
like
INSERT....
UNION
INSERT....
UNION
(I need to insert like 2-3000 rows)
INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of column values, each enclosed within parentheses and separated by commas.
Example:
INSERT INTO tbl_name
(a,b,c)
VALUES
(1,2,3),
(4,5,6),
(7,8,9);
Source
If you have your data in a text-file, you can use LOAD DATA INFILE.
When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.
Optimizing INSERT Statements
You can find more tips on how to speed up your insert statements on the link above.
Just use a SELECT statement to get the values for many lines of the chosen columns and put these values into columns of another table in one go. As an example, columns "size" and "price" of the two tables "test_b" and "test_c" get filled with the columns "size" and "price" of table "test_a".
BEGIN;
INSERT INTO test_b (size, price)
SELECT size, price
FROM test_a;
INSERT INTO test_c (size, price)
SELECT size, price
FROM test_a;
COMMIT;
The code is embedded in BEGIN and COMMIT to run it only when both statements have worked, else the whole run up to that point gets withdrawn.
Here is a PHP solution ready for use with a n:m (many-to-many relationship) table :
// get data
$table_1 = get_table_1_rows();
$table_2_fk_id = 123;
// prepare first part of the query (before values)
$query = "INSERT INTO `table` (
`table_1_fk_id`,
`table_2_fk_id`,
`insert_date`
) VALUES ";
//loop the table 1 to get all foreign keys and put it in array
foreach($table_1 as $row) {
$query_values[] = "(".$row["table_1_pk_id"].", $table_2_fk_id, NOW())";
}
// Implode the query values array with a coma and execute the query.
$db->query($query . implode(',',$query_values));
EDIT : After #john's comment I decided to enhance this answer with a more efficient solution :
divides the query to multiple smaller queries
use rtrim() to delete last coma instead of implod()
// limit of query size (lines inserted per query)
$query_values = "";
$limit = 100;
$table_1 = get_table_1_rows();
$table_2_fk_id = 123;
$query = "INSERT INTO `table` (
`table_1_fk_id`,
`table_2_fk_id`,
`insert_date`
) VALUES ";
foreach($table_1 as $row) {
$query_values .= "(".$row["table_1_pk_id"].", $table_2_fk_id, NOW()),";
// entire table parsed or lines limit reached :
// -> execute and purge query_values
if($i === array_key_last($table_1)
|| fmod(++$i / $limit) == 0) {
$db->query($query . rtrim($query_values, ','));
$query_values = "";
}
}
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO product_cate (site_title, sub_title)
VALUES ('$site_title', '$sub_title')";
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO menu (menu_title, sub_menu)
VALUES ('$menu_title', '$sub_menu', )";
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO blog_post (post_title, post_des, post_img)
VALUES ('$post_title ', '$post_des', '$post_img')";
This question's answers are a community effort. Edit existing answers to improve this post. It is not currently accepting new answers or interactions.
Is the database query faster if I insert multiple rows at once:
like
INSERT....
UNION
INSERT....
UNION
(I need to insert like 2-3000 rows)
INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of column values, each enclosed within parentheses and separated by commas.
Example:
INSERT INTO tbl_name
(a,b,c)
VALUES
(1,2,3),
(4,5,6),
(7,8,9);
Source
If you have your data in a text-file, you can use LOAD DATA INFILE.
When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.
Optimizing INSERT Statements
You can find more tips on how to speed up your insert statements on the link above.
Just use a SELECT statement to get the values for many lines of the chosen columns and put these values into columns of another table in one go. As an example, columns "size" and "price" of the two tables "test_b" and "test_c" get filled with the columns "size" and "price" of table "test_a".
BEGIN;
INSERT INTO test_b (size, price)
SELECT size, price
FROM test_a;
INSERT INTO test_c (size, price)
SELECT size, price
FROM test_a;
COMMIT;
The code is embedded in BEGIN and COMMIT to run it only when both statements have worked, else the whole run up to that point gets withdrawn.
Here is a PHP solution ready for use with a n:m (many-to-many relationship) table :
// get data
$table_1 = get_table_1_rows();
$table_2_fk_id = 123;
// prepare first part of the query (before values)
$query = "INSERT INTO `table` (
`table_1_fk_id`,
`table_2_fk_id`,
`insert_date`
) VALUES ";
//loop the table 1 to get all foreign keys and put it in array
foreach($table_1 as $row) {
$query_values[] = "(".$row["table_1_pk_id"].", $table_2_fk_id, NOW())";
}
// Implode the query values array with a coma and execute the query.
$db->query($query . implode(',',$query_values));
EDIT : After #john's comment I decided to enhance this answer with a more efficient solution :
divides the query to multiple smaller queries
use rtrim() to delete last coma instead of implod()
// limit of query size (lines inserted per query)
$query_values = "";
$limit = 100;
$table_1 = get_table_1_rows();
$table_2_fk_id = 123;
$query = "INSERT INTO `table` (
`table_1_fk_id`,
`table_2_fk_id`,
`insert_date`
) VALUES ";
foreach($table_1 as $row) {
$query_values .= "(".$row["table_1_pk_id"].", $table_2_fk_id, NOW()),";
// entire table parsed or lines limit reached :
// -> execute and purge query_values
if($i === array_key_last($table_1)
|| fmod(++$i / $limit) == 0) {
$db->query($query . rtrim($query_values, ','));
$query_values = "";
}
}
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO product_cate (site_title, sub_title)
VALUES ('$site_title', '$sub_title')";
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO menu (menu_title, sub_menu)
VALUES ('$menu_title', '$sub_menu', )";
// db table name / blog_post / menu / site_title
// Insert into Table (column names separated with comma)
$sql = "INSERT INTO blog_post (post_title, post_des, post_img)
VALUES ('$post_title ', '$post_des', '$post_img')";
There is MySQL table with 24500 rows of data, and there is text file with 26000 string of data needed to be inserted into MySQL, problem is what this 26000 strings duplicates data in MySQL table, so we need to compare them, and insert only new/unique.
cadastreArray - array from text file
districtArray - mysql array
When i try to do
foreach ($cadastreArray as $cadastreValue) {
$districtExist = false;
foreach ($districtArray as $districtData) {
if ($cadastreValue[0] == $districtData['1']) {
$districtExist = true;
break;
}
}
}
if(!$districtExist) { MySQL INSERT ... }
i am getting execution time error, and even 3 minutes are not enought.
Maybe you can offer better/faster way?
May be you can set the mysql field as unique so when you will be going to insert it will not insert and will generate error number and will continue execution. So you need not to compare.
One more thing that you can do is you can increase max_execution_time in php.ini
Another option. Load your 26k text file into a temp table (LOAD DATA INFILE... will do this quickly).
Then you can do an insert based on a query that takes your temp table and LEFT JOINs that against your full table, checking that a field on the full table is NULL.
Simple example script here:-
<?php
$file = "SomeTextFile.txt";
$sql = "CREATE TEMPORARY TABLE cadastre
(
field1 INT,
field2 VARCHAR(255),
etc...
)";
if(!($db->query($sql)))
{
die($db->error());// if error, stop script
}
if(!($db->query("LOAD DATA INFILE '$file' INTO TABLE cadastre")))
{
die($db->error());// if error, stop script
}
$sql = "INSERT INTO district (field1, field2, field3, ......)
SELECT a.field1, a.field2, a.field3
FROM cadastre a
LEFT OUTER JOIN district b
ON a.field1 = b.field1
WHERE b.field1 IS NULL";
if(!($db->query($sql)))
{
die($db->error());// if error, stop script
}
?>
Make sure the temp table and the table you are inserting into have useful indexes added.
I don't know if it can be done with just a sql query or it needs a php code
when a cid is missing
There exist many missing values which I can't handle manually
For example, here I don't have cid=1 and cid=6.
I want to insert a row:
cid=1 tcp_sport='undefined' tcp_dport='undefined'
and
cid=6 tcp_sport='undefined' tcp_dport='undefined'
It seems to me I should create a procedure and insert between lines
another solution that I thaught was that I will create a table with cid and undifined values with the respective order and then join this one with that one and this join should have for example ifnull(tcp_sport,'')
would you please help me?
First, use MAX for get the largest ID.
SELECT MAX(cid) as max FROM table
Then, create a for loop for checking if the individual IDs exist:
for ($i = 0; $i < $max; $i++) {
// $query = ... SELECT 1 FROM table WHERE cid = $i ...
// check if the number of rows for $query is greater than 0
// if not, INSERT INTO table VALUES ($i, DEFAULT, DEFAULT)
}
The whole idea of an auto increment ID is to have a value that only refers to one thing ever. By "inserting between the lines" you may be opening yourself up to a lot of unforeseen problems. Image you have another table that has some values that link to the CID of this table. What if that table already has an entry for CID=1, When you insert a new item with CID=1 it will then join to that supporting record. So Data that really belongs to the original item with CID=1 will show for the new item which it probably has nothing to do with.
You aren't going to run out of ID values (if you are approaching the limit of integer, switch it to bigInt), don't re-use IDs if you can avoid it.
You need to use PHP to automate this.
<?php
$link = mysql_connect("localhost", "mysql_user", "mysql_password");
mysql_select_db("database", $link);
while($i < max_value_cid)//replace max_value_cid by the numeric maximum value of cid (SELECT MAX(cid) as max FROM table)
{
$result = mysql_query("SELECT * FROM `table` WHERE cid=".$i, $link);
if(mysql_num_rows($result) == 0)
mysql_query("INSERT INTO `table` VALUES ($i, NULL, NULL);", $link);
$i++;
}
?>
Do test the query on a sample set before execution and remember to backup the entire table, just-in-case.
What is the easiest / most efficient way to get the entire row inserted after an INSERT statement?
I am pretty sure I could do this as follows:
$aQuery = "INSERT into myTable (a, b, c) VALUES (1, 'Hello', 'Goodbye')";
//the IDENTITY coloumn in myTable is named id
$result = sqlsrv_query($myConn, $aQuery );
if ($result) {
$res = sqlsrv_query('SELECT LAST_INSERT_ID()');
$row = sqlsrv_fetch_array($res);
$lastInsertId = $row[0];
$subQuery = "SELECT * FROM myTable where id = {$lastInsertId}";
$subResult = sqlsrv_query($myConn, $subQuery);
if ($subResult) {
while($subrow = sqlsrv_fetch_array($subResult)) {
echo($subrow ['id'] . ', '.
$subrow ['a'] . ', '.
$subrow ['b']); //etc...
}
}
}
However, I am concerned about the possibility of another insert occurring just before my SELECT LAST_INSERT_ID() and thus messing up my logic to boot. How can I be certain that the last inserted id is truly the INSERT I called previously, and not something happening somewhere else?
Is there a more appropriate way of doing this, perhaps a complete SQL solution (such that the query returns the row automatically rather than using PHP)?
UPDATE: myTable DOES have an explicitly defined (and auto-incremented) identity column, named id.
This will work:
"INSERT into myTable (a, b, c) OUTPUT Inserted.a, Inserted.b, Inserted.c VALUES (1, 'Hello', 'Goodbye')
In Sql Server, you would use select #lastID=SCOPE_IDENTITY()
And #LastID will have the last id inserted for the current scope; therefore, if there was another insertion in the middle, you would still get the correct record on your select.
Never use ##Identity for this or you may end up in a situation like you described.
If you were to use identity field (which maybe you should) there is a command called SCOPE_IDENTIY() which info you can find here:
http://msdn.microsoft.com/en-us/library/ms190315.aspx
Since you do not use it, you do not have to select latest data since you have it when you insert, so just use same data instead of selecting.