Currently, I am using prepared statements to insert a bunch of rows with underneath code:
UPDATED CODE:
$sql = $mysqli->prepare("INSERT INTO `TBLName` (Col1, Col2, Col3) Values(?, ?, ?)");
for ($i = 0; $i < 1000; $i++) {
$sql->bind_param('sss', $Col1Data, $Col2Data, $Col3Data);
$sql->execute();
}
$sql->close();
However, I have seen an article talking about Transaction being way faster... Could I use transaction in combination with the prepared statement or should I use it in a different way? Suppose that I have to insert 1000 rows per script run and the script get's triggered with an interval of 5 seconds, what is the best way to go for me?
I recommend 100 to 1000 rows being inserted in a single INSERT statement. (I don't think the code you presented will work without some more changes.)
And do the 100-1000 rows either with autocommit=1 or in a TRANSACTION. This has the benefit of not hogging the system and not collecting lots of data to potentially "undo". But the drawback of not doing all the rows atomically.
Less than 100 leads to extra overhead; more than 1000 gets into diminishing returns and other forms of overhead.
The 100-1000 is also Replication-friendly, rather than hogging the replication stream.
If you already have the data in a csv file, use LOAD DATA.
Related
I have a DB table which has approximately 40 columns and the main motive is to insert the records in the database as quickly as possible. I am using PHP for this.
The problems is, to create the insert statement, I have to loop through a for each. I am not sure if I doning this correctly. Please suggest me the best atlernative.. here is the example..
/// to loop through the available data ///
$sqc = "";
for ($i=1; $i<=100; $i++){
if ($sqc == ""){
$sqc = "('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
} else {
$sqc .= ",('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
}
}
/// finally the sql query ///
$sql_quyery = "INSERT INTO table_name (`col1`,.. till.. ,`col40`) values ".$sqc;
This concatenation of $sqc is taking a lot of time. and also the insertion in the DB, is there an alternate way of doing this.. i need to find a way to speed this up like 100X.. :(
Thank you
As suggested on MySQL Optimizing INSERT Statements page, there are some ways for this-
If you are inserting many rows from the same client at the same time, use INSERT statements with multiple VALUES lists to insert several rows at a time. This is considerably faster (many times faster in some cases) than using separate single-row INSERT statements. If you are adding data to a nonempty table, you can tune the bulk_insert_buffer_size variable to make data insertion even faster.
When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.
Find the link below-
[MySQL Guide]
[1] https://dev.mysql.com/doc/refman/5.7/en/insert-optimization.html
Is just a small contribute but you can avoid the concat using a binding
$stmt = mysqli_prepare($yuor_conn,
"INSERT INTO table_name (`col1`,.. till.. ,`col40`) VALUES (?, ... till.. ?)");
mysqli_stmt_bind_param($stmt, 'ss.......s',
$array_value["col1"], $array_value["col2"], .. till..,
$array_value["col40"]);
This is a sample pdo execute and commit example from php.net
<?php
/* Begin a transaction, turning off autocommit */
$dbh->beginTransaction();
/* Insert multiple records on an all-or-nothing basis */
$sql = 'INSERT INTO fruit
(name, colour, calories)
VALUES (?, ?, ?)';
$sth = $dbh->prepare($sql);
foreach ($fruits as $fruit) {
$sth->execute(array(
$fruit->name,
$fruit->colour,
$fruit->calories,
));
}
/* Commit the changes */
$dbh->commit();
/* Database connection is now back in autocommit mode */
?>
From what I understand, this enables me to faster insert thousand of rows in a table. Now, if I have a php file that is called
http://localhost/insertFruit.php?name=x&color=y&calories=z
which basically collects those values and insert them into database.
Now if I want to take advantage of that PDO loop and execute, I can't. If there is 100 of submission of those insertFruit.php, how can I collect all those values, combine into an array and then commit ? Using a separate class, calling object, using global values etc?
I am sure there is better way to do this.
how can i collect all those values, combine into an array and then commit?
Although you can, you shouldn't.
Neither transactions nor prepared statements are suitable for the separate requests.
Not to mention that your initial question is groundless. You don't need a faster insert. Your inserts are already fast.
There is no "better" way. Premature optimization is the root of all evil. By trying to devise a "better" way you'll ruin whole system.
All you can do is cluster some data into single request. If you have several fruits in one request, you can use prepared statements to process them in a loop.
I used to use
<?php
$sql = "insert into test (owner) values ('owen')";
$db->autocommit(false);
if (!$db->query($sql))
$db->rollback();
else
$db->commit();
$db->close();
?>
However, today I run two insert php files in a same tables, without any action. It is simple like:
<?php
$sql = "insert into test (owner) values ('owen')"; //the other php is the same but replacing 'owen' to 'huhu'
for ($i = 0; $i < 100 * 1000; $i++) {
$db->query($sql);
}
$db->close();
?>
I run two php files in two different consoles. Then I got 200,000 records without any error. Does that mean using transaction manually is really not needed. As there are no conflicts.
You do not need transactions for this.
Transactions exist to be able to roll back half-finished changes to the database. These only occur when you have a set of multiple statements changing the database which might be interrupted inbetween. Then often only some of the statements have been execute which might leave the database in a state which is not 'clean' from the applications point of view.
A simple and good example is a money transfer between two tables:
first a removal from one table
then it is added to a second table
If this process is interrupted inbetween the money has vanished. That is not intended, you probably want to be able to rollback.
In your case however all statements are 'atomic', meaning they succeed or fail, but the databases state is always 'clean'. It does not matter in this if it is a single or multiple clients running the statements.
I have only just begun learning about joining tables in MySQL. Now, I have a small project where I simply want to let the visitor insert data through a form. The data is then displayed in a HTML table with four rows, joining together two tables in my database. The "problem" is that the data should be submitted into those two different tables in my database.
I tried
$query = "INSERT INTO table1, table2 (col1, col2, col3, col4) VALUES ('value1', 'value2', 'value3', 'value4')";
but that doesn't seem to do it. What is the correct syntax for submitting form data to several database tables? Oh, and I read some similar threads mentioning using transactions. Is this necessary? My tables are run with MyISAM. Thanks!
You can read more about it from the MySQL Manual. In short, you cannot insert into multiple tables at once. This leaves you with three options:
Multiple INSERT statements
Triggers
Stored Procedures
The answer to this question: MySQL Insert into multiple tables? (Database normalization?) suggests using transactions, which will not work with MyISAM, but is a good FYI if you ever switch to InnoDB.
I really recommend you read up on Triggers. They can make your life a lot easier. But if you don't want to use them, look into PHP's mysqli_multi_query, which will allow you to execute two different queries at the same time, for example:
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2');";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4');";
$result = mysqli_multi_query($dbcon, $query);
You can perform this by using MySQL Transactions By:
Try:
BEGIN
INSERT INTO table1 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC)
INSERT INTO table2 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC);
COMMIT;
With MyISM you will need to execute the query for each table you want to insert into, I do not believe that in a single query you can add to multiple tables.
In your case you can not use Transactions because they are not supported by your engine.
Your only solution is to use several separate queries, preferably within a transaction. Transactions are necessary if you want to make sure that the data from each query is inserted, in which case you COMMIT the transaction; should one of the queries fail, you can ROLLBACK.
P.S. Use InnoDB. It's better in pretty much any environment where INSERT queries make up at least 5% of all queries (sadly I cannot give the source as I had read it several months ago and no longer remember where).
I may be wrong, but you don't insert into multiple tables at the same time. You split it into two or more commands, each handling the specific insertion, whats the big deal, that one extra line of code (which makes everything clearer) too much of a hassle to type?
Look at it this way, if you write a large script, for instance a routine to process some data, the more you segment the code, the easier it is to debug, and, if necessary, inoculate instructions that are problematic, it will end up saving you time in the long run.
I have this problem before You can use multiple query function
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2')";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4')";
$result = mysqli_multi_query($dbcon, $query);
I have to insert 50,000 to 16.000.000 rows in MySQL Table from php. It is taking at least 15-20 min to store in database.
To complete my project i have to do it more faster insertion. do anyone having any flexible opinion.
i am using this code
for($i=$d;$i<=$fd;$i++)
{ $j = $i-$d+1;
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','".
$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
mysql_query($sql1);
}
I agree with Cal. Try this:
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ";
<?php
for($i=$d;$i<=$fd;$i++)
{
$j = $i-$d+1;
$sql1 .= "('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','"
.$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
if($i<=$fd){
$sql1 .= ", ";
}
}
mysql_query($sql1);
?>
In this case you will perform only one insertion instead multiple. Also, as variant, you can store your data into text file (.csv, for example) and then import it into your database. But this is exotic method.
Use extended insert syntax to insert multiple rows per statement:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
http://dev.mysql.com/doc/refman/5.5/en/insert.html
Another trick is to disable/delete indexes and enable/add them after all data has been inserted.
DISABLE KEYS;
INSERT ......
ENABLE KEYS;
If the table supports transactions (InnoDB), you can speed up bulk inserts by doing everything inside a single transaction. Query BEGIN TRANSACTION before all the inserts, and COMMIT after you're done.
Obviously, this is irrelevant if your table doesn't support transactions (MyISAM).
Prepared statements are another way to speed up bulk inserts (with added security benefits), but it's difficult to do that in a project that uses mysql_query() everywhere, so I'll leave that out.
You're not really clear what your limits are, and especially WHY. Other answers assume you are on some sort of time-schedule, and need the insert completed before a certain moment
But when we assume you want to have control over your program back as quickly as possible, but not neccesairily need the insert to be finished, you could use delayed inserts?
http://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html
The DELAYED option for the INSERT statement is a MySQL extension to
standard SQL that is very useful if you have clients that cannot or
need not wait for the INSERT to complete. This is a common situation
when you use MySQL for logging and you also periodically run SELECT
and UPDATE statements that take a long time to complete.
When a client uses INSERT DELAYED, it gets an okay from the server at
once, and the row is queued to be inserted when the table is not in
use by any other thread.