I have a DB table which has approximately 40 columns and the main motive is to insert the records in the database as quickly as possible. I am using PHP for this.
The problems is, to create the insert statement, I have to loop through a for each. I am not sure if I doning this correctly. Please suggest me the best atlernative.. here is the example..
/// to loop through the available data ///
$sqc = "";
for ($i=1; $i<=100; $i++){
if ($sqc == ""){
$sqc = "('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
} else {
$sqc .= ",('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
}
}
/// finally the sql query ///
$sql_quyery = "INSERT INTO table_name (`col1`,.. till.. ,`col40`) values ".$sqc;
This concatenation of $sqc is taking a lot of time. and also the insertion in the DB, is there an alternate way of doing this.. i need to find a way to speed this up like 100X.. :(
Thank you
As suggested on MySQL Optimizing INSERT Statements page, there are some ways for this-
If you are inserting many rows from the same client at the same time, use INSERT statements with multiple VALUES lists to insert several rows at a time. This is considerably faster (many times faster in some cases) than using separate single-row INSERT statements. If you are adding data to a nonempty table, you can tune the bulk_insert_buffer_size variable to make data insertion even faster.
When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.
Find the link below-
[MySQL Guide]
[1] https://dev.mysql.com/doc/refman/5.7/en/insert-optimization.html
Is just a small contribute but you can avoid the concat using a binding
$stmt = mysqli_prepare($yuor_conn,
"INSERT INTO table_name (`col1`,.. till.. ,`col40`) VALUES (?, ... till.. ?)");
mysqli_stmt_bind_param($stmt, 'ss.......s',
$array_value["col1"], $array_value["col2"], .. till..,
$array_value["col40"]);
Related
Currently, I am using prepared statements to insert a bunch of rows with underneath code:
UPDATED CODE:
$sql = $mysqli->prepare("INSERT INTO `TBLName` (Col1, Col2, Col3) Values(?, ?, ?)");
for ($i = 0; $i < 1000; $i++) {
$sql->bind_param('sss', $Col1Data, $Col2Data, $Col3Data);
$sql->execute();
}
$sql->close();
However, I have seen an article talking about Transaction being way faster... Could I use transaction in combination with the prepared statement or should I use it in a different way? Suppose that I have to insert 1000 rows per script run and the script get's triggered with an interval of 5 seconds, what is the best way to go for me?
I recommend 100 to 1000 rows being inserted in a single INSERT statement. (I don't think the code you presented will work without some more changes.)
And do the 100-1000 rows either with autocommit=1 or in a TRANSACTION. This has the benefit of not hogging the system and not collecting lots of data to potentially "undo". But the drawback of not doing all the rows atomically.
Less than 100 leads to extra overhead; more than 1000 gets into diminishing returns and other forms of overhead.
The 100-1000 is also Replication-friendly, rather than hogging the replication stream.
If you already have the data in a csv file, use LOAD DATA.
I have an html page where I collect an array of values from checkboxes to insert in a database. The html page posts to a PHP page that collects the data and then stores in the database.
For each value, there are a few other fields I would like to include that are the same for all the values such as time entered. I can easily convert the captured array into a comma delimited list using implode. I use such a comma delimited list of ids to update and delete records. However, when I want to insert them, MYSQL does not seem to allow you to use a comma delimited list. My question is, what is the easiest way to insert records, one for each value in the c comma delimited list, without using a loop.
html page
<input type="checkbox" name="var[]" value=1>
<input type="checkbox" name="var{}" value=2>
PHP page
$vars = $_POST['var'];
This gives me an array that I can convert to a comma delimited list using implode.
To delete, I can go
$sql = "DELETE * from table WHERE id in '$vars'";
To update I can go
$sql = "UPDATE table, WHERE id in '$vars'";
But there does not seem to be an equivalent for Insert.
Following would work:
$sql = "INSERT into table (var, timeentered) values (1,now()) (2,now())";
However, that's not how I have my data. what I would like to do is something like
$sql = "INSERT into table (var,timeentered) values($vars), now()" but of course that doesn't work.
Do I have to convert my nice comma delimited list that works so well for update and delete into something that looks like (1,now) (2, now()) for inserting or is there an alternative?
Thanks for any suggestions.
Unfortunately you have to build whole query by yourself:
$sql ="insert into table (var, timeentered) values ";
$sql .= "(".implode(", now()), (", $vars).")";
You need to loop through your data set and create the multi-line insert query manually. There is no other alternative if you want to insert multiple rows with a single query. That is outside of using certain DB frameworks which might present a better interface for doing this. Of course at the end of the day, such a DB framework would in essence be building the multi-item insert query manually at some point.
The question might come done to one of how many items are you going to insert. If you are only going to be inserting a few records at a time, then you might want to consider just using prepared statements with individual inserts. However if you are going to be inserting hundreds of records at a time, that would probably not be a good idea.
In your mysql database you can set the default for the column "time_created" to be a TIMESTAMP default CURRENT_TIMESTAMP. This way you don't have to worry about it. Just use the regular insert and it will automatically set the "time_created" column.
For your other issue of multi-line inserts you can create an $update array and use a foreach loop to issue a sql insert command on every row of data.
Two options I can think of.
Build a dynamic insert query like you suggest. However do not call now() each time but just insert a single date ie
$time = gmdate();
$sql = "INSERT into table (var, timeentered) values (1,$time) (2,$time)";
Or use a prepared statement of the single insert below, turn off autocommit, start a transaction and execute the prepared statement in a for loop for the number of inserts needed, then commit the transaction.
$sql = "INSERT into table (var, timeentered) values (?,?)"
Mostly you will have to build your query using some type of looping structure. Convention and best practice aside if you just want to know how to make your array acceptable for a multiple insert statement then why not just do this:
$values = '('.implode('),(', $array).')';
or if already CSV then:
$values = '('.implode('),(', explode(',' $csv)).')';
then you can just use $values in your query using double quotes.
I have only just begun learning about joining tables in MySQL. Now, I have a small project where I simply want to let the visitor insert data through a form. The data is then displayed in a HTML table with four rows, joining together two tables in my database. The "problem" is that the data should be submitted into those two different tables in my database.
I tried
$query = "INSERT INTO table1, table2 (col1, col2, col3, col4) VALUES ('value1', 'value2', 'value3', 'value4')";
but that doesn't seem to do it. What is the correct syntax for submitting form data to several database tables? Oh, and I read some similar threads mentioning using transactions. Is this necessary? My tables are run with MyISAM. Thanks!
You can read more about it from the MySQL Manual. In short, you cannot insert into multiple tables at once. This leaves you with three options:
Multiple INSERT statements
Triggers
Stored Procedures
The answer to this question: MySQL Insert into multiple tables? (Database normalization?) suggests using transactions, which will not work with MyISAM, but is a good FYI if you ever switch to InnoDB.
I really recommend you read up on Triggers. They can make your life a lot easier. But if you don't want to use them, look into PHP's mysqli_multi_query, which will allow you to execute two different queries at the same time, for example:
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2');";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4');";
$result = mysqli_multi_query($dbcon, $query);
You can perform this by using MySQL Transactions By:
Try:
BEGIN
INSERT INTO table1 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC)
INSERT INTO table2 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC);
COMMIT;
With MyISM you will need to execute the query for each table you want to insert into, I do not believe that in a single query you can add to multiple tables.
In your case you can not use Transactions because they are not supported by your engine.
Your only solution is to use several separate queries, preferably within a transaction. Transactions are necessary if you want to make sure that the data from each query is inserted, in which case you COMMIT the transaction; should one of the queries fail, you can ROLLBACK.
P.S. Use InnoDB. It's better in pretty much any environment where INSERT queries make up at least 5% of all queries (sadly I cannot give the source as I had read it several months ago and no longer remember where).
I may be wrong, but you don't insert into multiple tables at the same time. You split it into two or more commands, each handling the specific insertion, whats the big deal, that one extra line of code (which makes everything clearer) too much of a hassle to type?
Look at it this way, if you write a large script, for instance a routine to process some data, the more you segment the code, the easier it is to debug, and, if necessary, inoculate instructions that are problematic, it will end up saving you time in the long run.
I have this problem before You can use multiple query function
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2')";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4')";
$result = mysqli_multi_query($dbcon, $query);
I have to insert 50,000 to 16.000.000 rows in MySQL Table from php. It is taking at least 15-20 min to store in database.
To complete my project i have to do it more faster insertion. do anyone having any flexible opinion.
i am using this code
for($i=$d;$i<=$fd;$i++)
{ $j = $i-$d+1;
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','".
$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
mysql_query($sql1);
}
I agree with Cal. Try this:
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ";
<?php
for($i=$d;$i<=$fd;$i++)
{
$j = $i-$d+1;
$sql1 .= "('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','"
.$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
if($i<=$fd){
$sql1 .= ", ";
}
}
mysql_query($sql1);
?>
In this case you will perform only one insertion instead multiple. Also, as variant, you can store your data into text file (.csv, for example) and then import it into your database. But this is exotic method.
Use extended insert syntax to insert multiple rows per statement:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
http://dev.mysql.com/doc/refman/5.5/en/insert.html
Another trick is to disable/delete indexes and enable/add them after all data has been inserted.
DISABLE KEYS;
INSERT ......
ENABLE KEYS;
If the table supports transactions (InnoDB), you can speed up bulk inserts by doing everything inside a single transaction. Query BEGIN TRANSACTION before all the inserts, and COMMIT after you're done.
Obviously, this is irrelevant if your table doesn't support transactions (MyISAM).
Prepared statements are another way to speed up bulk inserts (with added security benefits), but it's difficult to do that in a project that uses mysql_query() everywhere, so I'll leave that out.
You're not really clear what your limits are, and especially WHY. Other answers assume you are on some sort of time-schedule, and need the insert completed before a certain moment
But when we assume you want to have control over your program back as quickly as possible, but not neccesairily need the insert to be finished, you could use delayed inserts?
http://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html
The DELAYED option for the INSERT statement is a MySQL extension to
standard SQL that is very useful if you have clients that cannot or
need not wait for the INSERT to complete. This is a common situation
when you use MySQL for logging and you also periodically run SELECT
and UPDATE statements that take a long time to complete.
When a client uses INSERT DELAYED, it gets an okay from the server at
once, and the row is queued to be inserted when the table is not in
use by any other thread.
I have a array with a variable amount of values.
Is there a more efficient or better way to INSERT them into my DB besides a loop with a query inside it?
At this site, there is a nice example of MySQL with a multi-insert query. It is valid SQL to
INSERT INTO [table]
VALUES
(row1),
(row2),
...
On request: a php snippet:
$query="INSERT INTO mytable\nVALUES\n (".$values[0].")";
array_shift( $values );
foreach( $values as $value ) {
$query .= ",(".$value.")";
}
In my experience multi-row inserts are processed MUCH faster than an equivalent number of single row inserts, if you're inserting a large amount of data at a time, that's a good way to go. I've watched a process of entering thousands of rows of data be condensed from 5-10 minutes down to literally seconds using this method.
As far as the code part, I've been a fan of using implode() to join arrays of fields & values together. No reason you can't do the same for rows of data, you just need to be able to identify which fields need to be quoted, escaped, etc.
For the sake of argument assume $rows is an array of properly formatted SQL values...
$sql = "INSERT INTO `table` VALUES (" . implode("), (", $rows) . ")";
You could apply something similar to assemble the individual fields should you desire.
If the DB you are using allows multiple value insert, then you could create an multi-insert statement and send that to the DB - one connect with one command to do multiple inserts.
If you cannot do multiple inserts - (as MSSQL does not allow) - then I think you are stuck.