I have to insert 50,000 to 16.000.000 rows in MySQL Table from php. It is taking at least 15-20 min to store in database.
To complete my project i have to do it more faster insertion. do anyone having any flexible opinion.
i am using this code
for($i=$d;$i<=$fd;$i++)
{ $j = $i-$d+1;
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','".
$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
mysql_query($sql1);
}
I agree with Cal. Try this:
$sql1 = "INSERT INTO information(id, redirection, username, pc, date, time,
method,text, http, code, data, request, software) VALUES ";
<?php
for($i=$d;$i<=$fd;$i++)
{
$j = $i-$d+1;
$sql1 .= "('".$i."','"
.$_SESSION["redirection"][$j]."','".$_SESSION["username"]$j]."','"
.$_SESSION["pc"][$j]."','".$_SESSION["date"][$j]."','"
.$_SESSION["time"][$j]."','".$_SESSION["method"][$j]."','"
.$_SESSION["text"][$j]."','".$_SESSION["http"][$j]."','"
.$_SESSION["code"][$j]."','".$_SESSION["data"][$j]."','"
.$_SESSION["request"][$j]."','".$_SESSION["software"][$j]."')";
if($i<=$fd){
$sql1 .= ", ";
}
}
mysql_query($sql1);
?>
In this case you will perform only one insertion instead multiple. Also, as variant, you can store your data into text file (.csv, for example) and then import it into your database. But this is exotic method.
Use extended insert syntax to insert multiple rows per statement:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
http://dev.mysql.com/doc/refman/5.5/en/insert.html
Another trick is to disable/delete indexes and enable/add them after all data has been inserted.
DISABLE KEYS;
INSERT ......
ENABLE KEYS;
If the table supports transactions (InnoDB), you can speed up bulk inserts by doing everything inside a single transaction. Query BEGIN TRANSACTION before all the inserts, and COMMIT after you're done.
Obviously, this is irrelevant if your table doesn't support transactions (MyISAM).
Prepared statements are another way to speed up bulk inserts (with added security benefits), but it's difficult to do that in a project that uses mysql_query() everywhere, so I'll leave that out.
You're not really clear what your limits are, and especially WHY. Other answers assume you are on some sort of time-schedule, and need the insert completed before a certain moment
But when we assume you want to have control over your program back as quickly as possible, but not neccesairily need the insert to be finished, you could use delayed inserts?
http://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html
The DELAYED option for the INSERT statement is a MySQL extension to
standard SQL that is very useful if you have clients that cannot or
need not wait for the INSERT to complete. This is a common situation
when you use MySQL for logging and you also periodically run SELECT
and UPDATE statements that take a long time to complete.
When a client uses INSERT DELAYED, it gets an okay from the server at
once, and the row is queued to be inserted when the table is not in
use by any other thread.
Related
I have a DB table which has approximately 40 columns and the main motive is to insert the records in the database as quickly as possible. I am using PHP for this.
The problems is, to create the insert statement, I have to loop through a for each. I am not sure if I doning this correctly. Please suggest me the best atlernative.. here is the example..
/// to loop through the available data ///
$sqc = "";
for ($i=1; $i<=100; $i++){
if ($sqc == ""){
$sqc = "('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
} else {
$sqc .= ",('".$array_value["col1"]."'.. till .. '".$array_value["col40"]."',)";
}
}
/// finally the sql query ///
$sql_quyery = "INSERT INTO table_name (`col1`,.. till.. ,`col40`) values ".$sqc;
This concatenation of $sqc is taking a lot of time. and also the insertion in the DB, is there an alternate way of doing this.. i need to find a way to speed this up like 100X.. :(
Thank you
As suggested on MySQL Optimizing INSERT Statements page, there are some ways for this-
If you are inserting many rows from the same client at the same time, use INSERT statements with multiple VALUES lists to insert several rows at a time. This is considerably faster (many times faster in some cases) than using separate single-row INSERT statements. If you are adding data to a nonempty table, you can tune the bulk_insert_buffer_size variable to make data insertion even faster.
When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using INSERT statements.
Find the link below-
[MySQL Guide]
[1] https://dev.mysql.com/doc/refman/5.7/en/insert-optimization.html
Is just a small contribute but you can avoid the concat using a binding
$stmt = mysqli_prepare($yuor_conn,
"INSERT INTO table_name (`col1`,.. till.. ,`col40`) VALUES (?, ... till.. ?)");
mysqli_stmt_bind_param($stmt, 'ss.......s',
$array_value["col1"], $array_value["col2"], .. till..,
$array_value["col40"]);
I used to use
<?php
$sql = "insert into test (owner) values ('owen')";
$db->autocommit(false);
if (!$db->query($sql))
$db->rollback();
else
$db->commit();
$db->close();
?>
However, today I run two insert php files in a same tables, without any action. It is simple like:
<?php
$sql = "insert into test (owner) values ('owen')"; //the other php is the same but replacing 'owen' to 'huhu'
for ($i = 0; $i < 100 * 1000; $i++) {
$db->query($sql);
}
$db->close();
?>
I run two php files in two different consoles. Then I got 200,000 records without any error. Does that mean using transaction manually is really not needed. As there are no conflicts.
You do not need transactions for this.
Transactions exist to be able to roll back half-finished changes to the database. These only occur when you have a set of multiple statements changing the database which might be interrupted inbetween. Then often only some of the statements have been execute which might leave the database in a state which is not 'clean' from the applications point of view.
A simple and good example is a money transfer between two tables:
first a removal from one table
then it is added to a second table
If this process is interrupted inbetween the money has vanished. That is not intended, you probably want to be able to rollback.
In your case however all statements are 'atomic', meaning they succeed or fail, but the databases state is always 'clean'. It does not matter in this if it is a single or multiple clients running the statements.
I have only just begun learning about joining tables in MySQL. Now, I have a small project where I simply want to let the visitor insert data through a form. The data is then displayed in a HTML table with four rows, joining together two tables in my database. The "problem" is that the data should be submitted into those two different tables in my database.
I tried
$query = "INSERT INTO table1, table2 (col1, col2, col3, col4) VALUES ('value1', 'value2', 'value3', 'value4')";
but that doesn't seem to do it. What is the correct syntax for submitting form data to several database tables? Oh, and I read some similar threads mentioning using transactions. Is this necessary? My tables are run with MyISAM. Thanks!
You can read more about it from the MySQL Manual. In short, you cannot insert into multiple tables at once. This leaves you with three options:
Multiple INSERT statements
Triggers
Stored Procedures
The answer to this question: MySQL Insert into multiple tables? (Database normalization?) suggests using transactions, which will not work with MyISAM, but is a good FYI if you ever switch to InnoDB.
I really recommend you read up on Triggers. They can make your life a lot easier. But if you don't want to use them, look into PHP's mysqli_multi_query, which will allow you to execute two different queries at the same time, for example:
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2');";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4');";
$result = mysqli_multi_query($dbcon, $query);
You can perform this by using MySQL Transactions By:
Try:
BEGIN
INSERT INTO table1 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC)
INSERT INTO table2 (col1, col2...ETC)
VALUES('value1', 'value2'...ETC);
COMMIT;
With MyISM you will need to execute the query for each table you want to insert into, I do not believe that in a single query you can add to multiple tables.
In your case you can not use Transactions because they are not supported by your engine.
Your only solution is to use several separate queries, preferably within a transaction. Transactions are necessary if you want to make sure that the data from each query is inserted, in which case you COMMIT the transaction; should one of the queries fail, you can ROLLBACK.
P.S. Use InnoDB. It's better in pretty much any environment where INSERT queries make up at least 5% of all queries (sadly I cannot give the source as I had read it several months ago and no longer remember where).
I may be wrong, but you don't insert into multiple tables at the same time. You split it into two or more commands, each handling the specific insertion, whats the big deal, that one extra line of code (which makes everything clearer) too much of a hassle to type?
Look at it this way, if you write a large script, for instance a routine to process some data, the more you segment the code, the easier it is to debug, and, if necessary, inoculate instructions that are problematic, it will end up saving you time in the long run.
I have this problem before You can use multiple query function
$query = "INSERT INTO table1 (col1,col2) VALUES ('$value1','$value2')";
$query = "INSERT INTO table2 (col3,col4) VALUES ('$value3','$value4')";
$result = mysqli_multi_query($dbcon, $query);
I got two tables. One is account, another is Interest.
One account can have multi Interests and It can be edited.
Now, the process is deleting all Interest of this account then insert these insterests.
The QUERY IS:
"DELETE FROM Interests WHERE account_id='$id'"
"INSERT INTO Interests (account_id, interest_name) VALUES('$id', '$name')"
I use the both query when user update their account, but the insert is fail, there is nothing insert into the table (ps. the interests_id is auto_increment and this was be counted) but there is nothing new in the table. When I comment out the delete query. The insert will be successful.
Does any one know what can i do?
If you want to update your table records, you will do update operation.
like this:
UPDATE TABLE_NAME SET FIELD_NAME = 'VARIABLE_NAME'
WHERE PRIMERY_FIELD_NAME = 'VARIABLE_NAME' ;
you did not have to use these two queries, if you want to update data simply use the updat query of mysql.use this:
<?php
$query = "UPDATE Interests SET interest_name = '".$name."' WHERE account_id = '".$id."'" ;
mysql_query($query);
?>
If you want to update your table records then you may execute update operation. It like following
UPDATE Interests
SET
interest_name = '$name'
WHERE
accountno = '$id' ;
Try it. You may solve your problem by this way.
If you have queries failing, you should capture the error and see what went wrong. In all MySQL APIs for PHP, a query that fails returns a status code to indicate this. Examples of checking this status code are easy to find in the docs. But most developers fail to check the status.
Use transactions to ensure that both changes succeed together or neither are applied.
How to Decide to use Database Transactions
Definition of a transaction in MySQL: http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_transaction
Syntax for starting and committing transactions in MySQL: http://dev.mysql.com/doc/refman/5.5/en/commit.html
You need to use InnoDB. MyISAM does not support transactions. http://dev.mysql.com/doc/refman/5.5/en/innodb-storage-engine.html
In PHP, you need to stop using the old ext/mysql API and start using MySQLi or PDO.
http://php.net/manual/en/mysqli.quickstart.transactions.php
http://php.net/manual/en/pdo.begintransaction.php
This happens because the query are treated as two single transaction, so the order of execution is not guaranteed.
The effect you are describing is because the insert is processed before delete, so the interests_id is auto-incremented properly, then the row is deleted by delete statement.
You should change the query logic or perform both queries in one single transaction.
Im wondering if the way i use to retrieve the id of the last row inserted in a postgresql table is efficent..
It works, obviously, but referencing on the serial sequence currval value could be problematic when i have many users adding rows in the same table at the same time.
My actual way is:
$pgConnection = pg_connect('host=127.0.0.1 dbname=test user=myuser password=xxxxx')or die('cant connect');
$insert = pg_query("INSERT INTO customer (name) VALUES ('blabla')");
$last_id_query = pg_query("SELECT currval('customer_id_seq')");
$last_id_results = pg_fetch_assoc($last_id_query);
print_r($last_id_results);
pg_close($pgConnection);
Well, its just a test atm.
But anyway, i can see 3 issues with this way:
Referencing on the customer_id_seq, if two user do the same thing in the same time, could happen that them both get the same id from that way... or not?
I have to know the table's sequence name. Becose pg_get_serial_sequence dont works for me (im newbie on postgresql, probably is a configuration issue)
Any suggestion/better ways?
p.s: i can't use the PDO, becose seem lack a bit with the transaction savepoint; I wont use zend and, in the end, i'll prefer to use the php pg_* functions (maybe i'll build up my classes in the end)
EDIT:
#SpliFF(thet deleted his answer): this would works better?
$pgConnection = pg_connect('host=127.0.0.1 dbname=test user=myuser password=xxxxx')or die('cant connect');
pg_query("BEGIN");
$insert = pg_query("INSERT INTO customer (name) VALUES ('blabla')");
$last_id_query = pg_query("SELECT currval('customer_id_seq')");
$last_id_results = pg_fetch_assoc($last_id_query);
print_r($last_id_results);
//do somethings with the new customer id
pg_query("COMMIT");
pg_close($pgConnection);
If you use a newer version of PostgreSQL (> 8.1) you should use the RETURNING clause of INSERT (and UPDATE) command.
OTOH if you insist on using one of the sequence manipulation functions, please read the fine manual. A pointer: "Notice that because this is returning a session-local value, it gives a predictable answer whether or not other sessions have executed nextval since the current session did."
Insert and check curval(seq) inside one transaction. Before commiting transaction you'll see curval(seq) for your query and no matter who else inserted at the same time.
Don't remember the syntax exactly - read in manual (last used pgsql about 3 years ago), but in common it looks like this:
BEGIN TRANSACTION;
INSERT ...;
SELECT curval(seq);
COMMIT;
ex. minsert into log (desc,user_id) values ('drop her mind',6) returning id