I have a text file to read which has around 10000 points separated by,
x1,y1
x2,y2
x3,y3
.
.
10000 times
I read them using a loop in PHP and then store in an array and then I run a loop and insert one line at a time in my database. It takes really long time. Is there any way I can insert the whole array
for ($i=0; $i<10000; $i++)
{
$sql = '
INSERT INTO `firefly`.`FreeFormPoly`
(`markedObjectID`, `order`, `x`, `y`)
VALUES
('.$markedObjectsID.', '.$order.', '.$valuesx[i].','.$valuesy[i].')';
$db->query($sql, $markedObjectsID, $order, $values[1], $values[0]);
}
Try using multiple insert statement. Generate one insert and submit the entire statement
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
SO:
$sql = 'INSERT INTO `firefly`.`FreeFormPoly` (`markedObjectID`, `order`, `x`, `y`) VALUES';
for($i=0;$i<10000;$i++) {
if($i != 0) $sql .= ',';
$sql .= '('.$markedObjectsID.', '.$order.', '.$valuesx[i].','.$valuesy[i].')';
}
$db->query($sql);
I would do something like this:
$sql = 'INSERT INTO `firefly`.`FreeFormPoly` (`markedObjectID`, `order`, `x`, `y`) VALUES';
for($i=0;$i<length;$i++) {
$sql .= '('.$markedObjectsID.', '.$order.', .$valuesx[i].','.$valuesy[i].'),';
}
$sql = substr($sql,0,-1);
$db->query($sql);
Explanation:
The syntax to enter multiple records is
INSERT INTO TABLE_NAME VALUES(VAL1, VAL2, ....), (...), (...);
In the SQL you are concatenating (val1,val2,val3), every time you execute the loop hece you will get an extra , in the last position and substr() trims it off.
More preferably I would do
$sql = 'INSERT INTO `firefly`.`FreeFormPoly` (`markedObjectID`, `order`, `x`, `y`) VALUES ';
for($i=0;$i<length;$i++) {
$sql .= '('.$markedObjectsID.', '.$order.', .$valuesx[i].','.$valuesy[i].'),';
}
$sql = substr($sql,0,-1);
$result = mysqli_query($db,$sql)
or die('Error in querying the database');
You should be able to speed it up considerably by sending BEGIN TRANSACTION before the loop and COMMIT after the loop. One time I had to insert 14,000 data points on SQLite, and it took 20 minutes, but when I put the data in as a transaction it completed in 0.3 seconds.
First off, you can use prepared statements to reduce the network overhead. Assuming PDO:
$stmt = $db->prepare('INSERT INTO mytable (foo, bar, baz) VALUES (:foo, :bar, :baz)');
for ($i = 0; $i < $length; ++$i) {
$stmt->execute(array(
':foo' => $data[$i]['foo'],
':bar' => $data[$i]['bar'],
':baz' => $data[$i]['baz'],
));
}
Second, you could wrap the whole code inside $db->beginTransaction() and $db->commit().
Related
I have the following code which is working fine,
But I need a way to batch process it and
Insert it all in one go.
for($i=0; $i<sizeof($my_array); $i++)
{
$sql = "INSERT INTO exclude_resource (id,resource_id,config_id)
VALUES(DEFAULT, '$my_array[$i]' ,'$insert_id')";
$command = $connection->createCommand($sql);
$result = $command->execute();
}
Your query should resemble something like (see mysql docs):
INSERT INTO table (field1, field2, field3)
VALUES
(value1a, value2a, value3a),
(value1b, value2b, value3b),
(value1c, value2b, value3c),
...
So put the values in an array, join them with commas, and execute the resulting query. Incorporated in PHP:
$values = array();
for($i=0; $i<sizeof($my_array); $i++) {
$values[] = "(DEFAULT, '$my_array[$i]' ,'$insert_id')";
}
$sql =
"INSERT INTO exclude_resource (id,resource_id,config_id) VALUES '.
join(',', $values);
$command = $connection->createCommand($sql);
$result = $command->execute();
$insert_id is needs to have a value, but thats the same with your code snippet.
If you have more than 5k or 10k rows to insert, you should run an INSERT in between and reset the array.
So, I've got a few txt files, each container around 400,000 lines.
Each line is a word, I need to add to my database, if it isn't in there already.
Currently my code for checking/adding every word is
$sql = mysql_sql("SELECT `id` FROM `word_list` WHERE `word`='{$word}' LIMIT 1");
$num = mysql_num($sql);
if($num == '0'){
$length = strlen($word);
$timestamp = time();
#mysql_sql("INSERT INTO `word_list` (`word`, `length`, `timestamp`) VALUES ('{$word}', '{$length}', '{$timestamp}')");
}
and the functions being called are:
function mysql_sql($sql){
global $db;
$result = $db->query($sql);
return $result;
}
function mysql_num($result){
return $result->num_rows;
}
I'm looking for a better way to insert each word into the database.
Any ideas would be greatly appreciated.
I can think of some ways to do this.
First, if you have access to the MySQL server's file system you can use LOAD DATA INFILE to create a new table, then do an insert from that new table into your word_list table. This will most likely be your fastest option.
Second (if you don't have access to the MySQL server's file system), put a primary key or unique index on word_list.word. Then get rid of your SELECT query and use INSERT IGNORE INTO word_list .... That will allow MySQL automatically to skip the duplicate items without any need for you to do it with a query/insert operation.
Third, if your table uses an access method that handles transactions (InnoDB, not MyISAM), issue a BEGIN; statement before you start your insert loop. Then every couple of hundred rows issue COMMIT;BEGIN; . Then at the end issue COMMIT;. This will wrap your operations in multirow transactions so will speed things up a lot.
Try out this code. It will first create query with all your values and you will run query only ONCE ... Not again and again for ever row
$values = array();
$sql = mysql_sql("SELECT `id` FROM `word_list` WHERE `word`='{$word}' LIMIT 1");
$num = mysql_num($sql);
$insert_query = "INSERT INTO `word_list` (`word`, `length`, `timestamp`) VALUES ";
if ($num == '0') {
$length = strlen($word);
$timestamp = time();
$values[] = "('$word', '$length', '$timestamp')";
}
$insert_query .= implode(', ', $values);
#mysql_sql($insert_query);
how to insert random value in to database mysql ?
Ok, when i load this code i will have random number format like this
5,3,1,6,4,
and i want to insert 5,3,1,6,4, into database mysql ( 1 row 1 column) , how can i do ?
<?
$cards = array();
for ($i = 0; $i < 5; $i++)
{
$card = mt_rand(1, 6);
if(!in_array($card, $cards))
{
$cards[$i] = $card;
}
else
{
$i--;
}
}
foreach ($cards as $cards)
{
echo $cards.",";
}
?>
How does it differ from inserting non-random values ?
mysqli_query($link, "INSERT INTO yourtable (id, cards) VALUES (NULL, '$cards')");
Or something similar.
Pure (My)SQL solution:
INSERT INTO test( x, y )
SELECT 10*rand(),
10*rand()
FROM information_schema.columns
LIMIT 10;
Demo: http://www.sqlfiddle.com/#!2/be6e5/2
The easiest way to do what you describe would be as #Martin answered.
However this will probably screw you up later. Ideally you don't want to store a list in a RDBMS field; it defeats the whole purpose of having a database.
If you make a table like
ID, cardPosition, cardID
You can do this:
$insert = $db->prepare("INSERT INTO cardpositions (cardPosition, cardID) VALUES (:cardPosition, :cardID)");
foreach ($cards as $cardPosition=>$card)
{
$insert->execute(array(':cardPosition' => $cardPosition, ':cardID' => $card));
}
OR
The wrong way to do this, but the one you seem to want is:
$cardlist = implode(",",$cards); //now you have a comma separated list without the trailing comma
then you do this if PDO:
$insert = $db->prepare("INSERT INTO cardpositions (cardlist) VALUES (:cardlist)");
$insert->execute(array(':cardlist' => $cardlist));
or this if mysqli:
mysqli_query($link, "INSERT INTO yourtable (cardlist) VALUES ('$cardlist')");
Assuming I have an array as follows:
$array = array('first_value',
'second_value',
'thrid_value', 'and so on');
And a Column in which I'd want to insert those values, but each value in a separate row.
Would it it be possible to do that?
Obviously there are some answers to this one would be just loop thru the array elements and for every loop execute an insert statement, but that just seems unwise.
Or given that I'd have an ID column, that would help a lot(but I don't).
The amount of data to be introduced is not terribly large so the loop is perfectly viable, I just wanna make sure there isn't some easier way to do this that I may not be aware of.
You could use prepared statements; the first query will send the SQL statement and the subsequent calls will only send the data, thereby reducing the load:
$stmt = $db->prepare('INSERT INTO mytable (colname) VALUES (?)');
foreach ($array as $value) {
$stmt->execute(array($value));
}
If you're using PDO, such as the above example, make sure to disable prepared statement emulation.
// connect to database and store the resource in $connection
$array = array('first_value',
'second_value',
'thrid_value', 'and so on');
foreach($array as $value)
{
$value=mysqli_real_escape_string($connection,$value);
mysqli_query($connection,"INSERT INTO yourTABLE(columnName) VALUES('$value')");
}
You can put them all into a single INSERT statement with multiple VALUES lists.
$values = implode(',', array_map(function($v) use ($mysqli) {
return "'" . $mysqli->real_escape_string($v) . "'"; },
$array));
$query = "INSERT INTO yourTable (Column) VALUES $values";
$mysqli->execute($query) or die ($mysqli->error);
From mysql manual for insert,you may try this:
INSERT INTO yourtable (column_name) VALUES (value_a), (value_b), (value_c);
$array = array('first_value','second_value','third_value');
$SQL = "INSERT INTO `table` (column) VALUES('".implode("'),('",$array)."')";
OR
$values = '';
foreach($array as $val){
$values .= !empty($values)? ",('{$val}')" : "('{$val}')";
}
$SQL = "INSERT INTO `table` (column) VALUES{$values}";
I'm trying to load data from a few hundred text files into a database.
I believe MYSQL is exiting out of the loop without inserting all the rows.
Can anyone suggest how to insert blocks of 1000 rows to the end of data, with PHP code?
$filenames_array = array();
foreach($filenames_array as $filename)
{
$file_array = file($filename);
$file_value = $file_array[0];
$new_array = explode(",", $file_value);
$length = count($new_array);
for($i = 0; $i < $length; $i++)
{
$sql = "INSERT INTO `names`
(`id`, `name`)
VALUES
('',
'" . $new_array[$i] . "'
)";
$result = mysql_query($sql) or die(mysql_error());
echo $i . 'Row Inserted<br />';
}
}
you're probably trying to run too many INSERT statements in a single query.
look into PDO and prepared statements or use SQL syntax like this:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
Is it possible that one of the entries you're trying to insert contains a single quote '? In this case, an error would occur and the the loop wouldn't finish.
You should always escape the values you insert into the database with mysql_real_escape_string to prevent problems like that, and to make sure you're not vulnerable to sql injection.
$sql = "INSERT INTO `names`
(`id`, `name`)
VALUES
('',
'" . mysql_real_escape_string($new_array[$i]) . "'
)";
Why not combine every txt file into one big text file, and read it line by line? See the examples here http://php.net/manual/en/function.fgets.php
Mainly:
<?php
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
echo $buffer;
}
fclose($handle);
}
?>