I got something like this.
It's working, but have to do a mysql query like 40 times is not the best thing.
Can somebody help me get it in to one query?
$string_unique = 'Unique string';
$number = count($something);
for ($i=0; $i < $number; $i++)
{
if(!empty($some_input)) {
$string_one = 'Something_one' . $i;
$string_two = 'Something_two' . $i;
mysql_query("INSERT INTO `table` (`unique_string`, `string_one`, `string_two`) VALUES('$unique_string', '$string_one', '$string_two') ON DUPLICATE KEY UPDATE `string_one` = '$string_one', `string_two` = '$string_two'") or die(mysql_error());
} else {
$string_one = '';
$string_two = '';
mysql_query("INSERT INTO `table` (`unique_string`, `string_one`, `string_two`) VALUES('$unique_string', '$string_one', '$string_two') ON DUPLICATE KEY UPDATE `string_one` = '$string_one', `string_two` = '$string_two'") or die(mysql_error());
}
You can generate a single query in that loop and then execute it only once:
$query = 'INSERT INTO `table` (`unique_string`, `string_one`, `string_two`) VALUES ';
$queryValues = array();
$string_unique = 'Unique string';
$number = count($something);
for ($i=0; $i < $number; $i++)
{
if(!empty($some_input)) {
$string_one = 'Something_one' . $i;
$string_two = 'Something_two' . $i;
$queryValues[] = sprintf('("%s", "%s", "%s")', $unique_string, $string_one, $string_two);
} else {
// I can't understand what are you using this part for... Anyway.
$queryValues[] = sprintf('("%s", "", "")', $unique_string);
}
}
$query .= implode(', ', $queryValues);
// And here is the unique key updating.
// I don't know whole the structure, so I just guess you have the PRIMARY `id` row
// What you did in your ON DUPLICATE KEY was unnecessary.
$query .= ' ON DUPLICATE KEY UPDATE `id`=`id`';
// Don't use mysql_* functions in new code.
// See: http://stackoverflow.com/questions/12859942/why-shouldnt-i-use-mysql-functions-in-php
mysql_query($query);
You're a allowed to add multiple rows of data in an insert statement in MySQL. That way, you can build one big insert statement in the loop, and execute it in one go after the loop. That is way faster and much more efficient.
You can insert multi row with sql,
here an example
insert into table1 (First,Last) values ("Fred","Smith"),
("John","Smith"),
("Michael","Smith"),
("Robert","Smith");
Related
Uncaught mysqli_sql_exception: Truncated incorrect DOUBLE value: '0-' in ... Stack trace: #0 ...(68): mysqli->query('UPDATE `Results...') #1 {main} thrown in ... on line 68
That's the error I am getting with the ellipses representing the file pathway.
The php I have:
for($i = 1; $i < $maxQuestions; $i++){
$answer = $_POST["question$i"] == "Yes" ? "1-" : "0-";
$connection->query("UPDATE `Results`
SET `question$i` = `question$i` + '$answer'
WHERE `ID` = '$id'");
}
The problem is with the SET `question$i` = `question$i` + '$answer' as I am trying to basically keep a history of the question's answers. For example, I may want the stored data to go from "0-" to "0-1-". How do you += a string with php/sql?
EDIT:
I have tried SET `question$i` = concat(`question$i` + '$answer') and I am getting the same error.
I changed it to
for($i = 1; $i < $maxQuestions; $i++){
$answer = $_POST["question$i"] == "yes" ? "1-" : "0-";
$qColumn = "question" . strval($i);
$questionData = $data["question$i"] . $answer;
$connection->query("UPDATE `Results`
SET `$qColumn` = '$questionData'
WHERE `ID` = '$id'");
}
This worked, but I'm still not sure why my initial code did not work. If someone could post a better answer/solution and why my initial code didn't work, please do. Thanks!
In mysql You can do it dierctly using concat this way
SET `question$i` =concat(`question$i`, '$answer')
You should not have column names ending with numbers because this is an indicator that your schema is not correct. You should normalise the database and have questions as a separate tables joined to your results table. Your results should not be concatenated into a single column either. Have a separate row for each result.
However, if you want to achieve this with your current set-up you could use CONCAT(). The query should look similar to this:
$types = '';
$setString = [];
$values = [];
for($i = 1; $i < $maxQuestions; $i++){
// concat types
$types .= 's';
// gather values
$values[] = isset($_POST["question$i"]) ? "1-" : "0-";
// build SET statement
$columnName = "question" . $i;
$setString[] = $columnName.' = CONCAT('.$columnName.', ?)';
}
$sql = "UPDATE Results SET ".implode(', ', $setString).' WHERE ID = ?';
$stmt = $connection->prepare($sql);
$stmt->bind_param($types, ...$values);
$stmt->execute();
I´m trying to create just one new row in a MySQL table
The problem is that I´m getting two new rows in my database.
I really can´t see why this is happening. The debug_to_console( "makepass" ); - my debug function - only gets executed once? AND the two rows it creates is not identical/copies
debug_to_console( "makepass" );
$salt = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
$len = strlen($salt);
$makepassJBH = '';
for ($i = 57; $i < $len; $i ++) {
$makepassJBH .= $salt[mt_rand(0, $len -1)];
}
$newpassJBH = password_hash($makepassJBH, PASSWORD_BCRYPT );
$licensidentifierJBH = '';
for ($i = 57; $i < $len; $i ++) {
$licensidentifierJBH .= $salt[mt_rand(0, $len -1)];
}
$fullkey = $makepassJBH . $licensidentifierJBH;
try {
// Get a db connection.
$db = JFactory::getDbo();
// Create a new query object.
$query = $db->getQuery(true);
// Insert columns.
$columns = array('licenskey','licenskey_Identifier','dateCreated', 'printed');
// Insert values.
$values = array($db->quote($newpassJBH), $db->quote($licensidentifierJBH), $db->quote(date("Y-m-d")), $db->quote(0));
// Prepare the insert query.
$query
->insert($db->quoteName('#__licenskey'))
->columns($db->quoteName($columns))
->values(implode(',', $values));
$query .= ' ON DUPLICATE KEY UPDATE ' . $db->quoteName('licenskey_Identifier') . ' = VALUES(' . $db->quoteName('licenskey_Identifier') . ')';
// Set the query using our newly populated query object and execute it.
$db->setQuery($query);
echo $db->replacePrefix((string) $query);
$db->execute();
}
The SQL dump:
INSERT INTO `mdh_licenskey` (`licenskey`,`licenskey_Identifier`,`dateCreated`,`printed`) VALUES ('$2y$10$jXkjJX1OZ7Vu0okV/QlxcehF5T2SSPZFhVHIx.E64HhidgYY.3URS','juLEo','2018-07-23','0') ON DUPLICATE KEY UPDATE `licenskey_Identifier` = VALUES(`licenskey_Identifier`)
Since the license key is generated by the PHP code above, that code must be executed twice. This can happen for multiple reasons but it can be difficult for us to guess exactly why. Either check if your browser is calling it twice, or if it is called twice from within the PHP code.
Make sure your script isn't executed twice. I had this issue when using firebug for example. This plugin made my page execute twice but not outputting to the console.
Sort of like a "prefetch" of the page before showing, don't know for sure.
Maybe add a 'debug to console' point on every line, like 'here 1, here 2'..
want to insert or update nearly 10000 records. It takes too long to insert in the for loop. Trying for the aternative way it only inserts the values but on duplicate_key entry shows the errors.
$var=0;
if($method == 1)
{
// $sql5 is the current way trying but shows error
$sql5='insert into stu_regnum_hall(hall_name,row,column,register_number,subject_code,exam_date,session,unique_key) values';
for ($i = 0; $i <=$arr_length; $i++)
{
for ($j = 1; $j <= 5; $j++)
{
for($k=0;$k<=4;$k++)
{
if(isset($show[$var+$k]))
{
if($k==0)
{
$hall_name=$arr1[$i];
$row=$j;
}
$column=$k+1;
$reg_num=$show[$var+$k];
$sub_code=$newArr12[$var+$k];
$unique_key=$reg_num.$date.$session;
//current way to insert This is taking too long time to store nearly 10000 records
Yii::$app->db->createCommand('insert into stu_regnum_hall(hall_name,row,column,register_number,subject_code,exam_date,session,unique_key) values ("'.$hall_name.'","'.$row.'","'.$column.'","'.$reg_num.'","'.$sub_code.'","'.$date.'","'.$session.'","'.$unique_key.'") ON DUPLICATE KEY UPDATE hall_name ="'.$hall_name.'",row="'.$row.'",column="'.$column.'",register_number="'.$reg_num.'",subject_code="'.$sub_code.'",exam_date="'.$date.'",session="'.$session.'",unique_key="'.$unique_key.'"')->query();
//concatenation of $sql5
$sql5.='("'.$hall_name.'","'.$row.'","'.$column.'","'.$reg_num.'","'.$sub_code.'","'.$date.'","'.$session.'","'.$unique_key.'") ON DUPLICATE KEY UPDATE hall_name= "'.$hall_name.'",row="'.$row.'",column="'.$column.'",register_number="'.$reg_num.'",subject_code="'.$sub_code.'",exam_date="'.$date.'",session="'.$session.'",unique_ key="'.$unique_key.'",';
}
else
{
break;
}
}
$var=$var+5;
}
}
$sql5.='("");';
// This is the required answer but getting error
Yii::$app->db->createCommand($sql5)->query();
}
the $sql5 query should work in this
Try Yii2 batchInsert with a little custom use INSERT IGNORE, REPLACE or ON DUPLICATE KEY UPDATE. It will insert very fast.
$sql = Yii::$app->db->queryBuilder->batchInsert('your_table_name', ['column1', 'column2','column3',...], $dataArray);
// For IGNORE on Duplicate
//$sql = 'INSERT IGNORE' . mb_substr($sql, strlen('INSERT'));
// For Replace on Duplicate
//$sql = 'REPLACE' . mb_substr($sql, strlen('INSERT'));
// For Update on Duplicate
$sql .= ' ON DUPLICATE KEY UPDATE `column1` = VALUES(`column1`), `column2` = VALUES(`column2`), ...';
Yii::$app->db->createCommand($sql)->execute();
Refer: http://www.yiiframework.com/doc-2.0/yii-db-querybuilder.html#batchInsert()-detail
I am looking for the fastest way to INSERT 1-3,000 rows into a MySQL database using PHP. My current solution is taking around 42 seconds to insert the rows which I think that could be much faster.
I am using a self-written DB class, the insert() method takes two params (string) $table and (array) $vars. The $items array is an associative array where the key is the column name in the table and the value is the value to insert. This works really well for because I sometimes have 30 columns in a table and already have the data there in an array. The insert() method is below:
function insert($table,$vars) {
if(empty($this->sql_link)){
$this->connection();
}
$cols = array();
$vals = array();
foreach($vars as $key => $value) {
$cols[] = "`" . $key . "`";
$vals[] = "'" . $this->esc($value) . "'";
}
//join the columns and values to insert into sql
$fields = join(', ', $cols);
$values = join(', ', $vals);
$insert = mysql_query("INSERT INTO `$table` ($fields) VALUES ($values);", $this->sql_link);
return $insert;
}
It should be self-explanatory but basically I take the keys and values from $vars and create an INSERT statement. It works, I think the problem I am having is sending the queries one at a time.
Should I build a long query string?
INSERT INTO table (field, field2, etc) VALUES (1, 2, ect);INSERT INTO table (field, field2, etc) VALUES (1, 2, ect);INSERT INTO table (field, field2, etc) VALUES (1, 2, ect);INSERT INTO table (field, field2, etc) VALUES (1, 2, ect);INSERT INTO table (field, field2, etc) VALUES (1, 2, ect); and send it all at one time? If so can this handle 3,000 insert statements in one call?
Is there another way I am not looking at? Any info is appreciated.
Thanks
The most performant way is to use the multiple-row insert syntax:
INSERT INTO table (field, field2, etc) VALUES (1, 2, etc),(1, 2, etc),(1, 2, etc);
Manual:
INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of column values, each enclosed within parentheses and separated by commas. Example:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
The values list for each row must be enclosed within parentheses.
Two ways of improve insertion speeds:
At the start, before any INSERT, do a mysql_query("START TRANSACTION"); or a simpler mysql_query("BEGIN");. At the end, do a mysql_query("COMMIT");. These two lines, speeds up the bulk insertion a 5-10x performance.
If the table backend is MyISAM (NOT InnoDB), do the INSERTs followed with the word DELAYED. For example, instead of INSERT INTO table use INSERT DELAYED INTO table for an aditional 10-15x speed-up.
If you combine the 2 methods, is posible to achieve a speed-up of 100 times.
Mysql can import data directly from a file which can significantly speed up importing data. See:
LOAD DATA INFILE Syntax
<?php
$data = "data/fullz.txt";
$db = new PDO("sqlite:db/ssninfo.db");
$db->beginTransaction();
$stmt = $db->prepare('INSERT INTO ssninfo (fname,lname,ssn,address,city,state,zip,phone,birth,email) VALUES (?,?,?,?,?,?,?,?,?,?)');
if($file=fopen($data, "r")){
while(!feof($file)){
$line = fgets($file);
$part = explode('|', $line);
$stmt->execute($part);
}
}
$db->commit();
As usual, it depends; you don't even mention which engine you're using, which is a big determinant. But I've found the MySQL manual guidance pretty reliable.
http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html
Auto discovering the maximum ammount of inserts.
to insert that kind of ammounts (3000) there should not be any problem of doing something like (assuming you use pdo):
$stmt = $dbh->prepare("INSERT INTO yourtable(name, id) VALUES " . str_repeat('(?,?),', $amountOfRows - 1) . '(?,?)');
You can improve that to make create generic way to create big statements like the one above for tables with different ammount of fields:
$fields = array("name", "id");
$fieldList = implode(", ", $fields);
$params = '(' . str_repeat('?,', count($fields) - 1) . '?)';
$values = str_repeat($params . ',', $ammountOfRows - 1) . $params;
$stmt = $dbh->prepare("INSERT INTO $table($fieldList) VALUES " . $values);
but the problem with the above solution is that wont work with any combination of rows and ammount of fields.
Seems to be that mysql is not only limited by the ammount of rows but also the ammount of parameters is taken into account.
But you dont want to be changing your code whenever a new mysql release changes the limit of the parameters, rows or even the size of the sql sentence.
So, a much better approach to create a generic way to generate big statements would be trying to feat the underlaying database engine:
/**
* Creates an insert sql with the maximum allowed of parameters
* #param string $table
* #param string $attributeList
* #param int &$ammountInserts returns the ammount of inserts
* #return \PDOStatement
*/
public static function getBiggestInsertStatement($table, $attributeList, $max, &$ammountInserts)
{
$previousSize = null;
$size = 10;
$sql = 'INSERT INTO ' . $table . '(' . implode(',', $attributeList) . ') values ';
$return = null;
$params = '(' . str_repeat('?,', count($attributeList) - 1) . '?)';
do {
try {
$previousSize = $size;
$values = str_repeat($params . ',', $size - 1) . $params;
$return = Db::getInstance()->prepare($sql . $values);
if ($size > $max) {
$values = str_repeat($params . ',', $max - 1) . $params;
$return = Db::getInstance()->prepare($sql . $values);
$ammountInserts = $max;
break;
}
$ammountInserts = $size;
$size *= 2;
} catch(\Exception $e) {
}
} while($previousSize != $size);
return $return;
}
One thing that you must have in mind is that since you dont know that limits the query could be able to push a lower ammount of items that all that you need to insert.
So you would have to create a strategy like the one below to succesfuly achieve insert them all in any possible scenario:
$insert = Db::getBiggestInsertStatement($table, array('field1','field2'), $numrows, $maximumInserts);
$i = 0;
$values = array();
for ($j = 0; $j < $numrows; $j++) {
if ($i === $maximumInserts) {
$insert->execute($values);
$i = 0;
$values = array();
}
$values[] = "value1" . $j;
$values[] = "value2" . $j;
$i++;
});
if ($i > 0) {
$insertRemaining = Db::getBiggestInsertStatement($table, array('field1', 'field2'), $i, $maximumInserts);
$insertRemaining->execute($values);
}
I have tried to insert in a table with a single column 1000000 rows, and it's done within seconds, agains minutes that would take to insert them one by one.
The standard technique for speeding up bulk inserts in to use a prepared SQL statement inside a loop inside a transaction. That will make it pretty well optimal. After that you could try tweaking it in various ways, but you are probably wasting your time.
I have two arrays with anywhere from 1 to 5 set values. I want to insert these values into a table with two columns.
Here's my current query, given to me in another SO question:
INSERT INTO table_name (country, redirect)
VALUES ('$country[1]', '$redirect[1]'),
('$country[2]', '$redirect[2]'),
('$country[3]', '$redirect[3]'),
('$country[4]', '$redirect[4]'),
('$country[5]', '$redirect[5]')
ON DUPLICATE KEY UPDATE redirect=VALUES(redirect)
I'm a little concerned however with what happens if some of these array values aren't set, as I believe the above assumes there's 5 sets of values (10 values in total), which definitely isn't certain. If a value is null/0 does it automatically skip it?
Would something like this work better, would it be a lot more taxing on resources?
for($i = 0, $size = $sizeof($country); $i <= size; $i++) {
$query = "INSERT INTO table_name (country, redirect) VALUES ('$country[$i]', '$redirect[$i]) ON DUPLICATE KEY UPDATE redirect='$redirect[$i]'";
$result = mysql_query($query);
}
Questions highlighted in bold ;). Any answers would be very much appreciated :) :)!!
Do something like this:
$vals = array()
foreach($country as $key => $country_val) {
if (empty($country_val) || empty($redirect[$key])) {
continue;
}
$vals[] = "('" . mysql_real_escape_string($country_val) . "','" . mysql_real_escape_string($redirect[$key]) . "')";
}
$val_string = implode(',', $vals);
$sql = "INSERT INTO .... VALUES $val_string";
That'll built up the values section dynamically, skipping any that aren't set. Note, however, that there is a length limit to mysql query strings, set by the max_allowed_packet setting. If you're building a "huge" query, you'll have to split it into multiple smaller ones if it exceeds this limit.
If you are asking whether php will automatically skip inserting your values into the query if it is null or 0, the answer is no. Why dont you loop through the countries, if they have a matching redirect then include that portion of the insert statement.. something like this: (not tested, just showing an example). It's one query, all values. You can also incorporate some checking or default to null if they do not exist.
$query = "INSERT INTO table_name (country, redirect) VALUES ";
for($i = 0, $size = $sizeof($country); $i <= size; $i++) {
if(array_key_exists($i, $country && array_key_exists($i, $redirect)
if($i + 1 != $size){
$query .= "('".$country[$i]."', '".$redirect[$i]).",";
} else $query .= "('".$country[$i]."', '".$redirect[$i].")";
}
}
$query .= " ON DUPLICATE KEY UPDATE redirect=VALUES(redirect);"
$result = mysql_query($query);