I need to perform a batch insert in MySQL/MariaDB but since data is dynamic I need to build the proper SQL query. In a few steps:
I should find whether the current row exists or not in table - this is the first SELECT inside the loop
Right now I have 1454 but have to insert around 150k later, is better a batch query than 150k INSERT per item on the loop
If record already exists I should update it if doesn't then I should insert ,I just not care about UPDATE yet and the code you're seeing is only for INSERT
So here is what I am doing:
// Get values from Csv file as an array of values
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
// VALUES should be added only if row doesn't exists
if ($rows_returned === 0) {
// VALUES should be append until they reach 1000
while ($i % 1000 !== 0) {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW())";
++$i;;
}
// QUERY should be output to console to see if it's right or something is wrong
echo "DEBUG: ", $sqlInsert, "\n";
// QUERY should be executed if there are 1000 VALUES ready to add as a batch
/*$rs = $conn->query($sqlInsert);
if ($rs === false) {
echo 'Wrong SQL: '.$sqlInsert.' Error: '.$conn->error, E_USER_ERROR;*/
}
} else {
// UPDATE
echo "UPDATE";
}
}
}
But this line of code: echo "DEBUG: ", $sql, "\n"; is not outputting nothing to console. I must be doing something wrong but I can't find what. Can any help me to build the proper batch query and to execute it each 1000 values append?
Proper output should be:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008ReolAAC'
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='005800000039SIWAA2'
....
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES(...), VALUES(...), VALUES(...)
Obtained result:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RGg6AAG'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RQ4CAAW'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
.... // until reach 1454 results
The table is empty so it should never goes through ELSE condition (UPDATE one).
EDIT
With help from the answer this is how the code looks now:
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 1;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
if ($rows_returned === 0) {
if ($i % 1000 === 0) {
file_put_contents("output.log", $sqlInsert."\n", FILE_APPEND);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
} else {
$sqlInsert .= "($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW()), ";
}
$i++;
} else {
echo "UPDATE";
}
}
}
But still buggy because:
I have got a first empty INSERT query: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES
I have got a second INSERT query with 1000 VALUES() append, but what happened with the rest? The remaining 454?
Can any give me another tip? Help?
Since it looks like you are trying to load data from a CSV file, you might want to consider using LOAD DATA INFILE functionality which is designed specifically for this purpose.
Here is link to documentation: https://dev.mysql.com/doc/refman/5.6/en/load-data.html
consider using INSERT IGNORE INTO table to check if the record already exists. How to 'insert if not exists' in MySQL?
if you haven't already done so, make veeva_rep_id a PRIMARY key so the INSERT IGNORE will work
also check out using PDO for transactions, prepared statements and dynamically generating queries using PDO
PDO Prepared Inserts multiple rows in single query
<?php
$sql = 'INSERT IGNORE INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES ';
$insertQuery = array();
$insertData = array();
/*
assuming the array from the csv is like this
$data = array(
0 => array('name' => 'Robert', 'value' => 'some value'),
1 => array('name' => 'Louise', 'value' => 'another value')
);
*/
foreach ($data as $row) {
$insertQuery[] = '(:veeva_rep_id' . $n . ', :first' . $n . ', :last' . $n . ', :email' . $n . ', :username' . $n . ', :lastLoginAt' . $n . ', :lastSyncAt' . $n . ', :display_name' . $n . ', :rep_type' . $n . ', :avatar_url' . $n . ', :createdAt' . $n . ', :updatedAt' . $n . ')';
$insertData['veeva_rep_id' . $n] = $row['name'];
$insertData['first' . $n] = $row['value'];
$insertData['last' . $n] = $row['name'];
$insertData['email' . $n] = $row['value'];
$insertData['username' . $n] = $row['name'];
$insertData['lastLoginAt' . $n] = $row['value'];
$insertData['lastSyncAt' . $n] = $row['value'];
$insertData['display_name' . $n] = $row['name'];
$insertData['rep_type' . $n] = $row['value'];
$insertData['avatar_url' . $n] = $row['value'];
$insertData['createdAt' . $n] = $row['name'];
$insertData['updatedAt' . $n] = $row['value'];
$n++;
}
$db->beginTransaction();
if (!empty($insertQuery) and count($insertQuery)>1000) {
$sql .= implode(', ', $insertQuery);
$stmt = $db->prepare($sql);
$stmt->execute($insertData);
}
$db->commit();
print $sql . PHP_EOL;
let me know if it helps.
You should have something like:
// Try fetching data from table 1
// If there is no record available, then fetch some data from table 2
// and insert that data inito table 1
You just wrote
$sql = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
But from an insert no data is selected and second...you didn't run a select, where comes $data from?
update Use if ($i % 1000 === 0) { instead of while ($i % 1000 !== 0) {
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$veeva_rep_id = ...;
$first = ...;
$last = ...;
$email = ...;
// ...
// VALUES should be added only if row doesn't exists
if($rs->num_rows == 0) {
// Insert some data
$i++;
if ($i % 1000 === 0) {
echo "DEBUG: ", $sqlInsert, "\n";
// execSql($sqlInsert);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) "; // reset
} else {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,...) ";
}
} else {
echo "UPDATE";
}
}
}
I want to insert an array in the database. The array can be changed all the time. I want different rows in the database.
My code:
$var = file_get_contents("test2.txt");
$test = preg_replace('/\\\\/', '', $var);
$poep = explode(" ", $test);
Yeah, there is no database connection, because I want to know how to 'split' the array to insert it in the database.
I have tried this:
foreach($poep as $row) {
$row = $mysqli->real_escape_string($row);
if($mysqli->query("insert into data('array') VALUES ($row)") == false){
echo 'Doesnt works!';
}
It returns 'Doesnt works', so I think there is a problem with query?
#NadirDev Hi. Assuming that you are using Core PHP programming. After exploding the string by the space, run foreach loop and then insert individual rows. Look at this rough code to get idea:
foreach($poep as $row) {
// $row now contains one word. Add that in database.
$row = mysql_real_escape_string($row);
$query = mysql_query("insert into tableName('fieldName') VALUES ($row)");
}
here's some code I wrote. It processes a CSV file and stores separate rows into a db table (difference is just that you have a TXT file). It does the mysql insertion in batches of 250 rows. Hope it can help you!
// read all input rows into an array
echo "Processing input..<br /><br />";
$row = 0;
$input = array();
if (($handle = fopen($file['tmp_name'], "r")) !== FALSE) {
while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
$input[$row][] = addslashes($data[$c]);
}
$row++;
}
fclose($handle);
}
$count = 0;
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
foreach ($input as $inp) {
$q .= "('" . addslashes($inp[0]) . "', '" . addslashes($inp[1]) . "', '" . addslashes($inp[2]) . "', '" . date('Y-m-d H:i:s') . "'), ";
$count++;
if ($count >= 250) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
$count = 0;
}
}
if ($count > 0) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
}
echo "Successfully added " . count($input) . " rows to the input list.";
Whenever I import an excel file into MySQL using PHPExcel then I don't get some values and also values get inserted into irrespective columns.
BUT when I use echo for showing excel file then it shows exact report.
Also, I want only few selected columns to be imported from 10-15 columns.
Thanks in advance
My Code
<?php
//include the following 2 files
require 'C:\xampp\PHPExcel_1.7.9_doc\Classes\PHPExcel.php';
require_once 'C:\xampp\PHPExcel_1.7.9_doc\Classes\PHPExcel\IOFactory.php';
$conn = mysql_connect("localhost","root","");
mysql_select_db("invoice",$conn);
$file=$_POST['file'];
$srow=$_POST['srow'];
$objPHPExcel = PHPExcel_IOFactory::load($file);
foreach ($objPHPExcel->getWorksheetIterator() as $worksheet) {
$worksheetTitle = $worksheet->getTitle();
$highestRow = $worksheet->getHighestRow(); // e.g. 10
$highestColumn = $worksheet->getHighestColumn(); // e.g 'F'
$highestColumnIndex = PHPExcel_Cell::columnIndexFromString($highestColumn);
$nrColumns = ord($highestColumn) - 64;
echo "<br>The worksheet ".$worksheetTitle." has ";
echo $nrColumns . ' columns (A-' . $highestColumn . ') ';
echo ' and ' . $highestRow . ' row.';
}
echo 'Data: <table width="100%" cellpadding="3" cellspacing="0"><tr>';
for ($row = $srow; $row <= $highestRow-1; ++ $row) {
echo '<tr>';
for ($col = 0; $col < $highestColumnIndex; ++ $col) {
$cell = $worksheet->getCellByColumnAndRow($col, $row);
$val = $cell->getValue();
if($row === 1)
echo '<td style="background:#000; color:#fff;">' . $val . '</td>';
else
echo '<td>' . $val . '</td>';
}
echo '</tr>';
}
echo '</table>';
for ($row = $srow; $row <= $highestRow; ++ $row) {
$val=array();
for ($col = 0; $col <= $highestColumnIndex; ++ $col) {
$cell = $worksheet->getCellByColumnAndRow($col, $srow);
$val[] = $cell->getValue();
}
$dropdownval=$_POST['menu'];
if ($dropdownval!='none')
{
$sql="INSERT INTO $dropdownval(`Bkng Date`,`ID Number`,`Pax Name`,`Ticket Number`,`PNR`,`Sector`,`Travel Date`,`Basic`,`Tax`,`Net Payable`)
VALUES('".$val[1]."','".$val[2]."','".$val[3]."','" .$val[4]."','".$val[5]."','".$val[6]."','".$val[7]."','".$val[8]."','".$val[9]."','".$val[10]."')";
mysql_query($sql)or die('Invalid query: ' . mysql_error());
if(mysql_query($sql))
echo "Successfully Uploaded ".$file;
else
echo "Error while uploading ".$file;
//echo $sql."\n";
}
else
{
$tname=$_POST['tname'];
$createTable="CREATE TABLE $tname(
`Bkng Date` DATE NOT NULL ,
`ID Number` INT NOT NULL ,
`Pax Name` VARCHAR( 100 ) NOT NULL ,
`Ticket Number` VARCHAR( 100 ) NOT NULL ,
`PNR` VARCHAR( 50 ) NOT NULL ,
`Sector` VARCHAR( 50 ) NOT NULL ,
`Travel Date` DATE NOT NULL ,
`Basic` DECIMAL NOT NULL ,
`Tax` DECIMAL NOT NULL ,
`Net Payable` DECIMAL NOT NULL)";
mysql_query($createTable);
$sql1="INSERT INTO $tname(`Bkng Date`,`ID Number`,`Pax Name`,`Ticket Number`,`PNR`,`Sector`,`Travel Date`,`Basic`,`Tax`,`Net Payable`)
VALUES('".$val[1]."','".$val[2]."','".$val[3]."','" .$val[4]."','".$val[5]."','".$val[6]."','".$val[7]."','".$val[8]."','".$val[9]."','".$val[10]."')";
mysql_query($sql1)or die('Invalid query: ' . mysql_error());
if(mysql_query($sql1))
echo "Table $tname has been created...Successfully Uploaded ".$file;
else
echo "Error while uploading ".$file;
}
}
?>
I have an foreach loop that is creating a table with new column names from an array collected from a site. At the moment only the first element of the array is being set as a column name, can someone help?
H
Here is my code:
<?php
include 'simple_html_dom.php';
include 'connection.php';
function getsquad($url, $tablename){
$html = file_get_html($url);
$player_fromsite = array();
$space = ' ';
$replacespace = '_';
$player = array();
foreach($html->find('td[align=left]') as $element) {
if ($element->children(0)) { // work only when children exists
array_push($player_fromsite ,$element->children(0)->innertext);
}
}
//$player[] = str_replace($space, $replacespace, $player_fromsite);
//unset($player_fromsite);
$length = count($player);
foreach($player_fromsite as $player_name) {
mysql_query("CREATE TABLE " . $tablename . "(" . str_replace($space, $replacespace, $player_name) . " VARCHAR(30))") or die(mysql_error());
}
echo "Table Created!";
}
$Squad = new squad();
$Squad->getsquad('site', 'Ars');
?>
any spaces are replace with a "_", in the foreach loop
I am not quite sure if I correctly understand what you want: In $player_fromsite you have some strings, which should become columns of the new table with the name from $tablename, whereas all columns are VARCHAR(30).
If so, replace your last foreach loop with the following:
$columns = array();
foreach ($player_fromsite as $player_name) {
$columns[] = '`' . str_replace($space, $resplacespace, $player_name) . '` VARCHAR(30)';
}
$query = 'CREATE TABLE `' . $tablename . '` (' . implode(',', $columns) . ')';
mysql_query($query) or die(mysql_error());
The foreach loop basically prepares all the columns, which afterwards get concatenated to the actual query which is send to the database.
So for exmaple let $player_fromsite = array('foo', 'bar'), you would end up with the query
CREATE TABLE `Ars` (`foo` VARCHAR(30),`bar` VARCHAR(30));