I have been retrieving call logs from a cdr and dumping them into a database in MySQL. Of late the database crashes and the was giving me duplicates and junk characters so i modified to the below code.
modified code
$file1 = file_get_contents('file:///C:/Users/thy/Desktop/2011_0419_1531_v3.12R/cdr/'.$newname, FILE_USE_INCLUDE_PATH);
$arr1 = explode("\n", $file1);
foreach ($arr1 as $key => $value) {
$colArray = [];
$colArray['id'] = null;
$colArray['hashkey'] = md5(uniqid(rand(), true));
$split = explode(";", $value);
foreach ($split as $key => $val) {
# code...
$arr = (explode('=', $val));
$field = 'ch';
$item = '0';
$field = $arr[0];
$item = $arr[1];
$item = str_replace(str_split(')(\/'), '', $item);
$colArray[$field] = $item;
}
$columns = implode(', ', array_keys($colArray));
$values = implode(', ', $colArray);
$sql = "INSERT INTO `call`.`logs` (" . $columns . ") VALUES (" . $values . ")";
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
the above code keeps giving me errors
Error: INSERT INTO call.logs (id, hashkey, ) VALUES (,
797d8782a433b30e196fafc0ce01d09b, )You have an error in your SQL
syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near ') VALUES (,
797d8782a433b30e196fafc0ce01d09b, )' at line 1
Original code is below (the one i modified it from)
$file1 = file_get_contents('file:///C:/Users/thy/Desktop/2011_0419_1531_v3.12R/cdr/'.$newname, FILE_USE_INCLUDE_PATH);
$arr1 = explode("\n", $file1);
$data1 = array();
foreach ($arr1 as $key => $value) {
$split = explode(";", $value);
$keys = md5(uniqid(rand(), true));
//insert key to identify call.
$sql = "INSERT INTO `call`.`logs` (`id`, `hashkey`) VALUES (NULL, '{$keys}')";
if ($conn->query($sql) === TRUE) {
echo "New record created successfully";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
foreach ($split as $key => $val) {
# code...
$arr=(explode('=', $val));
$field='ch';
$item='0';
$field=$arr[0];
$item=$arr[1];
echo $field . " --";
echo "<br/>";
echo $item;
//sql
$sql = "UPDATE logs SET {$field}='{$item}' WHERE hashkey='{$keys}'";
if ($conn->query($sql) === TRUE) {
echo "Record updated successfully";
} else {
echo "Error updating record: " . $conn->error;
}
echo "done";
//$conn->close();
} echo "<br/>";
}
?>
<?php
I think the problem is with the generated/devised INSERt statement
INSERT INTO call.logs (" . $columns . ") VALUES (" . $values . ")";
There is an extra comma getting appended without column name?
If id is an autoincremental field. Try this:
INSERT INTO call.logs (id, hashkey) VALUES (default, '797d8782a433b30e196fafc0ce01d09b')
I want to use single for each for two array.
array 1 : has path of images (path is dynamic as per user need)
array 2: has values description field of each image (description is dynamic as per user need)
I want to insert both array in sql table using only one sql statement.
<?php $sql="INSERT INTO posts(title,description,category,createdBy,pictureURL,CreatedAt) VALUE ('$title',
'$description','$category','$creatdby','Admin/PostImages/$finalpath',now());";
$result = mysqli_query($conn,$sql);
if($result)
{
$upload_directory = 'PostImages/';
$field_values_array = $_REQUEST['desc'];
$x=0;
foreach($field_values_array as $value1){
foreach ( $_FILES['photo']['name'] AS $key => $value ){
//Move file to server directory
if(move_uploaded_file($_FILES["photo"]["tmp_name"][$x], $upload_directory . $_FILES["photo"]["name"][$x])){
$finalpath=$upload_directory . $_FILES["photo"]["name"][$x];
}
if (isset($_SESSION['p_id'])){
$p_id = $_SESSION["p_id"];
}
$sql1="INSERT INTO `postimages`(`p_id`,`description`, `img_path`) VALUES ('$p_id','$value1','$finalpath')";
$result1 = mysqli_query($conn,$sql1);
$x++;
}
}
header("Location: uploadpost_test.php");
}
else
{
echo "Error: " . $sql . "<br>" . $conn->error;
} ?>
I refactor and clean up your codes. Try this.
if(isset($_SESSION['p_id']))
{
$p_id = $_SESSION["p_id"];
}
$values = '';
$next = ',';
$len = count($field_values_array);
foreach($field_values_array as $x=>$value1)
{
if(move_uploaded_file($_FILES["photo"]["tmp_name"][$x], $upload_directory.$_FILES["photo"]["tmp_name"][$x]))
{
$tmp_name = $_FILES["photo"]["tmp_name"][$x];
$finalpath = $upload_directory.$tmp_name;
$next = $x >= $len-1 ? '' : $next;
$values .= "VALUES('$p_id','$value1','$finalpath')".$next;
}
}
$sql1="INSERT INTO `postimages`(`p_id`,`description`, `img_path`) $values";
$result1 = mysqli_query($conn,$sql1);
I need to perform a batch insert in MySQL/MariaDB but since data is dynamic I need to build the proper SQL query. In a few steps:
I should find whether the current row exists or not in table - this is the first SELECT inside the loop
Right now I have 1454 but have to insert around 150k later, is better a batch query than 150k INSERT per item on the loop
If record already exists I should update it if doesn't then I should insert ,I just not care about UPDATE yet and the code you're seeing is only for INSERT
So here is what I am doing:
// Get values from Csv file as an array of values
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
// VALUES should be added only if row doesn't exists
if ($rows_returned === 0) {
// VALUES should be append until they reach 1000
while ($i % 1000 !== 0) {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW())";
++$i;;
}
// QUERY should be output to console to see if it's right or something is wrong
echo "DEBUG: ", $sqlInsert, "\n";
// QUERY should be executed if there are 1000 VALUES ready to add as a batch
/*$rs = $conn->query($sqlInsert);
if ($rs === false) {
echo 'Wrong SQL: '.$sqlInsert.' Error: '.$conn->error, E_USER_ERROR;*/
}
} else {
// UPDATE
echo "UPDATE";
}
}
}
But this line of code: echo "DEBUG: ", $sql, "\n"; is not outputting nothing to console. I must be doing something wrong but I can't find what. Can any help me to build the proper batch query and to execute it each 1000 values append?
Proper output should be:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008ReolAAC'
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='005800000039SIWAA2'
....
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES(...), VALUES(...), VALUES(...)
Obtained result:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RGg6AAG'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RQ4CAAW'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
.... // until reach 1454 results
The table is empty so it should never goes through ELSE condition (UPDATE one).
EDIT
With help from the answer this is how the code looks now:
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 1;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
if ($rows_returned === 0) {
if ($i % 1000 === 0) {
file_put_contents("output.log", $sqlInsert."\n", FILE_APPEND);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
} else {
$sqlInsert .= "($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW()), ";
}
$i++;
} else {
echo "UPDATE";
}
}
}
But still buggy because:
I have got a first empty INSERT query: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES
I have got a second INSERT query with 1000 VALUES() append, but what happened with the rest? The remaining 454?
Can any give me another tip? Help?
Since it looks like you are trying to load data from a CSV file, you might want to consider using LOAD DATA INFILE functionality which is designed specifically for this purpose.
Here is link to documentation: https://dev.mysql.com/doc/refman/5.6/en/load-data.html
consider using INSERT IGNORE INTO table to check if the record already exists. How to 'insert if not exists' in MySQL?
if you haven't already done so, make veeva_rep_id a PRIMARY key so the INSERT IGNORE will work
also check out using PDO for transactions, prepared statements and dynamically generating queries using PDO
PDO Prepared Inserts multiple rows in single query
<?php
$sql = 'INSERT IGNORE INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES ';
$insertQuery = array();
$insertData = array();
/*
assuming the array from the csv is like this
$data = array(
0 => array('name' => 'Robert', 'value' => 'some value'),
1 => array('name' => 'Louise', 'value' => 'another value')
);
*/
foreach ($data as $row) {
$insertQuery[] = '(:veeva_rep_id' . $n . ', :first' . $n . ', :last' . $n . ', :email' . $n . ', :username' . $n . ', :lastLoginAt' . $n . ', :lastSyncAt' . $n . ', :display_name' . $n . ', :rep_type' . $n . ', :avatar_url' . $n . ', :createdAt' . $n . ', :updatedAt' . $n . ')';
$insertData['veeva_rep_id' . $n] = $row['name'];
$insertData['first' . $n] = $row['value'];
$insertData['last' . $n] = $row['name'];
$insertData['email' . $n] = $row['value'];
$insertData['username' . $n] = $row['name'];
$insertData['lastLoginAt' . $n] = $row['value'];
$insertData['lastSyncAt' . $n] = $row['value'];
$insertData['display_name' . $n] = $row['name'];
$insertData['rep_type' . $n] = $row['value'];
$insertData['avatar_url' . $n] = $row['value'];
$insertData['createdAt' . $n] = $row['name'];
$insertData['updatedAt' . $n] = $row['value'];
$n++;
}
$db->beginTransaction();
if (!empty($insertQuery) and count($insertQuery)>1000) {
$sql .= implode(', ', $insertQuery);
$stmt = $db->prepare($sql);
$stmt->execute($insertData);
}
$db->commit();
print $sql . PHP_EOL;
let me know if it helps.
You should have something like:
// Try fetching data from table 1
// If there is no record available, then fetch some data from table 2
// and insert that data inito table 1
You just wrote
$sql = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
But from an insert no data is selected and second...you didn't run a select, where comes $data from?
update Use if ($i % 1000 === 0) { instead of while ($i % 1000 !== 0) {
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$veeva_rep_id = ...;
$first = ...;
$last = ...;
$email = ...;
// ...
// VALUES should be added only if row doesn't exists
if($rs->num_rows == 0) {
// Insert some data
$i++;
if ($i % 1000 === 0) {
echo "DEBUG: ", $sqlInsert, "\n";
// execSql($sqlInsert);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) "; // reset
} else {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,...) ";
}
} else {
echo "UPDATE";
}
}
}
Following PHP code is working fine when only ONE set of array values in POST ie, on index value 0...When index value is greater than one, duplicate entries are getting inserted into the table..please help...
$sql = "INSERT INTO js (s_name, s_age, s_marks, s_school) VALUES ";
foreach($_POST as $objResult)
{
$i = 0;
foreach($objResult as $Result){
$i++;
if($i>1)// add ',' after first set of values in INSERT..
{
$sql .= ",";
}
$name = $Result['sname'];
$age = $Result['age'];
$mark = $Result['mark'];
$school = $Result['school'];
$sql .= "('".$name."','" .$age."','".$mark."','" .$school."')";
$result=$conn->query($sql);
}}
You can completely eliminate any issues with all of that iterating you have going on by doing something like this:
foreach($_POST as $objResult) {
foreach($objResult as $Result) {
$sql .= "(" . implode(', ', $Result) . "),";
}
}
$result=$conn->query($sql);
And now do you notice how I moved the query out of your loops? That makes sure that you run it properly and not on every iteration (loop) of the data.
Here is a working Example (You will have to press ctrl + enter to run the code)
You need to move the
$result=$conn->query($sql);
out of the inner loop, you are not clearing the string, so it will keep adding in the entries added already.
Thank you for your suggessions. And I changed my code as following..
$sql = "INSERT INTO js (s_name, s_age, s_marks, s_school) VALUES ";
foreach($_POST['student'] as $Result)
{
$name = $Result['sname'];
$age = $Result['age'];
$mark = $Result['mark'];
$school = $Result['school'];
$sql .= "('".$name."','" .$age."','".$mark."','" .$school."'),";
}
$sql = rtrim($sql,",");
$result=$conn->query($sql);
you can change code like this
foreach($_POST as $objResult) {
foreach($objResult as $Result) {
$sql .= "(" . implode(', ', $Result) . "),";
}
}
$result=$conn->query($sql);
i am making insert function that takes $table argument and $cols(as array)argument.
it inserts into given table given values:
$db->query("insert into $table({$cols[0]},{$cols[1]}) values('{$_POST[{$cols[0]}]}','{$_POST[{$cols[1]}]})");
this is all nice except i don't how long array is. how to do this??
One thing you haven't done is escaped the SQL using the correct escaping mechanism.
$postCols = $_POST['cols'];
foreach($postCols as &$col) {
$col = '"' . mysql_real_escape_string($col) . '"';
}
$db->query("insert into $table(" . implode(',', $cols) . ") values(" . implode(',', $postCols . ");
I would just use some foreach loops
<?php
$sql = "INSERT INTO $table (";
foreach ($cols as $col)
$sql .= "`$col`,";
$sql = substr($sql,0,-1);
$sql .= ") VALUES(";
foreach ($cols as $col)
$sql .= "'".$_POST[$col]."',";
$sql = substr($sql,0,-1);
$sql .= ");";
echo $sql;
?>