here is the code:
if (($handle = fopen($source_file, "r")) !== FALSE) {
$columns = fgetcsv($handle, $max_line_length, ",");
foreach ($columns as &$column) {
$column = str_replace(".","",$column);
}
while (($data = fgetcsv($handle, $max_line_length, ",")) !== FALSE) {
while(count($data) < count($columns)) {
array_push($data, NULL);
}
$c = count($data);
for($i = 0; $i < $c; $i++) {
$data[$i] = "'{$data[$i]}'";
}
$sql[] = '(' . implode(',',$data) . ','.$_POST[group].')';
}
$sql = implode(',',$sql);
$query = "INSERT INTO mytable (".mysql_real_escape_string(implode(",",$columns)).",Custgroup,user_id) VALUES "
. $sql . "\n";
mysql_query($query) or trigger_error(mysql_error());
fclose($handle);
}
}
If my csv file is:
lastname,firstname,gender
bob,ah,male
So now the query will be : INSERT INTO mytable (lastname,firstname,gender) VALUES ('bob','ah','male').
But how if I want to insert extra data into mysql together with the data in csv file?
Such as I have a value $_POST['group'] = 'family' and $_POST['user_id'] = '10', so I tried:
$sql[] = '(' . implode(',',$data) . ','.$_POST[group].','.$_POST[user_id].')';
$query = "INSERT INTO $target_table (".mysql_real_escape_string(implode(",",$columns)).",custgroup,user_id) VALUES "
. implode(',',$sql) . "\n";
But in the query it will become : INSERT INTO mytable (lastname,firstname,gender,custgroup,user_id) VALUES ('bob','ah','male',family,10).
It didn't have single quote , so I have error to insert the record.Can I know how to solve this problem?
try this:
$sql[] = '(' . implode(',', $data) . ", '" . $_POST['group'] . "'," . $_POST['user_id'] . ')';
$query = "INSERT INTO $target_table (" . mysql_real_escape_string(implode(',', $columns)) . ',custgroup,user_id) VALUES ' . implode(',', $sql);
for the sake of readability you might consider the following :
$sql = sprintf("(%s,'%s',%d)",
implode(',', $data),
$_POST['group'],
$_POST['user_id']
);
which gives you more flexibility and understanding in what your code actually performs.
Related
I need to perform a batch insert in MySQL/MariaDB but since data is dynamic I need to build the proper SQL query. In a few steps:
I should find whether the current row exists or not in table - this is the first SELECT inside the loop
Right now I have 1454 but have to insert around 150k later, is better a batch query than 150k INSERT per item on the loop
If record already exists I should update it if doesn't then I should insert ,I just not care about UPDATE yet and the code you're seeing is only for INSERT
So here is what I am doing:
// Get values from Csv file as an array of values
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
// VALUES should be added only if row doesn't exists
if ($rows_returned === 0) {
// VALUES should be append until they reach 1000
while ($i % 1000 !== 0) {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW())";
++$i;;
}
// QUERY should be output to console to see if it's right or something is wrong
echo "DEBUG: ", $sqlInsert, "\n";
// QUERY should be executed if there are 1000 VALUES ready to add as a batch
/*$rs = $conn->query($sqlInsert);
if ($rs === false) {
echo 'Wrong SQL: '.$sqlInsert.' Error: '.$conn->error, E_USER_ERROR;*/
}
} else {
// UPDATE
echo "UPDATE";
}
}
}
But this line of code: echo "DEBUG: ", $sql, "\n"; is not outputting nothing to console. I must be doing something wrong but I can't find what. Can any help me to build the proper batch query and to execute it each 1000 values append?
Proper output should be:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008ReolAAC'
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='005800000039SIWAA2'
....
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES(...), VALUES(...), VALUES(...)
Obtained result:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RGg6AAG'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RQ4CAAW'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
.... // until reach 1454 results
The table is empty so it should never goes through ELSE condition (UPDATE one).
EDIT
With help from the answer this is how the code looks now:
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 1;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
if ($rows_returned === 0) {
if ($i % 1000 === 0) {
file_put_contents("output.log", $sqlInsert."\n", FILE_APPEND);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
} else {
$sqlInsert .= "($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW()), ";
}
$i++;
} else {
echo "UPDATE";
}
}
}
But still buggy because:
I have got a first empty INSERT query: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES
I have got a second INSERT query with 1000 VALUES() append, but what happened with the rest? The remaining 454?
Can any give me another tip? Help?
Since it looks like you are trying to load data from a CSV file, you might want to consider using LOAD DATA INFILE functionality which is designed specifically for this purpose.
Here is link to documentation: https://dev.mysql.com/doc/refman/5.6/en/load-data.html
consider using INSERT IGNORE INTO table to check if the record already exists. How to 'insert if not exists' in MySQL?
if you haven't already done so, make veeva_rep_id a PRIMARY key so the INSERT IGNORE will work
also check out using PDO for transactions, prepared statements and dynamically generating queries using PDO
PDO Prepared Inserts multiple rows in single query
<?php
$sql = 'INSERT IGNORE INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES ';
$insertQuery = array();
$insertData = array();
/*
assuming the array from the csv is like this
$data = array(
0 => array('name' => 'Robert', 'value' => 'some value'),
1 => array('name' => 'Louise', 'value' => 'another value')
);
*/
foreach ($data as $row) {
$insertQuery[] = '(:veeva_rep_id' . $n . ', :first' . $n . ', :last' . $n . ', :email' . $n . ', :username' . $n . ', :lastLoginAt' . $n . ', :lastSyncAt' . $n . ', :display_name' . $n . ', :rep_type' . $n . ', :avatar_url' . $n . ', :createdAt' . $n . ', :updatedAt' . $n . ')';
$insertData['veeva_rep_id' . $n] = $row['name'];
$insertData['first' . $n] = $row['value'];
$insertData['last' . $n] = $row['name'];
$insertData['email' . $n] = $row['value'];
$insertData['username' . $n] = $row['name'];
$insertData['lastLoginAt' . $n] = $row['value'];
$insertData['lastSyncAt' . $n] = $row['value'];
$insertData['display_name' . $n] = $row['name'];
$insertData['rep_type' . $n] = $row['value'];
$insertData['avatar_url' . $n] = $row['value'];
$insertData['createdAt' . $n] = $row['name'];
$insertData['updatedAt' . $n] = $row['value'];
$n++;
}
$db->beginTransaction();
if (!empty($insertQuery) and count($insertQuery)>1000) {
$sql .= implode(', ', $insertQuery);
$stmt = $db->prepare($sql);
$stmt->execute($insertData);
}
$db->commit();
print $sql . PHP_EOL;
let me know if it helps.
You should have something like:
// Try fetching data from table 1
// If there is no record available, then fetch some data from table 2
// and insert that data inito table 1
You just wrote
$sql = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
But from an insert no data is selected and second...you didn't run a select, where comes $data from?
update Use if ($i % 1000 === 0) { instead of while ($i % 1000 !== 0) {
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$veeva_rep_id = ...;
$first = ...;
$last = ...;
$email = ...;
// ...
// VALUES should be added only if row doesn't exists
if($rs->num_rows == 0) {
// Insert some data
$i++;
if ($i % 1000 === 0) {
echo "DEBUG: ", $sqlInsert, "\n";
// execSql($sqlInsert);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) "; // reset
} else {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,...) ";
}
} else {
echo "UPDATE";
}
}
}
I want to insert an array in the database. The array can be changed all the time. I want different rows in the database.
My code:
$var = file_get_contents("test2.txt");
$test = preg_replace('/\\\\/', '', $var);
$poep = explode(" ", $test);
Yeah, there is no database connection, because I want to know how to 'split' the array to insert it in the database.
I have tried this:
foreach($poep as $row) {
$row = $mysqli->real_escape_string($row);
if($mysqli->query("insert into data('array') VALUES ($row)") == false){
echo 'Doesnt works!';
}
It returns 'Doesnt works', so I think there is a problem with query?
#NadirDev Hi. Assuming that you are using Core PHP programming. After exploding the string by the space, run foreach loop and then insert individual rows. Look at this rough code to get idea:
foreach($poep as $row) {
// $row now contains one word. Add that in database.
$row = mysql_real_escape_string($row);
$query = mysql_query("insert into tableName('fieldName') VALUES ($row)");
}
here's some code I wrote. It processes a CSV file and stores separate rows into a db table (difference is just that you have a TXT file). It does the mysql insertion in batches of 250 rows. Hope it can help you!
// read all input rows into an array
echo "Processing input..<br /><br />";
$row = 0;
$input = array();
if (($handle = fopen($file['tmp_name'], "r")) !== FALSE) {
while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
$input[$row][] = addslashes($data[$c]);
}
$row++;
}
fclose($handle);
}
$count = 0;
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
foreach ($input as $inp) {
$q .= "('" . addslashes($inp[0]) . "', '" . addslashes($inp[1]) . "', '" . addslashes($inp[2]) . "', '" . date('Y-m-d H:i:s') . "'), ";
$count++;
if ($count >= 250) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
$count = 0;
}
}
if ($count > 0) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
}
echo "Successfully added " . count($input) . " rows to the input list.";
This is my code on mysql query that I want to change to sqlsrv query to import into sql server database
$message = null;
$allowed_extensions = array('csv');
$upload_path = '../csv';
if (!empty($_FILES['file'])) {
if ($_FILES['file']['error'] == 0) {
// check extension
$file = explode(".", $_FILES['file']['name']);
$extension = array_pop($file);
if (in_array($extension, $allowed_extensions)) {
if (move_uploaded_file($_FILES['file']['tmp_name'], $upload_path.'/'.$_FILES['file']['name'])) {
if (($handle = fopen($upload_path.'/'.$_FILES['file']['name'], "r")) !== false) {
$keys = array();
$out = array();
$insert = array();
$line = 1;
while (($row = fgetcsv($handle, 0, ',', '"')) !== FALSE) {
foreach($row as $key => $value) {
if ($line === 1) {
$keys[$key] = $value;
} else {
$out[$line][$key] = $value;
}
}
$line++;
}
fclose($handle);
if (!empty($keys) && !empty($out)) {
$db = new PDO('mysql:host=localhost;dbname=project', 'root', '');
$db->exec("SET CHARACTER SET utf8");
foreach($out as $key => $value) {
$sql = "INSERT INTO `department` (`";
$sql .= implode("`, `", $keys);
$sql .= "`) VALUES (";
$sql .= implode(", ", array_fill(0, count($keys), "?"));
$sql .= ")";
$statement = $db->prepare($sql);
$statement->execute($value);
}
$message = '<span style="color:green">File has been uploaded successfully</span>';
}
}
}
} else {
$message = '<span style="color:red">Only .csv file format is allowed</span>';
}
} else {
$message = '<span style="color:red">There was a problem with your file</span>';
}
Hope it will help you to do it manually.
Sometime MS SQL table or column names are enclosed in square brackets in queries (e.g. if contains spaces or for some other reasons). MySQL does not allow square brackets around table of column names, they all must be replaced by symbol or cut off: [object] ->object`.
So you need to remove all "'" from the table name department and its columns too in your above case.
For more.
http://www.convert-in.com/mssql-to-mysql-queries.htm
Added:
Your SQL Query will be as below. Please remove ? mark too.
$sql = INSERT INTO department (;
$sql .= implode(",", $keys);
$sql .= ) VALUES (";
$sql .= implode(", ", array_fill(0, count($keys), "?"));
$sql .= ")";
$statement = $db->prepare($sql);
$statement->execute($value);
I am not well aware about PHP. I can only help to construct the SQL Query. Its should be
"INSERT INTO department (COLUMN1,COLUMN2,..) values(VALUE1,VALUE1,..)".
Added:
As I already told you I am not a PHP guy. But Tried for you. Please try your php code as :
$sql = "INSERT INTO department (";
$sql .= "implode(',', $keys)";
$sql .= ") VALUES (";
$sql .= "implode(',', array_fill(0, count($keys), ''))";
$sql .= ")";
$statement = $db->prepare($sql); $statement->execute($value);
But array_fill(0, count($keys), '') is empty data right?
I need to migrate data within our old bug tracker (Zentrack) to our new one and the only import method I can use is to import from CSV. To do so I need the data from two tables, tickets and logs so I wrote the script below in php.
<?php
error_reporting(E_ALL);
$host = "localhost";
$user = "dbuser";
$pass = "dbpass";
$db = "zentrk";
$users[1] = 'john';
$users[4] = 'sally';
$users[5] = 'nick';
$users[6] = 'ralph';
$r = mysql_connect($host, $user, $pass);
if (!$r) {
echo "Could not connect to server\n";
trigger_error(mysql_error(), E_USER_ERROR);
}
echo mysql_get_server_info() . "\n";
$r2 = mysql_select_db($db);
if (!$r2) {
echo "Cannot select database\n";
trigger_error(mysql_error(), E_USER_ERROR);
}
$query_tickets = "select ZENTRACK_TICKETS.id, ZENTRACK_TICKETS.title, ZENTRACK_TICKETS.priority, ZENTRACK_TICKETS.status, ZENTRACK_TICKETS.description, ZENTRACK_TICKETS.otime,
ZENTRACK_TICKETS.type_id, ZENTRACK_TICKETS.user_id, ZENTRACK_TICKETS.system_id, ZENTRACK_TICKETS.creator_id, ZENTRACK_TICKETS.proj_key
from ZENTRACK_TICKETS
where ZENTRACK_TICKETS.status = 'OPEN'
and ZENTRACK_TICKETS.system_id in ('18', '3', '6', '1', '16', '7', '9', '4', '20')
and ZENTRACK_TICKETS.type_id not in ('1', '10', '5')";
$rs = mysql_query($query_tickets);
if (!$rs) {
echo "Could not execute query: $query";
trigger_error(mysql_error(), E_USER_ERROR);
} else {
echo "Query: $query executed\n";
}
$export = array();
$export[] = 'id,title,created date,priority,type,assigned,description,system,creator,project key,log1,log2,log3,log4,log5,log6,log7,log8,log9,log10
';
while ($row = mysql_fetch_assoc($rs)) {
$line = '';
$count = 0;
$line .= $row['id'] . "," . $row['title'] . "," . date('d-M-y h:m a',$row['otime']) . "," . $row['priority'] . "," . $row['type_id'] . "," . $row['user_id'] . "," . $row['description'] . "," . $row['system_id'] . "," . $row['creator_id'] . "," . $row['proj_key'] . ",";
$logs = find_logs($id = $row['id']);
foreach($logs as $log_entry) {
$line .= $log_entry.",";
$count++;
}
while($count < 10) {
$line .= ",";
$count++;
}
$export[] = $line.'
';
}
mysql_close();
// print_r($export);
$file = 'tickets.csv';
file_put_contents($file, $export);
function find_logs($ticket) {
$content = array();
$query = "select ZENTRACK_LOGS.created, ZENTRACK_LOGS.user_id, ZENTRACK_LOGS.entry
from ZENTRACK_LOGS
where ZENTRACK_LOGS.ticket_id = $ticket
and ZENTRACK_LOGS.action <> 'EDIT' ";
$rs = mysql_query($query);
if (!$rs) {
echo "Could not execute query: $query";
trigger_error(mysql_error(), E_USER_ERROR);
}
while ($row = mysql_fetch_assoc($rs)) {
$date = date('d-M-y h:m a',$row['created']);
$content[] = $date . ";" . $row['user_id'] . ";" . $row['entry'];
}
return $content;
}
?>
I'm running into two problems with this script, that I'm sure are due to me being new to PHP.
1) I need to escape out the data in $row['description'] as it contains both carriage returns and , in the text that is incorrectly breaking the output into new rows when saved to CSV. I need to save the contents of this row within " " but I'm not sure how to do so.
2) The data returned in $row['user_id'], $row['creator_id'] and $row['user_id'] within the find_logs function returns a number, which I need to find that number and replace with the corresponding string in the $users array. What's the best way to do this?
When dealing with csv files use fgetcsv and fputcsv, provide it with an array and it will handle the escaping for you.
I'm having great trouble with "INSERT INTO"...
I have a variable part number so this my code...:
<?php
include ("db_conn.php");
$mem_id = "1";
$descript = "chair";
$qualifier = "sitting";
$major = "Y";
$value = "6";
//$mesh_cell_string = "tree_0,tree_1,tree_2,tree_3,tree_4";
//$mesh_values_string = "'C23','550','291','687','500'";
$part_number = "C23.550.291.687.500";
$parts = explode('.', $part_number);
$n = 0;
foreach ($parts as $something => $number)
{
$mesh_cell_string .= "tree_" . $n . ",";
$mesh_values_string .= "'" . $number . "'," ;
$n++;
}
$mesh_values_string = substr($mesh_values_string, 0, -1);
$mesh_cell_string = substr($mesh_cell_string, 0, -1);
$insert_string = "mem_id,mesh_heading_name," . $mesh_cell_string . ",qualifier_name,major,rank";
$values_string = "'$mem_id','$descript'," .$mesh_values_string. ",'$qualifier','$major','$value'";
$sql = "INSERT INTO mesh_table (" . $insert_string .") VALUES (" . $values_string .")";
$result = mysqli_query($cxn,$sql) or die ("couldn't execute the query");
?>
The strange thing is... i don't get an error ("couldn't execute the query") so i thought it went alright but when i look into my database there aren't any values written... when i un-comment the the 2 variables:
//$mesh_cell_string = "tree_0,tree_1,tree_2,tree_3,tree_4";
//$mesh_values_string = "'C23','550','291','687','500'";
And comment the foreach loop, it works...? So there goes something wrong in the foreach loop, but when i echo the $sql on both methods i get the same:
INSERT INTO mesh_table (mem_id,mesh_heading_name,tree_0,tree_1,tree_2,tree_3,tree_4,qualifier_name,major,rank) VALUES ('1','Chair','C23','550','291','687','500','sitting','Y','6')
I really don't know what i am doing wrong...?
Best regards,
Thijs
change $values_string = "'$mem_id','$descript'," .$mesh_values_string. ",'$qualifier','$major','$value'";
To
$values_string = "'".$mem_id."','".$descript."'," .$mesh_values_string. ",'".$qualifier."','".$major."','".$value."'";