I have a php code to upload a .csv file this one is working fine. But what i want is if i upload 10 records in a .csv file means it display how many records i upload and in that how many is correctly and how many's are wrong.
Here is my code :
if (isset($_POST["submitbutton"])) {
if (!empty($_FILES['file']['name'])) {
if (pathinfo($_FILES['file']['name'], PATHINFO_EXTENSION) == 'csv') {
$handle = fopen($_FILES['file']['tmp_name'], "r");
//Grab the header in csv
$headers = fgetcsv($handle, 1000, ",");
$not_valid = [];
$i = 1;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$aValid = false;
$field1 = mysql_real_escape_string($data[0]);
$field2 = mysql_real_escape_string($data[1]);
$field3 = mysql_real_escape_string($data[2]);
$field4 = mysql_real_escape_string($data[3]);
$field5 = mysql_real_escape_string($data[4]);
$field6 = mysql_real_escape_string($data[5]);
$field7 = mysql_real_escape_string($data[6]);
$field8 = mysql_real_escape_string($data[7]);
$field9 = mysql_real_escape_string($data[8]);
$sql2 = "select * from m_tl_mastercourse where LMS_Course_ID='$field9'";
$check1 = mysql_query($sql2);
$sql = "select * from mdl_course where id='".$field9."'";
$check = mysql_query($sql);
$row=mysql_fetch_array($check);
if (mysql_num_rows($check) > 0 && mysql_num_rows($check1) < 1) {
$aValid = true;
} else {
$not_valid[] = $i;
}
if ($aValid) {
echo count($i);
$import = "INSERT into m_tl_mastercourse values('','$field1','$field2','$field3','$field4','$field5','$field6','$field7','$field8','$field9')";
mysql_query($import);
$last_id = mysql_insert_id();
if($last_id){
$sql = "select m_tl_mastercourse.Id, Course_Type from m_tl_mastercourse inner join mdl_course where m_tl_mastercourse.LMS_Course_ID = mdl_course.id ";
$ret = get_records_sql($sql);
foreach ($ret as $index => $obj) {
$master_id = $obj->Id;
$course_type = $obj->Course_Type;
}
echo $master_id;
echo $course_type;
$query = mysql_query("update m_tl_course INNER JOIN m_tl_mastercourse ON m_tl_course.course_id = m_tl_mastercourse.LMS_Course_ID SET m_tl_course.course_type='$course_type',m_tl_course.master_id='$master_id' WHERE m_tl_course.master_id='' ");
}
}
$i++;
}
fclose($handle);
if (!empty($not_valid)) {
$v_ids = implode(', ', $not_valid);
echo count($v_ids);
echo "Please check the following csv Row Number's - " . $v_ids;
} else {
echo "Master Courses Uploaded Successfully";
}
} else {
echo "<script>alert('csv files only allowed to upload')</script>";
}
} else {
echo "<script>alert('Please select a file')</script>";
}
}
?>
How to get like this
The variables are already in your code: $i is incremented on each loop, and $not_valid is an array containing the row numbers for any rows that are evaluated as not valid.
You can use count($not_valid) to get the number.
// $i starts at 1 and is incremented on the completion of each loop so will end 1 more than the total rows
$total_rows = $i - 1;
$valid_rows = $i - 1 - count($not_valid);
$invalid_rows = count($not_valid);
I need to perform a batch insert in MySQL/MariaDB but since data is dynamic I need to build the proper SQL query. In a few steps:
I should find whether the current row exists or not in table - this is the first SELECT inside the loop
Right now I have 1454 but have to insert around 150k later, is better a batch query than 150k INSERT per item on the loop
If record already exists I should update it if doesn't then I should insert ,I just not care about UPDATE yet and the code you're seeing is only for INSERT
So here is what I am doing:
// Get values from Csv file as an array of values
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
// VALUES should be added only if row doesn't exists
if ($rows_returned === 0) {
// VALUES should be append until they reach 1000
while ($i % 1000 !== 0) {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW())";
++$i;;
}
// QUERY should be output to console to see if it's right or something is wrong
echo "DEBUG: ", $sqlInsert, "\n";
// QUERY should be executed if there are 1000 VALUES ready to add as a batch
/*$rs = $conn->query($sqlInsert);
if ($rs === false) {
echo 'Wrong SQL: '.$sqlInsert.' Error: '.$conn->error, E_USER_ERROR;*/
}
} else {
// UPDATE
echo "UPDATE";
}
}
}
But this line of code: echo "DEBUG: ", $sql, "\n"; is not outputting nothing to console. I must be doing something wrong but I can't find what. Can any help me to build the proper batch query and to execute it each 1000 values append?
Proper output should be:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008ReolAAC'
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='005800000039SIWAA2'
....
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES(...), VALUES(...), VALUES(...)
Obtained result:
DEBUG count(data): 1454
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RGg6AAG'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
DEBUG: SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='00580000008RQ4CAAW'
DEBUG: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt)
.... // until reach 1454 results
The table is empty so it should never goes through ELSE condition (UPDATE one).
EDIT
With help from the answer this is how the code looks now:
$data = convertCsvToArray($fileName);
echo "DEBUG count(data): ", count($data), "\n";
$i = 1;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$rows_returned = $rs->num_rows;
$veeva_rep_id = "'".$conn->real_escape_string($row['Id'])."'";
$first = "'".$conn->real_escape_string(ucfirst(strtolower($row['FirstName'])))."'";
$last = "'".$conn->real_escape_string(ucfirst(strtolower($row['LastName'])))."'";
$email = "'".$conn->real_escape_string($row['Email'])."'";
$username = "'".$conn->real_escape_string($row['Username'])."'";
$display_name = "'".$conn->real_escape_string(
ucfirst(strtolower($row['FirstName'])).' '.ucfirst(strtolower($row['LastName']))
)."'";
if ($rows_returned === 0) {
if ($i % 1000 === 0) {
file_put_contents("output.log", $sqlInsert."\n", FILE_APPEND);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES";
} else {
$sqlInsert .= "($veeva_rep_id,$first,$last,$email,$username,NOW(),NOW(),$display_name,'VEEVA','https://pdone.s3.amazonaws.com/avatar/default_avatar.png',NOW(),NOW()), ";
}
$i++;
} else {
echo "UPDATE";
}
}
}
But still buggy because:
I have got a first empty INSERT query: INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES
I have got a second INSERT query with 1000 VALUES() append, but what happened with the rest? The remaining 454?
Can any give me another tip? Help?
Since it looks like you are trying to load data from a CSV file, you might want to consider using LOAD DATA INFILE functionality which is designed specifically for this purpose.
Here is link to documentation: https://dev.mysql.com/doc/refman/5.6/en/load-data.html
consider using INSERT IGNORE INTO table to check if the record already exists. How to 'insert if not exists' in MySQL?
if you haven't already done so, make veeva_rep_id a PRIMARY key so the INSERT IGNORE will work
also check out using PDO for transactions, prepared statements and dynamically generating queries using PDO
PDO Prepared Inserts multiple rows in single query
<?php
$sql = 'INSERT IGNORE INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) VALUES ';
$insertQuery = array();
$insertData = array();
/*
assuming the array from the csv is like this
$data = array(
0 => array('name' => 'Robert', 'value' => 'some value'),
1 => array('name' => 'Louise', 'value' => 'another value')
);
*/
foreach ($data as $row) {
$insertQuery[] = '(:veeva_rep_id' . $n . ', :first' . $n . ', :last' . $n . ', :email' . $n . ', :username' . $n . ', :lastLoginAt' . $n . ', :lastSyncAt' . $n . ', :display_name' . $n . ', :rep_type' . $n . ', :avatar_url' . $n . ', :createdAt' . $n . ', :updatedAt' . $n . ')';
$insertData['veeva_rep_id' . $n] = $row['name'];
$insertData['first' . $n] = $row['value'];
$insertData['last' . $n] = $row['name'];
$insertData['email' . $n] = $row['value'];
$insertData['username' . $n] = $row['name'];
$insertData['lastLoginAt' . $n] = $row['value'];
$insertData['lastSyncAt' . $n] = $row['value'];
$insertData['display_name' . $n] = $row['name'];
$insertData['rep_type' . $n] = $row['value'];
$insertData['avatar_url' . $n] = $row['value'];
$insertData['createdAt' . $n] = $row['name'];
$insertData['updatedAt' . $n] = $row['value'];
$n++;
}
$db->beginTransaction();
if (!empty($insertQuery) and count($insertQuery)>1000) {
$sql .= implode(', ', $insertQuery);
$stmt = $db->prepare($sql);
$stmt->execute($insertData);
}
$db->commit();
print $sql . PHP_EOL;
let me know if it helps.
You should have something like:
// Try fetching data from table 1
// If there is no record available, then fetch some data from table 2
// and insert that data inito table 1
You just wrote
$sql = "INSERT INTO reps(veeva_rep_id,first,last,email,username,lastLoginAt,lastSyncAt,display_name,rep_type,avatar_url,createdAt,updatedAt) ";
// Processing on each row of data
foreach ($data as $row) {
But from an insert no data is selected and second...you didn't run a select, where comes $data from?
update Use if ($i % 1000 === 0) { instead of while ($i % 1000 !== 0) {
$i = 0;
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) ";
// Processing on each row of data
foreach ($data as $row) {
$sql = "SELECT id,lastSyncAt FROM reps WHERE veeva_rep_id='{$row['Id']}'";
echo "DEBUG: ", $sql, "\n";
$rs = $conn->query($sql);
if ($rs === false) {
echo 'Wrong SQL: '.$sql.' Error: '.$conn->error, E_USER_ERROR;
} else {
$veeva_rep_id = ...;
$first = ...;
$last = ...;
$email = ...;
// ...
// VALUES should be added only if row doesn't exists
if($rs->num_rows == 0) {
// Insert some data
$i++;
if ($i % 1000 === 0) {
echo "DEBUG: ", $sqlInsert, "\n";
// execSql($sqlInsert);
$sqlInsert = "INSERT INTO reps(veeva_rep_id,first,last,email,...) "; // reset
} else {
$sqlInsert .= "VALUES($veeva_rep_id,$first,$last,$email,...) ";
}
} else {
echo "UPDATE";
}
}
}
I want to insert an array in the database. The array can be changed all the time. I want different rows in the database.
My code:
$var = file_get_contents("test2.txt");
$test = preg_replace('/\\\\/', '', $var);
$poep = explode(" ", $test);
Yeah, there is no database connection, because I want to know how to 'split' the array to insert it in the database.
I have tried this:
foreach($poep as $row) {
$row = $mysqli->real_escape_string($row);
if($mysqli->query("insert into data('array') VALUES ($row)") == false){
echo 'Doesnt works!';
}
It returns 'Doesnt works', so I think there is a problem with query?
#NadirDev Hi. Assuming that you are using Core PHP programming. After exploding the string by the space, run foreach loop and then insert individual rows. Look at this rough code to get idea:
foreach($poep as $row) {
// $row now contains one word. Add that in database.
$row = mysql_real_escape_string($row);
$query = mysql_query("insert into tableName('fieldName') VALUES ($row)");
}
here's some code I wrote. It processes a CSV file and stores separate rows into a db table (difference is just that you have a TXT file). It does the mysql insertion in batches of 250 rows. Hope it can help you!
// read all input rows into an array
echo "Processing input..<br /><br />";
$row = 0;
$input = array();
if (($handle = fopen($file['tmp_name'], "r")) !== FALSE) {
while (($data = fgetcsv($handle, 0, ",")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
$input[$row][] = addslashes($data[$c]);
}
$row++;
}
fclose($handle);
}
$count = 0;
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
foreach ($input as $inp) {
$q .= "('" . addslashes($inp[0]) . "', '" . addslashes($inp[1]) . "', '" . addslashes($inp[2]) . "', '" . date('Y-m-d H:i:s') . "'), ";
$count++;
if ($count >= 250) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
$q = "INSERT INTO `inputs` (`keyword`, `percent`, `link`, `added_on`) VALUES ";
$count = 0;
}
}
if ($count > 0) {
$q = substr($q, 0, -2);
$q = mysqli_query($con, $q);
}
echo "Successfully added " . count($input) . " rows to the input list.";
This is my code on mysql query that I want to change to sqlsrv query to import into sql server database
$message = null;
$allowed_extensions = array('csv');
$upload_path = '../csv';
if (!empty($_FILES['file'])) {
if ($_FILES['file']['error'] == 0) {
// check extension
$file = explode(".", $_FILES['file']['name']);
$extension = array_pop($file);
if (in_array($extension, $allowed_extensions)) {
if (move_uploaded_file($_FILES['file']['tmp_name'], $upload_path.'/'.$_FILES['file']['name'])) {
if (($handle = fopen($upload_path.'/'.$_FILES['file']['name'], "r")) !== false) {
$keys = array();
$out = array();
$insert = array();
$line = 1;
while (($row = fgetcsv($handle, 0, ',', '"')) !== FALSE) {
foreach($row as $key => $value) {
if ($line === 1) {
$keys[$key] = $value;
} else {
$out[$line][$key] = $value;
}
}
$line++;
}
fclose($handle);
if (!empty($keys) && !empty($out)) {
$db = new PDO('mysql:host=localhost;dbname=project', 'root', '');
$db->exec("SET CHARACTER SET utf8");
foreach($out as $key => $value) {
$sql = "INSERT INTO `department` (`";
$sql .= implode("`, `", $keys);
$sql .= "`) VALUES (";
$sql .= implode(", ", array_fill(0, count($keys), "?"));
$sql .= ")";
$statement = $db->prepare($sql);
$statement->execute($value);
}
$message = '<span style="color:green">File has been uploaded successfully</span>';
}
}
}
} else {
$message = '<span style="color:red">Only .csv file format is allowed</span>';
}
} else {
$message = '<span style="color:red">There was a problem with your file</span>';
}
Hope it will help you to do it manually.
Sometime MS SQL table or column names are enclosed in square brackets in queries (e.g. if contains spaces or for some other reasons). MySQL does not allow square brackets around table of column names, they all must be replaced by symbol or cut off: [object] ->object`.
So you need to remove all "'" from the table name department and its columns too in your above case.
For more.
http://www.convert-in.com/mssql-to-mysql-queries.htm
Added:
Your SQL Query will be as below. Please remove ? mark too.
$sql = INSERT INTO department (;
$sql .= implode(",", $keys);
$sql .= ) VALUES (";
$sql .= implode(", ", array_fill(0, count($keys), "?"));
$sql .= ")";
$statement = $db->prepare($sql);
$statement->execute($value);
I am not well aware about PHP. I can only help to construct the SQL Query. Its should be
"INSERT INTO department (COLUMN1,COLUMN2,..) values(VALUE1,VALUE1,..)".
Added:
As I already told you I am not a PHP guy. But Tried for you. Please try your php code as :
$sql = "INSERT INTO department (";
$sql .= "implode(',', $keys)";
$sql .= ") VALUES (";
$sql .= "implode(',', array_fill(0, count($keys), ''))";
$sql .= ")";
$statement = $db->prepare($sql); $statement->execute($value);
But array_fill(0, count($keys), '') is empty data right?
I'm attempting the modify this Modx Snippet so that it will accept multiple values being returned from the db instead of the default one.
tvTags, by default, was only meant to be set to one variable. I modified it a bit so that it's exploded into a list of variables. I'd like to query the database for each of these variables and return the tags associated with each. However, I'm having difficulty as I'm fairly new to SQL and PHP.
I plugged in $region and it works, but I'm not really sure how to add in more WHERE clauses for the $countries variable.
Thanks for your help!
if (!function_exists('getTags')) {
function getTags($cIDs, $tvTags, $days) {
global $modx, $parent;
$docTags = array ();
$baspath= $modx->config["base_path"] . "manager/includes";
include_once $baspath . "/tmplvars.format.inc.php";
include_once $baspath . "/tmplvars.commands.inc.php";
if ($days > 0) {
$pub_date = mktime() - $days*24*60*60;
} else {
$pub_date = 0;
}
list($region, $countries) = explode(",", $tvTags);
$tb1 = $modx->getFullTableName("site_tmplvar_contentvalues");
$tb2 = $modx->getFullTableName("site_tmplvars");
$tb_content = $modx->getFullTableName("site_content");
$query = "SELECT stv.name,stc.tmplvarid,stc.contentid,stv.type,stv.display,stv.display_params,stc.value";
$query .= " FROM ".$tb1." stc LEFT JOIN ".$tb2." stv ON stv.id=stc.tmplvarid ";
$query .= " LEFT JOIN $tb_content tb_content ON stc.contentid=tb_content.id ";
$query .= " WHERE stv.name='".$region."' AND stc.contentid IN (".implode($cIDs,",").") ";
$query .= " AND tb_content.pub_date >= '$pub_date' ";
$query .= " AND tb_content.published = 1 ";
$query .= " ORDER BY stc.contentid ASC;";
$rs = $modx->db->query($query);
$tot = $modx->db->getRecordCount($rs);
$resourceArray = array();
for($i=0;$i<$tot;$i++) {
$row = #$modx->fetchRow($rs);
$docTags[$row['contentid']]['tags'] = getTVDisplayFormat($row['name'], $row['value'], $row['display'], $row['display_params'], $row['type'],$row['contentid']);
}
if ($tot != count($cIDs)) {
$query = "SELECT name,type,display,display_params,default_text";
$query .= " FROM $tb2";
$query .= " WHERE name='".$region."' LIMIT 1";
$rs = $modx->db->query($query);
$row = #$modx->fetchRow($rs);
$defaultOutput = getTVDisplayFormat($row['name'], $row['default_text'], $row['display'], $row['display_params'], $row['type'],$row['contentid']);
foreach ($cIDs as $id) {
if (!isset($docTags[$id]['tags'])) {
$docTags[$id]['tags'] = $defaultOutput;
}
}
}
return $docTags;
}
}
You don't add in more WHERE clauses, you use ANDs and ORs in the already existing where clause. I would say after the line $query .= " WHERE stv.name = '".$region... you put in
foreach ($countries as $country)
{
$query .= "OR stv.name = '{$country}', ";
}
but I don't know how you want the query to work.