CSV file formatting - php

I have problem: I need to be able to import CSV file to MS Access database with PHP. I have a script that can do it, but my CSV file format is not right.
The original format is:
But the script works with this format:
Both files are .csv. How can I convert the first file format to the needed one?
My import script below is not importing anything, and is not giving any errors:
if (isset($_POST['submit'])) {
$i=0;
require "connection.php";
if (is_uploaded_file($_FILES['csv']['tmp_name'])) {
echo "<h3>" . "File ". $_FILES['csv']['name'] ." uploaded successfully." . "</h3>";
}
//Import uploaded file to Database
$handle = fopen($_FILES['csv']['tmp_name'], "r");
$import=$db->prepare("INSERT INTO adherence(
dateandtime,
lastname,
firstname,
paidtime,
approvedtime,
notadhering)VALUES(
?,?,?,?,?,?)");
while (($data = fgetcsv($handle, 1000,"\t","'")) !== FALSE) {
if($i>0) {
$data = str_replace('"', '', $data);
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $data[1], PDO::PARAM_STR);
$import->bindParam(3, $data[2], PDO::PARAM_STR);
$import->bindParam(4, $data[3], PDO::PARAM_STR);
$import->bindParam(5, $data[4], PDO::PARAM_STR);
$import->bindParam(6, $data[5], PDO::PARAM_STR);
str_replace('"',' ',$data);
$import->execute();
}
$i++;
}
EDIT Current code:
<?php
if (isset($_POST['submit'])) {
$i=0;
require "connection.php";
if (is_uploaded_file($_FILES['csv']['tmp_name'])) {
echo "<h3>" . "File ". $_FILES['csv']['name'] ." uploaded successfully." . "</h3>";
}
//Import uploaded file to Database
$handle = fopen($_FILES['csv']['tmp_name'], "r");
$import=$db->prepare("INSERT INTO adherence(
dateandtime,
lastname,
firstname,
paidtime,
approvedtime,
notadhering)VALUES(
?,?,?,?,?,?)");
$badlines=[];
while (($data = fgetcsv($handle, 1000,"\t",'"')) !== FALSE) {
if($i>0) {
//skip but keep track of bad data
if(count($data) !== 5){
$badlines[] = $i;
continue;
}
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$names = explode(',', $data[1]);
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $names[0], PDO::PARAM_STR);
$import->bindParam(3, $names[1], PDO::PARAM_STR);
$import->bindParam(4, $data[2], PDO::PARAM_STR);
$import->bindParam(5, $data[3], PDO::PARAM_STR);
$import->bindParam(6, $data[4], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
$removal=$db->prepare("delete FROM adherence WHERE approvedtime = '0' OR notadhering IS NULL");
$removal->execute();
fclose($handle);
}

"name, lastname" is a single column, as it is surrounded by double quotes.
Your code originally worked as you misconfigured fgetscsv by passing a single quote as the enclosure parameter, causing it to ignore these double quotes and treat the single column as two as it contains a comma.
As your new file is tab delimetered, it does not split on the comma.
To fix, pass the correct enclosure parameter to fgetcsv, and split the column yourself:
$badlines=[];
while (($data = fgetcsv($handle, 1000,"\t",'"')) !== FALSE) {
if($i>0) {
//skip but keep track of bad data
if(count($data) !== 5){
$badlines[] = $i;
continue;
}
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$names = explode(',', $data[1]);
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $names[0], PDO::PARAM_STR);
$import->bindParam(3, $names[1], PDO::PARAM_STR);
$import->bindParam(4, $data[2], PDO::PARAM_STR);
$import->bindParam(5, $data[3], PDO::PARAM_STR);
$import->bindParam(6, $data[4], PDO::PARAM_STR);
$import->execute();
}
$i++;
}

Related

How to validate email in csv file

I have a csv file that I parse and display it by storing it in an array. I was wondering how can I validate the mail or make sure it's the right format before inserting it into mysql? And, if the mail is not validate it should display an error and not insert that row to mysql.
I know I can use that filter_var($email, FILTER_VALIDATE_EMAIL) !== false; but I am not sure how to use it in my code as it is inside an array (the value of email). Any help is appreciated.
My code:
<?php
require "database.php";
$lines = file('users.csv', FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);
$csv = array_map('str_getcsv', $lines);
$col_names = array_shift($csv);
$users = [];
try {
$stmt = $dbh->prepare("INSERT INTO users(name, surname, email)
VALUES(:name, :surname, :email)");
foreach($csv as $row) {
$users[] = [
$col_names[0] => ucfirst(strtolower($row[0])),
$col_names[1] => ucfirst(strtolower($row[1])),
$col_names[2] => strtolower(strtolower($row[2])), //that's the mail
];
$stmt->bindValue(':name', $col_names[0], PDO::PARAM_STR);
$stmt->bindValue(':surname', $col_names[1], PDO::PARAM_STR);
$stmt->bindValue(':email', $col_names[2], PDO::PARAM_STR);
$stmt->execute();
}
}catch(PDOException $e) {
echo "Error: " . $e->getMessage();
}
$dbh = null;
?>
You can simply check it within your loop something like below
foreach($csv as $row) {
if (filter_var(strtolower(strtolower($row[2])), FILTER_VALIDATE_EMAIL) === false) {
echo("".strtolower($row[2]))." is not a valid email address"); // You can break the loop and return from here
}
$users[] = [
$col_names[0] => ucfirst(strtolower($row[0])),
$col_names[1] => ucfirst(strtolower($row[1])),
$col_names[2] => strtolower(strtolower($row[2])), //that's the mail
];
$stmt->bindValue(':name', $col_names[0], PDO::PARAM_STR);
$stmt->bindValue(':surname', $col_names[1], PDO::PARAM_STR);
$stmt->bindValue(':email', $col_names[2], PDO::PARAM_STR);
$stmt->execute();
}
Apart from this I would suggest you to create a batch insert for this.. Never execute the insertion query in a loop

Inserting data from mysql from CSV using PHP PDO

I have list of data in CSV and need to insert this data into a MySQL database. These data should be safely inserted i.e sanitation. So, I have used PDO object to rectify SQL injection. But, it fails to get data from CSV file and inserts null values.
Here is the example,
<?php
$servername = "localhost";
$username = "root";
$password = "";
try {
$conn = new PDO("mysql:host=$servername;dbname=contact_list",$username,$password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "connection successfully";
}
catch(PDOException $e)
{
echo "connection Failed:" . $e -> getMessage();
}
// Create CSV to Array function
function csvToArray($filename = '', $delimiter = ',')
{
if (!file_exists($filename) || !is_readable($filename)) {
return false;
}
$header = NULL;
$result = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
if (!$header)
$header = $row;
else
$result[] = array_combine($header, $row);
}
fclose($handle);
}
return $result;
}
// Insert data into database
$all_data = csvToArray('contact.csv');
foreach ($all_data as $data) {
$data = array_map(function($row){
return filter_var($row, FILTER_SANITIZE_STRING, FILTER_SANITIZE_FULL_SPECIAL_CHARS);
}, $data);
$sql = $conn->prepare("INSERT INTO contact
(title, first_name,last_name,company_name,date_of_birth,notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
$sql->bindParam(':t', $data[1], PDO::PARAM_STR);
$sql->bindParam(':fname', $data[2], PDO::PARAM_STR);
$sql->bindParam(':lname', $data[3], PDO::PARAM_STR);
$sql->bindParam(':cname', $data[0], PDO::PARAM_STR);
$sql->bindParam(':dob', $data[4], PDO::PARAM_STR);
$sql->bindParam(':note', $data[15], PDO::PARAM_STR);
print_r($data);
$sql->execute();
}
?>
Can anyone help me to solve this?
If you take a look at the documentation for array_combine() you'll see that its purpose is to build an associative array. You use this function in csvToArray() but later in your code you are trying to get data using numeric keys. I wouldn't expect you'd ever have anything inserted.
On a side note, you are completely defeating the purpose of prepared statements by repeatedly preparing the same statement over and over again. Prepare once and execute many times. Individually binding parameters is rarely needed, in almost all cases you can provide the data to PDOStatement::execute() as an array. It's also bad form to store HTML entities in a database; if you need to output to HTML, you perform escaping at that point.
Something like this should work (adjust array key names as necessary.)
$all_data = csvToArray('contact.csv');
$sql = $conn->prepare("INSERT INTO contact
(title, first_name, last_name, company_name, date_of_birth, notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
foreach ($all_data as $data) {
$params = [
":t" => $data["t"],
":fname" => $data["fname"],
":lname" => $data["lname"],
":dob" => $data["dob"],
":note" => $data["note"],
];
$sql->execute($params);
}

Do While loop works well with echo but loops only once with a function inside loop

This code is supposed to insert 100 rows into the DB.
Yet when I run it, it loops only once, inserts one row and stops.
I replaced the function call with :
echo $keywords[4].'<br>';
It works perfectly. with no probleb
What is missing so that it will insert all rows into DB?
what should I change q add so that the code will insert all rows in the file
Here is the loop code:
do{
//Insert row content into array.
$keywords = preg_split("#\<(.*?)\>#", $row);
//Insert relevant data into DB
add_data($keywords);
}
else{
// If row is irrelevant - continue to next row
continue;
}
}while (strpos($row, 'Closed P/L') != true);
Here is the function
function add_data($keywords)
{
global $db;
$ticket =$keywords[2];
$o_time = $keywords[4];
$type = $keywords[6];
$size = $keywords[8];
$item = substr($keywords[10], 0, -1);
$o_price = $keywords[12];
$s_l = $keywords[14];
$t_p = $keywords[16];
$c_time = $keywords[18];
$c_price = $keywords[20];
$profit = $keywords[28];
try
{
$sql = "
INSERT INTO `data`
(ticket, o_time, type, size, item, o_price, s_l, t_p, c_time, c_price, profit)
VALUES
(:ticket, :o_time, :type, :size, :item, :o_price, :s_l, :t_p, :c_time, :c_price, :profit)";
$stmt = $db->prepare($sql);
$stmt->bindParam('ticket', $ticket, PDO::PARAM_STR);
$stmt->bindParam('o_time', $o_time, PDO::PARAM_STR);
$stmt->bindParam('type', $type, PDO::PARAM_STR);
$stmt->bindParam('size', $size, PDO::PARAM_STR);
$stmt->bindParam('item', $item, PDO::PARAM_STR);
$stmt->bindParam('o_price', $o_price, PDO::PARAM_STR);
$stmt->bindParam('s_l', $s_l, PDO::PARAM_STR);
$stmt->bindParam('t_p', $t_p, PDO::PARAM_STR);
$stmt->bindParam('c_time', $c_time, PDO::PARAM_STR);
$stmt->bindParam('c_price', $c_price, PDO::PARAM_STR);
$stmt->bindParam('profit', $profit, PDO::PARAM_STR);
$stmt->execute();
//return true;
}
catch(Exception $e)
{
return false;
echo 'something is wrong. Here is the system\'s message:<br>'.$e;
}
}

PHP imported data integrity

I will try to explain situation as well as possible:
I have script, that imports CSV file data to MS Access database.
I have 2 access Tables:
A) Users and their information(ID, name, last name etc.)
B) Table which contains data from CSV file
Problem is, data imported from file, (2nd table) contains Users name and lastname. I want to get idea, how to, while reading csv file line by line, check what name line contains, and assign userID from table 1 instead of name and lastname on table 2. It should be done while importing, because, on each import there are roughly 3k lines being imported. Any ideas appreciated. Images given bellow.
Import script:
<?php
function qualityfunction() {
error_reporting(0);
require_once '/Classes/PHPExcel.php'; // (this should include the autoloader)
require_once '/CLasses/PHPExcel/IOFactory.php';
$excel_readers = array(
'Excel5' ,
'Excel2003XML' ,
'Excel2007'
);
$files = glob('data files/quality/QA*.xls');
$sheetname= 'AvgScoreAgentComments';
if (count($files) >0 ) {
foreach($files as $flnam) {
$reader = PHPExcel_IOFactory::createReader('Excel5');
$reader->setReadDataOnly(true);
$reader->setLoadSheetsOnly($sheetname);
$path = $flnam;
$excel = $reader->load($path);
$writer = PHPExcel_IOFactory::createWriter($excel, 'CSV');
$writer->save('data files/quality/temp.csv');
/*
$filename = basename($path);
if (strpos($filename,'tes') !== false) {
echo 'true';
}*/
require "connection.php";
$handle = fopen("data files/quality/temp.csv", "r");
try {
$import= $db->prepare("INSERT INTO quality(
qayear,
qamonth,
lastname,
firstname,
score) VALUES(
?,?,?,?,?)");
$i = 0;
while (($data = fgetcsv($handle, 1000, ",", "'")) !== FALSE) {
if($i > 3) {
$data = str_replace('",', '', $data);
$data = str_replace('"', '', $data);
$import->bindParam(1, $data[1], PDO::PARAM_STR);
$import->bindParam(2, $data[2], PDO::PARAM_STR);
$import->bindParam(3, $data[3], PDO::PARAM_STR);
$import->bindParam(4, $data[4], PDO::PARAM_STR);
$import->bindParam(5, $data[7], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
fclose($handle);
$removal=$db->prepare("DELETE FROM quality WHERE score IS NULL;");
$removal->execute();
}
catch(PDOException $e) {
echo $e->getMessage()."\n";
}};
Data table 1 (Users info):
Data table 2 (In which data from CSV file is imported)
Found a solution. Thanks for help.
$lastname = "lastname";
$firstname = "firstname";
$showdata = $db->prepare("SELECT userID FROM users WHERE lastname= :lastname AND firstname= :firstname");
$showdata->bindParam(':lastname', $lastname);
$showdata->bindParam(':firstname', $firstname);
$showdata->execute();
$rowas= $showdata->fetch(PDO::FETCH_ASSOC);
echo $rowas['userID'];

Php Bulk insert

I have the code with php using bulk insert.,I run the code and there is no error., The Problem there is no OUTPUT with this code and blank page/screen appear .. All I want to do is to have the Output with the page and with the database using this code ..
<?php
$dbh = odbc_connect(
"DRIVER={SQL Server Native Client 10.0};Server=.;Database=ECPNWEB",
"sa", "ECPAY");
if (($handle = fopen("c:\\tblmcwd.txt", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 4096, "|")) !== FALSE) {
if (count($data) == 10) {
$sql = "INSERT INTO [dbo].[tblMCWD] (
[ID],
[ConsumerCode],
[ConsumerName],
[AccountStatus],
[AccountNumber],
[DueDate],
[CurrentBill],
[PreviousBill],
[TotalDiscount],
[TotalGrossAmountDue]
) VALUES (
?, ?, ?, ?, ?, ?, ?, ?, ?, ?
)";
$stmt = $dbh->prepare($sql);
$stmt->bindValue(1, $data[0]);
$stmt->bindValue(2, $data[1]);
$stmt->bindValue(3, $data[2]);
$stmt->bindValue(4, $data[3]);
$stmt->bindValue(5, $data[4]);
$stmt->bindValue(6, $data[5]);
$stmt->bindValue(7, $data[6]);
$stmt->bindValue(8, $data[7]);
$stmt->bindValue(9, $data[8]);
$stmt->bindValue(10, $data[9]);
$stmt->execute();
}
}
fclose($handle);
}
?>
Try this:
....
$sql = "INSERT INTO [dbo].[tblMCWD] (
[ID],
[ConsumerCode],
[ConsumerName],
[AccountStatus],
[AccountNumber],
[DueDate],
[CurrentBill],
[PreviousBill],
[TotalDiscount],
[TotalGrossAmountDue]
) VALUES (
:ID, :ConsumerCode, :ConsumerName, :AccountStatus, :AccountNumber, :DueDate, :CurrentBill, :PreviousBill, TotalDiscount, TotalGrossAmountDue
)";
$stmt = $dbh->prepare($sql);
$stmt->bindParam(':ID', $data[0]);
$stmt->bindParam(':ConsumerCode', $data[1],PDO::PARAM_STR);
$stmt->bindParam(':ConsumerName', $data[2],PDO::PARAM_STR);
$stmt->bindParam(':AccountStatus', $data[3],PDO::PARAM_STR);
$stmt->bindParam(':AccountNumber', $data[4],PDO::PARAM_STR);
$stmt->bindParam(':DuaDate' , $data[5],PDO::PARAM_STR);
$stmt->bindParam(':CurrentBill', $data[6],PDO::PARAM_STR);
$stmt->bindParam(':PreviousBill', $data[7],PDO::PARAM_STR);
$stmt->bindParam(':TotalDiscount', $data[8],PDO::PARAM_STR);
$stmt->bindParam(':TotalGrossAmountDue', $data[9],PDO::PARAM_STR);
$stmt->execute();
if(!$stmt){ // Check if the query executed succesfull, if not, print the data...
$printedString = "ID: %1$s ConsumerCode: %2$s ConsumerName: %3$s"; // and so on....
$printedString = sprintf($printedString , $data[0],$data[1],$data[2]); // and so on...
} else{
echo "Everything executed succesfully!<br />";
}
More info can you find here: Link
And here: Link 2
And here: Link 3
Use PDO::PARAM_STR for strings and PDO::PARAM_INT for an integer, and give it an try

Categories