PHP imported data integrity - php

I will try to explain situation as well as possible:
I have script, that imports CSV file data to MS Access database.
I have 2 access Tables:
A) Users and their information(ID, name, last name etc.)
B) Table which contains data from CSV file
Problem is, data imported from file, (2nd table) contains Users name and lastname. I want to get idea, how to, while reading csv file line by line, check what name line contains, and assign userID from table 1 instead of name and lastname on table 2. It should be done while importing, because, on each import there are roughly 3k lines being imported. Any ideas appreciated. Images given bellow.
Import script:
<?php
function qualityfunction() {
error_reporting(0);
require_once '/Classes/PHPExcel.php'; // (this should include the autoloader)
require_once '/CLasses/PHPExcel/IOFactory.php';
$excel_readers = array(
'Excel5' ,
'Excel2003XML' ,
'Excel2007'
);
$files = glob('data files/quality/QA*.xls');
$sheetname= 'AvgScoreAgentComments';
if (count($files) >0 ) {
foreach($files as $flnam) {
$reader = PHPExcel_IOFactory::createReader('Excel5');
$reader->setReadDataOnly(true);
$reader->setLoadSheetsOnly($sheetname);
$path = $flnam;
$excel = $reader->load($path);
$writer = PHPExcel_IOFactory::createWriter($excel, 'CSV');
$writer->save('data files/quality/temp.csv');
/*
$filename = basename($path);
if (strpos($filename,'tes') !== false) {
echo 'true';
}*/
require "connection.php";
$handle = fopen("data files/quality/temp.csv", "r");
try {
$import= $db->prepare("INSERT INTO quality(
qayear,
qamonth,
lastname,
firstname,
score) VALUES(
?,?,?,?,?)");
$i = 0;
while (($data = fgetcsv($handle, 1000, ",", "'")) !== FALSE) {
if($i > 3) {
$data = str_replace('",', '', $data);
$data = str_replace('"', '', $data);
$import->bindParam(1, $data[1], PDO::PARAM_STR);
$import->bindParam(2, $data[2], PDO::PARAM_STR);
$import->bindParam(3, $data[3], PDO::PARAM_STR);
$import->bindParam(4, $data[4], PDO::PARAM_STR);
$import->bindParam(5, $data[7], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
fclose($handle);
$removal=$db->prepare("DELETE FROM quality WHERE score IS NULL;");
$removal->execute();
}
catch(PDOException $e) {
echo $e->getMessage()."\n";
}};
Data table 1 (Users info):
Data table 2 (In which data from CSV file is imported)

Found a solution. Thanks for help.
$lastname = "lastname";
$firstname = "firstname";
$showdata = $db->prepare("SELECT userID FROM users WHERE lastname= :lastname AND firstname= :firstname");
$showdata->bindParam(':lastname', $lastname);
$showdata->bindParam(':firstname', $firstname);
$showdata->execute();
$rowas= $showdata->fetch(PDO::FETCH_ASSOC);
echo $rowas['userID'];

Related

Inserting data from mysql from CSV using PHP PDO

I have list of data in CSV and need to insert this data into a MySQL database. These data should be safely inserted i.e sanitation. So, I have used PDO object to rectify SQL injection. But, it fails to get data from CSV file and inserts null values.
Here is the example,
<?php
$servername = "localhost";
$username = "root";
$password = "";
try {
$conn = new PDO("mysql:host=$servername;dbname=contact_list",$username,$password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "connection successfully";
}
catch(PDOException $e)
{
echo "connection Failed:" . $e -> getMessage();
}
// Create CSV to Array function
function csvToArray($filename = '', $delimiter = ',')
{
if (!file_exists($filename) || !is_readable($filename)) {
return false;
}
$header = NULL;
$result = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
if (!$header)
$header = $row;
else
$result[] = array_combine($header, $row);
}
fclose($handle);
}
return $result;
}
// Insert data into database
$all_data = csvToArray('contact.csv');
foreach ($all_data as $data) {
$data = array_map(function($row){
return filter_var($row, FILTER_SANITIZE_STRING, FILTER_SANITIZE_FULL_SPECIAL_CHARS);
}, $data);
$sql = $conn->prepare("INSERT INTO contact
(title, first_name,last_name,company_name,date_of_birth,notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
$sql->bindParam(':t', $data[1], PDO::PARAM_STR);
$sql->bindParam(':fname', $data[2], PDO::PARAM_STR);
$sql->bindParam(':lname', $data[3], PDO::PARAM_STR);
$sql->bindParam(':cname', $data[0], PDO::PARAM_STR);
$sql->bindParam(':dob', $data[4], PDO::PARAM_STR);
$sql->bindParam(':note', $data[15], PDO::PARAM_STR);
print_r($data);
$sql->execute();
}
?>
Can anyone help me to solve this?
If you take a look at the documentation for array_combine() you'll see that its purpose is to build an associative array. You use this function in csvToArray() but later in your code you are trying to get data using numeric keys. I wouldn't expect you'd ever have anything inserted.
On a side note, you are completely defeating the purpose of prepared statements by repeatedly preparing the same statement over and over again. Prepare once and execute many times. Individually binding parameters is rarely needed, in almost all cases you can provide the data to PDOStatement::execute() as an array. It's also bad form to store HTML entities in a database; if you need to output to HTML, you perform escaping at that point.
Something like this should work (adjust array key names as necessary.)
$all_data = csvToArray('contact.csv');
$sql = $conn->prepare("INSERT INTO contact
(title, first_name, last_name, company_name, date_of_birth, notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
foreach ($all_data as $data) {
$params = [
":t" => $data["t"],
":fname" => $data["fname"],
":lname" => $data["lname"],
":dob" => $data["dob"],
":note" => $data["note"],
];
$sql->execute($params);
}

Parse large CSV file in a short time in PHP

I have been looking for how to find a value in one line and return the value of another column in a CSV file.
This is my function and it works fine but in small files:
function find_user($filename, $id) {
$f = fopen($filename, "r");
$result = false;
while ($row = fgetcsv($f, 0, ";")) {
if ($row[6] == $id) {
$result = $row[5];
break;
}
}
fclose($f);
return $result;
}
The problem is that the actual file with which I must work has a size of 4GB. And the time it takes to search is tremendous.
Navigating through Stack Overflow, I found the following post:
file_get_contents => PHP Fatal error: Allowed memory exhausted
There they give me the following function that (from what I understood) makes it easier for me to search for huge CSV values:
function file_get_contents_chunked($file,$chunk_size,$callback)
{
try
{
$handle = fopen($file, "r");
$i = 0;
while (!feof($handle))
{
call_user_func_array($callback,array(fread($handle,$chunk_size),&$handle,$i));
$i++;
}
fclose($handle);
}
catch(Exception $e)
{
trigger_error("file_get_contents_chunked::" . $e->getMessage(),E_USER_NOTICE);
return false;
}
return true;
}
And the way of using it seems to be the following:
$success = file_get_contents_chunked("my/large/file",4096,function($chunk,&$handle,$iteration){
/*
* Do what you will with the {&chunk} here
* {$handle} is passed in case you want to seek
** to different parts of the file
* {$iteration} is the section fo the file that has been read so
* ($i * 4096) is your current offset within the file.
*/
});
if(!$success)
{
//It Failed
}
The problem is that I do not know how to adapt my initial code to work with the raised function to speed up the search in large CSVs. My knowledge in PHP is not very advanced.
No matter how you read the file, there's no way to make search faster since you always have to scan every character while searching for the correct row and column. Worst case is when the row you're looking for is the last one in a file.
You should import your CSV to a proper indexed database and modify your application to further save new records to that database instead of a CSV file.
Here's a rudimentary example using SQLite. I created a CSV file with 100 million records (~5GB) and tested with it.
Create a SQLite database and import your CSV file into it:
$f = fopen('db.csv', 'r');
$db = new SQLite3('data.db');
$db->exec('CREATE TABLE "user" ("id" INT PRIMARY KEY, "name" TEXT,
"c1" TEXT, "c2" TEXT, "c3" TEXT, "c4" TEXT, "c5" TEXT)');
$stmt = $db->prepare('INSERT INTO "user"
("id", "name", "c1", "c2", "c3", "c4", "c5") VALUES (?, ?, ?, ?, ?, ?, ?)');
$stmt->bindParam(1, $id, SQLITE3_INTEGER);
$stmt->bindParam(2, $name, SQLITE3_TEXT);
$stmt->bindParam(3, $c1, SQLITE3_TEXT);
$stmt->bindParam(4, $c2, SQLITE3_TEXT);
$stmt->bindParam(5, $c3, SQLITE3_TEXT);
$stmt->bindParam(6, $c4, SQLITE3_TEXT);
$stmt->bindParam(7, $c5, SQLITE3_TEXT);
$db->exec('BEGIN TRANSACTION');
while ($row = fgetcsv($f, 0, ';')) {
list($c1, $c2, $c3, $c4, $c5, $name, $id) = $row;
$stmt->execute();
}
$db->exec('COMMIT');
This takes a long time, over 15 minutes on my computer, resulting a 6.5GB file.
Search from a database:
$id = 99999999;
$db = new SQLite3('data.db');
$stmt = $db->prepare('SELECT "name" FROM "user" WHERE "id" = ?');
$stmt->bindValue(1, $id, SQLITE3_INTEGER);
$result = $stmt->execute();
print_r($result->fetchArray());
This executes virtually instantenously.

CSV file formatting

I have problem: I need to be able to import CSV file to MS Access database with PHP. I have a script that can do it, but my CSV file format is not right.
The original format is:
But the script works with this format:
Both files are .csv. How can I convert the first file format to the needed one?
My import script below is not importing anything, and is not giving any errors:
if (isset($_POST['submit'])) {
$i=0;
require "connection.php";
if (is_uploaded_file($_FILES['csv']['tmp_name'])) {
echo "<h3>" . "File ". $_FILES['csv']['name'] ." uploaded successfully." . "</h3>";
}
//Import uploaded file to Database
$handle = fopen($_FILES['csv']['tmp_name'], "r");
$import=$db->prepare("INSERT INTO adherence(
dateandtime,
lastname,
firstname,
paidtime,
approvedtime,
notadhering)VALUES(
?,?,?,?,?,?)");
while (($data = fgetcsv($handle, 1000,"\t","'")) !== FALSE) {
if($i>0) {
$data = str_replace('"', '', $data);
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $data[1], PDO::PARAM_STR);
$import->bindParam(3, $data[2], PDO::PARAM_STR);
$import->bindParam(4, $data[3], PDO::PARAM_STR);
$import->bindParam(5, $data[4], PDO::PARAM_STR);
$import->bindParam(6, $data[5], PDO::PARAM_STR);
str_replace('"',' ',$data);
$import->execute();
}
$i++;
}
EDIT Current code:
<?php
if (isset($_POST['submit'])) {
$i=0;
require "connection.php";
if (is_uploaded_file($_FILES['csv']['tmp_name'])) {
echo "<h3>" . "File ". $_FILES['csv']['name'] ." uploaded successfully." . "</h3>";
}
//Import uploaded file to Database
$handle = fopen($_FILES['csv']['tmp_name'], "r");
$import=$db->prepare("INSERT INTO adherence(
dateandtime,
lastname,
firstname,
paidtime,
approvedtime,
notadhering)VALUES(
?,?,?,?,?,?)");
$badlines=[];
while (($data = fgetcsv($handle, 1000,"\t",'"')) !== FALSE) {
if($i>0) {
//skip but keep track of bad data
if(count($data) !== 5){
$badlines[] = $i;
continue;
}
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$names = explode(',', $data[1]);
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $names[0], PDO::PARAM_STR);
$import->bindParam(3, $names[1], PDO::PARAM_STR);
$import->bindParam(4, $data[2], PDO::PARAM_STR);
$import->bindParam(5, $data[3], PDO::PARAM_STR);
$import->bindParam(6, $data[4], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
$removal=$db->prepare("delete FROM adherence WHERE approvedtime = '0' OR notadhering IS NULL");
$removal->execute();
fclose($handle);
}
"name, lastname" is a single column, as it is surrounded by double quotes.
Your code originally worked as you misconfigured fgetscsv by passing a single quote as the enclosure parameter, causing it to ignore these double quotes and treat the single column as two as it contains a comma.
As your new file is tab delimetered, it does not split on the comma.
To fix, pass the correct enclosure parameter to fgetcsv, and split the column yourself:
$badlines=[];
while (($data = fgetcsv($handle, 1000,"\t",'"')) !== FALSE) {
if($i>0) {
//skip but keep track of bad data
if(count($data) !== 5){
$badlines[] = $i;
continue;
}
$myDate = date("Y/m/d",strtotime(str_replace('/','-',$data[0])));
$names = explode(',', $data[1]);
$import->bindParam(1, $myDate, PDO::PARAM_STR);
$import->bindParam(2, $names[0], PDO::PARAM_STR);
$import->bindParam(3, $names[1], PDO::PARAM_STR);
$import->bindParam(4, $data[2], PDO::PARAM_STR);
$import->bindParam(5, $data[3], PDO::PARAM_STR);
$import->bindParam(6, $data[4], PDO::PARAM_STR);
$import->execute();
}
$i++;
}

CSV import with progress bar

I'm using this script for importing a csv file to mysql database.
How can i display a progress bar for importing a csv file to db using jquery and php ?
I don't need te actual code, just some infos.
Thanks in advance.
if ( $request->get( $_POST["action"] ) == "import" ) {
$file = $upload->file_upload( "import", "media/import" );
if ( file_exists( DIR_UPLOAD_PHOTO . "/media/import/" . $file ) ) {
$file = DIR_UPLOAD_PHOTO . "/media/import/" . $file;
try {
$dbh = new PDO("mysql:host=".HOST."; dbname=".DATABASE, USER, PASSWORD);
}
catch(PDOException $e) {
echo $e->getMessage();
}
$handle = fopen( $file, "r" );
$delimiter = '|';
$dbh->beginTransaction();
$stmt = $dbh->prepare("INSERT INTO products SET title = :title, price = :price
ON DUPLICATE KEY UPDATE
title = :title, price = :price"
);
fgets($handle);
$rows = count(file($file));
while ($line = fgetcsv($handle, 1000, $delimiter)) {
$line = array_map('trim', $line);
$stmt->bindParam(':title', $line[0], PDO::PARAM_STR);
$stmt->bindParam(':price', $line[1], PDO::PARAM_STR);
$stmt->execute();
}
$dbh->commit();
fclose($handle);
$dbh = null;
}
}
At last in MariaDB you get some Status-Information this way (don't know if it works in MySQL as well):
Fork the INSERT INTO ... Statement in an other process
and run it.
Use SHOW PROCESSLIST in your main thread to get the Status of the INSERT Statement.

Import a CSV file into MySQL using PHP

I'm trying to import CSV data into a MySQL database using the fgetcsv function.
if(isset($_POST['submit'])) {
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv") {
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0) {
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is it gets stuck in between the import process and displays the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'S',0,0)' at line 1
The file is imported only partially each time. Only between 200-300 lines out of a 10000 line file are imported.
Here is the DDL of my table:
create table vpireport (
id int not null auto_increment,
item_code int,
company_id int,
purchase double,
primary key(id),
foreign key(company_id) references users(userid)
);
I haven't been able to find the problem so far, any help appreciated. Thanks.
You probably need to escape quotes, which you could accomplish using PDO and prepared statements.
I've skipped most of your code in the example for brevity and just focused on the for loop.
<?php
// Use PDO to connect to the DB
$dsn = 'mysql:dbname=YOUR_DB;host=localhost';
$user = 'DB_USERNAME';
$password = 'DB_PASSWORD';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
// The query uses placeholders for data
$sql = "INSERT INTO vpireport
(item_code,company_id,purchase,purchase_value)
VALUES
(:item_code,:company_id,:purchase,:purchase_value)";
$sth = $dbh->prepare($sql);
// The data is bound to the placeholders
$sth->bindParam(':item_code', $data[0]);
$sth->bindParam(':company_id', $data[1]);
$sth->bindParam(':purchase', $data[2]);
$sth->bindParam(':purhcase_value', $data[3]);
// The row is actually inserted here
$sth->execute();
$sth->closeCursor();
}
That won't get rid of any problem characters, though, so you may want to look at some kind of data sanitization if that poses a problem.
uncomment the //echo "$sql"; and look what is the last query (with error) - it may be that the csv data contains strange characters or the query is cut off.
BTW: you can also import csv file by mysql:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
$row = 1;
if (($handle = fopen("albums.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ',','"')) !== FALSE) {
if($row!=1){
$num = count($data);
$albumIndex=0;
//Insert into tbl_albums
$sqlAlbums="INSERT INTO tbl_albums(albumName) VALUES ('".$data[$albumIndex]."')";
$resultAlbums=mysql_query($sqlAlbums);
}
}
$row++;
}
}
fclose($handle);

Categories