Import a CSV file into MySQL using PHP - php

I'm trying to import CSV data into a MySQL database using the fgetcsv function.
if(isset($_POST['submit'])) {
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv") {
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0) {
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is it gets stuck in between the import process and displays the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'S',0,0)' at line 1
The file is imported only partially each time. Only between 200-300 lines out of a 10000 line file are imported.
Here is the DDL of my table:
create table vpireport (
id int not null auto_increment,
item_code int,
company_id int,
purchase double,
primary key(id),
foreign key(company_id) references users(userid)
);
I haven't been able to find the problem so far, any help appreciated. Thanks.

You probably need to escape quotes, which you could accomplish using PDO and prepared statements.
I've skipped most of your code in the example for brevity and just focused on the for loop.
<?php
// Use PDO to connect to the DB
$dsn = 'mysql:dbname=YOUR_DB;host=localhost';
$user = 'DB_USERNAME';
$password = 'DB_PASSWORD';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
// The query uses placeholders for data
$sql = "INSERT INTO vpireport
(item_code,company_id,purchase,purchase_value)
VALUES
(:item_code,:company_id,:purchase,:purchase_value)";
$sth = $dbh->prepare($sql);
// The data is bound to the placeholders
$sth->bindParam(':item_code', $data[0]);
$sth->bindParam(':company_id', $data[1]);
$sth->bindParam(':purchase', $data[2]);
$sth->bindParam(':purhcase_value', $data[3]);
// The row is actually inserted here
$sth->execute();
$sth->closeCursor();
}
That won't get rid of any problem characters, though, so you may want to look at some kind of data sanitization if that poses a problem.

uncomment the //echo "$sql"; and look what is the last query (with error) - it may be that the csv data contains strange characters or the query is cut off.
BTW: you can also import csv file by mysql:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html

$row = 1;
if (($handle = fopen("albums.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ',','"')) !== FALSE) {
if($row!=1){
$num = count($data);
$albumIndex=0;
//Insert into tbl_albums
$sqlAlbums="INSERT INTO tbl_albums(albumName) VALUES ('".$data[$albumIndex]."')";
$resultAlbums=mysql_query($sqlAlbums);
}
}
$row++;
}
}
fclose($handle);

Related

Inserting data from mysql from CSV using PHP PDO

I have list of data in CSV and need to insert this data into a MySQL database. These data should be safely inserted i.e sanitation. So, I have used PDO object to rectify SQL injection. But, it fails to get data from CSV file and inserts null values.
Here is the example,
<?php
$servername = "localhost";
$username = "root";
$password = "";
try {
$conn = new PDO("mysql:host=$servername;dbname=contact_list",$username,$password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "connection successfully";
}
catch(PDOException $e)
{
echo "connection Failed:" . $e -> getMessage();
}
// Create CSV to Array function
function csvToArray($filename = '', $delimiter = ',')
{
if (!file_exists($filename) || !is_readable($filename)) {
return false;
}
$header = NULL;
$result = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
if (!$header)
$header = $row;
else
$result[] = array_combine($header, $row);
}
fclose($handle);
}
return $result;
}
// Insert data into database
$all_data = csvToArray('contact.csv');
foreach ($all_data as $data) {
$data = array_map(function($row){
return filter_var($row, FILTER_SANITIZE_STRING, FILTER_SANITIZE_FULL_SPECIAL_CHARS);
}, $data);
$sql = $conn->prepare("INSERT INTO contact
(title, first_name,last_name,company_name,date_of_birth,notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
$sql->bindParam(':t', $data[1], PDO::PARAM_STR);
$sql->bindParam(':fname', $data[2], PDO::PARAM_STR);
$sql->bindParam(':lname', $data[3], PDO::PARAM_STR);
$sql->bindParam(':cname', $data[0], PDO::PARAM_STR);
$sql->bindParam(':dob', $data[4], PDO::PARAM_STR);
$sql->bindParam(':note', $data[15], PDO::PARAM_STR);
print_r($data);
$sql->execute();
}
?>
Can anyone help me to solve this?
If you take a look at the documentation for array_combine() you'll see that its purpose is to build an associative array. You use this function in csvToArray() but later in your code you are trying to get data using numeric keys. I wouldn't expect you'd ever have anything inserted.
On a side note, you are completely defeating the purpose of prepared statements by repeatedly preparing the same statement over and over again. Prepare once and execute many times. Individually binding parameters is rarely needed, in almost all cases you can provide the data to PDOStatement::execute() as an array. It's also bad form to store HTML entities in a database; if you need to output to HTML, you perform escaping at that point.
Something like this should work (adjust array key names as necessary.)
$all_data = csvToArray('contact.csv');
$sql = $conn->prepare("INSERT INTO contact
(title, first_name, last_name, company_name, date_of_birth, notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
foreach ($all_data as $data) {
$params = [
":t" => $data["t"],
":fname" => $data["fname"],
":lname" => $data["lname"],
":dob" => $data["dob"],
":note" => $data["note"],
];
$sql->execute($params);
}

PHP imported data integrity

I will try to explain situation as well as possible:
I have script, that imports CSV file data to MS Access database.
I have 2 access Tables:
A) Users and their information(ID, name, last name etc.)
B) Table which contains data from CSV file
Problem is, data imported from file, (2nd table) contains Users name and lastname. I want to get idea, how to, while reading csv file line by line, check what name line contains, and assign userID from table 1 instead of name and lastname on table 2. It should be done while importing, because, on each import there are roughly 3k lines being imported. Any ideas appreciated. Images given bellow.
Import script:
<?php
function qualityfunction() {
error_reporting(0);
require_once '/Classes/PHPExcel.php'; // (this should include the autoloader)
require_once '/CLasses/PHPExcel/IOFactory.php';
$excel_readers = array(
'Excel5' ,
'Excel2003XML' ,
'Excel2007'
);
$files = glob('data files/quality/QA*.xls');
$sheetname= 'AvgScoreAgentComments';
if (count($files) >0 ) {
foreach($files as $flnam) {
$reader = PHPExcel_IOFactory::createReader('Excel5');
$reader->setReadDataOnly(true);
$reader->setLoadSheetsOnly($sheetname);
$path = $flnam;
$excel = $reader->load($path);
$writer = PHPExcel_IOFactory::createWriter($excel, 'CSV');
$writer->save('data files/quality/temp.csv');
/*
$filename = basename($path);
if (strpos($filename,'tes') !== false) {
echo 'true';
}*/
require "connection.php";
$handle = fopen("data files/quality/temp.csv", "r");
try {
$import= $db->prepare("INSERT INTO quality(
qayear,
qamonth,
lastname,
firstname,
score) VALUES(
?,?,?,?,?)");
$i = 0;
while (($data = fgetcsv($handle, 1000, ",", "'")) !== FALSE) {
if($i > 3) {
$data = str_replace('",', '', $data);
$data = str_replace('"', '', $data);
$import->bindParam(1, $data[1], PDO::PARAM_STR);
$import->bindParam(2, $data[2], PDO::PARAM_STR);
$import->bindParam(3, $data[3], PDO::PARAM_STR);
$import->bindParam(4, $data[4], PDO::PARAM_STR);
$import->bindParam(5, $data[7], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
fclose($handle);
$removal=$db->prepare("DELETE FROM quality WHERE score IS NULL;");
$removal->execute();
}
catch(PDOException $e) {
echo $e->getMessage()."\n";
}};
Data table 1 (Users info):
Data table 2 (In which data from CSV file is imported)
Found a solution. Thanks for help.
$lastname = "lastname";
$firstname = "firstname";
$showdata = $db->prepare("SELECT userID FROM users WHERE lastname= :lastname AND firstname= :firstname");
$showdata->bindParam(':lastname', $lastname);
$showdata->bindParam(':firstname', $firstname);
$showdata->execute();
$rowas= $showdata->fetch(PDO::FETCH_ASSOC);
echo $rowas['userID'];

Create MySQL table dynamically from Excel CSV file

My goal is to create a MySQL table containing data from my CSV file.
I know how to create a MySQL table and how to load data from excel in it.
But the problem is:
I have a large CSV file containing long column names (questions labels for example "Q27 : Are you happy with the after sales service?") so it would be boring to create a MySQL table by copying all column names(almost 35) and add 'VARCHAR(100) NOT NULL'.
That's why I would like to write a small php script to create a MySQL table by getting the first row of my file, and then fill it with the rest of the csv file data.
For now, my script looks like this :
<?php
$host = 'localhost';
$user = 'root';
$pass = '';
$database = 'test';
$db = #mysql_connect($host, $user, $pass) or die('mysql connection pb');
#mysql_select_db($database) or die('database selection pb');
/********************************************************************************/
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) { $file = $argv[1]; }
else {
echo "Please provide a file name\n"; exit;
}
if($argv[2]) {
$table = $argv[2];
}
else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
/**************************************************************************** ****/
// Get the first row to create the column headings
$fp = fopen($file, 'r');
$frow = fgetcsv($fp,";");
$columns=false;
print_r($frow);
foreach($frow as $column) {
if($columns) $columns .= ', ';
$columns .= "`$column` VARCHAR(250) NOT NULL";
}
$create = "create table if not exists $table ($columns);";
#mysql_query($create, $db) or die('table creation pb');
/**************************************************************************** ****/
// Import the data into the newly created table.
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$q = "load data infile '$file' into table $table fields terminated by ',' ignore 1 lines";
#mysql_query($q, $db);
?>
And when i run in command line : php myscript.php csvfile.csv mytable, it appears that the problem is in the table creation query.
And on top of that the column names are not well identified even though they are separated by ";" in the csv.
As mentioned refrain from using mysql_ functions which as of PHP 7 (current version) this extension is no longer available. Use either mysqli or PDO.
Below is a PDO example with try/catch (more informative than die()). Also, the csv read is handled slightly different and its concatenation in SQL create table string.
<?php
$host="localhost";
$username="root";
$password="password";
$database="test"
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) {
$file = $argv[1];
} else {
echo "Please provide a file name\n";
exit;
}
if($argv[2]) {
$table = $argv[2];
} else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
// Read in only first row of CSV file
$handle = fopen($file, "r");
$row = 1;
$columns = [];
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE AND $row==1) {
$columns = $data;
$row++;
}
//SQL string commands
$createSQL = "CREATE TABLE IF NOT EXISTS $table
(".implode(" VARCHAR(255) NOT NULL, ", $columns). "
VARCHAR(255) NOT NULL);";
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$loadSQL = "LOAD DATA INFILE '$file'
INTO TABLE $table
FIELDS TERMINATED BY ','
IGNORE 1 LINES";
// Open database connection
try {
$dbh = new PDO("mysql:host=$host;dbname=$database",$username,$password);
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// Execute queries
$S1TH = $dbh->query($createSQL);
$S2TH = $dbh->query($loadSQL);
}
catch(PDOException $e) {
echo $e->getMessage();
}
# Close database connection
$dbh = null;
?>

CSV import with progress bar

I'm using this script for importing a csv file to mysql database.
How can i display a progress bar for importing a csv file to db using jquery and php ?
I don't need te actual code, just some infos.
Thanks in advance.
if ( $request->get( $_POST["action"] ) == "import" ) {
$file = $upload->file_upload( "import", "media/import" );
if ( file_exists( DIR_UPLOAD_PHOTO . "/media/import/" . $file ) ) {
$file = DIR_UPLOAD_PHOTO . "/media/import/" . $file;
try {
$dbh = new PDO("mysql:host=".HOST."; dbname=".DATABASE, USER, PASSWORD);
}
catch(PDOException $e) {
echo $e->getMessage();
}
$handle = fopen( $file, "r" );
$delimiter = '|';
$dbh->beginTransaction();
$stmt = $dbh->prepare("INSERT INTO products SET title = :title, price = :price
ON DUPLICATE KEY UPDATE
title = :title, price = :price"
);
fgets($handle);
$rows = count(file($file));
while ($line = fgetcsv($handle, 1000, $delimiter)) {
$line = array_map('trim', $line);
$stmt->bindParam(':title', $line[0], PDO::PARAM_STR);
$stmt->bindParam(':price', $line[1], PDO::PARAM_STR);
$stmt->execute();
}
$dbh->commit();
fclose($handle);
$dbh = null;
}
}
At last in MariaDB you get some Status-Information this way (don't know if it works in MySQL as well):
Fork the INSERT INTO ... Statement in an other process
and run it.
Use SHOW PROCESSLIST in your main thread to get the Status of the INSERT Statement.

Special characters can't be inserted into database

Im new to php. Im trying to read a text file and insert line by line data to database. My problem is for Some special character insert query does not works
For example Côte , d.ä. , d.y. , DAB-sändare these all are working. But cant insert d'affaires. If i remove d'affaires then the query will execute otherwise it will not insert any data to database. The php code i used to reaf and insert to database is
mysql_connect("localhost","root","");
mysql_select_db("testdb");
$query="INSERT INTO keywords (id, keyword) VALUES ";
$handle = fopen("Ordlista.txt", "r");
if ($handle) {
$i=1;
while (($line = fgets($handle)) !== false) {
// process the line read.
// echo $line.'<br>';
if($i==1)
{
$query.=" ( NULL , '".$line."') ";
$i++;
}
else {
$query.=" ,( NULL , '".$line."') ";
}
}
$query.=";";
// $qr=htmlspecialchars($query,ENT_QUOTES);
echo $query;
mysql_query($query);
} else {
echo 'error opening the file.';
// error opening the file.
}
fclose($handle);
UPDATED
I have used this code while creating a plugin in wordpress then the special characters are inserting as '?'. In the previous code it was working file the code change i done is
mysql_query("TRUNCATE TABLE $table");
// $structure = "INSERT INTO $table (`id`, `keyword`) VALUES (NULL, 'test1'), (NULL, 'test2');"; // Keywords for Testing
// $wpdb->query($structure);
//read text file & insert to database start
$query="INSERT INTO $table (id, keyword) VALUES ";
$fllocation=PLG_URL.'/Ordlista.txt';
$handle = fopen($fllocation, "r");
if ($handle) {
$i=1;
while (($line = fgets($handle)) !== false) {
// process the line read.
if($i==1)
{
$query.=" ( NULL , '".mysql_real_escape_string($line)."') ";
$i++;
}
else {
$query.=" ,( NULL , '".mysql_real_escape_string($line)."') ";
}
}
$query.=";";
$wpdb->query($query);
// echo $query;
// mysql_query($query);
} else {
echo 'error opening the file.';
// error opening the file.
}
fclose($handle);
Try mysql_real_escape_string();
mysql_connect("localhost","root","");
mysql_select_db("testdb");
$query="INSERT INTO keywords (id, keyword) VALUES ";
$handle = fopen("Ordlista.txt", "r");
if ($handle) {
$i=1;
while (($line = fgets($handle)) !== false) {
// process the line read.
// echo $line.'<br>';
if($i==1)
{
$query.=" ( NULL , '".mysql_real_escape_string($line)."') ";
$i++;
}
else {
$query.=" ,( NULL , '".mysql_real_escape_string($line)."') ";
}
}
$query.=";";
// $qr=htmlspecialchars($query,ENT_QUOTES);
echo $query;
mysql_query($query);
} else {
echo 'error opening the file.';
// error opening the file.
}
fclose($handle);
The best solution would be to upgrade from mysql_* to PDO or mysqli_*, as these allow you to run prepared queries with parameters. But if you can't do that, you have to escape the data:
while (($line = fgets($handle)) !== false) {
// process the line read.
// echo $line.'<br>';
$line = mysql_real_escape_string($line);
if($i==1)
{
$query.=" ( NULL , '".$line."') ";
$i++;
}
else {
$query.=" ,( NULL , '".$line."') ";
}
}
First, don't use the mysql extension. It has been officially deprecated.
Second, use a prepared statement with parameters to avoid any problems with SQL injection.
Third, make sure you're using a compatible connection, table and column encoding / character set.
For example, using mysqli...
$con = new mysqli('localhost', 'root', '', 'testdb');
if ($con->connect_errno) {
throw new Exception($con->connect_error, $con->connect_errno);
}
$con->set_charset('utf8');
$stmt = $con->prepare('INSERT INTO `keywords` (`keyword`) VALUES (?)');
if (!$stmt) {
throw new Exception($con->error, $con->errno);
}
$stmt->bind_param('s', $keyword);
foreach (file('Ordlista.txt') as $keyword) {
if (!$stmt->execute()) {
throw new Exception($stmt->error, $stmt->errno);
}
}
After reading your update, i think the problem is with the collate and charset of your table, execute this:
ALTER TABLE `keywords` CHARACTER SET = utf8 , COLLATE = utf8_unicode_ci ;

Categories