Error Importing CSV to MySQL with PHP - php

I am trying to use this code to import a csv file into my MySQL database:
... Thanks for solved that question!! ....
What am I doing wrong?
Guys I could solved that problem but now... I have an other problem, this is the code now:
<?php
$databasehost = "localhost";
$databasename = "cauctti";
$databasetable = "sample";
$databaseusername="root";
$databasepassword = "toor";
$fieldseparator = ";";
$lineseparator = "\r|n";
$csvfile = "Reporting.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
When I try to run it, only insert one row on the table...
Loaded a total of 1 records from this csv file.
Why it only gets one row when my csv file have like 5.000 entries...

As found here
You should set CHARACTER SET UTF8 in your query.
Example:
LOAD DATA INFILE 'file'
IGNORE INTO TABLE table
CHARACTER SET UTF8
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'

Have a look here --> Here
<?php
$table = "tableName";
$fileName = "States.csv";
$ignoreFirstRow = 1;
if (($handle = fopen($fileName, "r")) !== FALSE){
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
if($ignoreFirstRow != 1){
$sql = "insert into ".$table." values(";
$sql .= '"'.implode('","',$data).'"';
echo "".$sql.');';
$sql = "";
}
$ignoreFirstRow++;
}
fclose($handle);
}
?>

Related

How to get Row Number when insert records from File to DB with LOAD DATA LOCAL INFILE, Transaction and Commit?

How to get Row Number when insert records from File to DB with LOAD DATA LOCAL INFILE, Transaction and Commit?
However, only getting true or false(in fail) with the var dump present
Statement:
LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE tablename
FIELDS TERMINATED BY ','
LINES TERMINATED BY ' ' ( `Col1`, `Col2`, `Col3`)
Script:
public function PDO_UL_IUPD($dbUsing, $stmtpre) {
$started = microtime(true);
$DB = [];
$val = [];
$conn = new PDO(
"mysql:host=" . DB_HOST . ";dbname=" . DB_PRE . "" . $dbUsing . "",
DB_USERNAME,
DB_PASS,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => TRUE,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES UTF8",
PDO::ATTR_EMULATE_PREPARES => FALSE,
PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => TRUE,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
));
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
try {
$conn->beginTransaction();
$conn->exec("set names utf8");
$count = 0;
$RowReg = 0;
// our SQL statements
foreach ($stmtpre as $stmt) {
$count++;
$conn->exec($stmt);
}
$RowReg = $conn->commit();
echo var_dump($RowReg);
$DB['SMG'] = "Correct Execution, a csv file has been dumped.<br><br>";
$DB['R'] = true;
} catch (PDOException $e) {
$DB['SMG'] = "Error: " . $e->getMessage();
$DB['R'] = false;
}
return $DB;
}
Update1 Script:
Trying to get Row Number from EXEC return 0
Script:
public function PDO_UL_IUPD($dbUsing, $stmtpre) {
$started = microtime(true);
$DB = [];
$val = [];
$conn = new PDO(
"mysql:host=" . DB_HOST . ";dbname=" . DB_PRE . "" . $dbUsing . "",
DB_USERNAME,
DB_PASS,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => TRUE,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES UTF8",
PDO::ATTR_EMULATE_PREPARES => FALSE,
PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => TRUE,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
));
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$conn->beginTransaction();
try {
$conn->exec("set names utf8");
$count = 0;
$RowReg=0;
$cReg=0;
// our SQL statements
foreach ($stmtpre as $stmt) {
//echo $stmt."<br><br><br>";
$count++;
$cReg=$conn->exec($stmt);
$RowReg=$RowReg+$cReg;
}
$conn->commit();
echo var_dump($RowReg);
$DB['SMG'] = "Correct Execution, a csv file has been dumped.<br><br><br>";
$DB['R'] = true;
} catch (PDOException $e) {
$DB['SMG'] = "Error en Tiempo de Ejecucion: " . $e->getMessage();
$DB['R'] = false;
}
return $DB;
}
Update 2 statement:
adding this Return 1
SET #row=0;
LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE tablename
FIELDS TERMINATED BY ','
LINES TERMINATED BY ' ' ( `Col1`, `Col2`, `Col3`)
SET file_line_no = #row:=#row+1;
Solved with:
This script is full suported:
LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE tablename
FIELDS TERMINATED BY ','
LINES TERMINATED BY ' ' ( `Col1`, `Col2`, `Col3`)
But i merge it with other in some Query Like that:
DELETE FROM tablename WHERE date BETWEEN '$Date1' AND '$Date2';
ALTER TABLE tbalename AUTO_INCREMENT = 1;
LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE tablename
FIELDS TERMINATED BY ','
LINES TERMINATED BY ' ' ( `Col1`, `Col2`, `Col3`);
this is wrong, becouse $PDO->exec() Cant return the number of affected table or rows.
Instead of it, to solve i need past an array, becouse my script support it like that:
$stmtpre[1] = "DELETE FROM tablename WHERE date BETWEEN '$Date1' AND '$Date2';";
$stmtpre[2] = "ALTER TABLE tbalename AUTO_INCREMENT = 1;";
$stmtpre[3] = "LOAD DATA LOCAL INFILE 'file.csv' INTO TABLE tablename
FIELDS TERMINATED BY ','
LINES TERMINATED BY ' ' ( `Col1`, `Col2`, `Col3`);";

CSV uploading without browse button

I want to upload the contents of a CSV while one PHP page is running. I don't want any browse button to upload the CSV. Whenever the page is running the page should find the CSV which the path is already defined in the PHP page and contents should be inserted into the table. Now I am getting error related with fopen.
Here is my code
<?php
//database connection details
$connect = mysql_connect('localhost', 'root', '');
if (!$connect) {
die('Could not connect to MySQL: ' . mysql_error());
}
//your database name
$cid = mysql_select_db('test', $connect);
// path where your CSV file is located
define('CSV_PATH', 'D:/xamp/htdocs/test/');
// Name of your CSV file
$csv_file = CSV_PATH . "test.csv";
echo $csv_file;
if (($handle = fopen($csv_file, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($c = 0; $c < $num; $c++) {
$col[$c] = $data[$c];
}
$col1 = $col[0];
$col2 = $col[1];
$col3 = $col[2];
$col4 = $col[3];
$col5 = $col[4];
$col6 = $col[5];
// SQL Query to insert data into DataBase
$query = "INSERT INTO testcsv(Line,Part No,Make,Model,Year,Part Type) VALUES('" . $col1 . "','" . $col2 . "','" . $col3 . "','" . $col4 . "','" . $col5 . "','" . $col6 . "')";
$s = mysql_query($query, $connect);
}
fclose($handle);
}
echo "File data successfully imported to database!!";
mysql_close($connect);
?>
I am getting this error
Warning: fopen(D:/xamp/htdocs/test/test.csv): failed to open stream: No such file or directory in D:\xamp\htdocs\test\test.php on line 22
File data successfully imported to database!!
Can anyone help me?
I'm not sure why you are getting that particular error - one might assume that the file does not exist or that the directory is not readable but you are using the now deprecated mysql_ functions and directly embedding variables in the sql - thus making it vulnerable to sql injection. However, as this looks to be only a test that is probably not an issue.
The preferred method for this type of thing would be to use either mysqli or PDO in conjunction with prepared statements - below is an example of how you might implement that - I tested this with different data and database details and it seemed to work fine.
define('CSV_PATH','D:/xamp/htdocs/test/');
$filepath = CSV_PATH . "test.csv";
/* database connection details */
$host = 'localhost';
$uname = 'xxx';
$pwd = 'xxx';
$db = 'xxx';
/* create db connection */
$con = new mysqli( $host, $uname, $pwd, $db );
/* construct required sql statement */
$sql='insert into `testcsv` (`Line`,`Part No`,`Make`,`Model`,`Year`,`Part Type`) values (?,?,?,?,?,?)';
/* create prepared statement */
$stmt=$con->prepare( $sql );
if( !$stmt ){
echo 'error preparing sql statement!';
$con->close();
} else {
/* bind the columns to variables which will be populated later */
/* use "i" for integer and "s" for string values */
$stmt->bind_param( 'ssssss', $line,$part,$make,$model,$year,$type );
/* access csv file */
$file=new SplFileObject( $filepath );
/* Process each row of the csv file */
while( !$file->eof() ) {
/* read the line into a variable */
$data=$file->fgetcsv();
if( !empty( $data ) ){
/* assign a variable to each field value for this row */
list( $line,$part,$make,$model,$year,$type )=$data;
/* execute statement with the now defined variables */
$stmt->execute();
}
}
/* tidy up */
$stmt->close();
$con->close();
echo 'database updated with new records from csv';
}

Create MySQL table dynamically from Excel CSV file

My goal is to create a MySQL table containing data from my CSV file.
I know how to create a MySQL table and how to load data from excel in it.
But the problem is:
I have a large CSV file containing long column names (questions labels for example "Q27 : Are you happy with the after sales service?") so it would be boring to create a MySQL table by copying all column names(almost 35) and add 'VARCHAR(100) NOT NULL'.
That's why I would like to write a small php script to create a MySQL table by getting the first row of my file, and then fill it with the rest of the csv file data.
For now, my script looks like this :
<?php
$host = 'localhost';
$user = 'root';
$pass = '';
$database = 'test';
$db = #mysql_connect($host, $user, $pass) or die('mysql connection pb');
#mysql_select_db($database) or die('database selection pb');
/********************************************************************************/
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) { $file = $argv[1]; }
else {
echo "Please provide a file name\n"; exit;
}
if($argv[2]) {
$table = $argv[2];
}
else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
/**************************************************************************** ****/
// Get the first row to create the column headings
$fp = fopen($file, 'r');
$frow = fgetcsv($fp,";");
$columns=false;
print_r($frow);
foreach($frow as $column) {
if($columns) $columns .= ', ';
$columns .= "`$column` VARCHAR(250) NOT NULL";
}
$create = "create table if not exists $table ($columns);";
#mysql_query($create, $db) or die('table creation pb');
/**************************************************************************** ****/
// Import the data into the newly created table.
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$q = "load data infile '$file' into table $table fields terminated by ',' ignore 1 lines";
#mysql_query($q, $db);
?>
And when i run in command line : php myscript.php csvfile.csv mytable, it appears that the problem is in the table creation query.
And on top of that the column names are not well identified even though they are separated by ";" in the csv.
As mentioned refrain from using mysql_ functions which as of PHP 7 (current version) this extension is no longer available. Use either mysqli or PDO.
Below is a PDO example with try/catch (more informative than die()). Also, the csv read is handled slightly different and its concatenation in SQL create table string.
<?php
$host="localhost";
$username="root";
$password="password";
$database="test"
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) {
$file = $argv[1];
} else {
echo "Please provide a file name\n";
exit;
}
if($argv[2]) {
$table = $argv[2];
} else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
// Read in only first row of CSV file
$handle = fopen($file, "r");
$row = 1;
$columns = [];
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE AND $row==1) {
$columns = $data;
$row++;
}
//SQL string commands
$createSQL = "CREATE TABLE IF NOT EXISTS $table
(".implode(" VARCHAR(255) NOT NULL, ", $columns). "
VARCHAR(255) NOT NULL);";
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$loadSQL = "LOAD DATA INFILE '$file'
INTO TABLE $table
FIELDS TERMINATED BY ','
IGNORE 1 LINES";
// Open database connection
try {
$dbh = new PDO("mysql:host=$host;dbname=$database",$username,$password);
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// Execute queries
$S1TH = $dbh->query($createSQL);
$S2TH = $dbh->query($loadSQL);
}
catch(PDOException $e) {
echo $e->getMessage();
}
# Close database connection
$dbh = null;
?>

How should I escape the quotes in an SQL query?

Following is my code where I am trying to run the load query, but it's not running because of mismanaged quotes in the $qry string. Please explain how I can correct the query so that it can execute.
<?php
include 'connection.php';
$list=array();
//array_push($list,"304_updated_24may.csv");
array_push($list,"filename1.csv");
array_push($list,"filename2.csv");
array_push($list,"filename3.csv");
array_push($list,"filename4.csv");
try
{
foreach($list as $array)
{
echo 'hi';
$qry='LOAD DATA LOCAL INFILE '.$array.' INTO TABLE tablename FIELDS TERMINATED BY ',' ENCLOSED BY '/"' LINES TERMINATED BY '\n' IGNORE 1 ROWS';
print($qry);
print($qry);
$sqlvar= mysqli_query($mysqli, $qry) or printf("Errormessage2: %s\n", $mysqli->error);
}
}
catch(Exception $e)
{
var_dump($e);
}
?>
Don't Panic is rigth above. I did a similar solution. The following is what i did. I used pdo->quote() to escape my quotations. Should get around your problem.
$databasehost = "your database host";
$databasename = "your database name";
$databasetable = "table name";
$databaseusername = "database username";
$databasepassword = "database password";
$fieldseparator = ",";
$lineseparator = "\r\n";
$enclosedby = '\"'; // notice that we escape the double quotation mark
$csvfile = "your_csv_file_name.csv"; // this is your $list of csv files... replace as $list = array(); and array_push into list.
// check to see if you have the file in the first place
if(!file_exists( $csvfile )) {
die( "File not found. Make sure you specified the correct path." );
}
try {
$pdo = new PDO( "mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch ( PDOException $e ) {
die("database connection failed: ".$e->getMessage());
}
// Load your file into the database table, notice the quote() function, protects you from dangerous quotes
$qry = $pdo->exec( "
LOAD DATA LOCAL INFILE " . $pdo->quote( $csvfile ) . " INTO TABLE $databasetable FIELDS TERMINATED BY " . $pdo->quote( $fieldseparator ) .
" OPTIONALLY ENCLOSED BY " . $pdo->quote( $enclosedby ) .
" LINES TERMINATED BY " . $pdo->quote( $lineseparator ) );

Import a CSV file into MySQL using PHP

I'm trying to import CSV data into a MySQL database using the fgetcsv function.
if(isset($_POST['submit'])) {
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv") {
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0) {
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is it gets stuck in between the import process and displays the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'S',0,0)' at line 1
The file is imported only partially each time. Only between 200-300 lines out of a 10000 line file are imported.
Here is the DDL of my table:
create table vpireport (
id int not null auto_increment,
item_code int,
company_id int,
purchase double,
primary key(id),
foreign key(company_id) references users(userid)
);
I haven't been able to find the problem so far, any help appreciated. Thanks.
You probably need to escape quotes, which you could accomplish using PDO and prepared statements.
I've skipped most of your code in the example for brevity and just focused on the for loop.
<?php
// Use PDO to connect to the DB
$dsn = 'mysql:dbname=YOUR_DB;host=localhost';
$user = 'DB_USERNAME';
$password = 'DB_PASSWORD';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
// The query uses placeholders for data
$sql = "INSERT INTO vpireport
(item_code,company_id,purchase,purchase_value)
VALUES
(:item_code,:company_id,:purchase,:purchase_value)";
$sth = $dbh->prepare($sql);
// The data is bound to the placeholders
$sth->bindParam(':item_code', $data[0]);
$sth->bindParam(':company_id', $data[1]);
$sth->bindParam(':purchase', $data[2]);
$sth->bindParam(':purhcase_value', $data[3]);
// The row is actually inserted here
$sth->execute();
$sth->closeCursor();
}
That won't get rid of any problem characters, though, so you may want to look at some kind of data sanitization if that poses a problem.
uncomment the //echo "$sql"; and look what is the last query (with error) - it may be that the csv data contains strange characters or the query is cut off.
BTW: you can also import csv file by mysql:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
$row = 1;
if (($handle = fopen("albums.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ',','"')) !== FALSE) {
if($row!=1){
$num = count($data);
$albumIndex=0;
//Insert into tbl_albums
$sqlAlbums="INSERT INTO tbl_albums(albumName) VALUES ('".$data[$albumIndex]."')";
$resultAlbums=mysql_query($sqlAlbums);
}
}
$row++;
}
}
fclose($handle);

Categories