I'm using this script for importing a csv file to mysql database.
How can i display a progress bar for importing a csv file to db using jquery and php ?
I don't need te actual code, just some infos.
Thanks in advance.
if ( $request->get( $_POST["action"] ) == "import" ) {
$file = $upload->file_upload( "import", "media/import" );
if ( file_exists( DIR_UPLOAD_PHOTO . "/media/import/" . $file ) ) {
$file = DIR_UPLOAD_PHOTO . "/media/import/" . $file;
try {
$dbh = new PDO("mysql:host=".HOST."; dbname=".DATABASE, USER, PASSWORD);
}
catch(PDOException $e) {
echo $e->getMessage();
}
$handle = fopen( $file, "r" );
$delimiter = '|';
$dbh->beginTransaction();
$stmt = $dbh->prepare("INSERT INTO products SET title = :title, price = :price
ON DUPLICATE KEY UPDATE
title = :title, price = :price"
);
fgets($handle);
$rows = count(file($file));
while ($line = fgetcsv($handle, 1000, $delimiter)) {
$line = array_map('trim', $line);
$stmt->bindParam(':title', $line[0], PDO::PARAM_STR);
$stmt->bindParam(':price', $line[1], PDO::PARAM_STR);
$stmt->execute();
}
$dbh->commit();
fclose($handle);
$dbh = null;
}
}
At last in MariaDB you get some Status-Information this way (don't know if it works in MySQL as well):
Fork the INSERT INTO ... Statement in an other process
and run it.
Use SHOW PROCESSLIST in your main thread to get the Status of the INSERT Statement.
Related
I have list of data in CSV and need to insert this data into a MySQL database. These data should be safely inserted i.e sanitation. So, I have used PDO object to rectify SQL injection. But, it fails to get data from CSV file and inserts null values.
Here is the example,
<?php
$servername = "localhost";
$username = "root";
$password = "";
try {
$conn = new PDO("mysql:host=$servername;dbname=contact_list",$username,$password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "connection successfully";
}
catch(PDOException $e)
{
echo "connection Failed:" . $e -> getMessage();
}
// Create CSV to Array function
function csvToArray($filename = '', $delimiter = ',')
{
if (!file_exists($filename) || !is_readable($filename)) {
return false;
}
$header = NULL;
$result = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
if (!$header)
$header = $row;
else
$result[] = array_combine($header, $row);
}
fclose($handle);
}
return $result;
}
// Insert data into database
$all_data = csvToArray('contact.csv');
foreach ($all_data as $data) {
$data = array_map(function($row){
return filter_var($row, FILTER_SANITIZE_STRING, FILTER_SANITIZE_FULL_SPECIAL_CHARS);
}, $data);
$sql = $conn->prepare("INSERT INTO contact
(title, first_name,last_name,company_name,date_of_birth,notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
$sql->bindParam(':t', $data[1], PDO::PARAM_STR);
$sql->bindParam(':fname', $data[2], PDO::PARAM_STR);
$sql->bindParam(':lname', $data[3], PDO::PARAM_STR);
$sql->bindParam(':cname', $data[0], PDO::PARAM_STR);
$sql->bindParam(':dob', $data[4], PDO::PARAM_STR);
$sql->bindParam(':note', $data[15], PDO::PARAM_STR);
print_r($data);
$sql->execute();
}
?>
Can anyone help me to solve this?
If you take a look at the documentation for array_combine() you'll see that its purpose is to build an associative array. You use this function in csvToArray() but later in your code you are trying to get data using numeric keys. I wouldn't expect you'd ever have anything inserted.
On a side note, you are completely defeating the purpose of prepared statements by repeatedly preparing the same statement over and over again. Prepare once and execute many times. Individually binding parameters is rarely needed, in almost all cases you can provide the data to PDOStatement::execute() as an array. It's also bad form to store HTML entities in a database; if you need to output to HTML, you perform escaping at that point.
Something like this should work (adjust array key names as necessary.)
$all_data = csvToArray('contact.csv');
$sql = $conn->prepare("INSERT INTO contact
(title, first_name, last_name, company_name, date_of_birth, notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
foreach ($all_data as $data) {
$params = [
":t" => $data["t"],
":fname" => $data["fname"],
":lname" => $data["lname"],
":dob" => $data["dob"],
":note" => $data["note"],
];
$sql->execute($params);
}
I want to upload the contents of a CSV while one PHP page is running. I don't want any browse button to upload the CSV. Whenever the page is running the page should find the CSV which the path is already defined in the PHP page and contents should be inserted into the table. Now I am getting error related with fopen.
Here is my code
<?php
//database connection details
$connect = mysql_connect('localhost', 'root', '');
if (!$connect) {
die('Could not connect to MySQL: ' . mysql_error());
}
//your database name
$cid = mysql_select_db('test', $connect);
// path where your CSV file is located
define('CSV_PATH', 'D:/xamp/htdocs/test/');
// Name of your CSV file
$csv_file = CSV_PATH . "test.csv";
echo $csv_file;
if (($handle = fopen($csv_file, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($c = 0; $c < $num; $c++) {
$col[$c] = $data[$c];
}
$col1 = $col[0];
$col2 = $col[1];
$col3 = $col[2];
$col4 = $col[3];
$col5 = $col[4];
$col6 = $col[5];
// SQL Query to insert data into DataBase
$query = "INSERT INTO testcsv(Line,Part No,Make,Model,Year,Part Type) VALUES('" . $col1 . "','" . $col2 . "','" . $col3 . "','" . $col4 . "','" . $col5 . "','" . $col6 . "')";
$s = mysql_query($query, $connect);
}
fclose($handle);
}
echo "File data successfully imported to database!!";
mysql_close($connect);
?>
I am getting this error
Warning: fopen(D:/xamp/htdocs/test/test.csv): failed to open stream: No such file or directory in D:\xamp\htdocs\test\test.php on line 22
File data successfully imported to database!!
Can anyone help me?
I'm not sure why you are getting that particular error - one might assume that the file does not exist or that the directory is not readable but you are using the now deprecated mysql_ functions and directly embedding variables in the sql - thus making it vulnerable to sql injection. However, as this looks to be only a test that is probably not an issue.
The preferred method for this type of thing would be to use either mysqli or PDO in conjunction with prepared statements - below is an example of how you might implement that - I tested this with different data and database details and it seemed to work fine.
define('CSV_PATH','D:/xamp/htdocs/test/');
$filepath = CSV_PATH . "test.csv";
/* database connection details */
$host = 'localhost';
$uname = 'xxx';
$pwd = 'xxx';
$db = 'xxx';
/* create db connection */
$con = new mysqli( $host, $uname, $pwd, $db );
/* construct required sql statement */
$sql='insert into `testcsv` (`Line`,`Part No`,`Make`,`Model`,`Year`,`Part Type`) values (?,?,?,?,?,?)';
/* create prepared statement */
$stmt=$con->prepare( $sql );
if( !$stmt ){
echo 'error preparing sql statement!';
$con->close();
} else {
/* bind the columns to variables which will be populated later */
/* use "i" for integer and "s" for string values */
$stmt->bind_param( 'ssssss', $line,$part,$make,$model,$year,$type );
/* access csv file */
$file=new SplFileObject( $filepath );
/* Process each row of the csv file */
while( !$file->eof() ) {
/* read the line into a variable */
$data=$file->fgetcsv();
if( !empty( $data ) ){
/* assign a variable to each field value for this row */
list( $line,$part,$make,$model,$year,$type )=$data;
/* execute statement with the now defined variables */
$stmt->execute();
}
}
/* tidy up */
$stmt->close();
$con->close();
echo 'database updated with new records from csv';
}
I will try to explain situation as well as possible:
I have script, that imports CSV file data to MS Access database.
I have 2 access Tables:
A) Users and their information(ID, name, last name etc.)
B) Table which contains data from CSV file
Problem is, data imported from file, (2nd table) contains Users name and lastname. I want to get idea, how to, while reading csv file line by line, check what name line contains, and assign userID from table 1 instead of name and lastname on table 2. It should be done while importing, because, on each import there are roughly 3k lines being imported. Any ideas appreciated. Images given bellow.
Import script:
<?php
function qualityfunction() {
error_reporting(0);
require_once '/Classes/PHPExcel.php'; // (this should include the autoloader)
require_once '/CLasses/PHPExcel/IOFactory.php';
$excel_readers = array(
'Excel5' ,
'Excel2003XML' ,
'Excel2007'
);
$files = glob('data files/quality/QA*.xls');
$sheetname= 'AvgScoreAgentComments';
if (count($files) >0 ) {
foreach($files as $flnam) {
$reader = PHPExcel_IOFactory::createReader('Excel5');
$reader->setReadDataOnly(true);
$reader->setLoadSheetsOnly($sheetname);
$path = $flnam;
$excel = $reader->load($path);
$writer = PHPExcel_IOFactory::createWriter($excel, 'CSV');
$writer->save('data files/quality/temp.csv');
/*
$filename = basename($path);
if (strpos($filename,'tes') !== false) {
echo 'true';
}*/
require "connection.php";
$handle = fopen("data files/quality/temp.csv", "r");
try {
$import= $db->prepare("INSERT INTO quality(
qayear,
qamonth,
lastname,
firstname,
score) VALUES(
?,?,?,?,?)");
$i = 0;
while (($data = fgetcsv($handle, 1000, ",", "'")) !== FALSE) {
if($i > 3) {
$data = str_replace('",', '', $data);
$data = str_replace('"', '', $data);
$import->bindParam(1, $data[1], PDO::PARAM_STR);
$import->bindParam(2, $data[2], PDO::PARAM_STR);
$import->bindParam(3, $data[3], PDO::PARAM_STR);
$import->bindParam(4, $data[4], PDO::PARAM_STR);
$import->bindParam(5, $data[7], PDO::PARAM_STR);
$import->execute();
}
$i++;
}
fclose($handle);
$removal=$db->prepare("DELETE FROM quality WHERE score IS NULL;");
$removal->execute();
}
catch(PDOException $e) {
echo $e->getMessage()."\n";
}};
Data table 1 (Users info):
Data table 2 (In which data from CSV file is imported)
Found a solution. Thanks for help.
$lastname = "lastname";
$firstname = "firstname";
$showdata = $db->prepare("SELECT userID FROM users WHERE lastname= :lastname AND firstname= :firstname");
$showdata->bindParam(':lastname', $lastname);
$showdata->bindParam(':firstname', $firstname);
$showdata->execute();
$rowas= $showdata->fetch(PDO::FETCH_ASSOC);
echo $rowas['userID'];
i am trying to read a file and then i am trying to inserted to mysql table
i have tried the follow and i get the te text and i display the text from the file but when i trying to inserted into the db the row is created but i have nothing inside
any help?
This is my code
<?php
$file_handle = fopen("text.htm", "r");
while (!feof($file_handle)) {
$line = fgets($file_handle);
echo $line;
// make the DSN
$dsn = 'mysql:host=localhost;dbname=text;';
$user = 'text';
$password = 'text';
$name=$line;
}
try
{
$dbh = new PDO($dsn, $user, $password);
// set the error mode to exception
$dbh->setAttribute(PDO::ATTR_ERRMODE,
PDO::ERRMODE_EXCEPTION);
$sql = 'INSERT INTO text (db_text)
VALUES (:name)';
$stmt = $dbh->prepare($sql);
$stmt->bindParam(':name', $name);
$stmt->execute();
}
catch (PDOException $e)
{
echo 'PDO Exception Caught. ';
echo 'Error with the database: <br />';
echo 'SQL Query: ', $sql;
echo 'Error: ' . $e->getMessage();
}
fclose($file_handle);
?>
I think you should change
$line = fgets($file_handle);
to
$line .= fgets($file_handle);
inside the while loop.
Also, instantiate $line with $line = ""; somewhere before the while-loop;
fgets will probably return false the last time (or an empty line or something).
try declaring $name before the while... it could be a scope problem....
$name = "";
$file_handle = fopen("text.htm", "r");
while (!feof($file_handle)) {
$line = fgets($file_handle);
echo $line;
// make the DSN
$dsn = 'mysql:host=localhost;dbname=text;';
$user = 'text';
$password = 'text';
$name=$line;
}
I'm trying to import CSV data into a MySQL database using the fgetcsv function.
if(isset($_POST['submit'])) {
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv") {
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0) {
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is it gets stuck in between the import process and displays the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'S',0,0)' at line 1
The file is imported only partially each time. Only between 200-300 lines out of a 10000 line file are imported.
Here is the DDL of my table:
create table vpireport (
id int not null auto_increment,
item_code int,
company_id int,
purchase double,
primary key(id),
foreign key(company_id) references users(userid)
);
I haven't been able to find the problem so far, any help appreciated. Thanks.
You probably need to escape quotes, which you could accomplish using PDO and prepared statements.
I've skipped most of your code in the example for brevity and just focused on the for loop.
<?php
// Use PDO to connect to the DB
$dsn = 'mysql:dbname=YOUR_DB;host=localhost';
$user = 'DB_USERNAME';
$password = 'DB_PASSWORD';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
// The query uses placeholders for data
$sql = "INSERT INTO vpireport
(item_code,company_id,purchase,purchase_value)
VALUES
(:item_code,:company_id,:purchase,:purchase_value)";
$sth = $dbh->prepare($sql);
// The data is bound to the placeholders
$sth->bindParam(':item_code', $data[0]);
$sth->bindParam(':company_id', $data[1]);
$sth->bindParam(':purchase', $data[2]);
$sth->bindParam(':purhcase_value', $data[3]);
// The row is actually inserted here
$sth->execute();
$sth->closeCursor();
}
That won't get rid of any problem characters, though, so you may want to look at some kind of data sanitization if that poses a problem.
uncomment the //echo "$sql"; and look what is the last query (with error) - it may be that the csv data contains strange characters or the query is cut off.
BTW: you can also import csv file by mysql:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
$row = 1;
if (($handle = fopen("albums.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ',','"')) !== FALSE) {
if($row!=1){
$num = count($data);
$albumIndex=0;
//Insert into tbl_albums
$sqlAlbums="INSERT INTO tbl_albums(albumName) VALUES ('".$data[$albumIndex]."')";
$resultAlbums=mysql_query($sqlAlbums);
}
}
$row++;
}
}
fclose($handle);