I am using the below code to import a huge csv file.(almost 42452 lines):
<?php
$databasehost = "localhost";
$databasename = "framework_db";
$databasetable = "Main_CSV_Table";
$databaseusername="root";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "";
$csvfile = "main.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)."
INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."
IGNORE 1 LINES
(Code,Name,Trx,DocType,Date,Period,Ref,Description,Category,Dr,Cr)
;");
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
Table Structre:
But when I import this csv to mysql, sometimes the data is inserted in wrong columns like below:
Also there is an another issue. When I import the file of 42452 lines, it only imports 38777 lines.(upload_max_filesize = 500M is in php.ini)
I would be grateful if someone could help me on this problem?
Thanks in advance!
UPDATE
CSV File is give below:
Related
I am using the below php script and can't figure out why it is not working. I do get the echo that it processed with no errors in terminal, but the table remains empty. Is there something incorrect with the code causing this. I confirmed it is locating the file in the directory as well. Hopefully something simple is missing.
<?php
try {
$inputfile1 = '////MainDirectory/SubDiretory/data.csv';
$table1 = 'database.testTable1';
$time1 = microtime(true);
require_once("configFile.php"); //this changes the below as now this is the reference point
$mysql_host = DB_HOST;
$mysql_database = DB_NAME;
$mysql_username = DB_USER;
$mysql_password = DB_PASS;
$db = new PDO("mysql:host=$mysql_host; dbname=$mysql_database", $mysql_username, $mysql_password);
// set the PDO error mode to exception
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db->exec("SET CHARACTER SET utf8"); // Sets encoding UTF-8
} catch (Exception $e) {
die("Unable to connect: " . $e->getMessage());
}
try {
// Return errors
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// Begin transaction
$db->beginTransaction();
if(file_exists($inputfile1)) {
$Found = 'found';
// Query D2 Load Temp table
$tempA2 = $db->prepare("LOAD DATA INFILE :inputfile1
INTO TABLE $table1
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(column1, column2, column3);");
$tempA2->bindParam(':inputfile1', $inputfile1);
$tempA2 -> execute();
}
} catch (Exception $e) {
// If transaction fail, use checkpoint and rollback
$db->rollBack();
echo "Conversion failed: " . $e->getMessage().'<br />';
file_put_contents('PDOErrors.txt', $e->getMessage(), FILE_APPEND);
}
//time of script
//sleep(1);
$timeC = microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"];
echo "Full Process Time: {$timeC} $Found";
?>
CSV format:
"HeaderData1","HeaderData2","HeaderData3","HeaderData4","HeaderData5"
"DataA","DataB","DataC"
"Data2","Data2","Data2"
MySQL table format
id INT PK with AI and NN
column1 Varchar
column2 Varchar
column3 Varchar
Thank you for any help
I'm trying to Upload a CSV-File into a MySQL Database via the PHP PDO function. The CSV is uploaded by a html form and saved on the server. After this I do some validation (I have exclude this from the code below). However I was pretty sure it was functional and I uploaded several times successful a CSV-Files, but after a week it stopped working with the following error code:
Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 7890 Can't find file
This is the code im using:
$csv_upload_status = "none";
$sec_pass = "4220334";
$databasehost = "127.0.0.3";
$databasename = "database_name";
$databasetable = "table_name";
$databaseusername="database_user";
$databasepassword = "XXXXXXXXXXXX";
$fieldseparator = ",";
$lineseparator = "\r";
if(isset($_POST["submit"])) {
$valid = 1;
$target_dir = "schedule-uploads/";
$target_file = $target_dir . basename($_FILES["csv"]["name"]);
$FileType = pathinfo($target_file,PATHINFO_EXTENSION);
if ($valid == 1) {
move_uploaded_file($_FILES["csv"]["tmp_name"], $target_file);
$csvfile = basename( $_FILES["csv"]["name"]);
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."
(#date, show_time, show_title, show_plot)
SET show_date = STR_TO_DATE(#date, '%m/%d/%Y');");
$csv_upload_status = "success";
}
}
I am trying to import data from a CSV file to my table using PHP. I have tried using the exact same code without the backslash for ENCLOSED BY '"' on phpmyadmin and the import is successful.
I have also checked the user permissions for admin.
Here is my code:
<?php
error_reporting(E_ALL); ini_set('display_errors', 1);
$pdo = new PDO('mysql:dbname=mydatabase;host=localhost;charset=utf8', 'admin', 'admin');
$pdo->exec('SET CHARACTER SET utf8');
$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
try{
$query = $pdo->prepare("LOAD DATA LOCAL INFILE 'C:\feed.csv' IGNORE INTO TABLE tablename
fields terminated by ','
enclosed by '\"'
lines terminated by '\n'
IGNORE 1 LINES
(field1, field2, field3, field4)", array(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => true));
$query->execute();
$query->fetchAll();
} catch (PDOException $e) {
echo 'error: ' . $e->getMessage();
}
?>
When running this code I am seeing this error:
General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute.
I don't no what is wrong in your code but you can use my code it is help full to you I think
I have implement this code and it is tested code. I think it is very use full
You have follow some rule:-
1.your csv file according to database table name (ex: db table name is users then csv should be users.csv)
2.Your csv file's first row should be db table fields name (ex: Id, name etc) after the start your data entry
3.you can download data source class from :- http://code.google.com/p/php-csv-parser/ because i have require below the code: require_once 'CSV/DataSource.php';
<?php
ini_set('memory_limit','512M');
$dbhost = "localhost";
$dbname = "excel_import";
$dbuser = "root";
$dbpass = "";
$conn=mysql_connect ($dbhost, $dbuser, $dbpass) or die ("I cannot connect to the database because: " . mysql_error());
mysql_select_db($dbname) or die("Unable to select database because: " . mysql_error());
require_once 'CSV/DataSource.php';
$filename = "users.csv";
$ext = explode(".",$filename);
$path = "uploads/".$filename;
$dbtable = $ext[0];
import_csv($dbtable, $path);
function import_csv($dbtable, $csv_file_name_with_path)
{
$csv = new File_CSV_DataSource;
$csv->load($csv_file_name_with_path);
$csvData = $csv->connect();
$res='';
foreach($csvData as $key)
{
$myKey ='';
$myVal='';
foreach($key as $k=>$v)
{
$myKey .=$k.',';
$myVal .="'".$v."',";
}
$myKey = substr($myKey, 0, -1);
$myVal = substr($myVal, 0, -1);
$query="insert into ".$dbtable." ($myKey)values($myVal)";
$res= mysql_query($query);
}
if($res ==1)
{
echo "record successfully Import.";
}else{
echo "record not successfully Import.";
}
}
I have a php file that imports a CSV file into my database. Since my hosting provider upgraded to PHP 5.4, I get an error on the line 'LINES TERMINATED BY ', but I am not sure why. Here is my code:
<?php
require('phpsqlajax_dbinfo.php');
$databasehost = $db_host;
$databasename = $db_name;
$databasetable = "tbl_csvImport";
$databaseusername=$db_user;
$databasepassword = $db_pass;
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "doecsv.csv";
echo $lineseparator;
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$pdo->exec("DELETE FROM `$databasetable`");
try {
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".preg_replace('/"[^"]+"/','',$pdo->quote($csvfile))." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
OPTIONALLY ENCLOSED BY ".$pdo->quote('"')."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."IGNORE 1 LINES");
} catch (Exception $e) {
die ("CSV Parse Failed: ".$e->getMessage()." | Error on line: ".$e->getLine());
}
echo" Success. <br/>";
?>
Error Code: SQLSTATE[42000]: Syntax error or access violation: 1148 The used command is not allowed with this MySQL version | Error on line: 35
I found a workaround that works for me:
http://crewow.com/CSV-Importer-in-MySQL.php
This question already has answers here:
Import CSV file directly into MySQL
(6 answers)
Closed 9 years ago.
I have a CSV file that I'm wanting to import into a database, and then export the same data with omitted columns. How would I go about doing this? Any suggestions?
<?php
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>