I have a database that I am loading a large csv file into. Until recently the script below worked without any issues, and I have not made any changes to the script itself, however it has suddenly stopped working.
An example of the CSV file is
"Offer.ID","Offer.PhoneCost","Offer.TotalCost","Offer.MonthlyCost","Offer.FreeGift","Offer.FreeGiftImage","Offer.FreeGiftCategory","Offer.FullFreeGift","Offer.OfferCashback","Offer.AutoCashback","Offer.Clearance","Offer.OfferMins","Offer.OfferTxts","Offer.OfferRental","Offer.OfferLength","Offer.OfferText","Offer.Link"
"8676820","0.00","165.00","13.75","","","","15.00 Guaranteed Cashback","15.00","1","0","0","0","15.00","0","£15.00 Automatic Cashback","http://www.urmob.co.uk/t/a/psav89/URMOB-xmakex-xmodelx-xtariffx/track.php%253fid=8676820"
and the PHP script I'm using is...
<?php
require_once 'dbconnect.php';
$dbh = db_connect();
$log = fopen("database-log.txt", "w");
$time = date("Y-m-d H:i:s");
//start the log
$logstart = "------------------------------------------------\n" . $time . " log start\n------------------------------------------------\n\n";
fwrite($log, $logstart);
//clean the temp table
$query = "TRUNCATE TEMPDATA";
$trunc = mysqli_query($dbh, $query);
if ($trunc) {
fwrite($log, "TEMPDATA cleared\n");
} else {
fwrite($log, "TEMPDATA failed\n");
}
//load affiliate window csv
$query = "load data local infile '/home/data.csv' into table rim6jtvnox6vmwxk.TEMPDATA FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 ROWS";
$update = mysqli_query($dbh, $query);
fwrite($log, $update);
if ($update) {
fwrite($log, "TEMPDATA Table updated with CSV\n");
} else {
fwrite($log, "Failed CSV update - " . mysqli_error($dbh) . "\n");
}
mysqli_close($dbh);
$endtime = date("Y-m-d H:i:s");
$logend = "\n\n------------------------------------------------\n" . $endtime . " log end\n------------------------------------------------\n";
//end the log
fwrite($log, $logend);
fclose($log);
?>
I know that the connection to the database is working without any issues, and can see the database being cleared. But the load script doesn't seem to execute and doesn't give any error from mysqli_error either.
Related
'LINES TERMINATED BY' PROBLEM - PHP, SQL database, CSV file'
Im learning how to update database with PHP, from local file.
It is working fine, except just the last entry from my CSV file is inserted into all fields in database table.
I've tried inspecting CSV with HEX, and also all thinkable versions of LINES TERMINATED BY (\r\n, \n etc.).
Here is my CSV file
This image shows the code was executed properly.
https://imgur.com/p13OAoj
Here is my PHP code:
$dbhost = 'hidden';
$dbuser = 'hidden';
$dbpass = 'hidden';
$dbname = 'hidden';
// connect to the database
$conn = mysqli_connect($dbhost, $dbuser, $dbpass, $dbname);
mysqli_options($conn, MYSQLI_OPT_LOCAL_INFILE, true);
// check connection
if(!$conn){
echo 'Connection error: '. mysqli_connect_error();
}
// create temporary table
$create = 'CREATE TABLE tmp
(
name_id varchar(255),
phone varchar(255),
INDEX (name_id)
)';
// check if creation was successful
if(mysqli_query($conn, $create)){
echo "Records were updated successfully CREATE.";
} else {
echo "ERROR: Could not able to execute $create. " . mysqli_error($conn);
} ?><br><br><?php
// load data from local file into temp table
$load = "LOAD DATA LOCAL INFILE 'try1.csv'
INTO TABLE tmp
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n' IGNORE 1 LINES
(#name_id, #phone)" ;
// check if load was successful
if(mysqli_query($conn, $load)){
echo "Records were updated successfully LOAD.";
} else {
echo "ERROR: Could not able to execute $load. " . mysqli_error($conn);
} ?><br><br><?php
// set/update temp table with info from csv file
$set = "UPDATE tmp SET
name_id = #name_id,
phone = #phone "
;
// check if update was successful
if(mysqli_query($conn, $set)){
echo "Records were updated successfully SET.";
} else {
echo "ERROR: Could not able to execute $set. " . mysqli_error($conn);
}
mysqli_close($conn);
I would expect all 4 entries in CSV file to be inserted, instead I get the last rows values in all 4 entries in database.
This image shows the result in my database table.
https://i.imgur.com/2lKB2Ix.png
You need to modify two things in your code:
You don't need to update your data if you just have added the data.
The names of the columns in your load data are incorrect (you shoulkd remove the #), so you are inserting null lines. (If you only have those two columns in your table, you don't need to specify the names, so you can remove them too.)
Modify your code as:
$dbhost = 'hidden';
$dbuser = 'hidden';
$dbpass = 'hidden';
$dbname = 'hidden';
// connect to the database
$conn = mysqli_connect($dbhost, $dbuser, $dbpass, $dbname);
mysqli_options($conn, MYSQLI_OPT_LOCAL_INFILE, true);
// check connection
if(!$conn){
echo 'Connection error: '. mysqli_connect_error();
}
// create temporary table
$create = 'CREATE TABLE tmp
(
name_id varchar(255),
phone varchar(255),
INDEX (name_id)
)';
// check if creation was successful
if(mysqli_query($conn, $create)){
echo "Records were updated successfully CREATE.";
} else {
echo "ERROR: Could not able to execute $create. " . mysqli_error($conn);
} ?><br><br><?php
// load data from local file into temp table
$load = "LOAD DATA LOCAL INFILE 'try1.csv'
INTO TABLE tmp
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n' IGNORE 1 LINES
(name_id, phone)" ;
// check if load was successful
if(mysqli_query($conn, $load)){
echo "Records were updated successfully LOAD.";
} else {
echo "ERROR: Could not able to execute $load. " . mysqli_error($conn);
} ?><br><br><?php
mysqli_close($conn);
I am new to PHP
I am trying to load a large 14MB .csv into the mysql table.
But it is not fully uploaded into db, probably due to large file (~400000 rows). ERROR page took too long to respond.
Is there any faster way to do it.
My DB on Amazon RDS, PHP on EC2.
My current code is
<?php
header('Access-Control-Allow-Origin: *');
require "../config.php";
//$user_id = $_REQUEST['user_id'];
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
// path where your CSV file is located
define('CSV_PATH','./');
$csv_file = CSV_PATH . "data_unique.csv";
if (($handle = fopen($csv_file, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
$col[$c] = $data[$c];
}
$col1 = $col[0];
$col2 = $col[1];
$col3 = $col[2];
$col4 = $col[3];
$col5 = $col[4];
$col6 = $col[5];
// SQL Query to insert data into DataBase
$query = "INSERT INTO uniqueid_master(autoid,package_id,unique_id,user_id,issued,book_code) VALUES('".$col1."','".$col2."','".$col3."','".$col4."','".$col5."','".$col6."')";
$result = $conn->query($query);
}
fclose($handle);
}
echo "File data successfully imported to database!!";
$conn->close();
?>
I think you should try LOAD DATA Mysql statement.This will be very fast since you don't have to read everything into php.
mysqli_query($dblink, '
LOAD DATA LOCAL INFILE "'.$file.'"
INTO TABLE transactions
FIELDS TERMINATED by ","
OPTIONALLY ENCLOSED BY "\'"
LINES TERMINATED BY "\n"
');
could be write like this too:
$sql = "LOAD DATA LOCAL INFILE '/path/to/file.csv'
REPLACE INTO TABLE table_name FIELDS TERMINATED BY ','
ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES";
$result = $mysqli->query($sql);
OR
For an alternate method : Refer to this question too
This will improve some of PHP performance variables.
ini_set('memory_limit','-1');
ini_set('max_execution_time', 0);
This might solve your problem. But , it could still be possible, that, you reach a memory error. In that case, divide the csv file into multiple chunks and handle them one after another.
This is a good way to handle it.
How to extract data from csv file in PHP
I create a CSV file:
$newFileName = "myfile.csv";
$fileHandle = fopen($newFileName,"w");
foreach ($data as $line){
fputcsv($fileHandle, array($line['id'], $line['name'],$line['path']));
}
fclose($fileHandle);
if (!empty($fileHandle)) {
echo "CSV was successfully created";
} else {
echo "Error creating CSV";
}
and store it into the mySQL database:
$sql = "LOAD DATA LOCAL INFILE 'myfile.csv'
INTO TABLE test
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(id,name,path)";
$con=mysqli_connect("localhost","dbuser","dbpassword","dbname");
if (mysqli_connect_errno()) {
echo "Failed to connect to MySQL: " . mysqli_connect_error();
};
$result = mysqli_query($con, $sql);
if (mysqli_affected_rows($con) == 1) {
$message = "The data was successfully added!";
} else {
$message = "The user update failed: ";
$message .= mysqli_error($con);
};
echo $message;
mysqli_close($con);
To me the separation with "comma" doesn't seem to be very secure. I am afraid, if in the path for example, there will be a "comma" somewhere the data will be stored incorrect into the database. Can you suggest me a unique or more secure separation method of fields for the cvs file?
I am trying to load a 40 mg .CSV file into MYSQL database using MySQLi and PHP but I am getting only The user update failed: message (Witdout Error Message!) after loading the page
<?PHP
define ( 'DB_HOST', 'localhost' );
define ( 'DB_USER', 'root' );
define ( 'DB_PASS', '' );
define ( 'DB_NAME', 'map' );
$con = new mysqli(DB_HOST,DB_USER,DB_PASS,DB_NAME);
if (mysqli_connect_errno()) {
echo "Failed to connect to MySQL: " . mysqli_connect_error();
};
$sql = "LOAD DATA INFILE 'C:/wamp/www/UP/Modified_Single.csv'
INTO TABLE `single-tbl`
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES;";
$result = mysqli_query($con, $sql);
if (mysqli_affected_rows($con) == 1) {
$message = "The data was successfully added!";
} else {
$message = "The user update failed: ";
$message .= mysqli_error($con);
};
echo $message;
mysqli_close($con);
Can you please let me know why this is happening?
Your test for success is wrong. mysqli_affected_rows() returns the number of rows that were inserted into the table, which should be the same as the number of lines in the CSV file. I doubt a 40-meg file is only 1 line, so testing this for == 1 is wrong.
If you want to know if the query was successful, test $result.
if ($result) {
$message = "The data was successfully added!";
} else {
$message = "The user update failed: " . mysqli_error($con);
}
Hope someone can help me with what I think will be something minor (I'm still learning...). I'm trying to write the entire contents of a CSV File server based to an SQL database here is the code I presently have. The line // out writes perfectly and generates a new record. The $ar0 values generate no entries into the table named order - even though the csv file is about 100 lines long I just get
Error: INSERT INTO order (Picker,Order_Number,Timestamp,System)values ('','','','')
$file = "Pal.ORD.csv";
$tbl = "order";
$f_pointer=fopen("$file","r"); // file pointer
while(! feof($f_pointer)){
$ar=fgetcsv($f_pointer);
//$sql="INSERT INTO `order` (Picker,Order_Number,Timestamp,System)values ('Me','9999','23-01-2015','ORD')";
$sql="INSERT INTO `order` (Picker,Order_Number,Timestamp,System)values ('$ar[0]','$ar[1]','$ar[2]','$ar[3]')";
echo $sql;
echo "<br>";
}
if ($connect->query($sql) === TRUE) {
echo "New records created successfully";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
What I think may be going on is that your file probably has an empty line/carriage return as the last line in the file and is using that to insert the data as blank entries.
I can't be 100% sure about this since you have not provided a sample of your CSV file, however that is what my tests revealed.
Based on the following CSV test model: (Sidenote: blank lines will be ignored)
a1,a2,a3,a4
b1,b2,b3,b4
c1,c2,c3,c4
Use the following and replace with your own credentials.
This will create a new entry/row for each line found in a given file based on the model I have provide above.
<?php
$DB_HOST = 'xxx';
$DB_USER = 'xxx';
$DB_PASS = 'xxx';
$DB_NAME = 'xxx';
$db = new mysqli($DB_HOST, $DB_USER, $DB_PASS, $DB_NAME);
if($db->connect_errno > 0) {
die('Connection failed [' . $db->connect_error . ']');
}
$file = "Pal.ORD.csv";
$delimiter = ',';
if (($handle = fopen("$file", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
foreach($data as $i => $content) {
$data[$i] = $db->real_escape_string($content);
}
// echo $data[$i].""; // test only not required
$db->query("INSERT INTO `order`
(Picker, Order_Number, Timestamp, System)
VALUES ('" . implode("','", $data) . "');");
}
fclose($handle);
}
if($db){
echo "Success";
}
else {
echo "Error: " . $db->error;
}
At a quick glance it seems like this:
$f_pointer=fopen("$file","r"); // file pointer
Should be this:
$f_pointer=fopen($file,"r"); // file pointer
You might not be reading anything from the file. You can try outputting the file contents to see if that part is working, since you've confirmed that you can insert into the DB.