I'm using MySQL LOAD DATA INFILE to move CSV file data into a database table. I'm looking for a way to reference the original file line number (row) in the imported record.
So that a table like this:
CREATE TABLE tableName (
uniqueId INT UNSIGNED NOT NULL PRIMARY_KEY,
import_file VARCHAR(100),
import_line INT,
import_date = DATETIME,
fieldA VARCHAR(100),
fieldB VARCHAR(100),
fieldC VARCHAR(100)
);
Where import_file, import_line and import_date are meta data relevant to the specific file import. fieldA, fieldB and fieldC represent the actual data in the file.
Would be updated by a query like this:
LOAD DATA INFILE '$file'
REPLACE
INTO TABLE '$tableName'
FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATES BY '\n'
IGNORE 1 LINES # first row is column headers
(fieldA,fieldB,fieldC)
SET import_date = now(),
import_file = '" . addslashes($file) . "',
import_line = '???';
Is there a variable I can set 'import_line' to?
Thanks,
-M
You can do that by setting a user variable first, and increment this variable in your SET clause, i.e.
SET #a:=0; -- initialize the line count
LOAD DATA INFILE 'c:/tools/import.csv' -- my test import
REPLACE
INTO TABLE tableName
FIELDS TERMINATED BY ',' ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES # first row is column headers
(fieldA,fieldB,fieldC)
SET import_date = now(),
import_file = 'import.csv',
import_line = #a:=#a+1; -- save the incremented line count
Related
Sorry if this has been asked before, but I couldnt find anything that would relate to my case here on SE.
I am trying to import a CSV file into my Mysql database table with both the table the CSV having the exact same amount and order of columns, except that the table's column ID is not missing in the CSV file.
What I want to achieve is to import the CSV into the table while generating an ID number that automatically increases with each record. This does not seem possible as the CSV always seem to want to insert its data into the first colum in the table, but in my case I would need it to be the 2nd column.
How do I approach this and is there any reference code I can study? I currently am working off this PDO approach but am having the above mentioned difficulties.
PHP
<?php
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
Thank you
You can have MySQL set values for certain columns during import. If your id field is set to auto increment, you can set it to null during import and MySQL will then assign incrementing values to it.
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
SET id=null;
EDIT - In case the ID column is not present in CSV
The col1, col2, col3,... are names of actual columns in the DB table (without id column)
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
(col1, col2, col3,...)
SET id=null;
The AUTO_INCREMENT attribute can be used to generate a unique identity for new rows. Most version of mysql and engin support this. You need not worry about the ID and can use cron job to insert the needed field and AUTO_INCREMENT will take care of the id itself.
No value was specified for the AUTO_INCREMENT column, so MySQL assigned sequence numbers automatically. You can also explicitly assign 0 to the column to generate sequence numbers, unless the NO_AUTO_VALUE_ON_ZERO SQL mode is enabled. If the column is declared NOT NULL, it is also possible to assign NULL to the column to generate sequence numbers. When you insert any other value into an AUTO_INCREMENT column, the column is set to that value and the sequence is reset so that the next automatically generated value follows sequentially from the largest column value.
You can retrieve the most recent automatically generated AUTO_INCREMENT value with the LAST_INSERT_ID() SQL function or the mysql_insert_id() C API function. These functions are connection-specific, so their return values are not affected by another connection which is also performing inserts.
See example from official link :
[https://dev.mysql.com/doc/refman/5.7/en/example-auto-increment.html]
As you want to recreate the table over and over and want to manipulate the Data from the CSV, try this:
// You have to create the TABLE if not exists
$pdo->exec("TRUNCATE TABLE sample"); // No need to drop the table if columns don't change.
$csvContent = file_get_contents($csvfile); // Raw Data from file
$lines = explode("
", $csvContent); // The standard line separator is an ENTER
// Now you have each line separated
for($i = 0; $i < coount($lines); $i++) {
$col = explode(";", $lines[$i]); // Would be a comma
// Now you have each column separated
$pdo->exec("INSERT INTO sample (id, col1, col2, col3 ... coln) VALUES (NULL, '".$col[0]."', '".$col[1]."', '".$col[2]."' ... '".$col[n]."')");
}
This way you can dig into your Data and, besides setting an AUTO_INCREMENT ID, you can validate what is coming from the CSV and can correct/prevent importation errors.
I am using this query to create CSV of my mysql DB table
$path = getcwd() . '/uploads/data.csv' ;
$sql = "select email,username from `engine4_user` LIMIT 5000 into outfile '$path' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' ";
But now I need one extra field that is profile url .which doesn't exists on the table. So i wanna create a virtual url column using the ID of that user table. Like https://mysiteurl/profile/$user_id .
Please help. Thanks in advance.
Use "CONCAT" operator
$sql = "SELECT
email,
username,
CONCAT('https://mysiteurl/profile/',user_id) as profile_url
FROM `engine4_user`
LIMIT 5000
into outfile '$path' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' ";
See example here
As a workaround, I am trying to use this syntax to import my csv file into my database table instead of fgetcsv because files uploaded may be quite big. So I made this:
$connect = mysql_connect("localhost","root","");
mysql_select_db("dbtest",$connect);
$filename = $file_path;
$sql = "LOAD DATA INFILE '$filename'
INTO TABLE tbltest
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
(#rcode, #pcode, #mcode, #bcode, #ecode, #filetype, #rec_count, #ges_count, #ck_count, #ic_count, #ran_count);" ;
//echo $sql;
mysql_query ($sql,$connect) ;
Now, the problem is nothing is inserted in my table. When I do echo $sql, the result is this:
LOAD DATA INFILE 'uploads/testfile.csv' INTO TABLE tbltest FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n' (#rcode, #pcode, #mcode, #bcode, #ecode, #filetype, #rec_count,
#ges_count, #ck_count, #ic_count, #ran_count);
The testfile.csv looks like this:
01|01|01|01|0000|FORM1|5|5|5|||||||||
01|01|01|01|0000|FORM2|238|238||||||||||
01|01|01|01|0000|FORM4|0|||||||||||
01|01|01|01|0000|FORM5|1|||||||||||
01|01|01|01|0000|FORM3|3|3||||||||||
i am importing csv data into mysql database using codegniter framework. my database has two table promotion and promotion_product.promotion table is 3 fields name as promotion_id,name,start_dateand end_date.promotion_id is the primary key of promotion table.promotion_id is the auto incremment.promotion_product table is 5 fields id,promotion_id,sku,price,map. promotion_id is the foreign key of this table. i am importing csv file this table.csv field is sku,price and map.i am not getting how to insert promotion_id of promotion_porduct table.i am using last_insert_id.my code is below:
my model is:
public function save_csv($data,$data1) {
$this->db->insert('promotion', $data1);
$insert_id = $this->db->insert_id();
print_r($insert_id);
$infile = $data['upload_data']['full_path'];
//print_r($infile);
$sql = "LOAD DATA INFILE '" . $infile . "'
INTO TABLE promotion_product
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r'
IGNORE 1 LINES
(promotion_id,sku, price, map)";
print_r($sql);
$query = $this->db->query($sql);
//var_dump($query);
return $query;
}
Yo have to do manually.
First insert into database then you will get last_insert_id and then read CSV file and create array of CSV and last_insert_id dump that array into a database.
I have a PHP file that inserts data from a CSV file into the MYSQL DB using the LOAD DATA INFILE sql function.
If the CSV data is a duplicate it is not inserted because of the DB table indexing rules (unique).
$sql = "LOAD DATA LOW_PRIORITY LOCAL INFILE
'" . $makes_file . "' IGNORE
INTO TABLE make
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n' (make, img_url)";
$link->query($sql) or die(mysqli_error($link));
For some reason the Auto Increment of the table is not correct from this process.
So I have used the following code to correct this.
$get_max = $link->query("SELECT max(id) as max FROM `make` LIMIT 1");
while ($r = $get_max->fetch_assoc()) {
$link->query("ALTER TABLE `make` AUTO_INCREMENT = " . ($r['max'] + 1)) or
die(mysqli_error($link));
}
So if anybody knows:
Why the Auto Increment is incorrect from the LOAD DATA sql
or
If there is a 'prettier' way to set the auto increment to the max(id) +1
Thanks
For MyISAM tables you can just set AUTO_INCREMENT to 0 and MySQL will use the current maximum value plus one.
$link->query('ALTER TABLE make AUTO_INCREMENT = 0');
From the docs:
You cannot reset the counter to a value less than or equal to any that
have already been used. For MyISAM, if the value is less than or equal
to the maximum value currently in the AUTO_INCREMENT column, the value
is reset to the current maximum plus one.
This answer is about a prettier way to set the auto increment to the max(id) +1, You can do this with one Query only :
$link->query("ALTER TABLE `make` AUTO_INCREMENT = ( SELECT max(id)+1 FROM `make` LIMIT 1 )" );
But your first solution should work if AUTO_INCREMENT is unique , check your database sheme
Check your last id in your database manually and get it
Then insert new record and set id to
Your last id plus say 3 or 4
Example
If your last taken id is 5000
Insert new record and set id to 5003
Then auto increment will works again from 5003.