PHP - CSV import with auto-increase ID - php

Sorry if this has been asked before, but I couldnt find anything that would relate to my case here on SE.
I am trying to import a CSV file into my Mysql database table with both the table the CSV having the exact same amount and order of columns, except that the table's column ID is not missing in the CSV file.
What I want to achieve is to import the CSV into the table while generating an ID number that automatically increases with each record. This does not seem possible as the CSV always seem to want to insert its data into the first colum in the table, but in my case I would need it to be the 2nd column.
How do I approach this and is there any reference code I can study? I currently am working off this PDO approach but am having the above mentioned difficulties.
PHP
<?php
$databasehost = "localhost";
$databasename = "test";
$databasetable = "sample";
$databaseusername="test";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "filename.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
Thank you

You can have MySQL set values for certain columns during import. If your id field is set to auto increment, you can set it to null during import and MySQL will then assign incrementing values to it.
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
SET id=null;
EDIT - In case the ID column is not present in CSV
The col1, col2, col3,... are names of actual columns in the DB table (without id column)
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator))."
(col1, col2, col3,...)
SET id=null;

The AUTO_INCREMENT attribute can be used to generate a unique identity for new rows. Most version of mysql and engin support this. You need not worry about the ID and can use cron job to insert the needed field and AUTO_INCREMENT will take care of the id itself.
No value was specified for the AUTO_INCREMENT column, so MySQL assigned sequence numbers automatically. You can also explicitly assign 0 to the column to generate sequence numbers, unless the NO_AUTO_VALUE_ON_ZERO SQL mode is enabled. If the column is declared NOT NULL, it is also possible to assign NULL to the column to generate sequence numbers. When you insert any other value into an AUTO_INCREMENT column, the column is set to that value and the sequence is reset so that the next automatically generated value follows sequentially from the largest column value.
You can retrieve the most recent automatically generated AUTO_INCREMENT value with the LAST_INSERT_ID() SQL function or the mysql_insert_id() C API function. These functions are connection-specific, so their return values are not affected by another connection which is also performing inserts.
See example from official link :
[https://dev.mysql.com/doc/refman/5.7/en/example-auto-increment.html]

As you want to recreate the table over and over and want to manipulate the Data from the CSV, try this:
// You have to create the TABLE if not exists
$pdo->exec("TRUNCATE TABLE sample"); // No need to drop the table if columns don't change.
$csvContent = file_get_contents($csvfile); // Raw Data from file
$lines = explode("
", $csvContent); // The standard line separator is an ENTER
// Now you have each line separated
for($i = 0; $i < coount($lines); $i++) {
$col = explode(";", $lines[$i]); // Would be a comma
// Now you have each column separated
$pdo->exec("INSERT INTO sample (id, col1, col2, col3 ... coln) VALUES (NULL, '".$col[0]."', '".$col[1]."', '".$col[2]."' ... '".$col[n]."')");
}
This way you can dig into your Data and, besides setting an AUTO_INCREMENT ID, you can validate what is coming from the CSV and can correct/prevent importation errors.

Related

how to insert last_insert_id in sql query of csv query?

i am importing csv data into mysql database using codegniter framework. my database has two table promotion and promotion_product.promotion table is 3 fields name as promotion_id,name,start_dateand end_date.promotion_id is the primary key of promotion table.promotion_id is the auto incremment.promotion_product table is 5 fields id,promotion_id,sku,price,map. promotion_id is the foreign key of this table. i am importing csv file this table.csv field is sku,price and map.i am not getting how to insert promotion_id of promotion_porduct table.i am using last_insert_id.my code is below:
my model is:
public function save_csv($data,$data1) {
$this->db->insert('promotion', $data1);
$insert_id = $this->db->insert_id();
print_r($insert_id);
$infile = $data['upload_data']['full_path'];
//print_r($infile);
$sql = "LOAD DATA INFILE '" . $infile . "'
INTO TABLE promotion_product
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r'
IGNORE 1 LINES
(promotion_id,sku, price, map)";
print_r($sql);
$query = $this->db->query($sql);
//var_dump($query);
return $query;
}
Yo have to do manually.
First insert into database then you will get last_insert_id and then read CSV file and create array of CSV and last_insert_id dump that array into a database.

PHP & MySQL: Using a 'NOT NULL AUTO_INCREMENT' on a table that truncates

I'm really stuck here I have this PHP script:
<?php
$databasehost = "localhost";
$databasename = "";
$databasetable = "";
$databaseusername="";
$databasepassword = "";
$fieldseparator = ",";
$lineseparator = "\n";
$enclosedbyquote = '"';
$csvfile = "db-core/feed/csv/csv.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$pdo->exec("TRUNCATE TABLE `$databasetable`");
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." REPLACE INTO TABLE `$databasetable`
FIELDS OPTIONALLY ENCLOSED BY ".$pdo->quote($enclosedbyquote)."
TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."
IGNORE 1 LINES");
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
So as you can see that script is truncating my table with a CSV file. It replaces all the data currently in MySQL table.
This data is used to create a listing page that includes vehicles that are in stock, each row contains the data for one vehicle. This script is ran once a day to replace vehicles that are no longer in stock.
I now want to give each of my SQL rows it's own page, I've been told I will need to use a 'NOT NULL AUTO_INCREMENT' attribute on an 'id' key.
However seen as my table truncates wont the ID removed each time my script is ran?
Could I add anything to my script to combat this problem?
Thanks
Yes. Truncating a table is the equivalent of delete from table and nuking all the records. But the auto_increment won't reset to 0 again. The last used auto_inc value is part of the table's metadata and is not affected by a truncate operation.
If you had records 0->100, truncated the table, then added new records, they would start at #101 and climb from there.
If you want the auto_increment value to be reset after the truncation, you'd have to do:
TRUNCATE TABLE foo; // delete all records
ALTER TABLE foo SET auto_increment=1; // reset auto-increment to 1
INSERT ...
So yep... turns out truncate does reset the auto_increment value. Definitely seems wrong, since it also apparently won't process any cascade deletes while truncating. Repopulating a truncated table that is used as a foreign key elsewhere will undoubtedly lead to incorrect record joinings.

Can only insert into mysql table once

I have an issue where I can only insert data into my table once. If i delete the row and insert a new one, it works but if I already have a row and try to insert another one, it doesn't work. No errors in the console or network.
I'm inserting with this:
<?php
error_reporting(E_ALL);
include 'DB.php';
$con = mysql_connect($host,$user,$pass)
or die("Error: ".mysql_error());
$dbs = mysql_select_db($databaseName, $con);
$name = mysql_real_escape_string($_POST['name']);
$date = date('Y-m-d');
$amount = $_POST['amount'];
$timPaid = $_POST['timPaid'];
$rennyPaid = $_POST['rennyPaid'];
$sql = "INSERT INTO $tableName (`name`, `date`, `amount`, `timpaid`, `rennypaid`)
VALUES ('$name', '$date', '$amount', '$timPaid', '$rennyPaid')";
$result = mysql_query($sql, $con)
or die("Error: ".mysql_error());
mysql_close($con);
?>
I'm thinking it might have to do with how my table is set up, primary key and such. I have an id column which is the primary and I think it's auto-increment, can't tell.
Since you are not sure about whether the id field is auto-increment or not, you should alter your table like this,
ALTER TABLE `yourtable`
MODIFY COLUMN `id` int(11) NULL AUTO_INCREMENT FIRST;
the result FROM SHOW CREATE TABLE tableName would help.
I would guess you have a unique index on on of your fields and you are trying to insert a second record with the same value.
Also CHECK TABLE tablename could help identify the problem.
I had this... I had set my first column as 'unique' and my 'Insert' didn't involve that column.
As a result the 'Insert' added a value of zero into the 'Unique' column (I'd set that column to 'integer').
When I did another insert 'I THINK' that the 'Insert' wanted to add another zero in the 'Unique' column that I wasn't 'Inserting' into, so it tried to 'Insert' another zero, BUT because that column was 'unique' it wouldn't allow another zero and refused the 'Insert'.
I proved this by changing the first 'Inserts' entry into the 'Unique' column manually to another 'Integer' then the 'Insert; statement worked one more time.... repeat process above as described and my table allowed another 'Insert'.
Hope this makes sense and helps?.
I had a similar problem, however mine was where I was using the INT data type in my create table script for storing a 13-digit long number, and it only wanted to accept something 10-digits in size. Changing this to a VARCHAR(13) fixed the problem for me.

MySQL "LOAD LOCAL DATA INFILE"

I have hit a problem and i cant seem to get over it. I am taking information from a text file(two columns 29.12 23.42 for example separated by a space), using LOAD LOCAL DATA INFILE. Initially this was working with only one column in MySQL, however i have had to add an ID AUTO INCREMENT so i can use the most recent entry, in addition i want to now store the other column in my Database. Here is my code, its probably pretty obvious but I can't seem to find it:
<?PHP
$username = "root";
$password = "";
$hostname = "localhost";
$table = "Received_Data";
// Connect to Database and Table
$dbhandle = mysql_connect($hostname, $username, $password)
or die("Unable to connect to MySQL");
echo "Connected to MySQL<br>";
$selectdatabase = mysql_select_db("IWEMSData",$dbhandle)
or die("Could not select IWEMSData");
echo "Connected to IWEMSData<br>";
// Clear Current Table and ReCreate
$dt = /*"UPDATE Received_Data SET PotVal = ''";*/ "DROP TABLE Received_Data";
mysql_query($dt);
$ctb = "CREATE TABLE Received_Data
(
Current DECIMAL(30,2) NOT NULL,
PotVal DECIMAL(30,2) NOT NULL,
ID BIGINT NOT NULL AUTO_INCREMENT,
PRIMARY KEY(ID,PotVal,Current)
)";
mysql_query($ctb, $dbhandle);
echo "Received_Data Table has been created";
// Text File to read in
$Textfile = 'IWEMSData.txt';
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY ""
LINES TERMINATED BY "\\r\\n
(PotVal, Current)
";')
or die('Error Loading Data File.<br>' . mysql_error());
// Close Database connection when complete
mysql_close($dbhandle);
?>
So what i want is Column 1 is ID which will AUTO INCREMENT upon each entry. The second row is PotVal and the third row is Current. Then i want to only store in the second and third column.
The problem i am getting is that ID and Current are displaying the incorrect values and i can only seem to get one row.
Thanks in advance!
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY "" /*<- terminated by nothing? Shouldn't there be a space in it?*/
LINES TERMINATED BY "\\r\\n /*<- the closing " is missing*/
(PotVal, Current)
";') /*<- ah, here's the missing " What does it do here?*/
Write it like this:
mysql_query('
LOAD DATA LOCAL INFILE "IWEMSData.txt"
REPLACE INTO TABLE Received_Data
FIELDS TERMINATED BY " "
LINES TERMINATED BY "\\r\\n"
(PotVal, Current)
;')

mysql update from file insert if exist else update

I have a mysql table (my_table) which have 3 columns (Name, Age, Place). I have a new text file (data.txt tab separated data file) which have have 2 columns (Name and Place). I want to insert information from file "data.txt" into my_table with the condition if Name is same (in mysql table and in text file "data.txt") update row (insert Place value in 3rd column) otherwise create new row (in which Age will be NULL).
if NAME_my_tabe== Name_data.txt --> update this row (do Place_my_tabe = Place_data.txt)
if Name_data.txt not present in my_tabe --> create new row (do Name_my_table=Name_data.txt, Age_my_table=NULL, Place_my_tabe = Place_data.txt)
How to do this in mysql ??? (I want to insert values from text file)
First of all, make sure your table contains the following:
UNIQUE on Name
Age is NULLABLE
The following query will do what you need:
INSERT INTO my_table SET name = 'Name', place = 'Place'
ON DUPLICATE KEY UPDATE place = 'New Place'
With the code to read your file, you could use:
$file = new SplFileObject('data.txt');
while($file->eof() === true) {
$line = $file->fgets();
$data = explode("\t", $line);
// execute query:
$query = sprintf(
"INSERT INTO my_table SET name = '%1$s', place = '%2$s' ON DUPLICATE KEY UPDATE place = '%2$s'",
$data[0], // escape this
$data[1] // escape this
);
}
Note that you will need to execute that query using your mysql client library.

Categories