I have CSV file with header and data fields. How can I insert this via SQL query and remove headers. At the moment code that I wrote is working, but it is inserting header as a data into DB.
The code:
$files = directory_map('./assets/csv/');
foreach ($files as $file) :
$filefile = './assets/csv/' . $file;
$q = "LOAD DATA LOCAL INFILE '$filefile' INTO TABLE person FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n' (personal_name, personal_lastname,
personal_country,personal_address,contact_email,dateadded);";
$this->db->query($q);
endforeach;
You can use the ignore xx lines clause in load data (the doc on this)[http://dev.mysql.com/doc/refman/5.1/en/load-data.html] skip down about a third of the way:
$q = "LOAD DATA LOCAL INFILE '$filefile' INTO TABLE person FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n' ignore 1 lines (personal_name, personal_lastname,
personal_country,personal_address,contact_email,dateadded);";
This will cause the input to skip 1 line of the CSV - which I am assuming is your headers. Skip however many lines are needed to get to the data itself.
Try this: Hope so this will work for you..
$c = 1;
$files = directory_map('./assets/csv/');
foreach ($files as $file) :
if($c==2;) continue;
$filefile = './assets/csv/' . $file;
$q = "LOAD DATA LOCAL INFILE '$filefile' INTO TABLE person FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (personal_name, personal_lastname, personal_country,personal_address,contact_email,dateadded);";
$this->db->query($q);
$c++;
endforeach;
Related
I'm trying to upload my csv file using mysql's Load Data infile syntax, but upon uploading the data on my database it loads only half the rows and I notice its a patchy row. It's like all of even numbers only have inserted and the odd does not.
Here's my sample format of csv file:
storeid,txndate,productcategory1,qty,totalamt,uploaddate
"1100","19JAN2019","ADD ONS","1363","333.59","20JAN2019"
"1100","19JAN2019","KFC LP","58","1736.96","20JAN2019"
"1100","19JAN2019","KFC SP","269","1093.02","20JAN2019"
"1100","19JAN2019","LTO","26","495.39","20JAN2019"
"1100","19JAN2019","VALUE","71","534.13","20JAN2019"
"1102","19JAN2019","ADD ONS","244","32.5","20JAN2019"
"1102","19JAN2019","KFC LP","9","239.91","20JAN2019"
"1102","19JAN2019","KFC SP","70","277.63","20JAN2019"
"1102","19JAN2019","LTO","3","88.48","20JAN2019"
"1102","19JAN2019","VALUE","18","99.95","20JAN2019"
"1104","19JAN2019","ADD ONS","930","124.32","20JAN2019"
"1104","19JAN2019","KFC LP","21","680.79","20JAN2019"
"1104","19JAN2019","KFC SP","196","971.11","20JAN2019"
"1104","19JAN2019","LTO","17","338.84","20JAN2019"
Here's my load data infile script
$loadData = sprintf("
LOAD DATA local INFILE '%s' IGNORE INTO TABLE prodmix
CHARACTER SET UTF8 FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\"'
LINES TERMINATED BY '\\n' IGNORE 1 LINES
(
#storeid,
#txndate,
#productcategory1,
#qty,
#totalamt,
#uploaddate
)
SET
id=null,
storeid=#storeid,
txndate=#txndate,
productcategory1=#productcategory1,
qty=#qty,
totalamt=#totalamt,
uploaddate=#uploaddate,
unique_row=CONCAT(#txndate,'_',#storeid,'_',
#productcategory1,'_',#qty,'_',#totalamt,'_',#uploaddate),
created_at=now()
", addslashes($absolute_path));
if(DB::connection()->getpdo()->exec($loadData)){
//$total_success = $total_success +1;
$response['status'] = 'success';
}
What might be causing every other row to fail.
Are you sure your local data csv file line termination character is '\n' and not for example '\r\n' ?
I'm trying to take a TSV file and 'POST'ed inputs and load the TSV file's contents into a DB table, replacing any existing data for specified columns. The TSV may contain any number of columns and rows and the 1st row specifies the columns that are supposed to be modified.
My problem concerns data in columns that ARE NOT supposed to be modified when running the code-generated LOAD DATA INFILE ... REPLACE INTO TABLE ... MySQL statement. When I run my code (see below), data of columns that are NOT specified in $columnsText (which is generated from the 1st row of the TSV file) end-up getting set to NULL or their default value. On the other hand, data of columns that ARE specified in $columnsText have their contents replaced just as intended.
An example of the MySQL statement that is generated by my code and is working as described above is:
LOAD DATA INFILE 'C:\\MyProject\\public\\1459772537-cities7.tsv' REPLACE INTO TABLE cities FIELDS TERMINATED BY ' ' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES (id,UNLOCODE,name_english,UN_subdiv) -- for all TSV-file mentioned rows, this statement will update the mentioned columns (id,UNLOCODE,name_english,UN_subdiv) correctly, but then all unmentioned columns for that row will be set to NULL!
How do I modify this code to keep the data of unspecified columns from being set to their default/NULL values? Or more simply, getting to the root of the problem, how do I fix the MySQL statement that is being generated to achieve my objective?
I'm using PHP with Laravel.
// Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
echo "POST has file <br>";
$file = Input::file('file');
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "location: ".$pathName."<br>";
// Determine whether to use IGNORE OR REPLACE in MySQL query.
if (isset($_POST['replace']) && $_POST['replace'] == true){
$ignoreOrReplace = "REPLACE";
}
else {$ignoreOrReplace = "IGNORE";}
echo "ignore or replace: ".$ignoreOrReplace."<br>";
// Determine columns to insert in DB, based on values of input file's 1st row.
$columnsText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columns = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columns as $column){
if ($firstIteration){$firstIteration=false;}
else {$columnsText .= ",";}
$columnsText .= $column;
}
echo "DB columns to load: ".$columnsText;
fclose($handle);
}
$query = sprintf(
"LOAD DATA INFILE '%s' %s INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$ignoreOrReplace,$_POST['mytable'],$columnsText
);
echo "<br>Here's the query: ".$query."<br>";
echo "<br><br> Database update should be complete!<br><br>";
echo 'Return to Home Page<br>';
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // I forgot what this does.
return DB::connection()->getpdo()->exec($query);
The documentation states:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index as an existing row. See Section 13.2.8, “REPLACE Syntax”.
REPLACE is not UPDATE. REPLACE is a MySQL extension to the SQL that first deletes the row if it exists, then it inserts the new one.
On INSERT, MySQL uses the default values for the fields that are not provided in the query. These fields probably default to NULL in your case.
There is no way to update the existing rows using LOAD DATA INFILE.
I suggest you to create a working table and use it only for the purpose of loading data into it as follows:
TRUNCATE it before using it.
LOAD DATA INFILE in it.
Join it against the table you want to update and use UPDATE on the join to copy the fields you need from the working table to the final table.
Use INSERT ... SELECT to get from the join the rows that are not in the final table and insert them.
TRUNCATE it.
Don't delete the table after it is used, you'll need it again next time. The last step aims to keep its disk usage at a minimum; the table definition doesn't take much space.
The answer provided by axiac is broadly-speaking the correct answer.
In case it is useful to anyone, I have included below the specific code (PHP/Laravel/MySQL) that solved my problem. I can't necessarily say this is the most efficient way to solve this problem, but it it working! :)
// (1) setup
DB::connection()->disableQueryLog();
// (2) Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
$file = Input::file('file');
}
else {
echo "<br>Input file not found! Please review inputed information.<br>";
return null;
}
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "Input file location: ".$pathName."<br>";
// (3) Determine main table and staging table.
$mainTable = $_POST['mytable'];
$stagingTable = $_POST['mytable'].'_staging'; // All staging tables are named: 'standardtable_staging'.
// (4) Determine destination DB table's columns and columns to be inserted into that table (based on values of input file's 1st row).
$columnsMain = Schema::getColumnListing($mainTable);
$columnsInput = [];
$columnsInputText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columnsInput = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columnsInput as $columnInput){
if ($firstIteration){$firstIteration=false;}
else {$columnsInputText .= ",";}
$columnsInputText .= $columnInput;
}
echo "<br>DB columns to load: ".$columnsInputText."<br>";
fclose($handle);
}
// (5) Create a new empty staging table.
$statement = "DROP TABLE IF EXISTS ".$stagingTable; // we drop rather than truncate b/c we want to re-determine columns.
DB::connection()->getpdo()->exec($statement);
$statement = "CREATE TABLE ".$stagingTable." LIKE ".$mainTable;
DB::connection()->getpdo()->exec($statement);
// (6) The staging table only needs to have columns that exist in the TSV file, so let's minimize its columns.
$columnsToDrop = [];
foreach ($columnsMain as $columnMain){
if (! in_array($columnMain,$columnsInput)){
array_push($columnsToDrop,$columnMain);
}
}
if (count($columnsToDrop) > 0){
Schema::table($stagingTable, function($t) use ($columnsToDrop) {$t->dropColumn($columnsToDrop);});
}
// (7) Load data to the staging table.
$statement = sprintf(
"LOAD DATA INFILE '%s' INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$stagingTable,$columnsInputText
);
echo "<br>Here's the MySQL staging statement: <br>".$statement."<br>";
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // don't actually recall why I put this here.
DB::connection()->getpdo()->exec($statement);
// (8) 'INSERT...ON DUPLICATE KEY UPDATE' is used here to get data from staging table to the actually-used table.
// Note: Any new columns in the staging table MUST already be defined in the main table.
$statement = sprintf("INSERT INTO %s (%s) SELECT * FROM %s ON DUPLICATE KEY UPDATE ", $mainTable,$columnsInputText,$stagingTable);
$firstClause = true;
foreach ($columnsInput as $columnInput) {
if (strtoupper($columnInput) != "ID"){
if ($firstClause){$firstClause=false;}
else {$statement .= ", ";}
$clause = $mainTable.".".$columnInput." = IF (".$stagingTable.".".$columnInput." <=> NULL,".
$mainTable.".".$columnInput.",".
$stagingTable.".".$columnInput.")";
$statement .= $clause;
}
}
echo "<br>Here's the staging-to-actual-table statement:<br>".$statement."<br>";
DB::connection()->getpdo()->exec($statement);
echo "<br>New information added to database!<br>";
I am using this code to load a file in my database. It is a php file that I upload on putty:
$Db->query('LOAD DATA LOCAL INFILE \'/name 03.11.2015.csv\'
INTO TABLE '.$in_table.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
IGNORE 8 ROWS
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name= \'name\'
;');
Now I want to have a variable instead of a specific filename in the first line of the query :
$Db->query('LOAD DATA LOCAL INFILE \'/path/name 03.11.2015.csv\'
So, I am using that and it seems that it is working :
$date = '.....';
$name = '.....';
$Db->query('LOAD DATA INFILE \'/path/'.$name.' '.$date.'.csv\'
INTO TABLE '.$in_table.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
IGNORE 8 ROWS
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name= \'name\'
;');
But now I have to change my code because some files don't have this format in their filename : '$name $date'. Some examples are given below :
name 2015-10-10
rand name 2015-10-10
How can I check the filename given that the name is a specific value? I want to concat the filename and get the name and the date in order to find the file and then use the original filename in my code.
I found a solution for this question but I still have an unanswered which is related to this one (link in the end). The answer for this question is here in this code :
$searchString = 'aaaa';
$Dates = array();
// Get all the files in my folder with the extension ".xlsx"
$files = glob('/path/*.xlsx');
// I create an array where I save all the .xlsx files that contain "aaaa" in the filename
$filesFound = array();
foreach($files as $file) {
$name = pathinfo($file, PATHINFO_FILENAME);
// Determines if there is a date and if the search string is in the filename. If yes, it puts the date and the filename in the arrays I created before
if((strpos(strtolower($name),strtolower($searchString))) && (preg_match('~(\d{2}\.\d{2}\.\d{4})~', $name, $matches))) {
$filesFound[] = $name;
$Dates[] = $matches[1];
foreach($filesFound as $ftbu){
$sql = 'LOAD DATA LOCAL INFILE \'/path/'.$ftbu.'.xlsx\' INTO TABLE '.$dbtable.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name = \'AAA\'
';
$Db->query($sql);
echo $Db->error;
}
}
}
Link:
Unanswered Question
# download the file off the internet
$file = file_get_contents("http://localhost/sample.csv");
$filepath = "C:/xampp/htdocs/test/file/sample.csv";
file_put_contents($filepath, $file);
# load the data into sample table
$pt1 = "LOAD DATA LOCAL INFILE ";
$pt2 = "'/C:/xampp/htdocs/test/file/sample.csv' INTO TABLE sample ";
$pt3 = "FIELDS TERMINATED BY ',' ENCLOSED BY '\"' ";
$pt4 = "LINES TERMINATED BY '\r\n' ";
$pt5 = "(col1,col2,col3,col4)";
$sqlquerynew = $pt1.$pt2.$pt3.$pt4.$pt5;
mysql_query($sqlquerynew);
This piece of code works on non-csv (well, I tested it with a text file instead).
Before this part gets run, I have to create a table. The table is now created, but no data is loaded. The file stated in the path exists.
What could be the problem?
Thanks
This is a sample csv I found online
"REVIEW_DATE","AUTHOR","ISBN","DISCOUNTED_PRICE"
"1985/01/21","Douglas Adams",0345391802,5.95
"1990/01/12","Douglas Hofstadter",0465026567,9.95
....... etc
Two problems
$pt2 = "'/C:/xampp/htdocs/test/file/sample.csv' INTO TABLE sample ";
Remove the / in front of C.
and secondly,
$pt5 = "(col1,col2,col3,col4)";
Make sure you have the right name for the columns. If you want to import * all columns, just remove it. It is also a good idea to remove the header in your case, because removing $pt5 you will succeed, but the header row will be added to the table.
I'm trying to upload this csv file to mysql on my hosting.
The csv is built by two columns:
alias, id
and than each row contains the data.
Here is an image
But the mysql rejects me.
why is that?
Create a table "test_table " with two fields "alias" and "id", then execute this command:
LOAD DATA LOCAL INFILE '/importfile.csv'
INTO TABLE test_table
IGNORE 1 LINES
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(alias, id);
if your are using phpMyAdmin use the import option of phpMyAdmin. Choose CSV file format
You can also open up csv from excel and generate series of insert statements. Not the best solution but might be useful if you're looking for something quick and dirty.
Solution using PHP
$file = 'path/to.csv';
$lines = file($file);
$firstLine = $lines[0];
foreach ($lines as $line_num => $line) {
if($line_num==0) { continue; } //escape the header column
$arr = explode(",",$line);
$column1= $arr[0];
$column2= $arr[1];
echo $column1.$column2."<br />";
//put the mysql insert statement here
}