I spent the last few hours in a paradox. I use this query to import a CSV file into my database.
$query = $this->pdo->prepare('
LOAD DATA LOCAL INFILE
:file
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
:fields_terminated_by
OPTIONALLY ENCLOSED BY
:fields_optionally_enclosed_by
LINES TERMINATED BY
:lines_terminated_by
IGNORE 1 LINES
(
`aaa`,
`bbb`
)
SET
task_id = :task_id
');
$params = [
'file' => $this->path_to_file,
'fields_terminated_by' => $fields_terminated_by,
'fields_optionally_enclosed_by' => $fields_optionally_enclosed_by,
'lines_terminated_by' => $lines_terminated_by,
'task_id' => $this->task_id
];
$query->execute($params);
The entire file with its settings (separator, enclosure, line termination etc.) arrives from an OAuth-based API from end-users. I am telling you this to underline that obviously I cannot trust user input hence I must use placeholders.
The problem is that the way PDO escapes such parameters is odd. This is what the user sends via API:
'fields_terminated_by' => ';',
'fields_optionally_enclosed_by' => '"',
'lines_terminated_by' => '\n',
In essence we have ; " and \n which is pretty common for a CSV. I use these variables to feed my placeholders but PDO transforms them into this mess:
LOAD DATA LOCAL INFILE
'../../myfile.csv'
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
';'
OPTIONALLY ENCLOSED BY
'\"' <-------------- WRONG
LINES TERMINATED BY
'\\n' <-------------- WRONG
IGNORE 1 LINES
As you can see PDO adds an unnecessary \ in front of my ". Same goes for \n that becomes \\n. As result mysql fails to import the CSV because delimiters are different. The query should have been this:
LOAD DATA LOCAL INFILE
'../../myfile.csv'
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
';'
OPTIONALLY ENCLOSED BY
'"' <-------------- CORRECT
LINES TERMINATED BY
'\n' <-------------- CORRECT
IGNORE 1 LINES
I spent hours looking at other scripts but no one seems to use placeholders with LOAD DATA INFILE. Not for delimeters.
I know I could simply get rid of placeholders for delimiters and just use variables inside the query but I don't like this approach. Moreover I should rely on custom-made escapings to sanitize user-input.
Suggestions?
Related
I'm new here. I would like to ask question regarding with my codes. So my code is working perfectly but then after importing csv to mysql programmatically using LOAD INTO INFILE. I didn't know why my output kept getting this kind of format. Please see my codes below thanks!
$testing = $conn->prepare("LOAD DATA INFILE 'C:/xampp/mysql/data/mysql/usagetable.csv'
INTO TABLE trieinitialcountentry
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(Count_ID, InvItemLocID, OnHand, OnHandValue, StockCenter_ID)");
$testing->execute();
And also, some rows had been imported perfectly but some of them are not. Example of my output:
Count_ID InvItemLocID OnHand OnHandValue StockCenter_ID
737450 -2091889269 140.00 "2 788.80"
Apparently some of the values in the input CSV file are wrapped in quotes and you didn't pass this information to LOAD DATA INFILE.
Try this query:
$query = <<<'END'
LOAD DATA INFILE 'C:/xampp/mysql/data/mysql/usagetable.csv'
INTO TABLE trieinitialcountentry
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(Count_ID, InvItemLocID, OnHand, OnHandValue, StockCenter_ID)
END;
$testing = $conn->prepare($query);
$testing->execute();
Note the added OPTIONALLY ENCLOSED BY '"' clause.
I already written a query i have posted here but that is working with similar kinds of csv.I need a load data local infile query which will work with all kinds of csv file.Please look into it.
The below query works with all kinds of csv files in which fields separated by ,.
example csv
1,114300,1790,2,2,2,No,East
2,114200,2030,4,2,3,No,East
3,114800,1740,3,2,1,No,East
4,94700,1980,3,2,3,No,East
Query looks like this:
LOAD DATA LOCAL INFILE '$pathname'
IGNORE INTO TABLE $name
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\"'
LINES TERMINATED BY '\\n'
IGNORE 1 LINES"
Another csv is like this
Agfa ePhoto 1280;1997;1024.0;640.0;0.0;38.0;114.0;70.0;40.0;4.0;420.0;95.0;179.0
Agfa ePhoto 1680;1998;1280.0;640.0;1.0;38.0;114.0;50.0;0.0;4.0;420.0;158.0;179.0
Agfa ePhoto CL18;2000;640.0;0.0;0.0;45.0;45.0;0.0;0.0;2.0;0.0;0.0;179.0
Agfa ePhoto CL30;1999;1152.0;640.0;0.0;35.0;35.0;0.0;0.0;4.0;0.0;0.0;269.0
Agfa ePhoto CL30 Clik!;1999;1152.0;640.0;0.0;43.0;43.0;50.0;0.0;40.0;300.0;128.0;1299.0
In the query i just changed Fields terminated by ',' to ';'.It works for me.I need everything to be done by in a single query.I am looking for help.
I extracted the csv filename and its path and stored it in a variable named $filename.
$file = new SplFileObject($filename);//it holds the name and location means path to csv
$file->seek(0);//It holds the first line of csv
if (strpos($file, ';') == true){
$field = "FIELDS TERMINATED BY ';'";
}
if (strpos($file, ',') == true){
$field = "FIELDS TERMINATED BY ','";
}
Then edit the query like this
LOAD DATA LOCAL INFILE '$pathname'
IGNORE INTO TABLE $name
$field
OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\"'
LINES TERMINATED BY '\\n'
IGNORE 1 LINES"
It works for all the csv files which fields end by , and ;.
I have a text file which contain 2 field sepearated by | and record by new line
example:
L'EQUME|7A
Voley|18
L'olivier|158
i have a MySql Table with 3 column (id, name , val)
//id autoincrement...
so i would like to use the mysql load file feature to insert the value into name and val but my main problem is the apostrophe while loading file ...
How to addslahes while querying via load file ?
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\r\n'
(name, val);
You can use escaped by But remember escaped by and enclosed by should not be same
I am assuming that your want to enter single quotes value and your fields are enclosed by double quotes and first line should be ignore.
try something like this
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS
TERMINATED BY '|'
ENCLOSED BY '"'
ESCAPED BY ''
LINES
TERMINATED BY '\r\n'
IGNORE 1 LINES; //if you don't want to ignore first line than remove it
I'm fairly new to php and CakePHP, and I'm trying to execute the following code after uploading a text file (2kb):
$filefullname = $this->request->data['File']['file']['tmp_name'];
debug($filefullname);
move_uploaded_file($filefullname, WWW_ROOT.'tmp.txt');
$query = 'LOAD DATA LOW_PRIORITY INFILE "'.WWW_ROOT.'tmp.txt'.'" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES';
debug($query);
$this->Agency->query($query);
Though the file can be found, I get the following output:
'C:\Windows\Temp\phpB413.tmp'
'LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES'
Error: SQLSTATE[HY000]: General error: 29 File 'C:\Data\myphpapplication\app\webroot\tmp.txt' not found (Errcode: 13)
SQL Query: LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES
How can I fix this?
Most likely the file is there, but MySQL doesn't allow you to read files from that location. MySQL is fairly restrictive from where it reads LOAD DATA ... INFILE, with good reasons. You may need to move your file first to a path that MySQL can read from, or change the settings for your MySQL server.
From the MySQL 5.1 manual:
Note that, in the non-LOCAL case, these rules mean that a file named
as ./myfile.txt is read from the server's data directory, whereas the
file named as myfile.txt is read from the database directory of the
default database. [...]
And:
Windows path names are specified using forward slashes rather than
backslashes. If you do use backslashes, you must double them.
#JvO is guiding you down the right path. I think you're missing a couple of important elements, one of which is moving the uploaded file from its initial temp directory on the server to a place where MySQL can officially read it (it won't read it from a temp directory for security reasons).
Here's an example from a project I did, which uploaded a CSV file and then MOVED it to a readable directory. I first uploaded it with the move_uploaded_file function, and then created a variable describing the uploaded path of the new file for use in a LOAD DATA query.
move_uploaded_file($_FILES["fCSV"]["tmp_name"],"user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"]);
$csvfile = "user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"];
In your LOAD DATA query, you're missing the "LOCAL" keyword, which tells MySQL to read the file from a specified directory, otherwise it's looking for it in the database directory (the use of "Field1" etc. below is for demo purposes only and you should have your real table field names in there):
$sqlLoad = "LOAD DATA LOCAL INFILE '".$csvfile."' INTO TABLE my_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (Field1,Field2,Field3,Field4) SET ID =".$id
Lastly, you really need to read through this page in the MySQL manual. It covers many of the issues you're dealing with along with providing examples:
MySQL LOAD DATA INFILE Syntax
I am writing the following script up import a csv file into a mysql database.
The user has to log in first and their security rights to upload are checked by another file, if this evaluates to true, the file can then be uploaded by the file up-loader which will only allow csv's to be uploaded and then the following part imports the file.
Am I using the correct code for this below ?, also if the csv layout is wrong is it possible to refuse the import ?, im just concerned that this could go badly wrong if the csv is formatted correctly. This feature has been requested as a requirement for this project so I am just trying to make it as idiot proof as possible for them.
<?php
$uploadedcsv = './uploads/'.$filename.'';
$sql = 'LOAD DATA LOCAL INFILE "'.$uploadedcsv.'" INTO TABLE '.$table.' FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" IGNORE 1 LINES' or die(mysql_error());
?>
Unless you are 100% rely on the user and it seems that you are not,
it will be good not to blindly import uploaded file, but first to check if it's correct.
To do it, you'll need to open and to read the file, e.g. with fgetcsv and to check the data consistency line-by-line.
You can find a lot of examples on the web.
Here are just some:
http://www.damnsemicolon.com/php/php-upload-csv-mysql
http://www.programmingfacts.com/import-csvexcel-data-mysql-database/
It can be done via MySQL, but by default LOAD DATA INFILE expects a different format than CSV. From the documentation:
If you specify no FIELDS or LINES clause, the defaults are the same as if you had written this:
FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\'
LINES TERMINATED BY '\n' STARTING BY ''
The documentation notes, for when dealing with CSV files:
LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
Which is an example of dealing with comma separated values enclosed by "" and line-separated by \r\n
To use this to import a file, make sure your statement matches the CSV format of your upload files and you can import via this manner.