I have an file with Currency Sign delimeter :
20130217¤18122¤14
20130217¤62152¤14
20130217¤54512¤10
LOAD DATA INFILE '$file'
REPLACE INTO TABLE $my_table
FIELDS TERMINATED BY '¤'
IGNORE 1 LINES
The table has columns date, id, num.
The error is #1292 - Incorect date value: '20130217¤18122¤14' for column 'DATE' at row 1
The reason you're getting this is because non-ASCII terminators are not fully supported.
See this error message in the docs:
Error: 1638 SQLSTATE: HY000 (WARN_NON_ASCII_SEPARATOR_NOT_IMPLEMENTED)
Message: Non-ASCII separator arguments are not fully supported
The good news is that multi-character separators are supported, so you could use sed or something to replace your ¤ with #%&%# or something equally unique, and use that as a separator:
The FIELDS TERMINATED BY, LINES STARTING BY, and LINES TERMINATED BY values can be more than one character.
(see here)
Related
I spent the last few hours in a paradox. I use this query to import a CSV file into my database.
$query = $this->pdo->prepare('
LOAD DATA LOCAL INFILE
:file
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
:fields_terminated_by
OPTIONALLY ENCLOSED BY
:fields_optionally_enclosed_by
LINES TERMINATED BY
:lines_terminated_by
IGNORE 1 LINES
(
`aaa`,
`bbb`
)
SET
task_id = :task_id
');
$params = [
'file' => $this->path_to_file,
'fields_terminated_by' => $fields_terminated_by,
'fields_optionally_enclosed_by' => $fields_optionally_enclosed_by,
'lines_terminated_by' => $lines_terminated_by,
'task_id' => $this->task_id
];
$query->execute($params);
The entire file with its settings (separator, enclosure, line termination etc.) arrives from an OAuth-based API from end-users. I am telling you this to underline that obviously I cannot trust user input hence I must use placeholders.
The problem is that the way PDO escapes such parameters is odd. This is what the user sends via API:
'fields_terminated_by' => ';',
'fields_optionally_enclosed_by' => '"',
'lines_terminated_by' => '\n',
In essence we have ; " and \n which is pretty common for a CSV. I use these variables to feed my placeholders but PDO transforms them into this mess:
LOAD DATA LOCAL INFILE
'../../myfile.csv'
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
';'
OPTIONALLY ENCLOSED BY
'\"' <-------------- WRONG
LINES TERMINATED BY
'\\n' <-------------- WRONG
IGNORE 1 LINES
As you can see PDO adds an unnecessary \ in front of my ". Same goes for \n that becomes \\n. As result mysql fails to import the CSV because delimiters are different. The query should have been this:
LOAD DATA LOCAL INFILE
'../../myfile.csv'
INTO TABLE
product_feeds_raw
FIELDS TERMINATED BY
';'
OPTIONALLY ENCLOSED BY
'"' <-------------- CORRECT
LINES TERMINATED BY
'\n' <-------------- CORRECT
IGNORE 1 LINES
I spent hours looking at other scripts but no one seems to use placeholders with LOAD DATA INFILE. Not for delimeters.
I know I could simply get rid of placeholders for delimiters and just use variables inside the query but I don't like this approach. Moreover I should rely on custom-made escapings to sanitize user-input.
Suggestions?
I have CSV file which has 3 columns, and my table has 4 columns(one of it is a id autoincrement which I want to be autoincremented). I want to load the 3 columns from csv to the selected 3 columns in the table.
Tried code:
LOAD DATA LOCAL INFILE 'info.csv' INTO TABLE tbl_countryip (ipstart, ipend, countrycode) FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' ;
which throws error:
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'' at line 1
The list of columns is in the wrong place in your statement. Follow the syntax in the documentation.
You want something like this:
LOAD DATA LOCAL INFILE 'info.csv'
INTO TABLE tbl_countryip
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(ipstart, ipend, countrycode)
Reference: http://dev.mysql.com/doc/refman/5.5/en/load-data.html
Follow below steps-
1) connect with any gui like sqlyog.
2) select db > select table > right click on table > choose import > import csv data using load...
3) select all columns except primary key.
4) put 1 in ingnore line option if your csv file contains header.
5) click on import
If you want to do by command then command should be as per below-
LOAD DATA LOCAL INFILE 'info.csv' INTO TABLE tbl_countryip FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' (ipstart, ipend, countrycode);
Note: If your corrent dir not where csv file exist then you need to give full path.
If your csv file contains header line then you need to exclude it as per below-
LOAD DATA LOCAL INFILE 'info.csv' INTO TABLE tbl_countryip FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (ipstart, ipend, countrycode);
I removed the id column from the table, so both table and csv file had 3 columns, and then ran the load csv command, and then inserted a id column into the table with autoincrement.
I have a text file which contain 2 field sepearated by | and record by new line
example:
L'EQUME|7A
Voley|18
L'olivier|158
i have a MySql Table with 3 column (id, name , val)
//id autoincrement...
so i would like to use the mysql load file feature to insert the value into name and val but my main problem is the apostrophe while loading file ...
How to addslahes while querying via load file ?
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\r\n'
(name, val);
You can use escaped by But remember escaped by and enclosed by should not be same
I am assuming that your want to enter single quotes value and your fields are enclosed by double quotes and first line should be ignore.
try something like this
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS
TERMINATED BY '|'
ENCLOSED BY '"'
ESCAPED BY ''
LINES
TERMINATED BY '\r\n'
IGNORE 1 LINES; //if you don't want to ignore first line than remove it
I'm fairly new to php and CakePHP, and I'm trying to execute the following code after uploading a text file (2kb):
$filefullname = $this->request->data['File']['file']['tmp_name'];
debug($filefullname);
move_uploaded_file($filefullname, WWW_ROOT.'tmp.txt');
$query = 'LOAD DATA LOW_PRIORITY INFILE "'.WWW_ROOT.'tmp.txt'.'" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES';
debug($query);
$this->Agency->query($query);
Though the file can be found, I get the following output:
'C:\Windows\Temp\phpB413.tmp'
'LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES'
Error: SQLSTATE[HY000]: General error: 29 File 'C:\Data\myphpapplication\app\webroot\tmp.txt' not found (Errcode: 13)
SQL Query: LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES
How can I fix this?
Most likely the file is there, but MySQL doesn't allow you to read files from that location. MySQL is fairly restrictive from where it reads LOAD DATA ... INFILE, with good reasons. You may need to move your file first to a path that MySQL can read from, or change the settings for your MySQL server.
From the MySQL 5.1 manual:
Note that, in the non-LOCAL case, these rules mean that a file named
as ./myfile.txt is read from the server's data directory, whereas the
file named as myfile.txt is read from the database directory of the
default database. [...]
And:
Windows path names are specified using forward slashes rather than
backslashes. If you do use backslashes, you must double them.
#JvO is guiding you down the right path. I think you're missing a couple of important elements, one of which is moving the uploaded file from its initial temp directory on the server to a place where MySQL can officially read it (it won't read it from a temp directory for security reasons).
Here's an example from a project I did, which uploaded a CSV file and then MOVED it to a readable directory. I first uploaded it with the move_uploaded_file function, and then created a variable describing the uploaded path of the new file for use in a LOAD DATA query.
move_uploaded_file($_FILES["fCSV"]["tmp_name"],"user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"]);
$csvfile = "user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"];
In your LOAD DATA query, you're missing the "LOCAL" keyword, which tells MySQL to read the file from a specified directory, otherwise it's looking for it in the database directory (the use of "Field1" etc. below is for demo purposes only and you should have your real table field names in there):
$sqlLoad = "LOAD DATA LOCAL INFILE '".$csvfile."' INTO TABLE my_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (Field1,Field2,Field3,Field4) SET ID =".$id
Lastly, you really need to read through this page in the MySQL manual. It covers many of the issues you're dealing with along with providing examples:
MySQL LOAD DATA INFILE Syntax
How do we deal with field with comma when using load data infile? i have this query:
$sql = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE sales_per_pgs
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#user_id, #account_code, #pg_code, #sales_value)
SET
user_id = #user_id,
account_code = #account_code,
product_group_code = #pg_code,
sales_value = REPLACE(#sales_value, ',', ''),
company_id = {$company_id},
year = {$year},
month = {$month}";
and a line from the csv looks like this:
139, pg89898, op89890, 1,000,000.00
where 1,000,000.00 is a sales value.
Currently, what is inserted in my database is only "1.
EDIT
The user downloads a form with columns like:
user id, account id, pg id, sales value
where the first three columns user id, account id, pg id, were populated and the sales value column is blank because the user has to fill it up manually... the user uses MS excel to do that...
after the form is completed, he will now upload it, in which i am using the load data infile command...
Your content should really look like:
"139", "pg89898", "op89890", "1,000,000.00"
Then you could add the following to the command:
ENCLOSED BY '"' ESCAPED BY "\\"
And you won't have an issue.
Also, somethign you could try if you don't have any paragraphs or strings with , in them:
FIELDS TERMINATED BY ', '
You will have to alter the CSV file that is being input or alter the output that generates the CSV file - sounds the same but it isn't.
You can modify the data coming in by encapsulating fields with quotes and update your command so that it recognizes that fields are encapsulated with them using a command like ENCLOSED BY '"'
or
alter your output so that it formats the number as 1000000 rather than 1,000,000
had the same problem and used just ENCLOSED BY '"' which fixed my issue since i had mixed numbers and strings which is exctyly what ENCLOSED BY is for , from the manuall :
If you specify OPTIONALLY, the ENCLOSED BY character is used only to
enclose values from columns that have a string data type (such as
CHAR, BINARY, TEXT, or ENUM):
In a CSV, comas separate "columns". Since your last value is 1,000,000.00 it is regarded as 3 different columns instead one just one (as intended).
You can either quote each value(column) or change the number format, by removing the commas (,).
if your entire file is exactly as you wrote, then maybe you could use fields terminated by ', ' (comma + space), if and only if you don't have that string within any individual value. If you are using Linux (or any other Unix like system) and your field separator is comma + space, you can use sed to replace this separator with something else:
sed 's/, /|/g' myfile.csv > myfile.txt
However, I would recommend what has already been said: modify your input file enclosing each value with quotes or double quotes and use fields terminated by ',' optionally enclosed by '"'.
Remember that your field termination character must be unique, and must not be contained within any individual value.
As a workaround, try this one -
LOAD DATA INFILE
...
FIELDS TERMINATED BY ', '
...