How do we deal with field with comma when using load data infile? i have this query:
$sql = "LOAD DATA LOCAL INFILE '{$file}' INTO TABLE sales_per_pgs
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#user_id, #account_code, #pg_code, #sales_value)
SET
user_id = #user_id,
account_code = #account_code,
product_group_code = #pg_code,
sales_value = REPLACE(#sales_value, ',', ''),
company_id = {$company_id},
year = {$year},
month = {$month}";
and a line from the csv looks like this:
139, pg89898, op89890, 1,000,000.00
where 1,000,000.00 is a sales value.
Currently, what is inserted in my database is only "1.
EDIT
The user downloads a form with columns like:
user id, account id, pg id, sales value
where the first three columns user id, account id, pg id, were populated and the sales value column is blank because the user has to fill it up manually... the user uses MS excel to do that...
after the form is completed, he will now upload it, in which i am using the load data infile command...
Your content should really look like:
"139", "pg89898", "op89890", "1,000,000.00"
Then you could add the following to the command:
ENCLOSED BY '"' ESCAPED BY "\\"
And you won't have an issue.
Also, somethign you could try if you don't have any paragraphs or strings with , in them:
FIELDS TERMINATED BY ', '
You will have to alter the CSV file that is being input or alter the output that generates the CSV file - sounds the same but it isn't.
You can modify the data coming in by encapsulating fields with quotes and update your command so that it recognizes that fields are encapsulated with them using a command like ENCLOSED BY '"'
or
alter your output so that it formats the number as 1000000 rather than 1,000,000
had the same problem and used just ENCLOSED BY '"' which fixed my issue since i had mixed numbers and strings which is exctyly what ENCLOSED BY is for , from the manuall :
If you specify OPTIONALLY, the ENCLOSED BY character is used only to
enclose values from columns that have a string data type (such as
CHAR, BINARY, TEXT, or ENUM):
In a CSV, comas separate "columns". Since your last value is 1,000,000.00 it is regarded as 3 different columns instead one just one (as intended).
You can either quote each value(column) or change the number format, by removing the commas (,).
if your entire file is exactly as you wrote, then maybe you could use fields terminated by ', ' (comma + space), if and only if you don't have that string within any individual value. If you are using Linux (or any other Unix like system) and your field separator is comma + space, you can use sed to replace this separator with something else:
sed 's/, /|/g' myfile.csv > myfile.txt
However, I would recommend what has already been said: modify your input file enclosing each value with quotes or double quotes and use fields terminated by ',' optionally enclosed by '"'.
Remember that your field termination character must be unique, and must not be contained within any individual value.
As a workaround, try this one -
LOAD DATA INFILE
...
FIELDS TERMINATED BY ', '
...
Related
I'm new here. I would like to ask question regarding with my codes. So my code is working perfectly but then after importing csv to mysql programmatically using LOAD INTO INFILE. I didn't know why my output kept getting this kind of format. Please see my codes below thanks!
$testing = $conn->prepare("LOAD DATA INFILE 'C:/xampp/mysql/data/mysql/usagetable.csv'
INTO TABLE trieinitialcountentry
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(Count_ID, InvItemLocID, OnHand, OnHandValue, StockCenter_ID)");
$testing->execute();
And also, some rows had been imported perfectly but some of them are not. Example of my output:
Count_ID InvItemLocID OnHand OnHandValue StockCenter_ID
737450 -2091889269 140.00 "2 788.80"
Apparently some of the values in the input CSV file are wrapped in quotes and you didn't pass this information to LOAD DATA INFILE.
Try this query:
$query = <<<'END'
LOAD DATA INFILE 'C:/xampp/mysql/data/mysql/usagetable.csv'
INTO TABLE trieinitialcountentry
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(Count_ID, InvItemLocID, OnHand, OnHandValue, StockCenter_ID)
END;
$testing = $conn->prepare($query);
$testing->execute();
Note the added OPTIONALLY ENCLOSED BY '"' clause.
I have a text file which contain 2 field sepearated by | and record by new line
example:
L'EQUME|7A
Voley|18
L'olivier|158
i have a MySql Table with 3 column (id, name , val)
//id autoincrement...
so i would like to use the mysql load file feature to insert the value into name and val but my main problem is the apostrophe while loading file ...
How to addslahes while querying via load file ?
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\r\n'
(name, val);
You can use escaped by But remember escaped by and enclosed by should not be same
I am assuming that your want to enter single quotes value and your fields are enclosed by double quotes and first line should be ignore.
try something like this
LOAD DATA INFILE 'data.txt'
INTO TABLE table_name
FIELDS
TERMINATED BY '|'
ENCLOSED BY '"'
ESCAPED BY ''
LINES
TERMINATED BY '\r\n'
IGNORE 1 LINES; //if you don't want to ignore first line than remove it
There is a similar question already but I would like to know how to accomplish this for very large data sets.
The #dummy method etc will be take very long.
Also, what should I do if the column names are different in the spreadsheet and the table?
LOAD DATA INFILE '/tmp/actor.tsv'
REPLACE INTO TABLE actor
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '"'
(#f1, #f2, #f3)
SET first_name = #f1, last_name = #f3
UPDATE. If number of fields is big use this script to generate the LOAD command:
echo -n "LOAD DATA INFILE '/tmp/actor.tsv'
REPLACE INTO TABLE actor
FIELDS TERMINATED BY '\\t'
OPTIONALLY ENCLOSED BY '\"'
("
comma=""
for i in `seq 300`
do
echo -n "$comma #f$i"
comma=","
done
echo ")"
echo "SET first_name = #f1, last_name = #f3"
I have an file with Currency Sign delimeter :
20130217¤18122¤14
20130217¤62152¤14
20130217¤54512¤10
LOAD DATA INFILE '$file'
REPLACE INTO TABLE $my_table
FIELDS TERMINATED BY '¤'
IGNORE 1 LINES
The table has columns date, id, num.
The error is #1292 - Incorect date value: '20130217¤18122¤14' for column 'DATE' at row 1
The reason you're getting this is because non-ASCII terminators are not fully supported.
See this error message in the docs:
Error: 1638 SQLSTATE: HY000 (WARN_NON_ASCII_SEPARATOR_NOT_IMPLEMENTED)
Message: Non-ASCII separator arguments are not fully supported
The good news is that multi-character separators are supported, so you could use sed or something to replace your ¤ with #%&%# or something equally unique, and use that as a separator:
The FIELDS TERMINATED BY, LINES STARTING BY, and LINES TERMINATED BY values can be more than one character.
(see here)
Ok I know my database is not normalized but in my case, It is not possible to normalize the data as each user has different access level with access rights, everytime I modify something and I need to delete the rows and insert back so I went for comma-separated values, here the issue is if I remove a particular group, regex doesn't work, it throws me an error..am using php and the error goes like this
FUNCTION database_name.REGEXP_REPLACE does not exist
Table structure
allowed_group_ids
+----------------+
12345,34345,55454
My query, say for example $delete_id is 12345 or say 55454, I only pass one ID at a time and the id's has no space in between and it's a text field
UPDATE tbl_scripts SET allowed_group_ids = TRIM(BOTH ','
FROM REGEXP_REPLACE(allowed_group_ids, '(,(\s)?)?$detele_id', ''))
WHERE system_id = {$_SESSION['system_id']}
So what's wrong in here?
There is no such thing as regexp_replace in MySQL.
You could perform this query using regular replace:
UPDATE tbl_scripts SET allowed_group_ids =
trim(BOTH ',' FROM
replace(
replace(allowed_group_ids, '$delete_id',','),
',,',
','
)
)
WHERE system_id = {$_SESSION['system_id']}
This first removes the ID, and removes any doubled-up commas, then removes any commas at the start/end of the string. This should be sufficient but you can add another replace to remove whitespace if necessary.
Update: SQL Fiddle shows the query in action