I receive a csv file that looks fairly straightforward.
I run this on it and it tells me its ASCII.
echo mb_detect_encoding($fhandle , "auto");
However when i run my import code: It doesnt work correctly.
$sql= "LOAD DATA LOCAL INFILE '". $fhandle ."' INTO TABLE sys6_impBet FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n' IGNORE 1 LINES
( AccNo_1,
MtgDate,
Code,
Venue,
Location,
Pool,
EventNo,
Gross_Sales,
Refunds,
Turnover,
Dividends,
Profit_Loss);" ;
It brings in the correct number of records but puts a NULL or 0 in every field / record.
So it is reading the file as it sees the records but won't get the values.
Heres a small sample:
AccNo_1,MtgDate,Code,Venue,Location,Pool,EventNo,Gross_Sales,Refunds,Turnover,Dividends,Profit_Loss
66096159,12/07/2015,Gallops,Penola,SA,Treble,0,279.00,0.00,279.00,"1,955.70","1,676.70"
66096159,12/07/2015,Gallops,Warrnambool,VIC,Treble,0,"1,048.00",0.00,"1,048.00","2,672.80","1,624.80"
66096718,12/07/2015,Gallops,Kalgoorlie,WA,Win,2,783.00,0.00,783.00,"1,174.50",391.50
66096718,12/07/2015,Gallops,Penola,SA,Win,6,204.00,0.00,204.00,"1,143.00",939.00
66096718,12/07/2015,Gallops,Sha Tin,HK,Win,4,197.00,0.00,197.00,"2,064.00","1,867.00"
Is it an encoding problem.
IF I open the file in notepad and save as encoding UTF-8 and save it back down. Then the above code works and all is imported.
But I cant do that for every file every day??
Any ideas I can try?
I have tried this but no different:
$fhandle = mb_convert_encoding($fhandle, "UTF-8", "ASCII");
S
first check the internal encoding used by php
echo mb_internal_encoding();
then set it to required one
mb_internal_encoding("UTF-8");
Related
We are writing some php to move data from a Microsoft SQL database to a MySQL database. The Microsoft data contains user input text, with line breaks and all sorts of wacky stuff
The process we've been working with is as follows:
select from a Microsoft SQL database into php array (works fine)
processes this into a .csv by looping the fputcsv() function:
fputcsv($delimiter = ',', $enclosure = '"', $escape_char = "\\");
(the .csv file looks good when I open it with google sheets)
upload the .csv file to the MySQL server with the following code:
LOAD DATA LOCAL INFILE 'c:/filename.csv'
INTO TABLE tablename
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS; ```
The problem is that when step 3 is executed, the resulting MySQL table is all jumbled up because it appears to get confused with the line breaks in the data, thinking it's a new line in the table.
I noticed the MySQL LOAD DATA command doesn't take the escape character as an argument, but I can't seem to find a way to give it to it. Is this the issue? How can we make this work?
This is the code that is failing. The table on the data base works fine. But I am not able to load new data.
<!--php to upload csv file not working as written below-->
<?php
include("DBconnection.php");
$sql= "LOAD DATA LOCAL INFILE 'kb-listings.csv'
INTO TABLE `listings-table`
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
(MLS, STATUS, ENTRY, SUB, ADDRESS, ZIP, BEDS, BATHS, HBATHS, SQFT-LIV, LOT-SQFT, ORIG-LP, LIST-PRICE, POOL, DOM, SALE-PRICE, M-FEE, TAXES, CLOSING, VIEW, UNIT-VIEW, OFFICE, YEAR-BUILT, GARAGE, PARK, TYPE, WF, AMEN, REMARKS)";
if($result =mysqli_query($conn,$sql)){echo "success";}
else
echo "failure";
mysqli_close($conn);
?>
You don't have to escape the " you can just write like this OPTIONALLY ENCLOSED BY '"'
Thank you for all of the answers. Each answer helped me get to a solution in different ways.
1. I added mysqli_error() and that led me to - a syntax problem with my fields.
2. I checked online and found that using dashes in field names creates problems
because the dashes have to be escaped.
Once I fixed the field names everything worked perfect. I disagree with the usual comments of "this has been answered before in another post". My problem although pretty basic may trip up other beginner programmers out there.
The code below ignores the first five lines BUT I want to ignore last five lines of my CSV file
$query = <<<eof
LOAD DATA LOCAL INFILE '$fname'
INTO TABLE rawdata_01
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED AT '\n'
IGNORE 5 LINES
(SequenceNumber,CDRMajorVerision)
eof;
mySQL doesn't have support for ignoring ending lines.
From The MySQL 5.1 documentation:
The IGNORE number LINES option can be used to ignore lines at the
start of the file. For example, you can use IGNORE 1 LINES to skip
over an initial header line containing column names.
Instead, on a Linux system, you could go to a terminal window and use the command head to make a new file that omits the last 5 lines as follows:
head --lines=-5 oldfile > newfile
See: man page for head
Your other alternatives, (e.g. for Windows), are loading the entire file and deleting the last 5 lines somehow, and writing a script to do what head does in your language of your choice.
I'm fairly new to php and CakePHP, and I'm trying to execute the following code after uploading a text file (2kb):
$filefullname = $this->request->data['File']['file']['tmp_name'];
debug($filefullname);
move_uploaded_file($filefullname, WWW_ROOT.'tmp.txt');
$query = 'LOAD DATA LOW_PRIORITY INFILE "'.WWW_ROOT.'tmp.txt'.'" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES';
debug($query);
$this->Agency->query($query);
Though the file can be found, I get the following output:
'C:\Windows\Temp\phpB413.tmp'
'LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES'
Error: SQLSTATE[HY000]: General error: 29 File 'C:\Data\myphpapplication\app\webroot\tmp.txt' not found (Errcode: 13)
SQL Query: LOAD DATA LOW_PRIORITY INFILE "C:\Data\myphpapplication\app\webroot\tmp.txt" INTO TABLE agencies FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" LINES TERMINATED BY "\r" IGNORE 1 LINES
How can I fix this?
Most likely the file is there, but MySQL doesn't allow you to read files from that location. MySQL is fairly restrictive from where it reads LOAD DATA ... INFILE, with good reasons. You may need to move your file first to a path that MySQL can read from, or change the settings for your MySQL server.
From the MySQL 5.1 manual:
Note that, in the non-LOCAL case, these rules mean that a file named
as ./myfile.txt is read from the server's data directory, whereas the
file named as myfile.txt is read from the database directory of the
default database. [...]
And:
Windows path names are specified using forward slashes rather than
backslashes. If you do use backslashes, you must double them.
#JvO is guiding you down the right path. I think you're missing a couple of important elements, one of which is moving the uploaded file from its initial temp directory on the server to a place where MySQL can officially read it (it won't read it from a temp directory for security reasons).
Here's an example from a project I did, which uploaded a CSV file and then MOVED it to a readable directory. I first uploaded it with the move_uploaded_file function, and then created a variable describing the uploaded path of the new file for use in a LOAD DATA query.
move_uploaded_file($_FILES["fCSV"]["tmp_name"],"user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"]);
$csvfile = "user_uploads/SG".$dapID ."/". $_FILES["fCSV"]["name"];
In your LOAD DATA query, you're missing the "LOCAL" keyword, which tells MySQL to read the file from a specified directory, otherwise it's looking for it in the database directory (the use of "Field1" etc. below is for demo purposes only and you should have your real table field names in there):
$sqlLoad = "LOAD DATA LOCAL INFILE '".$csvfile."' INTO TABLE my_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (Field1,Field2,Field3,Field4) SET ID =".$id
Lastly, you really need to read through this page in the MySQL manual. It covers many of the issues you're dealing with along with providing examples:
MySQL LOAD DATA INFILE Syntax
I am writing the following script up import a csv file into a mysql database.
The user has to log in first and their security rights to upload are checked by another file, if this evaluates to true, the file can then be uploaded by the file up-loader which will only allow csv's to be uploaded and then the following part imports the file.
Am I using the correct code for this below ?, also if the csv layout is wrong is it possible to refuse the import ?, im just concerned that this could go badly wrong if the csv is formatted correctly. This feature has been requested as a requirement for this project so I am just trying to make it as idiot proof as possible for them.
<?php
$uploadedcsv = './uploads/'.$filename.'';
$sql = 'LOAD DATA LOCAL INFILE "'.$uploadedcsv.'" INTO TABLE '.$table.' FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """" IGNORE 1 LINES' or die(mysql_error());
?>
Unless you are 100% rely on the user and it seems that you are not,
it will be good not to blindly import uploaded file, but first to check if it's correct.
To do it, you'll need to open and to read the file, e.g. with fgetcsv and to check the data consistency line-by-line.
You can find a lot of examples on the web.
Here are just some:
http://www.damnsemicolon.com/php/php-upload-csv-mysql
http://www.programmingfacts.com/import-csvexcel-data-mysql-database/
It can be done via MySQL, but by default LOAD DATA INFILE expects a different format than CSV. From the documentation:
If you specify no FIELDS or LINES clause, the defaults are the same as if you had written this:
FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\'
LINES TERMINATED BY '\n' STARTING BY ''
The documentation notes, for when dealing with CSV files:
LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
Which is an example of dealing with comma separated values enclosed by "" and line-separated by \r\n
To use this to import a file, make sure your statement matches the CSV format of your upload files and you can import via this manner.