# download the file off the internet
$file = file_get_contents("http://localhost/sample.csv");
$filepath = "C:/xampp/htdocs/test/file/sample.csv";
file_put_contents($filepath, $file);
# load the data into sample table
$pt1 = "LOAD DATA LOCAL INFILE ";
$pt2 = "'/C:/xampp/htdocs/test/file/sample.csv' INTO TABLE sample ";
$pt3 = "FIELDS TERMINATED BY ',' ENCLOSED BY '\"' ";
$pt4 = "LINES TERMINATED BY '\r\n' ";
$pt5 = "(col1,col2,col3,col4)";
$sqlquerynew = $pt1.$pt2.$pt3.$pt4.$pt5;
mysql_query($sqlquerynew);
This piece of code works on non-csv (well, I tested it with a text file instead).
Before this part gets run, I have to create a table. The table is now created, but no data is loaded. The file stated in the path exists.
What could be the problem?
Thanks
This is a sample csv I found online
"REVIEW_DATE","AUTHOR","ISBN","DISCOUNTED_PRICE"
"1985/01/21","Douglas Adams",0345391802,5.95
"1990/01/12","Douglas Hofstadter",0465026567,9.95
....... etc
Two problems
$pt2 = "'/C:/xampp/htdocs/test/file/sample.csv' INTO TABLE sample ";
Remove the / in front of C.
and secondly,
$pt5 = "(col1,col2,col3,col4)";
Make sure you have the right name for the columns. If you want to import * all columns, just remove it. It is also a good idea to remove the header in your case, because removing $pt5 you will succeed, but the header row will be added to the table.
Related
How can I import a CSV file into a MySQL table? I would like for the first row of data be used as the column names.
I read How do I import CSV file into a MySQL table?, but the only answer was to use a GUI and not a shell?
Instead of writing a script to pull in information from a CSV file, you can link MYSQL directly to it and upload the information using the following SQL syntax.
To import an Excel file into MySQL, first export it as a CSV file. Remove the CSV headers from the generated CSV file along with empty data that Excel may have put at the end of the CSV file.
You can then import it into a MySQL table by running:
load data local infile 'uniq.csv' into table tblUniq fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(uniqName, uniqCity, uniqComments)
as read on: Import CSV file directly into MySQL
EDIT
For your case, you'll need to write an interpreter first, for finding the first row, and assigning them as column names.
EDIT-2
From MySQL docs on LOAD DATA syntax:
The IGNORE number LINES option can be used to ignore lines at the
start of the file. For example, you can use IGNORE 1 LINES to skip
over an initial header line containing column names:
LOAD DATA INFILE '/tmp/test.txt' INTO TABLE test IGNORE 1 LINES;
Therefore, you can use the following statement:
LOAD DATA LOCAL INFILE 'uniq.csv'
INTO TABLE tblUniq
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(uniqName, uniqCity, uniqComments)
Here's a simple PHP command line script that will do what you need:
<?php
$host = 'localhost';
$user = 'root';
$pass = '';
$database = 'database';
$db = mysqli_connect($host, $user, $pass) or die ("could not connect to mysql");
mysqli_select_db($db, $database) or die ("no database");
/********************************************************************************/
// Parameters: filename.csv table_name
$argv = $_SERVER[argv];
if($argv[1]) { $file = $argv[1]; }
else {
echo "Please provide a file name\n"; exit;
}
if($argv[2]) { $table = $argv[2]; }
else {
$table = pathinfo($file);
$table = $table['filename'];
}
/********************************************************************************/
// Get the first row to create the column headings
$fp = fopen($file, 'r');
$frow = fgetcsv($fp);
foreach($frow as $column) {
if($columns) $columns .= ', ';
$columns .= "`$column` varchar(250)";
}
$create = "create table if not exists $table ($columns);";
mysqli_query($db, $create) or die(mysqli_error($db));
/********************************************************************************/
// Import the data into the newly created table.
$file = $_SERVER['PWD'].'/'.$file;
$q = "LOAD DATA INFILE '$file' INTO TABLE $table FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n' ignore 1 lines;";
mysqli_query($db, $q) or die(mysqli_error($db));
?>
It will create a table based on the first row and import the remaining rows into it. Here is the command line syntax:
php csv_import.php csv_file.csv table_name
if you have the ability to install phpadmin there is a import section where you can import csv files to your database there is even a checkbox to set the header to the first line of the file contains the table column names (if this is unchecked, the first line will become part of the data
First create a table in the database with same numbers of columns that are in the csv file.
Then use following query
LOAD DATA INFILE 'D:/Projects/testImport.csv' INTO TABLE cardinfo
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
If you start mysql as "mysql -u -p --local-infile ", it will work fine
To load data from text file or csv file the command is
load data local infile 'file-name.csv'
into table table-name
fields terminated by '' enclosed by '' lines terminated by '\n' (column-name);
In above command, in my case there is only one column to be loaded so there is no "terminated by" and "enclosed by" so I kept it empty else programmer can enter the separating character . for e.g . ,(comma) or " or ; or any thing.
**for people who are using mysql version 5 and above **
Before loading the file into mysql must ensure that below tow line are added in side etc/mysql/my.cnf
to edit my.cnf command is
sudo vi /etc/mysql/my.cnf
[mysqld]
local-infile
[mysql]
local-infile
I wrote some code to do this, i'll put in a few snippets:
$dir = getcwd(); // Get current working directory where this .php script lives
$fileList = scandir($dir); // scan the directory where this .php lives and make array of file names
Then get the CSV headers so you can tell mysql how to import (note: make sure your mysql columns exactly match the csv columns):
//extract headers from .csv for use in import command
$headers = str_replace("\"", "`", array_shift(file($path)));
$headers = str_replace("\n", "", $headers);
Then send your query to the mysql server:
mysqli_query($cons, '
LOAD DATA LOCAL INFILE "'.$path.'"
INTO TABLE '.$dbTable.'
FIELDS TERMINATED by \',\' ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
IGNORE 1 LINES
('.$headers.')
;
')or die(mysql_error());
I wrestled with this for some time. The problem lies not in how to load the data, but how to construct the table to hold it. You must generate a DDL statement to build the table before importing the data.
Particularly difficult if the table has a large number of columns.
Here's a python script that (almost) does the job:
#!/usr/bin/python
import sys
import csv
# get file name (and hence table name) from command line
# exit with usage if no suitable argument
if len(sys.argv) < 2:
sys.exit('Usage: ' + sys.argv[0] + ': input CSV filename')
ifile = sys.argv[1]
# emit the standard invocation
print 'create table ' + ifile + ' ('
with open(ifile + '.csv') as inputfile:
reader = csv.DictReader(inputfile)
for row in reader:
k = row.keys()
for item in k:
print '`' + item + '` TEXT,'
break
print ')\n'
The problem it leaves to solve is that the final field name and data type declaration is terminated with a comma, and the mySQL parser won't tolerate that.
Of course it also has the problem that it uses the TEXT data type for every field. If the table has several hundred columns, then VARCHAR(64) will make the table too large.
This also seems to break at the maximum column count for mySQL. That's when it's time to move to Hive or HBase if you are able.
Here's how I did it in Python using csv and the MySQL Connector:
import csv
import mysql.connector
credentials = dict(user='...', password='...', database='...', host='...')
connection = mysql.connector.connect(**credentials)
cursor = connection.cursor(prepared=True)
stream = open('filename.csv', 'rb')
csv_file = csv.DictReader(stream, skipinitialspace=True)
query = 'CREATE TABLE t ('
query += ','.join('`{}` VARCHAR(255)'.format(column) for column in csv_file.fieldnames)
query += ')'
cursor.execute(query)
for row in csv_file:
query = 'INSERT INTO t SET '
query += ','.join('`{}` = ?'.format(column) for column in row.keys())
cursor.execute(query, row.values())
stream.close()
cursor.close()
connection.close()
Key points
Use prepared statements for the INSERT
Open the file.csv in 'rb' binary
Some CSV files may need tweaking, such as the skipinitialspace option.
If 255 isn't wide enough you'll get errors on INSERT and have to start over.
Adjust column types, e.g. ALTER TABLE t MODIFY `Amount` DECIMAL(11,2);
Add a primary key, e.g. ALTER TABLE t ADD `id` INT PRIMARY KEY AUTO_INCREMENT;
Import CSV Files into mysql table
LOAD DATA LOCAL INFILE 'd:\\Site.csv' INTO TABLE `siteurl` FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n';
Character Escape Sequence
\0 An ASCII NUL (0x00) character
\b A backspace character
\n A newline (linefeed) character
\r A carriage return character
\t A tab character.
\Z ASCII 26 (Control+Z)
\N NULL
visits :
http://www.webslessons.com/2014/02/import-csv-files-using-php-and-mysql.html
Use TablePlus application:
Right-Click on the table name from the right panel
Choose Import... > From CSV
Choose CSV file
Review column matching and hit Import
All done!
As others have mentioned, the load data local infile works just fine. I tried the php script that Hawkee posted, but it didn't work for me. Rather than debugging it, here's what I did:
1) Copy/paste the header row of the CSV file into a txt file and edit it with Emacs. Add a comma and CR between each field to get each on its own line.
2) Save that file as FieldList.txt.
3) Edit the file to include definitions for each field (most were varchar, but quite a few were int(x). Add create table *tablename* (to the beginning of the file and) to the end of the file. Save it as CreateTable.sql.
4) Start the mysql client with input from the Createtable.sql file to create the table.
5) Start the mysql client, copy/paste in most of the 'LOAD DATA INFILE' command substituting my table name and csv file name. Paste in the FieldList.txt file. Be sure to include the 'IGNORE 1 LINES' before pasting in the field list.
It sounds like a lot of work, but it's easy with Emacs...
So I attempted to use the script give by Hawkee but some of the commands are outdated. Using mysql_X is depreciated and needs to be replaced by mysqli_x. After doing some troubleshooting I wrote the following script and it is working nicely.
Please note: the following code assumes that you are entering floats. I used this script to import percentiles from the WHO for stats related to growth.
use -drop (before the file name) if you want to drop the table
<?php
//This script is for importing the percentile values.
//Written by Daniel Pflieger # GrowlingFlea Software
$host = 'localhost';
$user = 'root';
$pass = '';
$database = '';
//options. This is what we need so the user can specify whether or not to drop the table
$short_options = "d::";
$options = getopt($short_options);
//check if the flag "-drop" is entered by the end user.
if (!empty($options) && $options['d'] != "rop"){
echo "The only available argument is -drop \n";
exit;
} else if (!empty($options)){
$dropTable = true;
} else {
$dropTable = false;
}
//we use mysqli_* since this is required with newer versions of php
$db = mysqli_connect($host, $user, $pass, $database);
// argv changes if the drop flag is used. here we read in the name of the .csv file we want to import
if (isset($argv[1]) && empty($options) ) {
$file = $argv[1];
} else if (isset($argv[2]) && $options[1] = "rop" ) {
$file = $argv[2];
}
//we call the table name the name of the file. Since this script was used to import who growth chart info
//I appended the '_birth_to_5yrs' to the string. You probably want to remove this and add something that
//makes sense to you
$table = pathinfo($file);
$table = "who_" . $table['filename'] . "_birth_to_5yrs";
$table = str_replace('-', '_', $table);
// We read the first line of the .csv file. It is assumed that these are the headers.
$fp = fopen($file, 'r');
$frow = fgetcsv($fp);
$columns = '';
//we get the header names and for this purpose we make every value 'float'. If you are unsure of
//the datatype you can probably use varchar(250).
foreach($frow as $column) {
$columns .= "`" .$column . "` float,";
}
//drop the table to prevent data issues, if that is what the end user selects
if ($dropTable) {
mysqli_query($db, "drop table if exists $table");
}
// here we form the create statement and we create the table.
// we use the mysqli_real_escape_string to make sure we dont damage the DB
$create = "create table if not exists $table ($columns);";
$create = str_replace(',)', ')', $create);
$create = mysqli_real_escape_string($db, $create);
mysqli_query($db, $create);
// We read the values line-by-line in the .csv file and insert them into the table until we are done.
while ($frow = fgetcsv($fp)){
$insert = implode(", ", $frow);
$insert = "Insert into $table VALUES ( $insert )";
$insert = mysqli_real_escape_string($db, $insert);
$insert = mysqli_query($db, $insert);
}
An example of how to run the script:
php ../git/growlingflea-dev-tools/importCSV.php -drop wfh-female-percentiles-expanded-tables.csv
I have google search many ways to import csv to mysql, include " load data infile ", use mysql workbench, etc.
when I use mysql workbench import button, first you need to create the empty table on your own, set each column type on your own. Note: you have to add ID column at the end as primary key and not null and auto_increment, otherwise, the import button will not visible at later. However, when I start load CSV file, nothing loaded, seems like a bug. I give up.
Lucky, the best easy way so far I found is to use Oracle's mysql for excel. you can download it from here mysql for excel
This is what you are going to do:
open csv file in excel, at Data tab, find mysql for excel button
select all data, click export to mysql.
Note to set a ID column as primary key.
when finished, go to mysql workbench to alter the table,
such as currency type should be decimal(19,4) for large amount decimal(10,2) for regular use.
other field type may be set to varchar(255).
I'm trying to take a TSV file and 'POST'ed inputs and load the TSV file's contents into a DB table, replacing any existing data for specified columns. The TSV may contain any number of columns and rows and the 1st row specifies the columns that are supposed to be modified.
My problem concerns data in columns that ARE NOT supposed to be modified when running the code-generated LOAD DATA INFILE ... REPLACE INTO TABLE ... MySQL statement. When I run my code (see below), data of columns that are NOT specified in $columnsText (which is generated from the 1st row of the TSV file) end-up getting set to NULL or their default value. On the other hand, data of columns that ARE specified in $columnsText have their contents replaced just as intended.
An example of the MySQL statement that is generated by my code and is working as described above is:
LOAD DATA INFILE 'C:\\MyProject\\public\\1459772537-cities7.tsv' REPLACE INTO TABLE cities FIELDS TERMINATED BY ' ' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES (id,UNLOCODE,name_english,UN_subdiv) -- for all TSV-file mentioned rows, this statement will update the mentioned columns (id,UNLOCODE,name_english,UN_subdiv) correctly, but then all unmentioned columns for that row will be set to NULL!
How do I modify this code to keep the data of unspecified columns from being set to their default/NULL values? Or more simply, getting to the root of the problem, how do I fix the MySQL statement that is being generated to achieve my objective?
I'm using PHP with Laravel.
// Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
echo "POST has file <br>";
$file = Input::file('file');
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "location: ".$pathName."<br>";
// Determine whether to use IGNORE OR REPLACE in MySQL query.
if (isset($_POST['replace']) && $_POST['replace'] == true){
$ignoreOrReplace = "REPLACE";
}
else {$ignoreOrReplace = "IGNORE";}
echo "ignore or replace: ".$ignoreOrReplace."<br>";
// Determine columns to insert in DB, based on values of input file's 1st row.
$columnsText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columns = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columns as $column){
if ($firstIteration){$firstIteration=false;}
else {$columnsText .= ",";}
$columnsText .= $column;
}
echo "DB columns to load: ".$columnsText;
fclose($handle);
}
$query = sprintf(
"LOAD DATA INFILE '%s' %s INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$ignoreOrReplace,$_POST['mytable'],$columnsText
);
echo "<br>Here's the query: ".$query."<br>";
echo "<br><br> Database update should be complete!<br><br>";
echo 'Return to Home Page<br>';
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // I forgot what this does.
return DB::connection()->getpdo()->exec($query);
The documentation states:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index as an existing row. See Section 13.2.8, “REPLACE Syntax”.
REPLACE is not UPDATE. REPLACE is a MySQL extension to the SQL that first deletes the row if it exists, then it inserts the new one.
On INSERT, MySQL uses the default values for the fields that are not provided in the query. These fields probably default to NULL in your case.
There is no way to update the existing rows using LOAD DATA INFILE.
I suggest you to create a working table and use it only for the purpose of loading data into it as follows:
TRUNCATE it before using it.
LOAD DATA INFILE in it.
Join it against the table you want to update and use UPDATE on the join to copy the fields you need from the working table to the final table.
Use INSERT ... SELECT to get from the join the rows that are not in the final table and insert them.
TRUNCATE it.
Don't delete the table after it is used, you'll need it again next time. The last step aims to keep its disk usage at a minimum; the table definition doesn't take much space.
The answer provided by axiac is broadly-speaking the correct answer.
In case it is useful to anyone, I have included below the specific code (PHP/Laravel/MySQL) that solved my problem. I can't necessarily say this is the most efficient way to solve this problem, but it it working! :)
// (1) setup
DB::connection()->disableQueryLog();
// (2) Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
$file = Input::file('file');
}
else {
echo "<br>Input file not found! Please review inputed information.<br>";
return null;
}
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "Input file location: ".$pathName."<br>";
// (3) Determine main table and staging table.
$mainTable = $_POST['mytable'];
$stagingTable = $_POST['mytable'].'_staging'; // All staging tables are named: 'standardtable_staging'.
// (4) Determine destination DB table's columns and columns to be inserted into that table (based on values of input file's 1st row).
$columnsMain = Schema::getColumnListing($mainTable);
$columnsInput = [];
$columnsInputText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columnsInput = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columnsInput as $columnInput){
if ($firstIteration){$firstIteration=false;}
else {$columnsInputText .= ",";}
$columnsInputText .= $columnInput;
}
echo "<br>DB columns to load: ".$columnsInputText."<br>";
fclose($handle);
}
// (5) Create a new empty staging table.
$statement = "DROP TABLE IF EXISTS ".$stagingTable; // we drop rather than truncate b/c we want to re-determine columns.
DB::connection()->getpdo()->exec($statement);
$statement = "CREATE TABLE ".$stagingTable." LIKE ".$mainTable;
DB::connection()->getpdo()->exec($statement);
// (6) The staging table only needs to have columns that exist in the TSV file, so let's minimize its columns.
$columnsToDrop = [];
foreach ($columnsMain as $columnMain){
if (! in_array($columnMain,$columnsInput)){
array_push($columnsToDrop,$columnMain);
}
}
if (count($columnsToDrop) > 0){
Schema::table($stagingTable, function($t) use ($columnsToDrop) {$t->dropColumn($columnsToDrop);});
}
// (7) Load data to the staging table.
$statement = sprintf(
"LOAD DATA INFILE '%s' INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$stagingTable,$columnsInputText
);
echo "<br>Here's the MySQL staging statement: <br>".$statement."<br>";
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // don't actually recall why I put this here.
DB::connection()->getpdo()->exec($statement);
// (8) 'INSERT...ON DUPLICATE KEY UPDATE' is used here to get data from staging table to the actually-used table.
// Note: Any new columns in the staging table MUST already be defined in the main table.
$statement = sprintf("INSERT INTO %s (%s) SELECT * FROM %s ON DUPLICATE KEY UPDATE ", $mainTable,$columnsInputText,$stagingTable);
$firstClause = true;
foreach ($columnsInput as $columnInput) {
if (strtoupper($columnInput) != "ID"){
if ($firstClause){$firstClause=false;}
else {$statement .= ", ";}
$clause = $mainTable.".".$columnInput." = IF (".$stagingTable.".".$columnInput." <=> NULL,".
$mainTable.".".$columnInput.",".
$stagingTable.".".$columnInput.")";
$statement .= $clause;
}
}
echo "<br>Here's the staging-to-actual-table statement:<br>".$statement."<br>";
DB::connection()->getpdo()->exec($statement);
echo "<br>New information added to database!<br>";
I am using this code to load a file in my database. It is a php file that I upload on putty:
$Db->query('LOAD DATA LOCAL INFILE \'/name 03.11.2015.csv\'
INTO TABLE '.$in_table.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
IGNORE 8 ROWS
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name= \'name\'
;');
Now I want to have a variable instead of a specific filename in the first line of the query :
$Db->query('LOAD DATA LOCAL INFILE \'/path/name 03.11.2015.csv\'
So, I am using that and it seems that it is working :
$date = '.....';
$name = '.....';
$Db->query('LOAD DATA INFILE \'/path/'.$name.' '.$date.'.csv\'
INTO TABLE '.$in_table.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
IGNORE 8 ROWS
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name= \'name\'
;');
But now I have to change my code because some files don't have this format in their filename : '$name $date'. Some examples are given below :
name 2015-10-10
rand name 2015-10-10
How can I check the filename given that the name is a specific value? I want to concat the filename and get the name and the date in order to find the file and then use the original filename in my code.
I found a solution for this question but I still have an unanswered which is related to this one (link in the end). The answer for this question is here in this code :
$searchString = 'aaaa';
$Dates = array();
// Get all the files in my folder with the extension ".xlsx"
$files = glob('/path/*.xlsx');
// I create an array where I save all the .xlsx files that contain "aaaa" in the filename
$filesFound = array();
foreach($files as $file) {
$name = pathinfo($file, PATHINFO_FILENAME);
// Determines if there is a date and if the search string is in the filename. If yes, it puts the date and the filename in the arrays I created before
if((strpos(strtolower($name),strtolower($searchString))) && (preg_match('~(\d{2}\.\d{2}\.\d{4})~', $name, $matches))) {
$filesFound[] = $name;
$Dates[] = $matches[1];
foreach($filesFound as $ftbu){
$sql = 'LOAD DATA LOCAL INFILE \'/path/'.$ftbu.'.xlsx\' INTO TABLE '.$dbtable.'
FIELDS TERMINATED BY \',\'
ENCLOSED BY \'"\'
LINES TERMINATED BY \'\n\'
(#date, number, #name)
set date=str_to_date(#date,\'%Y-%m-%d\'),
name = \'AAA\'
';
$Db->query($sql);
echo $Db->error;
}
}
}
Link:
Unanswered Question
I am trying to export sql query result to csv file it works but the out csv data does not look good and strange.
MySQL Table Download Link:
https://www.dropbox.com/s/vtr215bcxqo3wsy/data.sql?dl=0
CSV Generated by sql query:
Download Link to original Generated CSV File: https://www.dropbox.com/s/fnjf7ycmh08hd22/data.csv?dl=0
I am using following code:
$query = <<<EOL
SELECT * FROM data ORDER BY FN ASC limit 3
INTO OUTFILE 'folder/data.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
EOL;
$result = mysql_query($query);
Why csv format is looking so weird and unacceptable?
If I try same code for some other table then everything works like charm then what's wrong?
See final answer below
It looks like your lines are terminated by \\n and it is throwing the extra slashes in random places.
Instead try a double slash followed by an n (\\n) and see what happens:
$query = <<<EOL
SELECT * FROM data ORDER BY FN ASC limit 3
INTO OUTFILE 'folder/data.csv'
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\n'
EOL;
$result = mysql_query($query);
EDIT
Final Answer
Another observation: I noticed that in your PROP_TYPE field, there are \r\n characters. Is there any way you can filter them out in your query using the REPLACE() function?
I know you are looking for a solution that is SQL based, and this is a hard issue because of the massive amount of data. Hope this leads you to the correct solution.
As you mentioned, using update data set PROP_TYPE = replace(PROP_TYPE, '"','') fixed the issue.
Consider simply using PHP to connect to MySQL, run query, then output to csv.
<?php
$host="localhost";
$username="user";
$password="password";
$database="dbName";
# open connection
try {
$dbh = new PDO("mysql:host=$host;dbname=$database",$username,$password);
}
catch(PDOException $e) {
echo $e->getMessage();
}
$sql = "SELECT * FROM data ORDER BY FN ASC limit 3;";
$STH = $dbh->query($sql);
$STH->setFetchMode(PDO::FETCH_ASSOC);
while($row = $STH->fetch()) {
# write to csv file
$fs = fopen("folder/data.csv","a");
fputcsv($fs, $row);
fclose($fs);
}
# close connection
$dbh = null;
?>
Finally I fixed my issue.
Actually #Terry is right. There was some issue with a field PROP_TYPE in table.
PROP_TYPE field had double quotes " in its values that was causing issue.
For example
PROP_TYPE
"Value 1"
"Value 2" ....
So first of all I had to remove extra double quotes using update data set PROP_TYPE = replace(PROP_TYPE, '"','') so now my issue is fixed.
Thanks all of you for your efforts.
I really appreciate.
I am quite new to PHP and need your help for an issue. I will explain my problem in the following steps:
1- I have a CSV file and I would like to import it to MySQL database with PHP.
2- When I add commas at the end of each lines in the CSV file, I could import it by using IMPORT option in Mysql db. I can not import it without adding commas at the end of lines.
3- I want to add a new line into my PHP code which will add commas at the end of each record/line and will enable the data imported without any error.
Here is my code:
<?php
//read file
$csvfile=file_get_contents("/samba/import/Tbl_HRList.csv");
$lines = explode(PHP_EOL, $csvfile);
$array = array();
foreach ($lines as $line) {
$field = str_getcsv($line);
if ($field[0] != ''){
$sql="INSERT INTO HR_Tbl (Employee_ID, First Name, Prefix, Surname, Location, Organizational Code, Organizational Unit, Team, Team Code, Function, Function code, T24 Department Code, Date in service (GBI), Company email, End Date Contract, End Date Systems, Temp. Leave Date, Temp. Leave End, Temp. Leave Code)
VALUES
('$field[0]','$field[1]','$field[2]','$field[3]','$field[4]','$field[5]','$field[6]','$field[7]','$field[8]','$field[9]','$field[10]','$field[11]','$field[12]','$field[13]','$field[14]','$field[15]','$field[16]','$field[17]','$field[18])";
}
//insert record to database
if ($conn->query($sql) === TRUE) {
echo "New record created successfully". PHP_EOL;
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
else
{
continue;
}
}
$conn->close();
?>
Thanks in advance
Alex
Use Load Data Infile and completely abandon the above idea.
You can have csv lines terminate with \n or \r\n
You can have row one of that import file either have or not have column names.
if you have column names, you skip the first row.
If you have column names on row1, you can have, say, 12 columns, and just use 4 if you want.
It is a very flexible construct. Check it out.
https://dev.mysql.com/doc/refman/5.1/en/load-data.html
LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;