I tried to change from insert to to load data infile for uploading a large CSV file.
I explored some examples on how to use load data infile. But it's not working and giving this Error:
SQLSTATE[HY000]: General error: 29 File '/direct1/#tmp/phpFZLLYA' not found (Errcode: 13 "Permission denied").
This is my code:
$target_dir = '/direct1/#tmp/';
$target_file = $target_dir . basename($_FILES["file"]["tmp_name"]);
$stmt = $dbcon->prepare("LOAD DATA INFILE '$target_file' INTO TABLE Rain FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES(id, date, vol, day)");
$stmt->execute();
When I try to change to load data local infile based on related discussion, error code does not appear but the data is not inserted to the database. I am using INNODB storage engine.
I solved the error by using move_uploaded_file function and the file can be access.
Related
I'm trying to insert data into MySQL table using the load data infile query. I have an HTML form that accepts a text file for upload, and then tries to process.
File
00|Oakleaf Forest|Norfolk|VA|United States|PC
01|Igo|Lowell|MA|United States|PC
02|Mint|Lawrence|MA|United States|PC
03|Hosa|Boston|MA|United States|PC
HTML
<p>Please upload file below:</p>
<form action="process.php" method="post" enctype="multipart/form-data">
<input type="file" name="file" accept="text/plain">
<input type="submit">
</form>
PHP
/* processFile
* Process the submitted file
*
* #param Array - file for processing
*/
function processFile($file) {
// Get global variable
global $DB;
// file contents
$file_contents = $file["tmp_name"];
$SQL_statement = "LOAD DATA INFILE '$file_contents' INTO TABLE organization FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n' IGNORE 2 LINES
(ein, legal_name, city_name, state_name, country_name, description);";
// Run SQL query
$DB->execute($SQL_statement);
}
When I upload the file, nothing is inserted into the table.
I've tried debugging with echo($SQL_statement);, which returns the following query:
LOAD DATA INFILE '/Applications/XAMPP/xamppfiles/temp/phpOtL2q1' INTO TABLE organization FIELDS TERMINATED BY '|' LINES TERMINATED BY ' ' IGNORE 2 LINES (ein, legal_name, city_name, state_name, country_name, description);
What am I missing here?
LOAD DATA INFILE requires that the file be readable by the database process. I doubt that the temporary upload file is world-readable, it's probably only readable by the webserver userid.
One thing you can try is changing the permissions on the temp file first:
chmod($file["tmp_name"], 0444);
However, this still might not work if the directory containing the temp files is not world-readable. Another solution is to use LOAD DATA LOCAL INFILE. This reads the file in PHP, not the database server, so any file that the PHP process can read can be loaded. It's not as efficient, since all the file data has to be sent over the database connection, but it avoids permission problems.
So try the first method first, if it doesn't work try the second.
this code with the path works fine for me in localhost
$sql = "LOAD DATA INFILE 'C://xampp/htdocs/xampp/www/tool/".#mysql_escape_string($this->file_name).
"' INTO TABLE `".$this->table_name.
"` FIELDS TERMINATED BY '".#mysql_escape_string($this->field_separate_char).
"' OPTIONALLY ENCLOSED BY '".#mysql_escape_string($this->field_enclose_char).
"' ESCAPED BY '".#mysql_escape_string($this->field_escape_char).
"' LINES TERMINATED BY '" .#mysql_escape_string("\r\n") . "' ".
($this->use_csv_header ? " IGNORE 1 LINES " : "")
."(`".implode("`,`", $this->arr_csv_columns)."`)";
//echo $sql;
mysqli_query(DB::cxn(),$sql) or die("Import Error: " . mysqli_error(DB::cxn()));
// Delete the file now it's been imported
unlink("C:/xampp/htdocs/xampp/www/tool/". $this->file_name);
now when i run in centos server i give path like
$sql = "LOAD DATA INFILE 'var/www/html/tool/".#mysql_escape_string($this->file_name).
unlink("var/www/html/tool/". $this->file_name);
Cant get stat of 'var/lib/mysql/var/www/html/tool/file.csv (Errcode:2)
now when i use LOAD DATA LOCAL INFILE 'var/www/html/tool it is giving
cant find file 'var/www/html/file.scv
my web root folder is in /var/www/html/tool and i access this via xxx.xxx.xx.xx/tool
everytime a csv file is uploaded in server location /var/www/html/tool with OWNER as apache-Apache having Read and Write
Group as apache having only Read Only permisssion
i tried like chmod 777 -R /var/www/html/tool but the file is still uploading in apache as group and apache-Apache as Owner
Yes I have seen the other post simillar to this and used LOAD DATA LOCAL INFILE
Plz Suggest
Just Add LOCAL before INFILE in your Query
$sql = LOAD DATA LOCAL INFILE 'C://xampp/htdocs/xampp/www/tool/"
You are using var/www/html/tool/ as a path and not /var/www/html/tool/, thus the path is relative to the mysql directory in /var/lib/mysql. Just add another slash at the front.
I have this method that uploads a file of a CSV format, but now i want to know how to upload it into columns in my database.
My method:
public function postUpload ()
{
if (Input::hasFile('file')){
$file = Input::file('file');
$name = time() . '-' . $file->getClientOriginalName();
// Moves file to folder on server
$file->move(public_path() . '/uploads/CSV', $name);
return 'Ok'; // return for testing
}
}
So my question would be within this method how can i can put it into the database ?
this one should work for you, it uses PDO + mySql's "LOAD DATA" approach
private function _import_csv($path, $filename)
{
$csv = $path . $filename;
//ofcourse you have to modify that with proper table and field names
$query = sprintf("LOAD DATA local INFILE '%s' INTO TABLE your_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\"' LINES TERMINATED BY '\\n' IGNORE 0 LINES (`filed_one`, `field_two`, `field_three`)", addslashes($csv));
return DB::connection()->getpdo()->exec($query);
}
So combined with your code, it could be something like the below
public function postUpload ()
{
if (Input::hasFile('file')){
$file = Input::file('file');
$name = time() . '-' . $file->getClientOriginalName();
//check out the edit content on bottom of my answer for details on $storage
$storage = '/some/world/readible/dir';
$path = $storage . '/uploads/CSV';
// Moves file to folder on server
$file->move($path, $name);
// Import the moved file to DB and return OK if there were rows affected
return ( $this->_import_csv($path, $name) ? 'OK' : 'No rows affected' );
}
}
EDIT
One thing to be noted, as per the error you report in comments which is probably some permissions issue (OS error code 13: Permission denied)
Please see: http://dev.mysql.com/doc/refman/5.1/en/load-data.html
"For security reasons, when reading text files located on the server,
the files must either reside in the database directory or be readable
by all. Also, to use LOAD DATA INFILE on server files, you must have
the FILE privilege. See Section 5.7.3, “Privileges Provided by
MySQL”."
As reported on mySql bug tracker (http://bugs.mysql.com/bug.php?id=31670) it seems that you need particular permission for all the folders in the csv file path:
All parent directories of the infile need world-readable I think
aswell as just the directory and infile...
So for an infile here: /tmp/imports/site1/data.file
you would need (I think, 755 worked) r+x for 'other' on these
directories: /tmp /tmp/imports
as well as the main two: /tmp/imports/site1
/tmp/imports/site1/data.file
To sum up:
To solve the "sqlstate hy000 general error 13 can't get stat of..." issue you have to move the uploaded file to a location with proper permissions (so not neccessarily the current one you are using) try something like "/tmp/import".
While load data infile is the quickest way, I prefer to use a lib like https://github.com/ddeboer/data-import or https://github.com/goodby/csv for 2 reasons.
It is extensible, what if your data source changes to excel files or a mongo db or some other method?
It is mallable, if you need to convert dates, or strings or numbers you can do it conditionally which cannot be done with a batch command.
my 2c
Originally $filepath is like this
$filepath=$_SERVER['DOCUMENT_ROOT']."/".SCRIPTFOLDER."somefile.txt";
$sqlquerynew = "LOAD DATA INFILE '".$filepath."' INTO TABLE mytable FIELDS TERMINATED BY ','";
mysql_query($sqlquery);
That works. It sits on the server.
But what if I want to put somefile.txt outside the same server? If I put that in dropbox, and reuse the code like the following, it doesn't do anything.
$sqlquerycreate = / ** Here insert DB scheme code **/
$filepath="http://example.com/somefile.txt";
$sqlquerynew = "LOAD DATA INFILE '".$filepath."' INTO TABLE mytable FIELDS TERMINATED BY ','";
mysql_query($sqlquery);
What should I do instead? The text file contains the data for the table.
Thanks.
$data_file = file_get_contents('http://dl.dropbox.com/u/14655573/john/nyccrash.txt');
file_put_contents('C:\xampp\htdocs\import\nyccrash.txt', $data_file);
$filepath = "C:\xampp\htdocs\import\nyccrash.txt";
echo $filepath;
$sqlquery = "LOAD DATA INFILE '".$filepath."' INTO TABLE nyccrash FIELDS TERMINATED BY ',' (crash_year, accident_type, collision_type,weather_condition,light_condition,x_coordinate,y_coordinate)";
Okay. I see this file now. But it didn't get import to the DB. This is XAMPP. But I will, eventually, move to Ubuntu. What's the problem with this code?
Try this:
$file = file_get_contents("http://example.com/somefile.txt");
$filepath=$_SERVER['DOCUMENT_ROOT']."/".SCRIPTFOLDER."somefile.txt";
file_put_contents($filepath, $file);
$sqlquerycreate = / ** Here insert DB scheme code **/
$sqlquerynew = "LOAD DATA INFILE '".$filepath."' INTO TABLE mytable FIELDS TERMINATED BY ','";
mysql_query($sqlquery);
Hope it helps..
The LOAD DATA INFILE reads a file from the client or from the server, not from the internet.
You could use copy() to first download the file to a temporary directory, and then point LOAD DATA INFILE to that directory.
Create first a temporary file on your server:
$tmpFile = tempnam('my-tmp-dir', 'TMP');
$ok = copy('http://example.com/somefile.txt', $tmpFile);
if ( !$ok ) exit('ERROR!');
// Then use $filepath in your SQL query
$sqlquerynew = "LOAD DATA INFILE '".$filepath."' INTO TABLE mytable FIELDS TERMINATED BY ','";
mysql_query($sqlquery);
unlink($tmpFile);
You can first import data in local file with
//read data from remote file
$data_file = file_get_contents($http_url);
// put data in a local file
$temp_file = tempnam(sys_get_temp_dir(),'TMP');
file_put_contents($temp_file,$data_file);
//now use $temp_file into the SQL query
....
//Delete temp file
unlink($temp_file);
Please note that PHP has functions to get the system's TEMP path, and to generate a temporary filename.
In the /tmp you should not have permissions issues.
I have csv file exported from access. (dates formatted to match mysql)
I need to import the data into a mysql DB through code.
when I import the file through PhpMyAdmin it worked beautifully.
$fname = $_FILES['sel_file']['name'];
$filename = $_FILES['sel_file']['tmp_name'];
$sql="LOAD DATA INFILE '../filepath/data.txt' INTO TABLE table1 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"';";
mysql_query($sql)or die(mysql_error());
I tried using the file path or using the variable $filename and both gave me the same error
I get the following error:
Access denied for user 'uname'#'%' (using password: YES)
I set the permissions to be 777, The databse was created allowing direct access. (I am able to opload the file and read from it using an "INSERT" statement, but not load data.)
A. Is the LOAD DATA statement wrong?
B. What other things do I have to take care of in order to use the LOAD DATA command.
Thank you all!
Try LOAD DATA LOCAL INFILE
It may be a authentication problem. Please check the permissions for Import/export data in mysql.