Using PHP to import CSV data into mySQL, Part 2 - php

Following up on my last thread. Trying to import a user-generated CSV into MySQL via a PHP upload script. Uploads successfully, but I am not able to use LOAD DATA due to a permissions problem. Here is what I am trying to do instead:
$row = 1;
if (($handle = fopen($target_path, "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++)
{
$fullRow = $fullRow . $data[$c] . "," ;
}
echo $fullRow;
mysql_query("INSERT INTO foo (field1, field2, field3, field4) VALUES ('$fullRow')");
$fullRow = NULL;
}
fclose($handle);
}
echo $fullRow spits out a verbatim copy of the line from the CSV file, except for an additional comma on the end. Is this why the Insert is not working correctly? When I do a manual upload via phpMyAdmin, the CSV file is imported without issue. Or is there a problem with the VALUE ('$fullRow') bit of the code?

You can simply remove the last comma.
for ($c=0; $c < $num; $c++)
{
$fullRow = $fullRow . $data[$c] . "," ;
}
echo $fullRow;
$fullRow = substr($fullRow,0,-1);
And also you script is not ok.
mysql_query(" INSERT INTO foo (field1, field2, field3, field4) VALUES ('$fullRow') " );
$fullRow = NULL;

Paolo_NL_FR's fixes should get you up and running. The script could use some TLC though, and does not have even basic sql injection protection. Try something like this perhaps:
if (($handle = fopen($target_path, "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$values = implode(",", array_map('mysql_real_escape_string', $data));
mysql_query("INSERT INTO foo (field1, field2, field3, field4) VALUES ($values);");
}
fclose($handle);
}

Related

PHP fgetcsv with not valid CSV file

I have a CSV file with content:
Центральный;Московская область;117036;PC;JEEP;GRAND CHEROKEE;ДЖИП ГРАНД ЧЕРОКИ;В;01.01.0001;08.05.2014;2004;1J8G2Е8N94У161064;-;1J8G2Е8N94У161064;-;94;2;227;4701;1;1;29;2495;2073;ФИЗ ЛИЦО;-;АКАДЕМИЧЕСКИЙ (ЮЗАО) Р-Н;-;-;-
Центральный;Московская область;117036;PC;VAZ;LARGUS;ЛАДА ЛАРГУС FS015L;В;01.01.0001;08.05.2014;2014;ХТАFS015LЕ0811430;UА46515;ХТАFS015LЕ0811430;-;11;1;84;1598;1;1;09;2010;1260;"АЗИЯ МАТЕРИАЛ ХЭНДЛИНГ ООО;7728814946;АКАДЕМИЧЕСКИЙ (ЮЗАО) Р-Н;-;КРЖИЖАНОВСКОГО;2|21
Центральный;Московская область;117208;PC;TOYOTA;LANDCRUISER;ТОЙОТА ЛЕНД КРУЗЕР ПРАДО;В;01.01.0001;08.05.2014;2001;JТЕВN99J100077420;1122027;JТЕВN99J100077420;JТЕВN99J100077420;94;2;178;3378;1;1;21;2680;1900;ФИЗ ЛИЦО;-;ЧЕРТАНОВО СЕВЕРНОЕ (ЮАО) Р-Н;-;-;-
I tried import this CSV to SQL DB with this while loop:
while (($csv_data = fgetcsv($csv_file, 10000, ';', '"')) !== false) {
//some stuff
}
In this CSV file I have 3 records, but in SQL I see only 2. Problem in this line:
;1260;"АЗИЯ МАТЕРИАЛ ХЭНДЛИНГ ООО;7728814946;
Problen in extra character " before АЗИЯ. How I can correctly parse this CSV file? File is more then 200 Mb.
===== EDIT =======
Okay, obviously your CSV gets not parsed right I used different data and thought CSV is CSV and that example worked just as fine idk. Now it works with your data. Only problem is your ", this is causing problems
(I replaced your first element because I don't speak Russian and I cannot see differences in words)
$file_data = array_filter(explode("\n", file_get_contents("data.csv")));
$data = array();
foreach ($file_data as $key => $row) {
$data[] = str_getcsv($row, ';', '', "\\");
}
print_r($data);
===== OLD MESSAGE =======
I tried this PHP example and it worked just as fine.
/*
$row = 1;
if (($handle = fopen("data.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in roe $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
*/

how to extend the length of time when fgetcsv function of php is reading csv file?

am having a problem here, i read a csv file using fgetcsv function then loop it, and write it into an xml file...when am using the script in a not so big data, everything works perfectly and writes the xml file....but when i use it in a huge csv, the xml can't finished being written and end up in an unclosed token thing
if (($handle = fopen("C:\\xampp\htdocs\yii\branch\seo_tagsdev.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 10000000, ",")) !== FALSE) {
$num = count($data);
// echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++)
{
$url = $doc->createElement('url');
$root->appendChild($url);
$loc = $doc->createElement('loc',htmlentities($data[$c]));
$url->appendChild($loc);
$lastmod = $doc->createElement('lastmod',date('Y-m-d'));
$url->appendChild($lastmod);
$freq = $doc->createElement('changefreq','daily');
$url->appendChild($freq);
$priority = $doc->createElement('priority',0.9);
$url->appendChild($priority);
// echo $data[$c] . "<br />\n";
}
$doc->save("C:\\xampp\htdocs\yii\branch\seo_tagsdev.xml");
}
i even adjusted the length parameter of fgetcsv and yet it doesn't work at all

Tools / Scripts for importing csv to db table

In tried importing a csv into a table using phpMyAdmin and I'm getting permission errors so I'm looking for any php scripts available around that I can use.
What scripts are available? Can ane point to any free one's?
fgetcsv is the php function you should use.
For details see here.
Sample code here..
$row = 1;
if (($handle = fopen("test.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
// write insert query here
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
Thanks.

Importing large CSV into mysql database

I'm having a really troublesome time trying to import a large CSV file into mysql on localhost.
The CSV is about 55 MB and has about 750,000 rows.
I've rewritten the script so that it parses the CSV and dumps the rows one by one.
Here's the code:
$row = 1;
if (($handle = fopen("postal_codes.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++)
{
$arr = explode('|', $data[$c]);
$postcode = mysql_real_escape_string($arr[1]);
$city_name = mysql_real_escape_string($arr[2]);
$city_slug = mysql_real_escape_string(toAscii($city_name));
$prov_name = mysql_real_escape_string($arr[3]);
$prov_slug = mysql_real_escape_string(toAscii($prov_name));
$prov_abbr = mysql_real_escape_string($arr[4]);
$lat = mysql_real_escape_string($arr[6]);
$lng = mysql_real_escape_string($arr[7]);
mysql_query("insert into cities (`postcode`, `city_name`, `city_slug`, `prov_name`, `prov_slug`, `prov_abbr`, `lat`, `lng`)
values ('$postcode', '$city_name', '$city_slug', '$prov_name', '$prov_slug', '$prov_abbr', '$lat', '$lng')") or die(mysql_error());
}
}
fclose($handle);
}
The problem is that it's taking forever to execute. Any suuggested solutions would be appreciated.
You are reinventing the wheel. Check out the mysqlimport tool, which comes with MySQL. It is an efficient tool for importing CSV data files.
mysqlimport is a command-line interface for the LOAD DATA LOCAL INFILE SQL statement.
Either should run 10-20x faster than doing INSERT row by row.
Your problem is likely that you have autocommit on (by default) so MySQL is committing a new transaction for each insert. You should turn autocommit off with SET autocommit=0;. If you can switch to using the mysqli library (and you should if possible), you can use mysqli::autocommit(false) to turn off autocommitting.
$mysqli = new mysqli('localhost','db_user','my_password','mysql');
$mysqli->autocommit(false);
$stmt=$mysqli->prepare("insert into cities (`postcode`, `city_name`, `city_slug`, `prov_name`, `prov_slug`, `prov_abbr`, `lat`, `lng`)
values (?, ?, ?, ?, ?, ?, ?, ?);")
$row = 1;
if (($handle = fopen("postal_codes.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++)
{
$arr = explode('|', $data[$c]);
$stmt->bind_param('ssssssdd', $arr[1], $arr[2], toAscii(arr[2]), $arr[3], toAscii($arr[3]), $arr[4], $arr[6], $arr[7]);
$stmt->execute();
}
}
}
$mysqli->commit();
fclose($handle);
It will be much faster to use LOAD DATA if you can
try to do it in one query.
It could be limited by your my.cnf (mysql configuration) though
<?php
$row = 1;
$query = ("insert into cities ");
if (($handle = fopen("postal_codes.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++)
{
$arr = explode('|', $data[$c]);
$postcode = mysql_real_escape_string($arr[1]);
$city_name = mysql_real_escape_string($arr[2]);
$city_slug = mysql_real_escape_string(toAscii($city_name));
$prov_name = mysql_real_escape_string($arr[3]);
$prov_slug = mysql_real_escape_string(toAscii($prov_name));
$prov_abbr = mysql_real_escape_string($arr[4]);
$lat = mysql_real_escape_string($arr[6]);
$lng = mysql_real_escape_string($arr[7]);
$query .= "(`postcode`, `city_name`, `city_slug`, `prov_name`, `prov_slug`, `prov_abbr`, `lat`, `lng`)
values ('$postcode', '$city_name', '$city_slug', '$prov_name', '$prov_slug', '$prov_abbr', '$lat', '$lng'),";
}
}
fclose($handle);
}
mysql_query(rtrim($query, ","));
if it won't work, you can try this (disable automatical commit)
mysql_query("SET autocommit = 0");
$row = 1;
if (($handle = fopen("postal_codes.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++)
{
$arr = explode('|', $data[$c]);
$postcode = mysql_real_escape_string($arr[1]);
$city_name = mysql_real_escape_string($arr[2]);
$city_slug = mysql_real_escape_string(toAscii($city_name));
$prov_name = mysql_real_escape_string($arr[3]);
$prov_slug = mysql_real_escape_string(toAscii($prov_name));
$prov_abbr = mysql_real_escape_string($arr[4]);
$lat = mysql_real_escape_string($arr[6]);
$lng = mysql_real_escape_string($arr[7]);
mysql_query("insert into cities (`postcode`, `city_name`, `city_slug`, `prov_name`, `prov_slug`, `prov_abbr`, `lat`, `lng`)
values ('$postcode', '$city_name', '$city_slug', '$prov_name', '$prov_slug', '$prov_abbr', '$lat', '$lng')") or die(mysql_error());
}
}
fclose($handle);
}
I did this with SQL server:
I used SQL Bulkinsert command combined with data tables.
Data Tables reside in memory and are built from reading rows inside the file.
Each data table is built from a chunk of rows, not the entire file.
Keep track from the chunk processed by keeping pointers from last row read and max size of chunk.
When you are reading the file. exit the loop when the row id > last row + chunk size.
Keeping on looping and keep on inserting.
Also sometimes when you are using Load data if there are warnings the import will stop. You can use the keyword ignore.
LOAD DATA INFILE 'file Path' IGNORE INTO TABLE YOUR_Table
I had a similar situation where is was NOT feasible to use LOAD DATA. Transactions were at times unacceptable as well, as data needed to be checked for duplicates. Yet, the following drastically improved the process time for some of my import data files.
Before your while loop (CSV Lines) set autocommit to 0 and start a transaction (InnoDB only):
mysql_query('SET autocommit=0;');
mysql_query('START TRANSACTION;');
After your loop, commit and reset autocommit back to 1 (default):
mysql_query('COMMIT;');
mysql_query('SET autocommit=1;');
Replace mysql_query() with whatever Database object your code is using. I hope this helps others.

Reading CSV in PHP and out put its contents by array value

I want to display the contents of a CSV file by assigning it to a variable and then display it using echo statement.
This code is not working could someone point the error in it
$fh = fopen('db.csv', 'r');
$now = time();
$data=fgetcsv($fh);
$data[0]=$name;
echo $name;
Am a newbie to coding and scripting.
Thanks
This is the CSV line that I want to be printed
katz,26-11-2011,http//www.google.com
Why you first valorize the $data with an array of your CSV and then you overwrite the first position with the $name variable (which apparently is null)?
$data[0] = $name;
should be
$name = $data[0];
You could use the handy fgetcvs function for this.
Might as well post an example for want of a complete answer, shamelessly ripped from that same
manpage:
<?php
$row = 1;
if (($handle = fopen("test.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
echo "<p> $num fields in line $row: <br /></p>\n";
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}
?>
Hope that fits your needs. Happy coding.

Categories