Processing a large csv file limiting up to a line - php

Question

So after encountering the memory limits I copied the following code.
It worked great. My only issue now is in the context of processing a csv file, the structure limits based on the size of chunk, but this would mean it could cut a row mid way through.
What could I do to ensure when a chunk is made, that it ends up to the /n.

Answer 1

I'm interested to know what others do.
I changed the limit to based on a certain amount of lines to read instead of the size limit. Instead of using fread, i used fgets to get the whole line.
Once again, this is all derived from the code linked in the question.
<?php
function file_get_contents_chunked($file,$chunk_size,$callback)
{
try
{
$handle = fopen($file, "r");
$i = 0;
$x = 0;
$chunk = array();
while (!feof($handle)) {
while ($row = fgets($handle)) {
// can parse further $row by usingstr_getcsv
$x ++;
$chunk[] = $row;
if ($x == $chunk_size) {
call_user_func_array($callback, array($chunk, &$handle, $i));
unset($chunk);
$x = 0;
}
}
}
fclose($handle);
}
catch(Exception $e)
{
trigger_error("file_get_contents_chunked::" . $e->getMessage(),E_USER_NOTICE);
return false;
}
return true;
}
?>
//Fixed for what I actually intended, limit by x amount of lines

Answer 2

You can first try setting the memory limit with :
ini_set('memory_limit', '32MB');
Then, to read a line at once :
$handle = fopen("inputfile.csv", "r");
if ($handle) {
while (($line = fgets($handle)) !== false) {
// process the line read.
}
} else {
// error opening the file.
}
fclose($handle);
Based on your code example, this would give :
function file_get_contents_chunked($file,$chunk_size,$callback)
{
$handle = fopen($file, "r");
$i = 0;
if ($handle) {
while (($line = fgets($handle)) !== false) {
call_user_func_array($callback,array($line,&$handle,$i));
$i++;
}
} else {
return false;
}
fclose($handle);
return true;
}
Note that $chunk_size parameter is not needed anymore... so you can remove it if you want.
You can also use this function to read a sav file line by line :
fgetcsv(file,length,separator,enclosure);
Example 1
<?php
$file = fopen("contacts.csv","r");
print_r(fgetcsv($file));
fclose($file);
?>
The CSV file:
Kai Jim, Refsnes, Stavanger, Norway
Hege, Refsnes, Stavanger, Norway
The output of the code above will be:
Array
(
[0] => Kai Jim
[1] => Refsnes
[2] => Stavanger
[3] => Norway
)