i have some trouble for a very simple loop because i goes out of memory and i really dont know why.
Hope someone will be able to help me.
Here is my code:
$full_list = array();
$fp = fopen($file_name, 'r');
while (($line = fgetcsv($fp, 0, $delimiter)) !== FALSE)
{
$val = array_slice($line, 0, 1);
$line = NULL;
unset($line);
if (in_array($val, $full_list) === FALSE)
$full_list[] = $val;
$val = NULL;
unset($val);
}
fclose($fp);
I tried the $line = NULL && then unset it as u can see but even like that it doesnt work, if the file is too big i'll get out of memory...
To be honest, i dont even understand why the memory even increase during the loop....
yes, you will always run out of memory, because, variable $full_list will always grow, till you will reach out of memory.
you need to do your stuff directly in while cycle.
and instead use array_slice($line, 0, 1) just use: $line[0]
Related
I am reading a file containing around 50k lines using the file() function in Php. However, its giving a out of memory error since the contents of the file are stored in the memory as an array. Is there any other way?
Also, the lengths of the lines stored are variable.
Here's the code. Also the file is 700kB not mB.
private static function readScoreFile($scoreFile)
{
$file = file($scoreFile);
$relations = array();
for($i = 1; $i < count($file); $i++)
{
$relation = explode("\t",trim($file[$i]));
$relation = array(
'pwId_1' => $relation[0],
'pwId_2' => $relation[1],
'score' => $relation[2],
);
if($relation['score'] > 0)
{
$relations[] = $relation;
}
}
unset($file);
return $relations;
}
Use fopen, fread and fclose to read a file sequentially:
$handle = fopen($filename, 'r');
if ($handle) {
while (!feof($handle)) {
echo fread($handle, 8192);
}
fclose($handle);
}
EDIT after update of question and comments to answer of fabjoa:
There is definitely something fishy if a 700kb file eats up 140MB of memory with that code you gave (you could unset $relation at the end of the each iteration though). Consider using a debugger to step through it to see what happens. You might also want to consider rewriting the code to use SplFileObject's CSV functions as well (or their procedural cousins)
SplFileObject::setCsvControl example
$file = new SplFileObject("data.csv");
$file->setFlags(SplFileObject::READ_CSV);
$file->setCsvControl('|');
foreach ($file as $row) {
list ($fruit, $quantity) = $row;
// Do something with values
}
For an OOP approach to iterate over the file, try SplFileObject:
SplFileObject::fgets example
$file = new SplFileObject("file.txt");
while (!$file->eof()) {
echo $file->fgets();
}
SplFileObject::next example
// Read through file line by line
$file = new SplFileObject("misc.txt");
while (!$file->eof()) {
echo $file->current();
$file->next();
}
or even
foreach(new SplFileObject("misc.txt") as $line) {
echo $line;
}
Pretty much related (if not duplicate):
How to save memory when reading a file in Php?
If you don't know the maximum line length and you are not comfortable to use a magic number for the max line length then you'll need to do an initial scan of the file and determine the max line length.
Other than that the following code should help you out:
// length is a large number or calculated from an initial file scan
while (!feof($handle)) {
$buffer = fgets($handle, $length);
echo $buffer;
}
Old question but since I haven't seen anyone mentioning it, PHP generators is a great way to reduce save memory consumption.
For example:
function read($fileName)
{
$fileHandler = fopen($fileName, 'rb');
while(($line = fgets($fileHandler)) !== false) {
yield rtrim($line, "\r\n");
}
fclose($fileHandler);
}
foreach(read(__DIR__ . '/filenameHere') as $line) {
echo $line;
}
allocate more memory during the operation, maybe something like ini_set('memory_limit', '16M');. Don't forget to go back to initial memory allocation once operation is done
Using PHP, how can I get the total number of rows that are in a CSV file? I'm using this method but cannot get it to work properly.
if (($fp = fopen("test.csv", "r")) !== FALSE) {
while (($record = fgetcsv($fp)) !== FALSE) {
$row++;
}
echo $row;
}
Create a new file reference using SplFileObject:
$file = new SplFileObject('test.csv', 'r');
Try to seek to the highest Int PHP can handle:
$file->seek(PHP_INT_MAX);
Then actually it will seek to the highest line it could in the file, there is your last line and the last line + 1 is equals to your total lines:
echo $file->key() + 1;
Tricky, but this will avoid you from loading the file contents into memory, which is a very cool thing to do when dealing with really large files.
Here's another option using file() to read the entire file into an array, automatically parsing new lines etc:
$fp = file('test.csv');
echo count($fp);
Also, since PHP5, you can pass in the FILE_SKIP_EMPTY_LINES... to skip empty lines, if you want to:
$fp = file('test.csv', FILE_SKIP_EMPTY_LINES);
Manual: http://php.net/manual/en/function.file.php
Try
$c =0;
$fp = fopen("test.csv","r");
if($fp){
while(!feof($fp)){
$content = fgets($fp);
if($content) $c++;
}
}
fclose($fp);
echo $c;
I know that this is pretty old, but actually I ran into the same question.
As a solution I would assume to use linux specific logic:
$rows = shell_exec('$(/bin/which cat) file.csv | $(/bin/which tr) "\r" "\n" | $(which wc) -l');
NOTE: this only works for linux only and this only should be used if you are 100% certain that your file has no multiline-cells
CSV rows are separated by line breaks. Therefore, split the rows by line breaks, and you will get an array of rows, which is countable.
if (($fp = fopen("test.csv", "r")) !== FALSE) {
$rows = explode("\n", $fp);
$length = count($rows);
echo $length;
}
Note; none of higher-upvoted solutions that count lines in the file are reliable, as they are only counting the lines, not the csv entries (which can contain newline characters)
I'm using a similar solution to op, and it works perfectly, but with op's code the while part can break on empty lines, which is potentially his problem.
So it looks like this (edited op's code)
$rowCount=0;
if (($fp = fopen("test.csv", "r")) !== FALSE) {
while(!feof($fp)) {
$data = fgetcsv($fp , 0 , ',' , '"', '"' );
if(empty($data)) continue; //empty row
$rowCount++;
}
fclose($fp);
}
echo $rowCount;
I find this the most reliable:
$file = new SplFileObject('file.csv', 'r');
$file->setFlags(
SplFileObject::READ_CSV |
SplFileObject::READ_AHEAD |
SplFileObject::SKIP_EMPTY |
SplFileObject::DROP_NEW_LINE
);
$file->seek(PHP_INT_MAX);
$lineCount = $file->key() + 1;
I know this is an old post, but I've been googling this issue, and found that the only problem with the original code was that you need to define $row outside the while loop, like this:
if (($fp = fopen("test.csv", "r")) !== FALSE) {
$row = 1;
while (($record = fgetcsv($fp)) !== FALSE) {
$row++;
}
Just in case it helps someone :)
echo $row;
}
In case you are getting the file from a form
$file = $_FILES['csv']['tmp_name'];
$fp = new SplFileObject($file, 'r');
$fp->seek(PHP_INT_MAX);
echo $fp->key() + 1;
$fp->rewind();
Works like charm!!!!!!!!!!!!!!!!!!
$filename=$_FILES['sel_file']['tmp_name'];
$file=fopen($filename,"r");
$RowCount=0;
while ((fgetcsv($file)) !== FALSE)
{
$RowCount++;
}
echo $RowCount;
fclose($file);
I am new at php programming but I have been stuck with this code for some time.
I would like to read a .csv file line by line and then save its values in a list of arrays.
$file = fopen('Sub-Companies.csv', 'r');
while (($line =
fgetcsv($file)) !== FALSE) {
print_r($line);
list($customer_id[],$company_name[],$department[],$employee[],$country[],$zipcode[],$address[],$city[],
$smth1[], $smth2[], $phone_no1[],$phone_no2[],$email[],$website[],
$customer_no[],$problem1[],$problem2[]) = explode(";",$line); }
fclose($file); var_dump($customer_id);
The problem is that, although it is read correctly the file, then the explode is not working and the arrays appear to be null.
One thing that I am considering is that some arrays have more ";" than others, so that might be a problem, that is why I have the arrays $problem1 and $problem2, in order to store the values of this arrays.
Any help would be great!
You're using fgetcsv() in the wrong way.
We've come to this solution while chatting here on StackOverflow.
<?php
// Create file data.csv with your data
$handle = fopen('Sub-Companies.csv', 'r');
$customer_id = array();
$xyz_array = array();
// ...
// Better use a specified length (second parameter) instead of 0
// It slows down the whole process of reading the data!
while (($line = fgetcsv($handle, 0, ';')) !== FALSE) {
$customer_id[] = $line[0];
$xyz_array[] = $line[1];
}
I have a csv file that has like 30,000 rows in it. It also has like 9 columns. In the interest of speeding up the processing of everything I want to reduce the file to the two columns that I need and remove the rest. here is what I have done.
$retardment=1;//17;// 151; //499;// 991;// 1877
if (($handle = fopen($source, "r")) !== FALSE) {
$stock_handle = fopen($source_stock, "w+");
$row=0;
$col=array();
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
unset($line[1]);
unset($line[2]);
unset($line[3]);
unset($line[4]);
unset($line[5]);
unset($line[6]);
unset($line[8]);
unset($line[9]);
if($row%$retardment<1){
fputcsv($stock_handle, $line);
}
unset($line);
$row++;
}
fclose($handle);
fclose($stock_handle);
}
I am coping it to a new file and this works... but it seems to be pretty slow. Any ideas on how to make it faster? Thank you for the help.
Cheers -Jeremy
{EDIT}
So far this seems to take just as long. But works just fine
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row%$retardment<1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
$row++;
}
You could replace those unset() calls with...
$line = array($line[0], $line[7]);
Alternatively, remember that unset() takes multiple arguments...
unset($line[1], $line[2], ...);
You can speed it up fractionally more, but again it's a microtime()-measurable improvement: perception is that it won't be noticeably faster.
while (($line = fgetcsv($handle, 100000, ",")) !== FALSE) {
if($row++ % $retardment < 1){
fputcsv($stock_handle, array($line[0],$line[7]));
}
}
but as your script is IO-bound, it's the actual reads and writes that are the slowest functions, and you can't speed those up.
Using stream_copy_to_stream() with a stream input filter might be another approach, but you won't see much noticeable improvement unless you can reduce disk access times
I am reading a file containing around 50k lines using the file() function in Php. However, its giving a out of memory error since the contents of the file are stored in the memory as an array. Is there any other way?
Also, the lengths of the lines stored are variable.
Here's the code. Also the file is 700kB not mB.
private static function readScoreFile($scoreFile)
{
$file = file($scoreFile);
$relations = array();
for($i = 1; $i < count($file); $i++)
{
$relation = explode("\t",trim($file[$i]));
$relation = array(
'pwId_1' => $relation[0],
'pwId_2' => $relation[1],
'score' => $relation[2],
);
if($relation['score'] > 0)
{
$relations[] = $relation;
}
}
unset($file);
return $relations;
}
Use fopen, fread and fclose to read a file sequentially:
$handle = fopen($filename, 'r');
if ($handle) {
while (!feof($handle)) {
echo fread($handle, 8192);
}
fclose($handle);
}
EDIT after update of question and comments to answer of fabjoa:
There is definitely something fishy if a 700kb file eats up 140MB of memory with that code you gave (you could unset $relation at the end of the each iteration though). Consider using a debugger to step through it to see what happens. You might also want to consider rewriting the code to use SplFileObject's CSV functions as well (or their procedural cousins)
SplFileObject::setCsvControl example
$file = new SplFileObject("data.csv");
$file->setFlags(SplFileObject::READ_CSV);
$file->setCsvControl('|');
foreach ($file as $row) {
list ($fruit, $quantity) = $row;
// Do something with values
}
For an OOP approach to iterate over the file, try SplFileObject:
SplFileObject::fgets example
$file = new SplFileObject("file.txt");
while (!$file->eof()) {
echo $file->fgets();
}
SplFileObject::next example
// Read through file line by line
$file = new SplFileObject("misc.txt");
while (!$file->eof()) {
echo $file->current();
$file->next();
}
or even
foreach(new SplFileObject("misc.txt") as $line) {
echo $line;
}
Pretty much related (if not duplicate):
How to save memory when reading a file in Php?
If you don't know the maximum line length and you are not comfortable to use a magic number for the max line length then you'll need to do an initial scan of the file and determine the max line length.
Other than that the following code should help you out:
// length is a large number or calculated from an initial file scan
while (!feof($handle)) {
$buffer = fgets($handle, $length);
echo $buffer;
}
Old question but since I haven't seen anyone mentioning it, PHP generators is a great way to reduce save memory consumption.
For example:
function read($fileName)
{
$fileHandler = fopen($fileName, 'rb');
while(($line = fgets($fileHandler)) !== false) {
yield rtrim($line, "\r\n");
}
fclose($fileHandler);
}
foreach(read(__DIR__ . '/filenameHere') as $line) {
echo $line;
}
allocate more memory during the operation, maybe something like ini_set('memory_limit', '16M');. Don't forget to go back to initial memory allocation once operation is done