PHP - Zero length (empty) CSV but only for larger datasets - php

Question

Although my application is built using the Yii framework this is more of a general PHP issue (I think).
I have some code which takes a Yii CActiveDataProvider, loops over it and builds a CSV export. This works great and correctly builds and sends the CSV.
I encounter a problem when trying to export a larger dataset. I have successfully output ~2500 records without any problem but when I run the exact same code for a larger set of data (~5000 records) the script appears to run ok but sends a zero length/blank CSV. I can't figure out why... it seems to run for a while and then sends the CSV, no errors or warnings in the logs. Could it be that the output is being flushed or similar before it's ready?
Code is as follows (added a couple of inline comments here for clarity):
<?php
header('Content-type: text/csv');
header('Content-Disposition: attachment; filename="vacancies.csv"');
set_time_limit(240); // I know this is arbitrarily long, it's just to avoid any timeout
$outstream = fopen("php://output", 'w');
$headings = array(
$vacancy->getAttributeLabel('vacancy_id'), // this is a Yii method that returns the active record attribute as a string
...
);
fputcsv($outstream, $headings, ',', '"');
foreach($dp->getData() as $vacancy){ // the getData() method pulls the next active record model out of the Yii dataprovider and the values for various attributes are set below
$row = array(
$vacancy->vacancy_id,
...
);
fputcsv($outstream, $row, ',', '"');
}
fclose($outstream);
?>
Any thoughts on why this is working ok up to a certain number of records?
Update
After re-checking the logs as suggested below I've found I am in fact running out of memory, doh!
I can write out to the filesystem and that gets me up to about 3000 records but then runs out of memory. Any idea of the best way to alter my code to avoid running out of memory?

Answer 1

Thanks very much for the suggestions to check the error logs, I had somehow missed an out of memory error I was getting.
The problem was in fact caused by the way I was using the CActiveDataProvider from the Yii Framework. Reading straight from the DataProvider as I was doing in my question was reading each row into memory, as the script ran on this meant I eventually ran out of memory available to PHP.
There are a couple of ways to fix this, one is to set pagination on the dataprovider to a smaller number of records and to manually iterate over the data, loading only the pagesize into memory each iteration.
The option I went for is to use a CDataProviderIterator to handle this for me $iterator = new CDataProviderIterator($dp); this prevents memory filling up with the records I'm retrieving.
Note that I also had to add an ob_flush(); call to prevent the output buffer from filling up with the CSV contents iteslf.
For reference I ended up with the following:
<?php
header('Content-type: text/csv');
header('Content-Disposition: attachment; filename="vacancies.csv"');
set_time_limit(240);
$outstream = fopen("php://output", 'w');
$headings = array(
$vacancy->getAttributeLabel('vacancy_id'),
...
);
fputcsv($outstream, $headings, ',', '"');
$iterator = new CDataProviderIterator($dp); // create an iterator instead of just using the dataprovider
foreach($iterator as $vacancy){ // use the new iterator here
$row = array(
$vacancy->vacancy_id,
...
);
fputcsv($outstream, $row, ',', '"');
ob_flush(); // explicitly call a flush to avoid filling the buffer
}
fclose($outstream);
?>
Would not have thought to go back and look at the logs again without the suggestion so many thanks :)