Reading from a big table - php

Question

I want to read data from a table (or view) on SQL server 2008R2 using PHP 5.4.24 and freetds 0.91.
In PHP, I write:
$ret = mssql_query( 'SELECT * FROM mytable', $Conn ) ;
Then I read rows one at a time, process them, all is ok. Except, when the table is really big, then I get an error:
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 4625408 bytes)
in /home/prove/test.php on line 43
The error happens in the mssql_query(), so it does not matter how I query.
Altering the query to return less rows or less columns is not viable, because I must read a lot of data from many tables in a limited time.
What can I do to convince PHP to read in memory one row at a time, or a reasonable number at a time?

Answer 1

I agree with #James in the Question comments, if you need to read a table so big it exhausts your memory to stash the results before even handing back to PHP, it probably just means you need to figure out a better way. However, here is a possible solution (untested and I've only used MSSQL a couple of times; but if not perfect, I tried my best):
$ret = mssql_query( 'SELECT COUNT(*) as TotalRows FROM mytable', $Conn);
$row = mssql_fetch_row($ret);
$offset = 0;
$increment = 50;
while ($offset < $row['TotalRows'])
{
$ret_2 = mssql_query("SELECT * FROM mytable ORDER BY Id ASC OFFSET {$offset} ROWS FETCH NEXT {$increment} ROWS ONLY", $Conn);
//
// loop over those 50 rows, do your thing...
//
$offset += $increment;
}
//
// there will probably be a remainder, so you'll have to account for that in the post-iteration; probably making the interior loop a function or method would be wise.
//