How to ocassionally shorten files? - php

Question

This is the code I use:
if(mt_rand(0,20000)==0)
{
$lines = file($fileName);
if (count($lines)>50000)
{
$lines=array_slice($lines, -50000, 50000, true);
}
$result=implode("\n",lines);
file_put_contents($fileName, $result . "\n",FILE_APPEND);
}
I often got this error:
[25-Nov-2013 23:20:40 UTC] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 33 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 02:41:54 UTC] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 27 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 09:56:49 UTC] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 72 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 12:44:32 UTC] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 2097152 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
[26-Nov-2013 13:53:31 UTC] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 2097152 bytes) in /home/datetrng/public_html/checkblocked.php on line 40
I guess reading the whole file may not be a good idea if we just want to shorten a file by erasing the beginning of the file.
Knows any alternative?

Answer 1

fopen fwrite fseek will probably come in handy for you

Answer 2

I think you only want the last 50000 lines in your file.
if(mt_rand(0,20000)==0)
{
$tmp_file = $fileName . '.tmp';
$cmd = "tail -n 50000 $fileName > $tmp_file";
exec($cmd);
rename($tmp_file, $fileName);
}
update for pure php
I make a file about 100,000 lines:
<?php
$file_name = 'tmp.dat';
$f = fopen($file_name, 'w');
for ($i = 0; $i < 1000000; $i++)
{
fwrite($f, str_pad($i, 100, 'x') . "\n");
}
fclose($f);
This file is about 97M.
[huqiu#localhost home]$ ll -h tmp.dat
-rw-rw-r-- 1 huqiu huqiu 97M Nov 27 06:08 tmp.dat
read the last 50000 lines
<?php
$file_name = 'tmp.dat';
$remain_count = 50000;
$begin_time = microtime(true);
$temp_file_name = $file_name . '.tmp';
$fp = fopen($file_name, 'r');
$total_count = 0;
while(fgets($fp))
{
$total_count++;
}
echo 'total count: ' . $total_count . "\n";
if ($total_count > $remain_count)
{
$start = $total_count - $remain_count;
echo 'start: ' . $start . "\n";
$temp_fp = fopen($temp_file_name, 'w');
$index = 0;
rewind($fp);
while($line = fgets($fp))
{
$index++;
if ($index > $start)
{
fwrite($temp_fp, $line);
}
}
fclose($temp_fp);
}
fclose($fp);
echo 'time: ' . (microtime(true) - $begin_time), "\n";
rename($temp_file_name, $file_name);
elapsed time: 0.63908791542053
total count: 1000000
start: 950000
time: 0.63908791542053
the result:
[huqiu#localhost home]$ ll -h tmp.dat
-rw-rw-r-- 1 huqiu huqiu 4.9M Nov 27 06:23 tmp.dat

Answer 3

Why not fseek the pointer to a position past the point you want to eliminate? You might also have better luck using fpassthru to save some memory.