Related
So, i have a database with big data. The data to use is currently about 2,6 GB.
All the data need to be written to a text file for later use in another scripts.
The data is being limited per file and splitted in multiple parts. 100 results per file (around 37MB each file). Thats about 71 files.
The data is json data that is being serialized and then encrypted with openssl.
The data is correctly being written to the files, untill the max execution time is reached after 240 seconds. That's after about 20 files...
Well, i can just extend that time, but thats not the problem.
The problem is the following:
Writing file 1-6: +/- 5 seconds
Writing file 7-8: +/- 7 seconds
Writing file 9-11: +/- 12 seconds
Writing file 12-14: +/- 17 seconds
Writing file 14-16: +/- 20 seconds
Writing file 16-18: +/- 23 seconds
Writing file 19-20: +/- 27 seconds
Note: time is needed time per file
In other words, with every file im writing, the writing time per file goes significantly up, what causes the script to be slow offcourse.
The structure of the script is a bit like this:
$needed_files = count needed files/parts
for ($part=1; $part<=$needed_files; $part++) { // Loop throught parts
$query > mysqli select data
$data > json_encode > serialize > openssl_encrypyt
file_put_contents($filename.$part, $data, LOCK_EX);
}
WORKING CODE AFTER HELP
$notchDetails = mysqli_query($conn, "SELECT * FROM notches WHERE projectid = ".$projectid."");
$rec_count = 0;
$limit = 100;
$part = 1;
while ($notch = mysqli_fetch_assoc($notchDetails)) {
$data1[] = $notch;
$rec_count++;
if ($rec_count >= $limit) {
$data = json_encode($data1);
$data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
$filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
file_put_contents($filename, $data, LOCK_EX);
$part++;
$rec_count = 0;
$data = $data1 = "";
}
}
if ($data1 != "") {
$data = json_encode($data1);
$data = openssl_encrypt(bin2hex($data), "aes128", $pass, false, $iv);
$filename = $mainfolder."/".$projectfolder."/".$subfolder."/".$fname.".part".$part."".$fext;
file_put_contents($filename, $data, LOCK_EX);
}
mysqli_free_result($notchDetails);
Personally I would have coded this as a single SELECT with no LIMIT and then based on a $rec_per_file = ?; write the outputs from within the single while get results loop
Excuse the cryptic code, you didnt give us much of a clue
<?php
//ini_set('max_execution_time', 600); // only use if you have to
$filename = 'something';
$filename_suffix = 1;
$rec_per_file = 100;
$sql = "SELECT ....";
Run query
$rec_count = 0;
while ( $row = fetch a row ) {
$data[] = serialize > openssl_encrypyt
$rec_count++;
if ( $rec_count >= $rec_per_file ) {
$json_string = json_encode($data);
file_put_contents($filename.$filename_suffix,
$json_string,
LOCK_EX);
$filename_suffix++; // inc the suffix
$rec_count = 0; // reset counter
$data = array(); // clear data
// add 30 seconds to the remaining max_execution_time
// or at least a number >= to the time you expect this
// while loop to get back to this if statement
set_time_limit(30);
}
}
// catch the last few rows
$json_string = json_encode($data);
file_put_contents($filename.$filename_suffix, $data, LOCK_EX);
Also I am not sure why you would want to serialize() and json_encode()
I had a thought, based on your comment about execution time. If you place a set_time_limit(seconds) inside the if inside the while loop it might be cleaner, and you would not have to set ini_set('max_execution_time', 600); to a very large number, which if you have a real error in here may cause PHP continue processing for a long time before kicking the script out.
From the manual:
Set the number of seconds a script is allowed to run. If this is reached, the script returns a fatal error. The default limit is 30 seconds or, if it exists, the max_execution_time value defined in the php.ini.
When called, set_time_limit() restarts the timeout counter from zero. In other words, if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out.
Is there a PHP equivalent to setting timeouts in JavaScript?
In JavaScript you can execute code after certain time has elapsed using the set time out function.
Would it be possible to do this in PHP?
PHP is single-threaded, and in general PHP is focused on the HTTP request cycle, so this would be tricky to allow a timeout to run code, potentially after the request is done.
I can suggest you look into Gearman as a solution to delegate work to other PHP processes.
You can use the sleep() function:
int sleep ( int $seconds )
// Delays the program execution for the given number of seconds.
Example:
public function sleep(){
sleep(1);
return 'slept for 1 second';
}
This is ugly, but basically works:
<?php
declare(ticks=1);
function setInterval($callback, $ms, $max = 0)
{
$last = microtime(true);
$seconds = $ms / 1000;
register_tick_function(function() use (&$last, $callback, $seconds, $max)
{
static $busy = false;
static $n = 0;
if ($busy) return;
$busy = true;
$now = microtime(true);
while ($now - $last > $seconds)
{
if ($max && $n == $max) break;
++$n;
$last += $seconds;
$callback();
}
$busy = false;
});
}
function setTimeout($callback, $ms)
{
setInterval($callback, $ms, 1);
}
// user code:
setInterval(function() {
echo microtime(true), "\n";
}, 100); // every 10th of a second
while (true) usleep(1);
The interval callback function will only be called after a tickable PHP statement. So if you try to call a function 10 times per second, but you call sleep(10), you'll get 100 executions of your tick function in a batch after the sleep has finished.
Note that there is an additional parameter to setInterval that limits the number of times it is called. setTimeout just calls setInterval with a limit of one.
It would be better if unregister_tick_function was called after it expired, but I'm not sure if that would even be possible unless there was a master tick function that monitored and unregistered them.
I didn't attempt to implement anything like that because this is not how PHP is designed to be used. It's likely that there's a much better way to do whatever it is you want to do.
Without knowing a use-case for your question it's hard to answer it:
If you want to send additional data to the client a bit later you can do a JS timeout on the client side with a handler that will make a new HTTP request to PHP.
If you want to schedule some task for a later time you can store that in a database and poll the DB in regular intervalls. It's not the best peforming solution but relatively easy to implement.
if ($currenturl != $urlto)
exit( wp_redirect( $urlto ) );
You can replace above two line with below code inside your function
if ($currenturl != $urlto)
header( "refresh:10;url=$urlto" );
I have just found out that my script gives me a fatal error:
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 440 bytes) in C:\process_txt.php on line 109
That line is this:
$lines = count(file($path)) - 1;
So I think it is having difficulty loading the file into memeory and counting the number of lines, is there a more efficient way I can do this without having memory issues?
The text files that I need to count the number of lines for range from 2MB to 500MB. Maybe a Gig sometimes.
Thanks all for any help.
This will use less memory, since it doesn't load the whole file into memory:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo $linecount;
fgets loads a single line into memory (if the second argument $length is omitted it will keep reading from the stream until it reaches the end of the line, which is what we want). This is still unlikely to be as quick as using something other than PHP, if you care about wall time as well as memory usage.
The only danger with this is if any lines are particularly long (what if you encounter a 2GB file without line breaks?). In which case you're better off doing slurping it in in chunks, and counting end-of-line characters:
$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
$line = fgets($handle, 4096);
$linecount = $linecount + substr_count($line, PHP_EOL);
}
fclose($handle);
echo $linecount;
Using a loop of fgets() calls is fine solution and the most straightforward to write, however:
even though internally the file is read using a buffer of 8192 bytes, your code still has to call that function for each line.
it's technically possible that a single line may be bigger than the available memory if you're reading a binary file.
This code reads a file in chunks of 8kB each and then counts the number of newlines within that chunk.
function getLines($file)
{
$f = fopen($file, 'rb');
$lines = 0;
while (!feof($f)) {
$lines += substr_count(fread($f, 8192), "\n");
}
fclose($f);
return $lines;
}
If the average length of each line is at most 4kB, you will already start saving on function calls, and those can add up when you process big files.
Benchmark
I ran a test with a 1GB file; here are the results:
+-------------+------------------+---------+
| This answer | Dominic's answer | wc -l |
+------------+-------------+------------------+---------+
| Lines | 3550388 | 3550389 | 3550388 |
+------------+-------------+------------------+---------+
| Runtime | 1.055 | 4.297 | 0.587 |
+------------+-------------+------------------+---------+
Time is measured in seconds real time, see here what real means
True line count
While the above works well and returns the same results as wc -l, if the file ends without a newline, the line number will be off by one; if you care about this particular scenario, you can make it more accurate by using this logic:
function getLines($file)
{
$f = fopen($file, 'rb');
$lines = 0; $buffer = '';
while (!feof($f)) {
$buffer = fread($f, 8192);
$lines += substr_count($buffer, "\n");
}
fclose($f);
if (strlen($buffer) > 0 && $buffer[-1] != "\n") {
++$lines;
}
return $lines;
}
Simple Oriented Object solution
$file = new \SplFileObject('file.extension');
while($file->valid()) $file->fgets();
var_dump($file->key());
#Update
Another way to make this is with PHP_INT_MAX in SplFileObject::seek method.
$file = new \SplFileObject('file.extension', 'r');
$file->seek(PHP_INT_MAX);
echo $file->key();
If you're running this on a Linux/Unix host, the easiest solution would be to use exec() or similar to run the command wc -l $path. Just make sure you've sanitized $path first to be sure that it isn't something like "/path/to/file ; rm -rf /".
There is a faster way I found that does not require looping through the entire file
only on *nix systems, there might be a similar way on windows ...
$file = '/path/to/your.file';
//Get number of lines
$totalLines = intval(exec("wc -l '$file'"));
If you're using PHP 5.5 you can use a generator. This will NOT work in any version of PHP before 5.5 though. From php.net:
"Generators provide an easy way to implement simple iterators without the overhead or complexity of implementing a class that implements the Iterator interface."
// This function implements a generator to load individual lines of a large file
function getLines($file) {
$f = fopen($file, 'r');
// read each line of the file without loading the whole file to memory
while ($line = fgets($f)) {
yield $line;
}
}
// Since generators implement simple iterators, I can quickly count the number
// of lines using the iterator_count() function.
$file = '/path/to/file.txt';
$lineCount = iterator_count(getLines($file)); // the number of lines in the file
If you're under linux you can simply do:
number_of_lines = intval(trim(shell_exec("wc -l ".$file_name." | awk '{print $1}'")));
You just have to find the right command if you're using another OS
Regards
This is an addition to Wallace Maxter's solution
It also skips empty lines while counting:
function getLines($file)
{
$file = new \SplFileObject($file, 'r');
$file->setFlags(SplFileObject::READ_AHEAD | SplFileObject::SKIP_EMPTY |
SplFileObject::DROP_NEW_LINE);
$file->seek(PHP_INT_MAX);
return $file->key() + 1;
}
The most succinct cross-platform solution that only buffers one line at a time.
$file = new \SplFileObject(__FILE__);
$file->setFlags($file::READ_AHEAD);
$lines = iterator_count($file);
Unfortunately, we have to set the READ_AHEAD flag otherwise iterator_count blocks indefinitely. Otherwise, this would be a one-liner.
private static function lineCount($file) {
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
if (fgets($handle) !== false) {
$linecount++;
}
}
fclose($handle);
return $linecount;
}
I wanted to add a little fix to the function above...
in a specific example where i had a file containing the word 'testing' the function returned 2 as a result. so i needed to add a check if fgets returned false or not :)
have fun :)
Based on dominic Rodger's solution,
here is what I use (it uses wc if available, otherwise fallbacks to dominic Rodger's solution).
class FileTool
{
public static function getNbLines($file)
{
$linecount = 0;
$m = exec('which wc');
if ('' !== $m) {
$cmd = 'wc -l < "' . str_replace('"', '\\"', $file) . '"';
$n = exec($cmd);
return (int)$n + 1;
}
$handle = fopen($file, "r");
while (!feof($handle)) {
$line = fgets($handle);
$linecount++;
}
fclose($handle);
return $linecount;
}
}
https://github.com/lingtalfi/Bat/blob/master/FileTool.php
Counting the number of lines can be done by following codes:
<?php
$fp= fopen("myfile.txt", "r");
$count=0;
while($line = fgetss($fp)) // fgetss() is used to get a line from a file ignoring html tags
$count++;
echo "Total number of lines are ".$count;
fclose($fp);
?>
You have several options. The first is to increase the availble memory allowed, which is probably not the best way to do things given that you state the file can get very large. The other way is to use fgets to read the file line by line and increment a counter, which should not cause any memory issues at all as only the current line is in memory at any one time.
There is another answer that I thought might be a good addition to this list.
If you have perl installed and are able to run things from the shell in PHP:
$lines = exec('perl -pe \'s/\r\n|\n|\r/\n/g\' ' . escapeshellarg('largetextfile.txt') . ' | wc -l');
This should handle most line breaks whether from Unix or Windows created files.
TWO downsides (at least):
1) It is not a great idea to have your script so dependent upon the system its running on ( it may not be safe to assume Perl and wc are available )
2) Just a small mistake in escaping and you have handed over access to a shell on your machine.
As with most things I know (or think I know) about coding, I got this info from somewhere else:
John Reeve Article
public function quickAndDirtyLineCounter()
{
echo "<table>";
$folders = ['C:\wamp\www\qa\abcfolder\',
];
foreach ($folders as $folder) {
$files = scandir($folder);
foreach ($files as $file) {
if($file == '.' || $file == '..' || !file_exists($folder.'\\'.$file)){
continue;
}
$handle = fopen($folder.'/'.$file, "r");
$linecount = 0;
while(!feof($handle)){
if(is_bool($handle)){break;}
$line = fgets($handle);
$linecount++;
}
fclose($handle);
echo "<tr><td>" . $folder . "</td><td>" . $file . "</td><td>" . $linecount . "</td></tr>";
}
}
echo "</table>";
}
I use this method for purely counting how many lines in a file. What is the downside of doing this verses the other answers. I'm seeing many lines as opposed to my two line solution. I'm guessing there's a reason nobody does this.
$lines = count(file('your.file'));
echo $lines;
this is a bit late but...
Here is my solution for a text log file I have which uses \n to separate each line.
$data = file_get_contents("myfile.txt");
$numlines = strlen($data) - strlen(str_replace("\n","",$data));
It does load the file into memory but doesn't need to cycle through an unknown number of lines. It may be unsuitable if the file is GB in size but for smaller files with short lines of data it works a treat for me.
It just removes the "\n" from the file and compares how many have been removed by comparing the length of the data in the file to the length after removing all the line breaks ("\n" chars n my case). If your line delineator is a different char, replace the "\n" with whatever is your line delineation character.
I know it is not the best answer for all occasions but is something I have found quick and simple for my purposes where each line of the log is only a few hundred chars and total log file is not too large.
For just counting the lines use:
$handle = fopen("file","r");
static $b = 0;
while($a = fgets($handle)) {
$b++;
}
echo $b;
What is the best way to see how long it takes for your PHP script to run?
I was thinking something like this:
$start_time = time(); //this at the beginning
$end_time = time(); //this at the end
echo = $end_time-$start_time;
But how can I make it into something that is readable to me and make sense to me?
If you want any further granularity to your timing than seconds, you'll need to use microtime() (Return current Unix timestamp with microseconds)
<?php
$time_start = microtime(true);
// Sleep for a while
usleep(100);
$time_end = microtime(true);
$time = $time_end - $time_start;
echo "Did nothing in $time seconds\n";
?>
** Below was added later **
As far as formatting this result further:
Well, depending on what you're doing, you typically don't have scripts going past a minute. You definitely shouldn't have anything exceeding an hour. (If you do, you need to ask yourself what you are doing with your life)
With that in mind, all you need is simple calculations:
$tmp = floor($time);
$minutes = $tmp / 60;
$seconds = ($tmp % 60) + ($time - $tmp);
$output = 'Script took ';
if ($minutes > 0) $output .= $minutes . ' minutes and ';
$output .= $seconds . ' seconds to complete.';
echo $output;
(This isn't tested, and it could potentially be optimized, but should start you in the right direction)
I would use microtime(TRUE) instead. That'll give you better results with microsecond resolution. There's also the PECL APD extension which profiles scripts.
Expanding a bit on APD, assuming one has APD installed, you just need to add the line
apd_set_pprof_trace();
to the top (or whenever you want to start tracing) of your script. It generates a profiling file with pprofp which generates very readable output
Real User System secs/ cumm
%Time (excl/cumm) (excl/cumm) (excl/cumm) Calls call s/call Memory Usage Name
--------------------------------------------------------------------------------------
100.0 0.00 0.00 0.00 0.00 0.00 0.00 1 0.0000 0.0009 0 main
56.9 0.00 0.00 0.00 0.00 0.00 0.00 1 0.0005 0.0005 0 apd_set_pprof_trace
28.0 0.00 0.00 0.00 0.00 0.00 0.00 10 0.0000 0.0000 0 preg_replace
The example output is from the PHP manual.
I personally find APD extremely useful and use it quite frequently in heavily-hit scripts.
Knowing how long something takes to run is one thing, finding out where (in your php operations) it slows down and why is another question.
If you're serious about find an answer to the latter question then install xdebug, and you get the bonus of not having to call the likes of microtime() which itself slows down your script.
http://xdebug.org/docs/profiler
http://xdebug.org/docs/
Is this a web-accessible script (i.e. http://www.yoursite.com/BLA.php), or a script that you're running through the command line? If it's a command line script, you can use the time command:
time php FILE.php
Otherwise, microtime is probably your best bet.
I find this class to be useful to time the scripts, microtime turn out to be friend in that:
class timer
{
private $start_time = NULL;
private $end_time = NULL;
private function getmicrotime()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
function start()
{
$this->start_time = $this->getmicrotime();
}
function stop()
{
$this->end_time = $this->getmicrotime();
}
function result()
{
if (is_null($this->start_time))
{
exit('Timer: start method not called !');
return false;
}
else if (is_null($this->end_time))
{
exit('Timer: stop method not called !');
return false;
}
return round(($this->end_time - $this->start_time), 4);
}
# an alias of result function
function time()
{
$this->result();
}
}
What is it about fgets() implementation that makes it so horrible on large files vs fread?
To demonstrate, run this code:
<?php
$line = str_repeat('HelloWorld', 100000) . "\n";
for($i=0; $i<10000; ++$i)
file_put_contents('myfile', $line, FILE_APPEND);
//now we have roughly a 1gig file
// We'll even let fread go first incase
// the subsequent test would have any caching benefits
$fp = fopen('myfile2','r');
$start = microtime(true);
while (fread($fp,4096));
$end = microtime(true);
print ($end-$start) . " using fread\n";
fseek($fp, 0);
$start = microtime(true);
while (fgets($fp));
$end = microtime(true);
print ($end-$start) . " using fgets\n";
?>
It might have something to do with how you're calling fgets. From the manual, it says that if you leave the second parameter out, then:
If no length is specified, it will keep reading from the stream until it reaches the end of the line.
If the data your working with has some very long lines (eg: "HelloWorld" 100,000 times = 1,000,000 characters), then it has to read that whole thing.
Again, from the manual:
If the majority of the lines in the file are all larger than 8KB, it is more resource efficient for your script to specify the maximum line length.