% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12.4M 100 12.4M 0 0 4489k 0 0:00:02 0:00:02 --:--:-- 4653k
The above is a CURL output from the command line when download the file. I have captured this using PHP like so, but I am having trouble working out how to use pre_match to extract the percentage done.
$handle = popen('curl -o '.VIDEOPATH.$fileName.'.flv '.$url, 'rb');
while(!feof($handle))
{
$progress = fread($handle, 8192);
//I don't even know what I was attempting here
$pattern = '/(?<Total>[0-9]{1,3}\.[0-9]{1,2})% of (?<Total>.+) at/';
//divide received by total somehow, then times 100
if(preg_match_all($pattern, $progress, $matches)){
fwrite($fh, $matches[0][0]."\r\n");
}
}
How can I do this? Please note, I have no idea what I am doing with the above preg_match_all!
Thanks
Update
Thanks to the help of ylebre. I have this so far.
$handle = popen('curl -o '.VIDEOPATH.$fileName.'.flv '.$url.' 2>&1', 'rb');//make sure its saved to videos
while(!feof($handle))
{
$line = fgets($handle, 4096); // Get a line from the input handle
echo '<br>Line'.$line.'<br>';
$line = preg_replace("/s+/", " ", $line); // replace the double spaces with one
$fields = explode(" ", $line); // split the input on spaces into fields array
echo '<br>Fields: '.$fields[0];
fwrite($fh, $fields[0]); // write a part of the fields array to the output file
}
I get this output to the browser:
Line % Total % Received % Xferd Average Speed Time Time Time Current
Fields:
Line Dload Upload Total Spent Left Speed
Fields:
Line 0 1340k 0 4014 0 0 27342 0 0:00:50 --:--:-- 0:00:50 27342 41 1340k 41 552k 0 0 849k 0 0:00:01 --:--:-- 0:00:01 1088k 100 1340k 100 1340k 0 0 1445k 0 --:--:-- --:--:-- --:--:-- 1711k
Fields:
Line
How do I extract the percentage part only? Maybe CURL can do this by itself - hmm will ask a question on this.
The progress that is showing up is probably updating the information in the same spot, so it will help if you know what you are parsing exactly.
The next step I recommend is taking one line of input, and trying to get the regexp to work on that.
You could also just split the string at the spaces if I'm reading the output correctly. If you start out by replacing all the double spaces into one. After that you can use explode() to get an array with the values, which you can print_r to take a peek what is inside.
This would be something like:
$line = fgets($handle, 4096); // Get a line from the input handle
$line = preg_replace("/s+/", " ", $line); // replace the double spaces with one
$fields = explode(" ", $line); // split the input on spaces into fields array
fwrite($fh, $fields[0]); // write a part of the fields array to the output file
As long as the ordering in the fields remains the same, your resulting array should give you a consistent result.
Hope this helps!
If you have access to PHP 5.3, you can use the CURL_PROGRESSFUNCTION option, which results in a much more elegant solution (no parsing output). Here's an example of how to use it:
function callback($download_size, $downloaded, $upload_size, $uploaded)
{
$percent=$downloaded/$download_size;
// Do something with $percent
}
$ch = curl_init('http://www.example.com');
// Turn off the default progress function
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
// Set up the callback
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, 'callback');
// You'll want to tweak the buffer size. Too small could affect performance. Too large and you don't get many progress callbacks.
curl_setopt($ch, CURLOPT_BUFFERSIZE, 128);
$data = curl_exec($ch);
Related
file: data.txt (11617 lines)
user datetime
23 2015-03-01 08:04:15
15 2015-05-01 08:05:20
105 2015-05-01 08:07:10
15 2015-06-01 08:08:29
105 2015-06-01 08:12:48
I only need data in 2015-06, I'm using fget and check each line's datetime but really slow, more than 50s.
$d='data.txt';
import($d);
function import($d){
$handle = fopen($d, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
$line=explode("\t",$buffer);
if(date("Y-m",strtotime($line[1])=="2015-06"){
mysql_query("INSERT INTO `table` ....");
}
else{
//break? when month>6
}
}
fclose($handle);
}
}
SOLUTION: less than 2s!!!! (thanks to Kevin P. and Dragon)
if(substr($line[1],0,7)=="2015-06"){
$sql.=empty($sql)?"":","."(`".$line[1]."`.........)";
}
elseif(substr($line[1],0,7)>"2015-06"){
break;// when month>6
}
mysql_query("INSERT INTO `table` ....".$sql);
Can't be helped, use something faster than PHP. For instance, you can use grep or awk to read the file and filter it quickly. For example:
$lines = explode("\n", `awk '$2 ~ /^2015-06/ { print }' data.txt`);
EDIT: Also, fgets is not guaranteed to give you whole lines. You are getting 4096 bytes at a time; the boundary could be in the middle of a line, which will make the line not match if you are lucky, or break your code due to missed assumptions (such as the length of the $line array when constructing the SQL) if not.*
*) Or vice versa - it would be better for it to break completely, that is at least an obvious error yelling to be fixed; as opposed to silent data droppage.
Maybe insert multiple entries in to the DB at once instead of calling it every time you find a desired time?
In which case it's similar to this
Maybe you should use grep to filter out the lines you do not need.
I have a very large file with only single line. It contains about 2.6 million of numbers. The file is about 15 mb.
My goal is to find the nth number in this single line string.
I tried to read the file into a string (remember it is single line file). Then I exploded the strings into an array which I ran out of memory. (Allowed memory size of 268435456 bytes exhausted (tried to allocate 71 bytes)
Am I doing it right? Or is there another easier way to find the nth value in a very large string?
$file = file_get_contents ('a.txt', true);
$array = explode(" ", $file, -1);
echo $array[$nth];
Create a counter variable; read the file using fopen and loop it in a while with feof and fgets (with the desired buffer size); within the loop, check how many spaces are present in the bit you just read (I'm assuming your entries are separated by spaces, it could be commas or whatever); finally increment the counter and go on until you reach the part you want (after a n number of spaces, you have the [n+1]th entry you are looking for).
I include some tested (with a 16 MB file) proof-of-concept code. I don't know if there are better ways to do it; this is the only one that came to my mind and it works. memory_get_usage reports a memory usage of ~8 kb.
<?php
$counter;
$nth = 49959;
$handle = #fopen('numbers.txt', 'r'); // File containing numbers from 1 to 2130829, size ~16 MB.
if ($handle) {
while (($buffer = fgets($handle, 128)) !== false) {
$spaces = substr_count($buffer, ' ');
if ($counter + $spaces > $nth) {
$numbers = explode(' ', $buffer);
$key = $nth - $counter;
echo $numbers[$key]; // print '49959'
exit;
}
else {
$counter += $spaces;
}
}
if (!feof($handle)) {
echo "Error: unexpected fgets() fail\n";
}
fclose($handle);
}
?>
I need to login to a production server retrieve a file and update my data base with the data in this file. Since this is a production database, I don't want to get the whole file every 5 minutes since the file may be huge and this may impact the server. I need to get the last 30 lines of this file every 5 minutes interval and have as little impact as possible.
The following is my current code, I would appreciate any insight to how best accomplish this:
<?php
$user="id";
$pass="passed";
$c = curl_init("sftp://$user:$pass#server1.example.net/opt/vmstat_server1");
curl_setopt($c, CURLOPT_PROTOCOLS, CURLPROTO_SFTP);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($c);
curl_close($c);
$data = explode("\n", $data);
?>
Marc B is wrong. SFTP is perfectly capable of partial file transfers. Here's an example of how to do what you want with phpseclib, a pure PHP SFTP implementation:
<?php
include('Net/SFTP.php');
$sftp = new Net_SFTP('www.domain.tld');
if (!$sftp->login('username', 'password')) {
exit('Login Failed');
}
$size = $sftp->size('filename.remote');
// outputs the last ten bytes of filename.remote
echo $sftp->get('filename.remote', false, $size - 10);
?>
In fact I'd recommend an approach like this anyway since some SFTP servers don't let you run commands via the system shell. Plus, SFTP can work on Windows SFTP servers whereas tail is unlikely to do so even if you do have shell access. ie. overall, it's a lot more portable a solution.
If you want to get the last x lines of a file you could loop repeatedly, reading however many bytes each time, until you encounter 10x new line characters. ie. get the last 10 bytes, then the next to last 10 bytes, then the ten bytes before those ten bytes, etc.
An answer by #Sammitch to a duplicate question Get last 15 lines from a large file in SFTP with phpseclib:
The following should result in a blob of text with at least 15 lines from the end of the file that you can then process further with your existing logic. You may want to tweak some of the logic depending on if your file ends with a trailing newline, etc.
$filename = './file.txt'
$filesize = $sftp->size($filename);
$buffersize = 4096;
$offset = $filesize; // start at the end
$result = '';
$lines = 0;
while( $offset > 0 && $lines < 15 ) {
// work backwards
if( $offset < $buffersize ) {
$offset = 0;
} else {
$offset -= $buffer_size;
}
$buffer = $sftp->get($filename, false, $offset, $buffer_size));
// count the number of newlines as we go
$lines += substr_count($buffer, "\n");
$result = $buffer . $result;
}
SFTP is not capable of partial file transfers. You might have better luck using a fullblowin SSH connection and use a remote 'tail' operation to get the last lines of the file, e.g.
$lines = shell_exec("ssh user#remote.host 'tail -30 the_file'");
Of course, you might want to have something a little more robust that can handle things like net.glitches that prevent ssh from getting through, but as a basic starting point, this should do the trick.
I have some text files that are very large - 100MB each that contain a single-line string (just 1 line). I want to extract the last xx bytes / characters from each of them. I know how to do this by reading them in a string and then searching by strpos() or substr() but that would require a large chunk of the RAM which isn't desirable for such a small action.
Is there any other way I can just extract, say, the last 50 bytes / characters of the text file in PHP before executing the search?
Thank you!
You can use fseek:
$fp = fopen('somefile.txt', 'r');
fseek($fp, -50, SEEK_END); // It needs to be negative
$data = fgets($fp, 50);
You can do this with file_get_contents by playing with the fourth parameter offset.
PHP 7.1.0 onward:
In PHP 7.1.0 the fourth parameter offset can be negative.
// only negative seek if it "lands" inside the file or false will be returned
if (filesize($filename) > 50) {
$data = file_get_contents($filename, false, null, -50);
}
else {
$data = file_get_contents($filename);
}
Pre PHP 7.1.0:
$fsz = filesize($filename);
// only negative seek if it "lands" inside the file or false will be returned
if ($fsz > 50) {
$data = file_get_contents($filename, false, null, $fsz - 50);
}
else {
$data = file_get_contents($filename);
}
Thank you for taking the time to read this and I will appreciate every single response no mater the quality of content. :)
Using php, I'm trying to create a script which will delete several lines within a text file (.txt) if required, based upon whether the line starts with a 0 or a negative number. Each line within the file will always start with a number, and I need to erase all the neutral and/or negative numbers.
The main part I'm struggling with is that the content within the text file isn't static (e.g. contain x number of lines/words etc.) Infact, it is automatically updated every 5 minutes with several lines. Therefore, I'd like all the lines containing a neutral or negative number to be removed.
The text file follows the structure:
-29 aullah1
0 name
4 username
4 user
6 player
If possible, I'd like Line 1 and 2 removed, since it begins with a neutral/negative number. At points, there maybe times when there are more than two neutral/negative numbers.
All assistance is appreciated and I look forward to your replies; thank you. :) If I didn't explain anything clearly and/or you'd like me to explain in more detail, please reply. :)
Thank you.
Example:
$file = file("mytextfile.txt");
$newLines = array();
foreach ($file as $line)
if (preg_match("/^(-\d+|0)/", $line) === 0)
$newLines[] = chop($line);
$newFile = implode("\n", $newLines);
file_put_contents("mytextfile.txt", $newFile);
It is important that you chop() the newline character off of the end of the line so you don't end up with empty space. Tested successfully.
Something on these lines i guess, it is untested.
$newContent = "";
$lines = explode("\n" , $content);
foreach($lines as $line){
$fChar = substr($line , 0 , 1);
if($fChar == "0" || $fChar == "-") continue;
else $newContent .= $line."\n";
}
If the file is big, its better to read it line by line as:
$fh_r = fopen("input.txt", "r"); // open file to read.
$fh_w = fopen("output.txt", "w"); // open file to write.
while (!feof($fh_r)) { // loop till lines are left in the input file.
$buffer = fgets($fh_r); // read input file line by line.
// if line begins with num other than 0 or -ve num write it.
if(!preg_match('/^(0|-\d+)\b/',$buffer)) {
fwrite($fh_w,$buffer);
}
}
fclose($fh_r);
fclose($fh_w);
Note: Err checking not included.
file_put_contents($newfile,
implode(
preg_grep('~^[1-9]~',
file($oldfile))));
php is not particularly elegant, but still...
Load whole line into variable trim it and then check if first letter is - or 0.
$newContent = "";
$lines = explode("\n" , $content);
foreach($lines as $line){
$fChar = $line[0];
if(!($fChar == '0' || $fChar == '-'))
$newContent .= $line."\n";
}
I changed malik's code for better performance and quality.
Here's another way:
class FileCleaner extends FilterIterator
{
public function __construct($srcFile)
{
parent::__construct(new ArrayIterator(file($srcFile)));
}
public function accept()
{
list($num) = explode(' ', parent::current(), 2);
return ($num > 0);
}
public function write($file)
{
file_put_contents($file, implode('', iterator_to_array($this)));
}
}
Usage:
$filtered = new FileCleaner($src_file);
$filtered->write($new_file);
Logic and methods can be added to the class for other stuff, such as sorting, finding the highest number, converting to a sane storage method such as csv, etc. And, of course, error checking.