How can I delete some lines("\r\n") from a binary data:
for example if I have $data which is the content of a binary file I'm getting from my database and wanted to delete the first five lines from it.
How can this be done in php?
// Load the file into memory
$fileData = file_get_contents('myfile.bin');
// Split it by CRLF sequences
$fileData = explode("\r\n", $fileData);
// Remove the first 5 lines
$fileData = array_slice($fileData, 5);
// Turn it back into a string
$fileData = implode("\r\n", $fileData);
// Write it back to the file
file_put_contents('myfile.bin', $fileData);
Or, in one line:
file_put_contents('myfile.bin', implode("\r\n", array_slice(explode("\r\n", file_get_contents('myfile.bin')), 5)));
But if you want to do this with "binary data", I would question whether it really is binary data.
If you simply want to get rid of all empty lines at the start of the string you could use ltrim() PHP Manual ltrim but if the lines aren't empty and you know you want to get rid of exactly 5 lines then this preg_replace should do the trick:
$newData = preg_replace('/(.*)\r\n/', '', $data, 5);
[Edit:]
Come to think of it, this might not work for binary data at all but you can give it a try.
Related
Using file_get_contents() I retrieve CSV data from an external URL. In this data there are breaks visible for every new record in the CSV. However, when I try to explode the lines, I only get one item in my array, containing the full CSV dataset. This is the code I use:
$data = file_get_contents('http://example.url/csvdata.asp?id=20'); // has 12 records
$rows = explode("\n", $data);
echo count($rows); // returns 1
Could this have to do with that an ASP script generates the CSV? Do they use other new line characters?
Use file and let PHP handle the new lines
file — Reads entire file into an array
You can simply do
$rows = file('http://example.url/csvdata.asp?id=20');
\n can work but there are more ways for new lines. This should work:
$lines = preg_split("/\\r\\n|\\r|\\n/", $data);
I had the same issue with mine. I found this somewhere on SO:
$rows = explode(PHP_EOL, $data);
PHP_EOL finds the end of the line.
So i have a little issue with some PHP read functionality. What I am trying to do is basically grab data into an array from a file that is being continuously updated from a python script reading values from a micro controller. So basically, the file would look something like this.
ID, Datetime, Count, Name
ID, Datetime, Count, Name
ID, Datetime, Count, Name
What i need is for it to read the new line that is being added in (eof) and store it into an array. So what i have so far is allowing read access into the file
<?php
$myfile = fopen("read.txt", "r")
For the storing the lines in an array i figured something like an array map would be efficient
$result = array();
// some loop
$parts = array_map('trim', explode(':', $line_of_text, 2)));
$result[$parts[0]] = $parts[1];
However i am not to sure on how to structure the loop to have it read the new line that is being updated in the file without exiting the loop.
while (feof($file)) {
}
fclose($file);
?>
Any help would be appreciated!!
Can you do this?
Read the lines of the file to an array using $lines = file("filename");.
Use the $lines[count($lines) - 1] to get the last line?
You can even trim off the empty lines before you wanna do this.
Trim Empty Lines
Use this function:
$lines = array_filter($lines);
Since the file is continually being appended, you'd have to read until you hit the end of file, sleep for a while to let more data be appended, then read again.
e.g.
while(true) {
while(!feof($file)) {
... process data
}
sleep(15); // pause to let more data be appended
}
However, I'm not sure if PHP will cache the fact that it hit eof, and not try again once the sleep() finishes. It may be necessary to record your current position ftell(), close the file, reopen it, then fseek() to the stored location.
I've came up with this solution
$filename = "file.txt";
$file = fopen($filename, "r");
$lines = explode("/n", fread($file, filesize($filename)));
$last = $lines[count($lines)-1];
If the file is going to get big, it could take some time to parse, so its also possible to adjust the fread() function so it only reads the last 100 characters for example.
This works
$arr = array_merge(array_diff($words, array("the","an"));
Why doesn't this work?
$common consists of 40 words in an array.
$arr = array_merge(array_diff($words, $common));
Is there another solution for this?
For Reference:
<?php
error_reporting(0);
$str1= "the engine has two ways to run: batch or conversational. In batch, expert system has all the necessary data to process from the beginning";
common_words($str1);
function common_words(&$string) {
$file = fopen("common.txt", "r") or exit("Unable to open file!");
$common = array();
while(!feof($file)) {
array_push($common,fgets($file));
}
fclose($file);
$words = explode(" ",$string);
$arr = array_merge(array_diff($words, array("the","an")));
print_r($arr);
}
?>
White-spaces are evil, sometimes..
fgets with only one parameter will return one line of data from the filehandle provided.
Though, it will not strip off the trailing new-line ("\n" or whatever EOL character(s) is used) in the line returned.
Since common.txt seems to have one word per line, this is the reason why php won't find any matching elements when you use array_diff.
PHP: fgets - Manual
parameter: length
Reading ends when length - 1 bytes have been read, on a newline (which is included in the return value), or on EOF (whichever comes first). If no length is specified, it will keep reading from the stream until it reaches the end of the line.
Rephrase:
All entries off $common will have a trailing line-break the way you are doing it now.
Alternative solutions 1
If you are not going to process the entries in common.txt I'd recommend you to take a look at php's function file, and use that in conjunction with array_map to rtrim the lines for you.
$common = array_map ('rtrim', file ('common.txt')); // will do what you want
Alternative solutions 2
After #MarkBaker saw the solution above he made a comment saying that you might as well pass a flag to file to make it work in the same manner, there is no need to call array_map to "fix" the entries returned.
$common = file ('common.txt', FILE_IGNORE_NEW_LINES);
I do have two text files and want to loop through both files then combine both line (line 1 of first test file and line1 of second text file. like that for thousands of lines) and do some function
I am familiar with loop through one file and for that code is given below:
$lines = file('data.txt');
foreach ($lines as $line) {
//some function
}
but how will I do for two files and combine bothe lines?
Not sure what you mean by search through the table, but to open both files and do stuff with them:
$file1 = fopen("/path/to/file1.txt","r"); //Open file with read only access
$file2 = fopen("/path/to/file2.txt","r");
$combined = fopen("/path/to/combined.txt","w"); //in case you want to write the combined lines to a new file
while(!feof($file1) && !feof($file2))
{
$line1 = trim(fgets($file1)); //Grab a line of the first file, note the trim will clip off the carriage return/new line at the end of the line, can remove it if you don't need it.
$line2 = trim(fgets($file2)); //Grab a line of the second file
$combline = $line1 . $line2;
fwrite($combined,$combline . "\r\n"); //Write to new combined file, and add a new carriage return/newline at the end of the combined line to replace the one trimmed off.
//You can do whatever with data from $line1, $line2, or the combined $combline after getting them.
}
Note: You might run into trouble if you hit the end of file on one file before the other, which would only happen if they aren't the same length, might need some if control statements to set $line1 or $line2 to "" or something else if feof() their respective files, once both hit the end of file, the while loop will end.
You can do this programmatically as Crayon and Tim have shown. If both files have the same number of lines, it should work. If the line number is different you will have to loop over the larger file to make sure you get all lines or check EOF on both.
To combine line by line, I often use the unix command paste which is very fast. This also accounts for files with different lengths. Run this on the command line:
paste file1 file2 > output.txt
See the manpage for paste for command line options, field delimiters.
man paste
Example:
$file1 = fopen("file1.txt", "rb");
$file2 = fopen("file2.txt", "rb");
while (!feof($file1)) {
$combined = fread($file1, 8192) . " " . fread($file2, 8192);
// now insert $combined into db
}
fclose($file1);
fclose($file2);
you will want to use the longer of the two files in the while condition.
you may need to adjust the bytes read in fread depending on how long your lines are
change " " to whatever delimiter you want
I have a form that allows the user to either upload a text file or copy/paste the contents of the file into a textarea. I can easily differentiate between the two and put whichever one they entered into a string variable, but where do I go from there?
I need to iterate over each line of the string (preferably not worrying about newlines on different machines), make sure that it has exactly one token (no spaces, tabs, commas, etc.), sanitize the data, then generate an SQL query based off of all of the lines.
I'm a fairly good programmer, so I know the general idea about how to do it, but it's been so long since I worked with PHP that I feel I am searching for the wrong things and thus coming up with useless information. The key problem I'm having is that I want to read the contents of the string line-by-line. If it were a file, it would be easy.
I'm mostly looking for useful PHP functions, not an algorithm for how to do it. Any suggestions?
preg_split the variable containing the text, and iterate over the returned array:
foreach(preg_split("/((\r?\n)|(\r\n?))/", $subject) as $line){
// do stuff with $line
}
I would like to propose a significantly faster (and memory efficient) alternative: strtok rather than preg_split.
$separator = "\r\n";
$line = strtok($subject, $separator);
while ($line !== false) {
# do something with $line
$line = strtok( $separator );
}
Testing the performance, I iterated 100 times over a test file with 17 thousand lines: preg_split took 27.7 seconds, whereas strtok took 1.4 seconds.
Note that though the $separator is defined as "\r\n", strtok will separate on either character - and as of PHP4.1.0, skip empty lines/tokens.
See the strtok manual entry:
http://php.net/strtok
If you need to handle newlines in diferent systems you can simply use the PHP predefined constant PHP_EOL (http://php.net/manual/en/reserved.constants.php) and simply use explode to avoid the overhead of the regular expression engine.
$lines = explode(PHP_EOL, $subject);
It's overly-complicated and ugly but in my opinion this is the way to go:
$fp = fopen("php://memory", 'r+');
fputs($fp, $data);
rewind($fp);
while($line = fgets($fp)){
// deal with $line
}
fclose($fp);
Potential memory issues with strtok:
Since one of the suggested solutions uses strtok, unfortunately it doesn't point out a potential memory issue (though it claims to be memory efficient). When using strtok according to the manual, the:
Note that only the first call to strtok uses the string argument.
Every subsequent call to strtok only needs the token to use, as it
keeps track of where it is in the current string.
It does this by loading the file into memory. If you're using large files, you need to flush them if you're done looping through the file.
<?php
function process($str) {
$line = strtok($str, PHP_EOL);
/*do something with the first line here...*/
while ($line !== FALSE) {
// get the next line
$line = strtok(PHP_EOL);
/*do something with the rest of the lines here...*/
}
//the bit that frees up memory
strtok('', '');
}
If you're only concerned with physical files (eg. datamining):
According to the manual, for the file upload part you can use the file command:
//Create the array
$lines = file( $some_file );
foreach ( $lines as $line ) {
//do something here.
}
foreach(preg_split('~[\r\n]+~', $text) as $line){
if(empty($line) or ctype_space($line)) continue; // skip only spaces
// if(!strlen($line = trim($line))) continue; // or trim by force and skip empty
// $line is trimmed and nice here so use it
}
^ this is how you break lines properly, cross-platform compatible with Regexp :)
Kyril's answer is best considering you need to be able to handle newlines on different machines.
"I'm mostly looking for useful PHP functions, not an algorithm for how
to do it. Any suggestions?"
I use these a lot:
explode() can be used to split a string into an array, given a
single delimiter.
implode() is explode's counterpart, to go from array back to string.
Similar as #pguardiario, but using a more "modern" (OOP) interface:
$fileObject = new \SplFileObject('php://memory', 'r+');
$fileObject->fwrite($content);
$fileObject->rewind();
while ($fileObject->valid()) {
$line = $fileObject->current();
$fileObject->next();
}
SplFileObject doc: https://www.php.net/manual/en/class.splfileobject.php
PHP IO streams: https://www.php.net/manual/en/wrappers.php.php