Handling text file with unknown newline positions - php

Question

My problem is simple: I have a text file, which i handle and insert all the data in a database and also do stuff with it for each new line. The problem is that the text file is a log for sms'es received in my gateway and depending on the text that is being sent I would have a line corresponding to each sms. If an SMS does not have any new lines in its body, everything is alright, on the other hand, if and SMS is sent like this:
"Test
TestOnANewLine"
I get a log file that breaks and with a new line everytime. A sample follows:
2012-01-01 10:10:10,4C64DCD6.req,192.168.999.999,[phone],OK -- SMPP - 999.999.999.999:9999,SubmitUser=user;Sender=sender;SMSCMsgId=999999999;Text="Test1
NewLineTest
AnotherNEwLineTEst"
The log file is interpreted like this:
date time, smsid, ip that processed it, number that is being sent to, status --connection type - ip that is sent from, user that submitted; sender name that is displayed; sms connection id; body of the sms
As for the language I am using PHP and for the functions used its a simple
foreach($lines as $line)
{ explode and do stuff }
How do I handle this situation? At this point any help is appreciated
Thanks in advance!!

Answer 1

fgetcsv could handle the linebreaks enclosed in '"' but with an additional '"' character in the body it would fail...
So what about some unresponsible regexp usage?
preg_match_all(#^(\d{4}-\d{2}-\d{2}[^,]+),([^,]+),([^,]+),([^,]+),([^,]+),SubmitUser=([^;])+;Sender=([^;])+;SMSCMsgId=([^;])+;Text="([\w\d\s\.\-,:;'"]+)"$#im', $file, $matches);
should do the job, for not too crazy texts, maybe you should adpot the \w\d\s.-,:;'" expression more to your needs

Answer 2

Couldn't you loop through the newlines until you can parse a date from it?
Maybe take into account that the previous line ended with a double quote ?
I know its not fool proof but without some recognisable "end of message" character(s). This is the best i could think of :P

Answer 3

First of all, thank you for all the feedback, it was really precious and it helped me on solving this issue. Also, for all the other people that will look through this post and would want a solution here is mine:
I changed the way I would interpret the end of line /r/n from the regular one to /r/n2 which means that ill consider a new line in my file reading if and only if there is a regular new line /r/n and on the new physical line there is a 2 (which is the beginning of the year)
The actual solved part is:
$data = file_get_contents($backup_file);
$lines=explode("\r\n2",$data);
foreach($lines as $line)
{
//explode and do stuff
}

Answer 4

Try this to get all the log entries normalized into a single array item per log entry (i.e. combine entries across multiple line breaks into a single item)
$line_array = file('/path/to/file');
$log_array = array();
$i = -1;
$date_pattern = '/^[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}/';
foreach ($line_array as $line) {
if (1 === preg_match($date_pattern, $line)) {
// this is a new log entry
// let's trim the whitespace from the end of the last log array entry since we are done with it
if(isset($log_array[$i])) {
$log_array[$i] = rtrim($log_array[$i]);
}
// start a new log array entry
$i++;
$log_array[$i] = $line;
} else {
// this is not a new log entry
$log_array[$i] .= $line;
}
}
After that you should be able to work with $log_array to extract the data you need. By the way I should note that when you loop through the $log_array. It would probably be helpful to extract the msg text first. If you do a greedy preg_match on the double quotes, you shouldn't have any problems with messages that have quotes within them as the greedy match will find the largest possible matching string, which in your case would be everything between the quotes bounding the message content.