Writing to JSON file from PHP fseek()/rewind() not working - php

I'm working on a logger class , the JSON that is being added has the following format
{"log_owner" : "test123","log_message" : "Has logged in","log_timestamp" : "1397921556","log_type" : "1"}
To retrieve it I require square brackets around all the different JSON Objects as the following :
[
{"log_owner" : "test456","log_message" : "Has logged in","log_timestamp" : "1397921856","log_type" : "2"}
{"log_owner" : "test123","log_message" : "Has logged in","log_timestamp" : "1397921556","log_type" : "1"}
]
I've managed to insert it at the beginning of the file whenever the file didn't existed, but the main issue resides on moving the closing bracket to the end of the file as I'm adding new objects , I've tried to move my file pointer 2 places before to be able to overwrite the last bracket keep adding the closing bracket to every new entry, I'm trying to accomplish this with :
if(!$content_new) {
$pos=ftell($handle);
fseek($handle,$pos-2);
}
fwrite($handle, $content);
fclose($handle);
But seems that It does not work with .json files, as I'm not able to move my file pointer to any other line or rewind it.
How could I accomplish this ? Any kind of guidance or suggestions for improvement is highly appreciated .
Thank you.

Direct solution for your problem - quick and dirty.
<?php
function writeLog($path, $newLine)
{
$exists = file_exists($path);
$handle = fopen($path, 'c');
if (!$exists) {
// first write to log file
$line = "[" . PHP_EOL . $newLine . PHP_EOL . "]";
} else {
// file exists so it has one or more logged lines
$line = "," . PHP_EOL . $newLine . PHP_EOL . "]";
fseek($handle , -(strlen(PHP_EOL) + 1) , SEEK_END);
}
fwrite($handle, $line);
fclose($handle);
}
$path = __DIR__ . '/file.json';
// delete file if exists - for tests
if (file_exists($path)) {
unlink($path);
}
$line = '{"log_owner" : "test123","log_message" : "Has logged in","log_timestamp" : "1397921556","log_type" : "1"}';
for ($i = 0; $i < 10; $i++) {
writeLog($path, $line);
}
Problems
concurrency
scaling
JSON is not easiest to edit and view
no easy filtering
Use CSV, output JSON
<?php
function writeLogCSV($path, $newLine)
{
$handle = fopen($path, 'a');
fputcsv($handle, $newLine);
fclose($handle);
}
function readLogCsv ($path)
{
$handle = fopen($path, 'r');
$rows = [];
while (false !== ($line = fgetcsv($handle))) {
$rows[] = array_combine(
["log_owner", "log_message", "log_timestamp", "log_type"],
$line
);
}
fclose($handle);
echo json_encode($rows);
}
$path = __DIR__ . '/file.csv';
// delete file if exists - for tests
if (file_exists($path)) {
unlink($path);
}
$line = ["test123", "Has logged in", "1397921556", "1"];
for ($i = 0; $i < 10; $i++) {
writeLogCSV($path, $line);
}
readLogCsv($path);
Good parts:
easy to read and write
Problems:
scaling
concurrency
no easy filtering
Store your log in database or use logging service, output JSON
Good parts
no concurrency issues
easy filtering
speed
good for scaling

Related

Reading large text files efficiently

I have a couple of huge (11mb and 54mb) files that I need to read to process the rest of the script. Currently I'm reading the files and storing them in an array like so:
$pricelist = array();
$fp = fopen($DIR.'datafeeds/pricelist.csv','r');
while (($line = fgetcsv($fp, 0, ",")) !== FALSE) {
if ($line) {
$pricelist[$line[2]] = $line;
}
}
fclose($fp);
.. but I'm constantly getting memory overload messages from my webhost. How do I read it more efficiently?
I don't need to store everything, I already have the keyword which exactly matches the array key $line[2] and I need to read just that one array/line.
If you know the key why don't you filter out by the key? And you can check memory usage with memory_get_usage() function to see how much memory allocated after you fill your $pricelist array.
echo memory_get_usage() . "\n";
$yourKey = 'some_key';
$pricelist = array();
$fp = fopen($DIR.'datafeeds/pricelist.csv','r');
while (($line = fgetcsv($fp, 0, ",")) !== FALSE) {
if (isset($line[2]) && $line[2] == $yourKey) {
$pricelist[$line[2]] = $line;
break;
/* If there is a possiblity to have multiple lines
we can store each line in a separate array element
$pricelist[$line[2]][] = $line;
*/
}
}
fclose($fp);
echo memory_get_usage() . "\n";
You can try this (I have not checked if it works properly)
$data = explode("\n", shell_exec('cat filename.csv | grep KEYWORD'));
You will get all the lines containing the keyword, each line as an element of array.
Let me know if it helps.
I join what user2864740 said : "The problem is the in-memory usage caused by the array itself and is not about "reading" the file"
My Solution is :
Split your `$priceList` array
Load only 1 at time a splitted Array in memory
Keep the other splitted Arrays in an intermediate file
N.B: i did not test what i've written
<?php
define ("MAX_LINE", 10000) ;
define ("CSV_SEPERATOR", ',') ;
function intermediateBuilder ($csvFile, $intermediateCsvFile) {
$pricelist = array ();
$currentLine = 0;
$totalSerializedArray = 0;
if (!is_file()) {
throw new Exception ("this is not a regular file: " . $csv);
}
$fp = fopen ($csvFile, 'r');
if (!$fp) {
throw new Exception ("can not read this file: " . $csv);
}
while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) {
if ($line) {
$pricelist[$line[2]] = $line;
}
if (++$currentLine == MAX_LINE) {
$fp2 = fopen ($intermediateCsvFile, 'a');
if (!$fp) throw new Exception ("can not write in this intermediate csv file: " . $intermediateCsvFile);
fputs ($fp2, serialize ($pricelist) . "\n");
fclose ($fp2);
unset ($pricelist);
$pricelist = array ();
$currentLine = 0;
$totalSerializedArray++;
}
}
fclose($fp);
return $totalSerializedArray;
}
/**
* #param array : by reference unserialized array
* #param integer : the array number to read from the intermediate csv file; start from index 1
* #param string : the (relative|absolute) path/name of the intermediate csv file
* #throw Exception
*/
function loadArray (&$array, $arrayNumber, $intermediateCsvFile) {
$currentLine = 0;
$fp = fopen ($intermediateCsvFile, 'r');
if (!$fp) {
throw new Exception ("can not read this intermediate csv file: " . $intermediateCsvFile);
}
while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) {
if (++$currentLine == $arrayNumber) {
fclose ($fp);
$array = unserialize ($line);
return;
}
}
throw new Exception ("the array number argument [" . $arrayNumber . "] is invalid (out of bounds)");
}
Usage example
try {
$totalSerializedArray = intermediateBuilder ($DIR . 'datafeeds/pricelist.csv',
$DIR . 'datafeeds/intermediatePricelist.csv');
$priceList = array () ;
$arrayNumber = 1;
loadArray ($priceList,
$arrayNumber,
$DIR . 'datafeeds/intermediatePricelist.csv');
if (!array_key_exists ($key, $priceList)) {
if (++$arrayNumber > $totalSerializedArray) $arrayNumber = 1;
loadArray ($priceList,
$arrayNumber,
$DIR . 'datafeeds/intermediatePricelist.csv');
}
catch (Exception $e) {
// TODO : log the error ...
}
You can drop the
if ($line) {
That only repeats the check from the loop condition. If your file is 54MB, and you are going to retain every line from the file, as an array, plus the key from column 3 (which is hashed for lookup)... I could see that requiring 75-85MB to store it all in memory. That isn't much. Most wordpress or magento pages using widgets run 150-200MB. But if your host is set low it could be a problem.
You can try filtering out some rows by changing the if($line) to a if($line[1] == 'book') to reduce how much you store. But the only sure way to handle storing that much content in memory is to have that much memory available to the script.
You can try set bigger memory using this. You can change limit how you want.
ini_set('memory_limit', '2048M');
But also depents how you want that script use.

create multiple directories using loop in php

I am taking data from text file( data is: daa1 daa2 daa3 on separate lines) then trying to make folders with exact name but only daa3 folders is created. Also when i use integer it creates all folders, same is the case with static string i.e "faraz".
$file = __DIR__."/dataFile.txt";
$f = fopen($file, "r");
$line =0;
while ( $line < 5 )
{
$a = fgets($f, 100);
$nl = mb_strtolower($line);
$nl = "checkmeck/".$nl;
$nl = $nl."faraz"; // it works for static value i.e for faraz
//$nl = $nl.$a; // i want this to be the name of folder
if (!file_exists($nl)) {
mkdir($nl, 0777, true);
}
$line++;
}
kindly help
use feof function its much better to get file content also line by line
Check this full code
$file = __DIR__."/dataFile.txt";
$linecount = 0;
$handle = fopen($file, "r");
$mainFolder = "checkmeck";
while(!feof($handle))
{
$line = fgets($handle);
$foldername = $mainFolder."/".trim($line);
//$line is line name daa1,daa2,daa3 etc
if (!file_exists($foldername)) {
mkdir($foldername, 0777, true);
}
$linecount++;
unset($line);
}
fclose($handle);
output folders
1countfaraz
2countfaraz
3countfaraz
Not sure why you're having trouble with your code, but I find it to be more straightforward to use file_get_contents() instead of fopen() and fgets():
$file = __DIR__."/dataFile.txt";
$contents = file_get_contents($file);
$lines = explode("\n", $contents);
foreach ($lines as $line) {
$nl = "checkmeck/". $line;
if (!file_exists($nl)) {
echo 'Creating file '. $nl . PHP_EOL;
mkdir($nl, 0777, true);
echo 'File '. $nl .' has been created'. PHP_EOL;
} else {
echo 'File '. $nl .' already exists'. PHP_EOL;
}
}
The echo statements above are for debugging so that you can see what your code is doing. Once it is working correctly, you can remove them.
So you get the entire file contents, split it (explode()) by the newline character (\n), and then loop through the lines in the file. If what you said is true, and the file looks like:
daa1
daa2
daa3
...then it should create the following folders:
checkmeck/daa1
checkmeck/daa2
checkmeck/daa3

File manupulation search and replace csv php

I need a script that is finding and then replacing a sertain line in a CSV like file.
The file looks like this:
18:110327,98414,127500,114185,121701,89379,89385,89382,92223,89388,89366,89362,89372,89369
21:82297,79292,89359,89382,83486,99100
98:110327,98414,127500,114185,121701
24:82297,79292,89359,89382,83486,99100
Now i need to change the line 21.
This is wat i got so far.
The first 2 to 4 digits folowed by : ar a catergory number. Every number after this(followed by a ,) is a id of a page.
I acces te id's i want (i.e. 82297 and so on) from database.
//test 2
$sQry = "SELECT * FROM artikelen WHERE adviesprijs <>''";
$rQuery = mysql_query ($sQry);
if ( $rQuery === false )
{
echo mysql_error ();
exit ;
}
$aResult = array ();
while ( $r = mysql_fetch_assoc ($rQuery) )
{
$aResult[] = $r['artikelid'];
}
$replace_val_dirty = join(",",$aResult);
$replace_val= "21:".$replace_val_dirty;
// file location
$file='../../data/articles/index.lst';
// read the file index.lst
$file1 = file_get_contents($file);
//strip eerde artikel id van index.lst
$file3='../../data/articles/index_grp21.lst';
$file3_contents = file_get_contents($file3);
$file2 = str_replace($file3_contents, $replace_val, $file1);
if (file_exists($file)) {
echo "The file $filename exists";
} else {
echo "The file $filename does not exist";
}
if (file_exists($file3)) {
echo "The file $filename exists";
} else {
echo "The file $filename does not exist";
}
// replace the data
$file_val = $file2;
// write the file
file_put_contents($file, $file_val);
//write index_grp98.lst
file_put_contents($file3, $replace_val);
mail('info#', 'Aanbieding catergorie geupdate', 'Aanbieding catergorie geupdate');
Can anyone point me in the right direction to do this?
Any help would be appreciated.
You need to open the original file and go through each line. When you find the line to be changed, change that line.
As you can not edit the file while you do that, you write a temporary file while doing this, so you copy over line-by-line and in case the line needs a change, you change that line.
When you're done with the whole file, you copy over the temporary file to the original file.
Example Code:
$path = 'file';
$category = 21;
$articles = [111182297, 79292, 89359, 89382, 83486, 99100];
$prefix = $category . ':';
$prefixLen = strlen($prefix);
$newLine = $prefix . implode(',', $articles);
This part is just setting up the basics: The category, the IDs of the articles and then building the related strings.
Now opening the file to change the line in:
$file = new SplFileObject($path, 'r+');
$file->setFlags(SplFileObject::DROP_NEW_LINE | SplFileObject::SKIP_EMPTY);
$file->flock(LOCK_EX);
The file is locked so that no other process can edit the file while it gets changed. Next to that file, the temporary file is needed, too:
$temp = new SplTempFileObject(4096);
After setting up the two files, let's go over each line in $file and compare if it needs to be replaced:
foreach ($file as $line) {
$isCategoryLine = substr($line, 0, $prefixLen) === $prefix;
if ($isCategoryLine) {
$line = $newLine;
}
$temp->fwrite($line."\n");
}
Now the $temporary file contains already the changed line. Take note that I used UNIX type of EOF (End Of Line) character (\n), depending on your concrete file-type this may vary.
So now, the temporary file needs to be copied over to the original file. Let's rewind the file, truncate it and then write all lines again:
$file->seek(0);
$file->ftruncate(0);
foreach ($temp as $line) {
$file->fwrite($line);
}
And finally you need to lift the lock:
$file->flock(LOCK_UN);
And that's it, in $file, the line has been replaced.
Example at once:
$path = 'file';
$category = 21;
$articles = [111182297, 79292, 89359, 89382, 83486, 99100];
$prefix = $category . ':';
$prefixLen = strlen($prefix);
$newLine = $prefix . implode(',', $articles);
$file = new SplFileObject($path, 'r+');
$file->setFlags(SplFileObject::DROP_NEW_LINE | SplFileObject::SKIP_EMPTY);
$file->flock(LOCK_EX);
$temp = new SplTempFileObject(4096);
foreach ($file as $line) {
$isCategoryLine = substr($line, 0, $prefixLen) === $prefix;
if ($isCategoryLine) {
$line = $newLine;
}
$temp->fwrite($line."\n");
}
$file->seek(0);
$file->ftruncate(0);
foreach ($temp as $line) {
$file->fwrite($line);
}
$file->flock(LOCK_UN);
Should work with PHP 5.2 and above, I use PHP 5.4 array syntax, you can replace [111182297, ...] with array(111182297, ...) in case you're using PHP 5.2 / 5.3.

Sort the unsorted text file and rewrite to same text file in sorted order

I have a question. I am in process of learning how to read/write files, but having little trouble trying to do both at the same time in same php script. I have a text file with words like this,
Richmond,Virginia
Seattle,Washington
Los Angeles,California
Dallas,Texas
Jacksonville,Florida
I wrote a code to sort them in order and this will display in sort order by City.
<?php
$file = file("states.txt");
sort($file);
for($i=0; $i<count($file); $i++)
{
$states = explode(",", $file[$i]);
echo $states[0], $states[1],"<br />";
}
?>
From this, how can I rewrite those sorted information back into the states.txt file?
The easiest way to write the contents of $file back to the file would be using file_put_contents in collaboration with implode.
file_put_contents("states.txt", implode($file));
Try using fopen and fwrite.
$fileWrite = fopen("filePah", "w");
for($i=0; $i<count($file); $i++)
{
fWrite($fileWrite, $file[i]);
}
fClose($fileWrite);
<?php
$file = file("states.txt");
sort($file);
$newContent = "";
for($i=0; $i<count($file); $i++)
{
$states = explode(",", $file[$i]);
$newContent .= $states[0] .', '. $states[1] . PHP_EOL;
}
file_put_contents('states.txt',$newContent);
?>
PHP: file_put_contents
Try something like this:
$fo = fopen("filename", "w");
$content = "";
for ($i = 0; $i < count($file); $i++) {
$states = explode(",", $file[$i]);
$content .= $states[0] . "," . $states[1] . "\n";
}
fwrite($fo, $content);
fclose($fo);
this is a little extended, however I thought it might be useful to smb. I have an m3u playlist and need only particular rows filtered, sorted and printed. Credits go to Devil:
<?php
//specify that the variable is of type array
$masiv = array();
//read the file
$file = '/home/zlobi/radio/pls/all.m3u';
$f = fopen($file, "r");
while ($line = fgets($f))
{
//skip rows with #EXT
if(strpos($line, "#EXT") !== false) continue;
$text = str_replace('.ogg', ' ', $line);
$text = str_replace('/home/zlobi/radio/',' ',$text);
//add the song as an element in an array
$masiv[] = $text;
}
$f = fclose($f);
//sort the array
sort($masiv);
//pass via the array, take each element and print it
foreach($masiv as $pesen)
print $pesen.'<br/>';
?>
masiv is array, pesen is song in Bulgarian :)
CAPital letters are sorted first.
Regads
Once you are done reading the file into the array by a call to file. You can open the file for writing by using the fopen function, write into the file using the fwrite and close the file handle using fclose:
<?php
$file = file("states.txt"); // read file into array.
$fh = fopen('states.txt','w') or die("..."); // open same file for writing.
sort($file);
for($i=0; $i<count($file); $i++)
{
$states = explode(",", $file[$i]);
echo $states[0], $states[1],"<br />";
fwrite($fh,"$states[0],$states[1] <br />"); // write to file.
}
fclose($fh); // close file.
?>
Open the file, write to it, close it (this assumes $file is the variable from your code):
$fp = fopen('states.txt', 'w');
for($i=0; $i<count($file); $i++)
fwrite($fp, $file[$i]);
}
fclose($fp);
And see http://php.net/manual/en/function.fwrite.php
This is by far the fastest and most elegant solution that I have found, when I had the same problem.
If you're on Linux (with exec allowed in PHP configuration) you can do the following (provided you want to sort files numerically):
exec("sort -n " . $pathToOriginalFile . " > " . $pathToSortedFile);
Basically, execute bash command sort that sorts the lines in a file numerically.
If you want to keep the data in the original file do this:
exec("sort -n " . $pathToOriginalFile . " > " . $pathToSortedFile);
exec("rm " . $pathToOriginalFile);
exec("mv " . $pathToSortedFile . " " . $pathToOriginalFile);
If you want an alphabetical sort just exclude -n (--numeric-sort) option.
exec("sort " . $pathToOriginalFile . " > " . $pathToSortedFile);
For me the command took about 3 seconds to sort 10 million lines in the file on the server.
You can find more about sort here http://www.computerhope.com/unix/usort.htm
Hope it helps.

PHP write to file

below is some code I am using to "translate" a map array into SQL code so I can easily update my database when I have updated my game map. As you can see it prints out the SQL code onto the screen so I can copy and paste it.
As my maps will get bigger this will become inefficient as it will crash the browser due to mass output, so instead I am wondering if it is possible to make it create a .txt file and write all of the data to it instead of printing onto the screen?
<?php
if (isset($_POST['code'])){
$map = $_POST['code'];
$map = preg_replace("/,\\s*}/i", "}", $map);
$map = str_replace("{", "[", $map);
$map = str_replace("}", "]", $map);
$map = json_decode('[' . $map . ']');
$arrayCount1 = 0;
$arrayCount2 = -1;
$H = sprintf('%05d', 00000);
$V = sprintf('%05d', 00000);
$id = 1;
echo "INSERT INTO `map` (`id`, `horizontal`, `verticle`, `image`) VALUES" . "<br />";
for ($count1 = 0; $count1 < sizeof($map[0]); $count1++){
$arrayCount2++;
$arrayCount1 = 0;
$V = sprintf('%05d', $V + 1);
$H = sprintf('%05d', 00000);
for ($count2 = 0; $count2 < sizeof($map); $count2++){
echo "(" . $id . ", '" . $H . "', '" . $V . "', '" . $map[$arrayCount1][$arrayCount2] . "')," . "<br />";
$arrayCount1++;
$id++;
$H = sprintf('%05d', $H + 1);
}
}
}
?>
That should be quite simple. Add
// second parameter 'a' stands for APPEND
$f = fopen('/path/to/the/file/you/want/to/write/to', 'a');
to the beginning of your script.
Add
fclose($f);
to the end fo your script to cleanly close the file handle (good pratice even though handles would be closed the the terminating script automatically).
And the change all your echo's and prints to
fwrite($f, '<<your string>>');
EDIT:
That way you can even compress the data on the fly using a compression stream wrapper if amnount of data gets really large.
There is an even simpler approach:
ob_start();
# Your code here ...
file_put_contents('yourfile.txt', ob_get_clean());
If this is something you plan on writing on a regular interval or by different scripts, look at using flock() to lock the file and prevent data corruption.
$fp = fopen("/tmp/lock.txt", "w+");
if (flock($fp, LOCK_EX)) { // do an exclusive lock
fwrite($fp, "Write something here\n");
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't lock the file !";
}
fclose($fp);
$str = <<<your string comes here>>>
if( $fh = #fopen( "myfile.txt", "a+" ) ) {
fputs( $fh, $str, strlen($str) );
fclose( $fh );
}
this should do...
write all the lines and then send the file to the client.
Check this post for further instructions
+my 2 cents:
You may check your database servers mass data loading features, as most of them can load files in batch faster than performing thousands of inserts.

Categories