PHP Simultaneous file access / flock() issue - php
I'm having trouble implementing a script which should parse a json string from a file, either overwrite the object with the same id or add it to the json array and write it back to the file. The script is called in a for loop so that I tried to use flock to prevent overwriting the file before the json string could be parsed, but the results look really strange and I don't know whats going wrong. Here is the script:
$method = $_SERVER['REQUEST_METHOD'];
$file = fopen($userid . '.json', 'c+');
flock($file, LOCK_EX);
$jsonStr = (filesize($userid . '.json') == 0 ) ? '{"success": true}' : fread($file, filesize($userid . '.json'));
$jsonObj = json_decode($jsonStr);
if($method === 'POST') {
$dataStr = file_get_contents('php://input');
$dataObj = json_decode($dataStr);
$replacements = array();
if(isset($jsonObj->images)) {
foreach($jsonObj->images as $idx=>$image) {
if($image->id == $dataObj->id) {
$replacements += array($idx => $dataObj);
}
}
if(count($replacements)==0) {
array_push($replacements, $dataObj);
}
$jsonObj->images = array_replace($jsonObj->images, $replacements);
} else {
//$jsonObj = new stdClass();
$jsonObj->images = new stdClass();
$jsonObj->images = array($dataObj);
}
fwrite($file, json_encode($jsonObj));
file_put_contents('log.json', json_encode($dataObj), FILE_APPEND);
echo '{"success":true, "images":' . $dataStr . '}';
} else if($method === 'GET') {
echo $jsonStr;
}
fclose($file);
As result I get sometime three files one just called ".json" and the other one correctly userid.json and the log.json. The log looks best, since all 26 objects are shown correctly, but without checking for exiting objects previously. The ".json" is just blank and the looks really odd:
{"success":true,"images":[{"id":"f9f4b6ee-8b7b-414e-98f4-47ed5ee3c594","top":1200,"left":4050,"clicks":0,"affinity":[]}]}
{"success":true,"images":[{"id":"034c7661-9466-4651-b860-7e049e1543ac","top":900,"left":600,"clicks":0,"affinity":[]}]}
{"images":[{"id":"eebc35ec-6c0e-416a-9516-061df821083d","top":1800,"left":3300,"clicks":0,"affinity":[]}]}
{"images":[{"id":"e09da5b9-2e65-4a12-94cd-4745b354a256","top":1200,"left":1200,"clicks":0,"affinity":[]}]}
{"images":[{"id":"b023a259-4554-48a5-acd8-cb4215e6a391","top":1350,"left":1500,"clicks":0,"affinity":[]}]}
{"images":[{"id":"6521d8c3-461c-4fcd-b69d-69f14ae37418","top":600,"left":3750,"clicks":0,"affinity":[]}]}
{"images":[{"id":"4c082c1a-d4ae-4c1c-bfc4-ed36980f5db2","top":3150,"left":600,"clicks":0,"affinity":[]}]}
{"images":[{"id":"22dbdea2-4936-4353-825e-9af6e6f2ca93","top":3000,"left":2100,"clicks":0,"affinity":[]}]}
{"images":[{"id":"a7ba32d4-912e-489d-8f9b-e412bc1629c1","top":2850,"left":3750,"clicks":0,"affinity":[]}]}
{"images":[{"id":"b258148d-1779-4f17-ae5e-1160aa0a34b9","top":750,"left":4050,"clicks":0,"affinity":[]}]}
{"images":[{"id":"21c1c9d1-a7a6-48d2-a4ba-174c0fe248a3","top":1650,"left":750,"clicks":0,"affinity":[]}]}
{"images":[{"id":"bfed2f56-7876-4eb2-a217-e0f71f4455ee","top":3300,"left":1500,"clicks":0,"affinity":[]}]}
{"images":[{"id":"7f13b674-5d07-444f-bf6d-463758c96788","top":1650,"left":1950,"clicks":0,"affinity":[]}]}
{"images":[{"id":"fb8711f2-ba10-44d7-9038-02cae6438506","top":1800,"left":450,"clicks":0,"affinity":[]}]}
{"images":[{"id":"ac83e869-e4fb-4eee-8d2e-4d26762101d9","top":750,"left":1800,"clicks":0,"affinity":[]}]}
{"images":[{"id":"069a7595-2271-4430-b324-974115df7b80","top":2550,"left":1800,"clicks":0,"affinity":[]}]}
{"images":[{"id":"ea9ab2fc-e179-4f56-9ce7-8af456911b33","top":1950,"left":750,"clicks":0,"affinity":[]}]}
{"images":[{"id":"9b997ae3-0c66-4f78-937a-2fde2470d7d7","top":300,"left":300,"clicks":0,"affinity":[]}]}
{"images":[{"id":"f860dea4-3b27-4401-999b-72ce45022110","top":1200,"left":1950,"clicks":0,"affinity":[]}]}
{"images":[{"id":"2633c107-8a51-4e9c-bc2c-f87c30f7359d","top":750,"left":2100,"clicks":0,"affinity":[]}]}
{"images":[{"id":"bebcb2bb-c2f0-4e4d-b202-b05090eacf49","top":3600,"left":3600,"clicks":0,"affinity":[]}]}
{"images":[{"id":"460f961e-f1dd-4329-95be-d30770a37b83","top":2400,"left":1200,"clicks":0,"affinity":[]}]}
{"images":[{"id":"ee9b254f-fe6a-4ece-a662-e934625e992a","top":3600,"left":600,"clicks":0,"affinity":[]}]}
{"images":[{"id":"4e047ea9-345e-4cf9-a066-d0b431820c61","top":2100,"left":1350,"clicks":0,"affinity":[]}]}
{"images":[{"id":"8bbb5327-6d39-40e1-919e-d7b70e13fc2b","top":2700,"left":1200,"clicks":0,"affinity":[]}]}
{"images":[{"id":"0c50c0ec-e18b-4917-9787-bad245eed798","top":1200,"left":2550,"clicks":0,"affinity":[]}]}
I also get an exception:
<b>Strict Standards</b>: Creating default object from empty value in <b>C:\xampp\htdocs\gallery\store.php</b> on line <b>38</b><br />
I guess I didn't understand the procedure correctly and I'm happy for any help.
EDIT: The result I want, looks like this:
{"success":true,"images":[{"id":"f9f4b6ee-8b7b-414e-98f4-47ed5ee3c594","top":1200,"left":4050,"clicks":0,"affinity":[]},
{"id":"034c7661-9466-4651-b860-7e049e1543ac","top":900,"left":600,"clicks":0,"affinity":[]},{"id":"eebc35ec-6c0e-416a-9516-061df821083d","top":1800,"left":3300,"clicks":0,"affinity":[]},
{"id":"e09da5b9-2e65-4a12-94cd-4745b354a256","top":1200,"left":1200,"clicks":0,"affinity":[]},
{"id":"b023a259-4554-48a5-acd8-cb4215e6a391","top":1350,"left":1500,"clicks":0,"affinity":[]},
{"id":"6521d8c3-461c-4fcd-b69d-69f14ae37418","top":600,"left":3750,"clicks":0,"affinity":[]},
{"id":"4c082c1a-d4ae-4c1c-bfc4-ed36980f5db2","top":3150,"left":600,"clicks":0,"affinity":[]},
{"id":"22dbdea2-4936-4353-825e-9af6e6f2ca93","top":3000,"left":2100,"clicks":0,"affinity":[]},
{"id":"a7ba32d4-912e-489d-8f9b-e412bc1629c1","top":2850,"left":3750,"clicks":0,"affinity":[]},
{"id":"b258148d-1779-4f17-ae5e-1160aa0a34b9","top":750,"left":4050,"clicks":0,"affinity":[]},
{"id":"21c1c9d1-a7a6-48d2-a4ba-174c0fe248a3","top":1650,"left":750,"clicks":0,"affinity":[]},
{"id":"bfed2f56-7876-4eb2-a217-e0f71f4455ee","top":3300,"left":1500,"clicks":0,"affinity":[]},
{"id":"7f13b674-5d07-444f-bf6d-463758c96788","top":1650,"left":1950,"clicks":0,"affinity":[]},
{"id":"fb8711f2-ba10-44d7-9038-02cae6438506","top":1800,"left":450,"clicks":0,"affinity":[]},
{"id":"ac83e869-e4fb-4eee-8d2e-4d26762101d9","top":750,"left":1800,"clicks":0,"affinity":[]},
{"id":"069a7595-2271-4430-b324-974115df7b80","top":2550,"left":1800,"clicks":0,"affinity":[]},
{"id":"ea9ab2fc-e179-4f56-9ce7-8af456911b33","top":1950,"left":750,"clicks":0,"affinity":[]},
{"id":"9b997ae3-0c66-4f78-937a-2fde2470d7d7","top":300,"left":300,"clicks":0,"affinity":[]},
{"id":"f860dea4-3b27-4401-999b-72ce45022110","top":1200,"left":1950,"clicks":0,"affinity":[]},
{"id":"2633c107-8a51-4e9c-bc2c-f87c30f7359d","top":750,"left":2100,"clicks":0,"affinity":[]},
{"id":"bebcb2bb-c2f0-4e4d-b202-b05090eacf49","top":3600,"left":3600,"clicks":0,"affinity":[]},
{"id":"460f961e-f1dd-4329-95be-d30770a37b83","top":2400,"left":1200,"clicks":0,"affinity":[]},
{"id":"ee9b254f-fe6a-4ece-a662-e934625e992a","top":3600,"left":600,"clicks":0,"affinity":[]},
{"id":"4e047ea9-345e-4cf9-a066-d0b431820c61","top":2100,"left":1350,"clicks":0,"affinity":[]},
{"id":"8bbb5327-6d39-40e1-919e-d7b70e13fc2b","top":2700,"left":1200,"clicks":0,"affinity":[]},
{"id":"0c50c0ec-e18b-4917-9787-bad245eed798","top":1200,"left":2550,"clicks":0,"affinity":[]}]}
Cheers,
Daniel
I finally have it working and for anyone else having issues with simultaneous file access, here's my solution:
Check if the file exists, if not, create it and set up the data structure (to prevent the strict standard error)
$filename = $userid . '.json';
if(!file_exists($filename)) {
$file = fopen($filename, 'w');
if(flock($file, LOCK_EX)) {
fwrite($file, '{"success": true, "images": []}');
flock($file, LOCK_UN);
fclose($file);
}
}
Open the file using c+ as second parameter (w+ would set the filesize to 0). Check if you can get the exclusive lock. Read the file and rewind in order to overwrite it later.
else {
$file = fopen($userid . '.json', 'c+');
if(flock($file, LOCK_EX)) {
$jsonStr = fread($file, filesize($userid . '.json'));
$jsonObj = json_decode($jsonStr);
rewind($file);
Manipulate the object how you need it and write it back as encoded json string using fwrite
if($method === 'POST') {
$dataStr = file_get_contents('php://input');
if(isset($dataStr) &&$dataStr != '') {
$dataObj = json_decode($dataStr);
$replace = -1;
foreach($jsonObj->images as $idx=>$image) {
if($image->id == $dataObj->id) {
$replace = $idx;
}
}
if($replace != -1) {
$img = $jsonObj->images;
$img[$replace] = $dataObj;
$jsonObj->images = $img;
} else {
array_push($jsonObj->images, $dataObj);
}
fwrite($file, json_encode($jsonObj));
}
echo '{"success":true, "images":' . $dataStr . '}';
} else if($method === 'GET') {
echo $jsonStr;
}
Release the lock and close the file.
flock($file, LOCK_UN);
fclose($file);
}
}
Related
PHP change only one value inside ini file
I'm trying to rewrite a config.ini file which looks like this dbhost=localhost dbname=phonebook dbuname=root dbpass= reinstall=2 I want to change the reinstall value to 1 like so dbhost=localhost dbname=phonebook dbuname=root dbpass= reinstall=1 I already wrote some lines, yet i am stuck and don't know how to change only one value $filepath = 'config.ini'; $data = #parse_ini_file("config.ini");; //update ini file, call function function update_ini_file($data, $filepath) { $content = ""; //parse the ini file to get the sections //parse the ini file using default parse_ini_file() PHP function $parsed_ini = parse_ini_file($filepath, true); foreach($data as $section => $values){ if($section === "submit"){ continue; } $content .= $section ."=". $values . "\n"; } //write it into file if (!$handle = fopen($filepath, 'w')) { return false; } $success = fwrite($handle, $content); fclose($handle); } update_ini_file($data, $filepath); header('location: '.ROOT_PATH.'/');
Got it fixed like this $filepath = 'config.ini'; $data = #parse_ini_file("config.ini"); $data['reinstall']='1'; //update ini file, call function function update_ini_file($data, $filepath) { $content = ""; //parse the ini file to get the sections //parse the ini file using default parse_ini_file() PHP function $parsed_ini = parse_ini_file($filepath, true); foreach($data as $section => $values){ if($section === "submit"){ continue; } $content .= $section ."=". $values . "\n"; } //write it into file if (!$handle = fopen($filepath, 'w')) { return false; } $success = fwrite($handle, $content); fclose($handle); } update_ini_file($data, $filepath); header('location: '.ROOT_PATH.'/');
PHP - How to write a line in text file if line already available in file then count the request
I want to write a PHP code which write a string line in text file if the line already available in text file then count the requests for example text file contain: red.apple:1 big.orange:1 green.banana:1 If some one request to add big.orange in file if its already available in file then count as big.orange:2 if not available then write new line big.orange:1 after execution code text file red.apple:1 big.orange:2 green.banana:1 I've written the following code but not working. <?PHP $name = $_GET['fname'] $file = fopen('request.txt', "r+") or die("Unable to open file!"); if ($file) { while (!feof($file)) { $entry_array = explode(":",fgets($file)); if ($entry_array[0] == $name) { $entry_array[1]==$entry_array[1]+1; fwrite($file, $entry_array[1]); } } fclose($file); } else{ fwrite($file, $name.":1"."\n"); fclose($file); } ?>
Instead of creating your own format which you need to parse manually, you can simply use json. Below is a suggestion about how it would work. It will add the requested fname value if it doesn't already exist and will also create the file if it doesn't already exists. $name = $_GET['fname'] ?? null; if (is_null($name)) { // The fname query param is missing so we can't really continue die('Got no name'); } $file = 'request.json'; if (is_file($file)) { // The file exists. Load it's content $content = file_get_contents($file); // Convert the contents (stringified json) to an array $data = json_decode($content, true); } else { // The file does not extst. Create an empty array we can use $data = []; } // Get the current value if it exists or start with 0 $currentValue = $data[$name] ?? 0; // Set the new value $data[$name] = $currentValue + 1; // Convert the array to a stringified json object $content = json_encode($data); // Save the file file_put_contents($file, $content);
If you still need to use this format (like, this is some exam test or legacy), try the function: function touchFile($file, $string) { if (!file_exists($file)) { if (is_writable(dirname($file))) { // create file (later) $fileData = ""; } else { throw new ErrorException("File '".$file."' doesn't exist and cannot be created"); } } else $fileData = file_get_contents($file); if (preg_match("#^".preg_quote($string).":(\d+)\n#m", $fileData, $args)) { $fileData = str_replace($args[0], $string.":".(intval($args[1])+1)."\n", $fileData); } else { $fileData .= $string.":1\n"; } if (file_put_contents($file, $fileData)) { return true; } else { return false; } }
PHP data processing not outputting
Hi I am trying to get this external text file to print inside my php document. The code looks fine to me however when I echo it does not output anything and I am not sure why this is. Can anybody help me out as I am new to this. $location = '/Applications/MAMP/htdocs/PHPLabs/branches.txt'; $fp = fopen($location, 'r'); if ($fp) { $readin = fread($fp); fclose($fp); } else { echo 'Can\'t open input.txt'; }
Not sure what you're trying to 'echo' but have you checked if the file exists in the first place? Your code could be written as: $location = '/Applications/MAMP/htdocs/PHPLabs/branches.txt'; if (file_exists($location) && $data = file_get_content($location)){ echo $data; } else { echo 'File not found'; } if (file_exists($location) && $file = fopen($location, 'r')){ $file_content = fread($file, filesize($location)); fclose($file); } esle { echo 'File not found'; } See here for more: http://php.net/manual/en/function.file-get-contents.php, http://php.net/manual/en/function.filesize.php
Reading textfiles from path in a text file [PHP]
I have a textfile ($file) that contains the path to other files. My goal is to read the content of those files and print it in a table. When reading the paths from $file, only the last line path works correctly. <?php $file = "log2.txt"; if(file_exists($file)) { $handler = fopen($file,'r'); while(!feof($handler)) { $lines = fgets($handler); $wordarray = explode(' ', $lines); #echo $wordarray[0]." ".$wordarray[1]." ".$wordarray[2]; if (strpos($lines, 'NOK') !== false) { echo "<tr><td>".$wordarray[0]."</td></tr>"; if(file_exists($wordarray[2])){ $log = fopen($wordarray[2], 'r'); #echo "FILE EXISTS ".$log; $logtext = fread($log,filesize($wordarray[2])); echo "<tr><td>".$logtext."</td></tr>"; fclose($log); } else { echo "<tr><td>"."FILE ".$wordarray[2]." FAILED TO LOAD"."</td></tr>"; } } } fclose($handler); } else { echo "FILE DOES NOT EXISTS"; } ?> Here is an exemple of how log2.txt would look like: POE NOK poelog.txt LINK-ERRORS OK OK LATENCIES NOK latencieslog.txt VOLATILE NOK volatilelog.txt I think that the problem could be about line endings or so, but cannot get the point.
You're right. The $wordarray[2] contains also new line character so before using it pass it through trim() function and store it in a variable ($filename in my case). Updated inner if: if (strpos($lines, 'NOK') !== false) { echo "<tr><td>".$wordarray[0]."</td></tr>"; $filename = trim($wordarray[2]); if (file_exists($filename)){ echo "FILE EXISTS ".$filename; $log = fopen($filename, 'r'); $logtext = fread($log, filesize($filename)); echo "<tr><td>".$logtext."</td></tr>"; fclose($log); } else{ echo "<tr><td>"."FILE ".$filename." FAILED TO LOAD"."</td></tr>"; } }
This short function works like a sharm: function displayFileContent( $fileName ) { $arrayWithFileNames = file ( $fileName ); echo "<table>"; foreach ( $arrayWithFileNames as $singleFileName ) { # remove the trailing \n on Linux - windows has 2 character as EOL $singleFileName = trim(preg_replace('/\s\s+/', ' ', $singleFileName)); $contentOfFile = file_get_contents( $singleFileName ); echo "<tr><td>{$contentOfFile}</td></tr>"; } echo "</table>"; } You use it like this: displayFileContent ("path-to-your-file"); Remark: There is no check if the file does exist....
Reading large text files efficiently
I have a couple of huge (11mb and 54mb) files that I need to read to process the rest of the script. Currently I'm reading the files and storing them in an array like so: $pricelist = array(); $fp = fopen($DIR.'datafeeds/pricelist.csv','r'); while (($line = fgetcsv($fp, 0, ",")) !== FALSE) { if ($line) { $pricelist[$line[2]] = $line; } } fclose($fp); .. but I'm constantly getting memory overload messages from my webhost. How do I read it more efficiently? I don't need to store everything, I already have the keyword which exactly matches the array key $line[2] and I need to read just that one array/line.
If you know the key why don't you filter out by the key? And you can check memory usage with memory_get_usage() function to see how much memory allocated after you fill your $pricelist array. echo memory_get_usage() . "\n"; $yourKey = 'some_key'; $pricelist = array(); $fp = fopen($DIR.'datafeeds/pricelist.csv','r'); while (($line = fgetcsv($fp, 0, ",")) !== FALSE) { if (isset($line[2]) && $line[2] == $yourKey) { $pricelist[$line[2]] = $line; break; /* If there is a possiblity to have multiple lines we can store each line in a separate array element $pricelist[$line[2]][] = $line; */ } } fclose($fp); echo memory_get_usage() . "\n";
You can try this (I have not checked if it works properly) $data = explode("\n", shell_exec('cat filename.csv | grep KEYWORD')); You will get all the lines containing the keyword, each line as an element of array. Let me know if it helps.
I join what user2864740 said : "The problem is the in-memory usage caused by the array itself and is not about "reading" the file" My Solution is : Split your `$priceList` array Load only 1 at time a splitted Array in memory Keep the other splitted Arrays in an intermediate file N.B: i did not test what i've written <?php define ("MAX_LINE", 10000) ; define ("CSV_SEPERATOR", ',') ; function intermediateBuilder ($csvFile, $intermediateCsvFile) { $pricelist = array (); $currentLine = 0; $totalSerializedArray = 0; if (!is_file()) { throw new Exception ("this is not a regular file: " . $csv); } $fp = fopen ($csvFile, 'r'); if (!$fp) { throw new Exception ("can not read this file: " . $csv); } while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) { if ($line) { $pricelist[$line[2]] = $line; } if (++$currentLine == MAX_LINE) { $fp2 = fopen ($intermediateCsvFile, 'a'); if (!$fp) throw new Exception ("can not write in this intermediate csv file: " . $intermediateCsvFile); fputs ($fp2, serialize ($pricelist) . "\n"); fclose ($fp2); unset ($pricelist); $pricelist = array (); $currentLine = 0; $totalSerializedArray++; } } fclose($fp); return $totalSerializedArray; } /** * #param array : by reference unserialized array * #param integer : the array number to read from the intermediate csv file; start from index 1 * #param string : the (relative|absolute) path/name of the intermediate csv file * #throw Exception */ function loadArray (&$array, $arrayNumber, $intermediateCsvFile) { $currentLine = 0; $fp = fopen ($intermediateCsvFile, 'r'); if (!$fp) { throw new Exception ("can not read this intermediate csv file: " . $intermediateCsvFile); } while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) { if (++$currentLine == $arrayNumber) { fclose ($fp); $array = unserialize ($line); return; } } throw new Exception ("the array number argument [" . $arrayNumber . "] is invalid (out of bounds)"); } Usage example try { $totalSerializedArray = intermediateBuilder ($DIR . 'datafeeds/pricelist.csv', $DIR . 'datafeeds/intermediatePricelist.csv'); $priceList = array () ; $arrayNumber = 1; loadArray ($priceList, $arrayNumber, $DIR . 'datafeeds/intermediatePricelist.csv'); if (!array_key_exists ($key, $priceList)) { if (++$arrayNumber > $totalSerializedArray) $arrayNumber = 1; loadArray ($priceList, $arrayNumber, $DIR . 'datafeeds/intermediatePricelist.csv'); } catch (Exception $e) { // TODO : log the error ... }
You can drop the if ($line) { That only repeats the check from the loop condition. If your file is 54MB, and you are going to retain every line from the file, as an array, plus the key from column 3 (which is hashed for lookup)... I could see that requiring 75-85MB to store it all in memory. That isn't much. Most wordpress or magento pages using widgets run 150-200MB. But if your host is set low it could be a problem. You can try filtering out some rows by changing the if($line) to a if($line[1] == 'book') to reduce how much you store. But the only sure way to handle storing that much content in memory is to have that much memory available to the script.
You can try set bigger memory using this. You can change limit how you want. ini_set('memory_limit', '2048M'); But also depents how you want that script use.