PHP File locking problem with multiple writes - php

I'm using a very simple script where some users data are stored in a text file beside a database because this file needs to be read by a specific process on the system .
The php expire date update code for a line in the following format like this :
U: w w { enddate=2022-10-18 }
The function code goes like this:
$date = "enddate=";
$date .= setExpireDate($duration);
$Read = file($fileDir);
foreach ($Read as $LineNum => $line) {
$LineParts = explode(' ', $line);
if (count($LineParts) == 1) {continue;}
if ($LineParts[1] == $username) {
$LineParts[4] = $date;
$Read[$LineNum] = implode(' ', $LineParts);
break;
}
}
file_put_contents($fileDir, $Read, LOCK_EX);
some users have reported problems , after doing some search i found that the data in the database is updated fine , but the not in the text file as the data is not updated for some users .
as i think it is a file locking problem , that prevents some users under heavy load to write on the file , but the database can handle multiple writes just fine .
So is there anyway to handle the multiple writes on the file at the same time better than this ?
and is there a way to simulate this in php like locking the file by a form submit ?
i tried this :
function lock(){
$fp = fopen("file.txt", "r+");
flock($fp, LOCK_EX);
echo "file is locked <br>";
}
but it is not working at all ,and others can write to the file .
P.S: I know there are a lot of questions about file locking ,but non of the answers did help !
Regards

Here is an example of the file locking approach
First you attempt to lock the Lock file, if you can all well and good proceed to edit the real file. If you cannot get a lock you retry until you can.
// this is not the file you are attempting to update, its just the lock file
// it must already exist and its bext if you place it in a specific directory,
// not accessable from the web.
$fplock = fopen("/some/path/lock.txt", "r+");
// attempt to get an exclusive lock on the lock file
while ( ! flock($fplock, LOCK_EX) ) {
// this has failed to get the lock so do nothing but retry,
// it should be only milli seconds until the other process finishes
// and releases the lock
}
// the lock must have been released by the other process when you get here
// or you got a lock on your first attempt and never ran the wait loop
// at this point this script has the lock
$date = "enddate=";
$date .= setExpireDate($duration);
$Read = file($fileDir);
foreach ($Read as $LineNum => $line) {
$LineParts = explode(' ', $line);
if (count($LineParts) == 1) {continue;}
if ($LineParts[1] == $username) {
$LineParts[4] = $date;
$Read[$LineNum] = implode(' ', $LineParts);
break;
}
}
file_put_contents($fileDir, $Read, LOCK_EX);
/*
closing the lock file has the effect of releasing the lock, or simply finishing this
script will do the same, but do it yourself as soon as you have finished updating the
real file, so other process's dont have to wait until the script finishes,
I assume you may have other code after this
*/
fclose($fplock);
Of course ALL scripts that access the Real File, must also attempt to lock the lock file before accessing the real file, or this all falls down in a mess.

Related

PHP Piping Data with f_open

I have a piece of code, the issue is the file "data" is over 8GB. This is very memory intensive. I want to reduce the usage of RAM and saw the f_load would be ideal, however, how could i explode this data?
This is my current code:
$data = file_get_contents("data");
$data = explode("|", $data);
foreach ($data as $d) { // rest of code
theoretically, i need to open a pipe, stream and close a pipe How would i go about this?
I've tried using f_open rather than file_get_contents but errors started popping up so i'm doing something wrong and would really like to learn.
You can use stream_get_line to read your data block per block (with | as the delimiter character) : PHP stream_get_line()
$fh = fopen('data', 'r'); // open the file in read-only mode
while( $d = stream_get_line($fh, 1000, '|') ) // read the file until the next |
{
echo $d . PHP_EOL ; // display one block of data
}
fclose($fh); // close the file

PHP File Handling (Download Counter) Reading file data as a number, writing it as that plus 1

I'm trying to make a download counter in a website for a video game in PHP, but for some reason, instead of incrementing the contents of the downloadcount.txt file by 1, it takes the number, increments it, and appends it to the end of the file. How could I just make it replace the file contents instead of appending it?
Here's the source:
<?php
ob_start();
$newURL = 'versions/v1.0.0aplha/Dungeon1UP.zip';
//header('Location: '.$newURL);
//increment download counter
$file = fopen("downloadcount.txt", "w+") or die("Unable to open file!");
$content = fread($file,filesize("downloadcount.txt"));
echo $content;
$output = (int) $content + 1;
//$output = 'test';
fwrite($file, $output);
fclose($file);
ob_end_flush();
?>
The number in the file is supposed to increase by one every time, but instead, it gives me numbers like this: 101110121011101310111012101110149.2233720368548E+189.2233720368548E+189.2233720368548E+18
As correctly pointed out in one of the comments, for your specific case you can use fseek ( $file, 0 ) right before writing, such as:
fseek ( $file, 0 );
fwrite($file, $output);
Or even simpler you can rewind($file) before writing, this will ensure that the next write happens at byte 0 - ie the start of the file.
The reason why the file gets appended it is because you're opening the file in append and truncate mode, that is "w+". You have to open it in readwrite mode in case you do not want to reset the contents, just "r+" on your fopen, such as:
fopen("downloadcount.txt", "r+")
Just make sure the file exists before writing!
Please see fopen modes here:
https://www.php.net/manual/en/function.fopen.php
And working code here:
https://bpaste.net/show/iasj
It will be much simpler to use file_get_contents/file_put_contents:
// update with more precise path to file:
$content = file_get_contents(__DIR__ . "/downloadcount.txt");
echo $content;
$output = (int) $content + 1;
// by default `file_put_contents` overwrites file content
file_put_contents(__DIR__ . "/downloadcount.txt", $output);
That appending should just be a typecasting problem, but I would not encourage you to handle counts the file way. In order to count the number of downloads for a file, it's better to make a database update of a row using transactions to handle concurrency properly, as doing it the file way could compromise accuracy.
You can get the content, check if the file has data. If not initialise to 0 and then just replace the content.
$fileContent = file_get_contents("downloadcount.txt");
$content = (!empty($fileContent) ? $fileContent : 0);
$content++;
file_put_contents('downloadcount.txt', $content);
Check $str or directly content inside the file

counter with IP block via files - PHP

I tried to write a counter for visits of my page, which only counts each IP once per day.
I transcribed a found code to understand how it works and to implement some other features later.
Unfortunately my new code doesn't work.
Code I made from it:
<?php
$timeReset = 86400; //24h
$ipPath = "ipsafe.txt";
$countPath = "counts.txt";
$timePath = "time.txt";
$ip = $REMOTE_ADDR;
$time = time();
$resetTime = $time+$timeReset;
$timeFile = fopen($timePath, "r+");
$timeData = fread($timeFile, filesize($timePath));
//if resetTime is passed -> Reset
if($time >= $timeData) {
//clear ipSafe
$ipFile1 = fopen($ipPath, "w+");
fclose($ipFile);
//set new resetTime
rewind($timeData);
fwrite ($timeData, $resetTime);
}
fclose($timeFile);
//creat IP-List
$ipData = file($ipPath);
while (list ($line_num, $line) = each ($ipData)) {
$digits = explode(",",$line);
}
//if IP was not saved in last timeframe
if(!in_array($ip, $digits))
{
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile1,2);
fwrite($ipFile2, "$ip".",");
#flock($ipFile1,3);
fclose($ipFile2);
//increment counter
$countFile = fopen($countPath,"r+");
$countData = fread($countFile, filesize($countPath);
rewind($countFile);
$countData++;
fwrite($countFile, $countData);
fclose($countFile);
}
?>
with the following questions:
what's wrong with my code?
flock is used to manage the access to the files, but why shall I use different names for the same file?
Thanks for your suggestions.
EDIT:
Sorry for being so unspecific in explaining the problem. After integrating a debugger, I changed "REMOTE_ADDR" to "$_SERVER['REMOTE_ADDR']" so I fixed 1 error. Now i got the following problems:
Warning: fopen(time.txt): failed to open stream: No such file or directory in .../counter.php on line 15
But the file is in the same directory as counter.php - of course the fread and filesize failed too
Warning: fopen(ipsafe.txt): failed to open stream: Permission denied in .../counter.php on line 20
The file doesn't exist yet, but I thought "fopen($ipPath, "w+");" creats it if it doesn't exist.
Fatal error: Call to undefined function fb() in .../counter.php on line 26
Doesn't "fb($timeFile);" send the value to firePHP?
Not doing a full code review (which would result in an almost full rewrite of your code), here are the things I would recommend changing to fix your listed issues:
The following line uses $REMOTE_ADDR, which is undefined.
$ip = $REMOTE_ADDR;
You most likely meant to use $_SERVER['REMOTE_ADDR'] (which you confirmed with your edit), so the line should read:
$ip = $_SERVER['REMOTE_ADDR'];
You state you're receiving an error attempting to open the time.txt file: "No such file or directory". You also say that the file exists in the same directory as counter.php - and yet you're still receiving this error. This can mean one of two things; the first, the file doesn't exist and you're mistaken; the second, your counter.php file is in a different directory than whatever script is executing it - this would mean that the time.txt file would need to be in the directory of the executing script. I would recommend two things for this.
The first, if the file actually exists, use an absolute path for it:
$timePath = '/path/to/your/files/time.txt';
The second, verify if the file exists (in code) and if not, create it. fopen() with r/r+ flags does not create a file if it doesn't exist, so this needs to be manual. Try changing the code to the following:
if (!file_exists($timePath)) touch($timePath);
$timeFile = fopen($timePath, 'r+');
if ($timeFile) {
// the rest of your code
}
You say you're receiving an error when opening the ipsafe.txt file: "Permission denied". This is a fun one; Though the line numbers don't match with your sample code, I believe these are the lines in question:
//clear ipSafe
$ipFile1 = fopen($ipPath, "w+");
fclose($ipFile);
The first issue is that you never use $ipFile1 in your code (except erroneously locking/unlocking it; see below for that info). The second is that you call fclose($ipFile);. There is no handle named $ipFile! You can remove these lines of code!
You attempt to rewind the file pointer in your time.txt file to write the new time, however, you're using the value you read-in from the file $timeData - not the file itself $timeFile. Update the lines to the following:
rewind($timeFile);
fwrite($timeFile, $resetTime);
You read in the file ipsafe.txt, again, to look through the full list of IP addresses - if the current user's IP isn't in it, you add it. This may be a logic issue, but you're only checking the IP addresses of the last line of the file - but you still iterate through every line of the item. Per the comment in the code: //if IP was not saved in last timeframe, I'm going to assume that this is what you're intending to do - but just in case - I would recommend moving the full if-statement block that appears below the loop to inside the loop.
Here's the fun block:
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile1,2);
fwrite($ipFile2, "$ip".",");
#flock($ipFile1,3);
fclose($ipFile2);
You open the file into $ipFile2 (2), then you lock a completely different file $ipFile1 (1), write to 2, unlock 1, and then close 2. I think that you're intentions were to lock/unlock 2, which would make perfect sense here, so I'd recommend changing $ipFile1 to $ipFile2 in those lines.
That should about do it (potentially). After everything, the code should resemble something similar to:
<?php
$timeReset = 86400; //24h
$ipPath = "ipsafe.txt";
$countPath = "counts.txt";
$timePath = "time.txt";
$ip = $_SERVER['REMOTE_ADDR'];
$time = time();
$resetTime = $time+$timeReset;
if (!file_exists($timePath)) touch($timePath);
$timeFile = fopen($timePath, "r+");
if ($timeFile) {
$timeData = fread($timeFile, filesize($timePath));
//if resetTime is passed -> Reset
if($time >= $timeData) {
//set new resetTime
rewind($timeFile);
fwrite ($timeFile, $resetTime);
}
fclose($timeFile);
//creat IP-List
$ipData = file($ipPath);
while (list ($line_num, $line) = each ($ipData)) {
$digits = explode(",",$line);
}
//if IP was not saved in last timeframe
if(!in_array($ip, $digits)) {
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile2, LOCK_EX);
fwrite($ipFile2, "$ip".",");
#flock($ipFile2, LOCK_UN);
fclose($ipFile2);
//increment counter
if (!file_exists($countPath)) touch($countPath);
$countFile = fopen($countPath,"r+");
if ($countFile) {
$countData = fread($countFile, filesize($countPath);
if (empty($countData)) $countData = 0; // initialize the counter to '0' if it hasn't been set
rewind($countFile);
$countData++;
fwrite($countFile, $countData);
fclose($countFile);
}
}
}
?>
Now, I also have one additional logic issue regarding your $resetTime variable. Ideally, incrementing by a single day is to add 24 hours to a given timestamp (24 hours = 86400 seconds). However, daylight savings time will break this. Instead of doing the following:
$resetTime = time() + 86400;
Try going a different route (if you're worried about this, that is):
$resetTime = strtotime('+1 day');

Removing Lines in php ? Is this possible?

I have been struggling to create a Simple ( really simple ) chat system for my website as my knowledge on Javascripting/AJAX are Limited after gather resources and help from many kind people I was able to create my simple chat system but left with one problem.
The messages are posted to a file called "msg.html" in this format :
<p><span id="name">$name</span><span id="Msg">$message</span></p>
And then using PHP and AJAX I will retrieve the messages instantly from the file using the
file(); function and a foreach(){} loop withing PHP here is the code :
<?php
$file = 'msg.html';
$data = file($file);
$max_lines = 20;
if(count($data) > $max_lines){
// here i want the data to be deleted from oldest until i only have 20 messages left.
}
foreach($data as $line_num => $line){
echo $line_num . " . " . $line;
}
?>
My Question is how can i delete the oldest messages so that i am only left with the latest 20 Messages ?
How does something like this seem to you:
$file = 'msg.html';
$data = file($file);
$max_lines = 20;
foreach($data as $line_num => $line)
{
if ($line_num < $max_lines)
{
echo $line_num . " . " . $line;
}
else
{
unset($data[$line_num]);
}
}
file_put_contents('msg.html', $data);
?>
http://www.php.net/manual/en/function.file-put-contents.php for more info :)
I suppose you can read the file, explode it into an array, chop off everything but last 20 fields and write it back to file, overwriting the old one... Perhaps not the best solution but one that comes to mind if you really cant use database as Delan suggested
That's called round-robin if I recall correctly.
As far as I know, you can't remove arbitrary portions of a file. You need to overwrite the file with the new contents (or create a new file and remove the old one). You could also store messages in individual files but of course that implies up to $max_lines files to read.
You should also use flock() to avoid data corruption. Depending on the platform it's not 100% reliable but it's better than nothing.

Split big files using PHP

I want to split huge files (to be specific, tar.gz files) in multiple part from php code. Main reason to do this is, php's 2gb limit on 32bit system.
SO I want to split big files in multiple part and process each part seperately.
Is this possible? If yes, how?
My comment was voted up twice, so maybe my guess was onto something :P
If on a unix environment, try this...
exec('split -d -b 2048m file.tar.gz pieces');
split
Your pieces should be pieces1, pieces2, etc.
You could get the number of resulting pieces easily by using stat() in PHP to get the file size and then do the simple math (int) ($stat['size'] / 2048*1024*1024) (I think).
A simple method (if using Linux based server) is to use the exec command and to run the split command:
exec('split Large.tar.gz -b 4096k SmallParts'); // 4MB parts
/* | | | | |
| | |______| |
App | | |_____________
The source file | |
The split size Out Filename
*/
See here for more details: http://www.computerhope.com/unix/usplit.htm
Or you can use: http://www.computerhope.com/unix/ucsplit.htm
exec('csplit -k -s -f part_ -n 3 LargeFile.tar.gz');
PHP runs within a single thread and the only way to increase this thread count is to create child process using the fork commands.
This is not resource friendly. What I would suggest is to look into a language that can do this fast and effectively. I would suggest using node.js.
Just install node on the server and then create a small script, called node_split for instance, that can do the job on its own for you.
But I do strongly advise that you do not use PHP for this job but use exec to allow the host operating system to do this.
HJSPLIT
http://www.hjsplit.org/php/
PHP itself might not be able to...
If you can figure out how to do this from your computers' command line,
You should be able to then execute these commands using exec();
function split_file($source, $targetpath='/split/', $lines=1000){
$i=0;
$j=1;
$date = date("m-d-y");
$buffer='';
$handle = fopen ($_SERVER['DOCUMENT_ROOT'].$source, "r");
while (!feof ($handle)) {
$buffer .= fgets($handle, 4096);
$i++;
if ($i >= $lines) {
$fname = $_SERVER['DOCUMENT_ROOT'].$targetpath."part_".$date.$j.".txt";
$fhandle = fopen($fname, "w") or die($php_errormsg);
if (!$fhandle) {
echo "Cannot open file ($fname)";
//exit;
}
if (!fwrite($fhandle, $buffer)) {
echo "Cannot write to file ($fname)";
//exit;
}
fclose($fhandle);
$j++;
$buffer='';
$i=0;
$line+=10; // add 10 to $lines after each iteration. Modify this line as required
}
}
fclose ($handle);
}
$handle = fopen('source/file/path','r');
$f = 1; //new file number
while(!feof($handle))
{
$newfile = fopen('newfile/path/'.$f.'.txt','w'); //create new file to write to with file number
for($i = 1; $i <= 5000; $i++) //for 5000 lines
{
$import = fgets($handle);
//print_r($import);
fwrite($newfile,$import);
if(feof($handle))
{break;} //If file ends, break loop
}
fclose($newfile);
$f++; //Increment newfile number
}
fclose($handle);
If you want to split files which are
already on server, you can do it
(simply use the file functions fread,
fopen, fwrite, fseek to read/write
part of the file).
If you want to
split files which are uploaded from
the client, I am afraid you cannot.
This would probably be possible in php, but php was built for web development and trying to this whole operation in one request will result in the request timing out.
You could however use another language like java or c# and build a background process that you can notify from php to perform the operation. Or even run from php, depending on your Security settings on the host.
Splits are named as filename.part0 filename.part1 ...
<?php
function fsplit($file,$buffer=1024){
//open file to read
$file_handle = fopen($file,'r');
//get file size
$file_size = filesize($file);
//no of parts to split
$parts = $file_size / $buffer;
//store all the file names
$file_parts = array();
//path to write the final files
$store_path = "splits/";
//name of input file
$file_name = basename($file);
for($i=0;$i<$parts;$i++){
//read buffer sized amount from file
$file_part = fread($file_handle, $buffer);
//the filename of the part
$file_part_path = $store_path.$file_name.".part$i";
//open the new file [create it] to write
$file_new = fopen($file_part_path,'w+');
//write the part of file
fwrite($file_new, $file_part);
//add the name of the file to part list [optional]
array_push($file_parts, $file_part_path);
//close the part file handle
fclose($file_new);
}
//close the main file handle
fclose($file_handle);
return $file_parts;
}
?>

Categories