In some languages C# or .NET this would be a static variable, but in PHP the memory is cleared after each request. I want the value to persist across all requests. I don't wan't $_SESSION because that is different for each user.
To help explain here is an example:
I want to have a script like this that will count up. No matter which user/browser opens the url.
<?php
function getServerVar($name){
...
}
function setServerVar($name,$val){
...
}
$count = getServerVar("count");
$count++;
setServerVar("count", $count);
echo $count;
I want the value stored in memory. It will not be something that needs to persist when apache restarts and the data is not that important that it needs to be thread safe.
UPDATE: I'm fine if it holds different values per server in a loadbalanced environment. Static variables in C# or Java will not be in sync either.
You would typically use a database to store the count.
However as an alternative you could do so using a file:
<?php
$file = 'count.txt';
if (!file_exists($file)) {
touch($file);
}
//Open the File Stream
$handle = fopen($file, "r+");
//Lock File, error if unable to lock
if(flock($handle, LOCK_EX)) {
$size = filesize($file);
$count = $size === 0 ? 0 : fread($handle, $size); //Get Current Hit Count
$count = $count + 1; //Increment Hit Count by 1
echo $count;
ftruncate($handle, 0); //Truncate the file to 0
rewind($handle); //Set write pointer to beginning of file
fwrite($handle, $count); //Write the new Hit Count
flock($handle, LOCK_UN); //Unlock File
} else {
echo "Could not Lock File!";
}
//Close Stream
fclose($handle);
In php your going to have to use an external store that all servers share. The most commonly used tool is memcached, but sql and redis both work fine for this use case.
The only way to do taht is, like bspates said, a tool that does not depend on any resource on your server. If you have various servers, you cannot rely on memory-based mechanisms on each machine.
You have to store this number outside the servers, because each server will store the value for its own file or memory.
File writting, like $_SESSION, will work if you have only one server to receive your requests. For more than one server, you need any type of database where all your servers will communicate with.
Related
I have a log file maintained by a PHP script. The PHP script is subject to parallel processing. I cannot get the flock() mechanism to work on the log file: in my case, flock() does not prevent the log file shared by PHP scripts running in parallel from being accessed at the same time and being sometimes overwritten.
I want to be able to read a file, do some processing, modify the data and write back without the same code running in parallel on the server doing the same at the same time. The read modify write has to be in sequence.
On one of my shared hostings (OVH France), it does not work as expected. In that case, we see that the counter $c has the same value in different iframes, which should not be possible if the lock works as expected, which it does on an other shared hosting.
Any suggestions to make this work, or for an alternative method?
Googling "read modify write" php or fetch and add or test and set did not provide useful information: all solutions are based on a working flock().
Here is some standalone running demo code to illustrate. It generates a number of parallel requests from the browser to the server and displays the results. It is easy to visually observe a disfunction: if your webserver does not support flock() like one of mine, the counter value and the number of log lines will be the same in some frames.
<!DOCTYPE html>
<html lang="en">
<title>File lock test</title>
<style>
iframe {
width: 10em;
height: 300px;
}
</style>
<?php
$timeStart = microtime(true);
if ($_GET) { // iframe
// GET
$time = $_GET['time'] ?? 'no time';
$instance = $_GET['instance'] ?? 'no instance';
// open file
// $mode = 'w+'; // no read
// $mode = 'r+'; // does not create file, we have to lock file creation also
$mode = 'c+'; // read, write, create
$fhandle = fopen(__FILE__ .'.rwtestfile.txt', $mode) or exit('fopen');
// lock
flock($fhandle, LOCK_EX) or exit('flock');
// start of file (optional, only some modes like require it)
rewind($fhandle);
// read file (or default initial value if new file)
$fcontent = fread($fhandle, 10000) or ' 0';
// counter value from previous write is last integer value of file
$c = strrchr($fcontent, ' ') + 1;
// new line for file
$fcontent .= "<br />\n$time $instance $c";
// reset once in a while
if ($c > 20) {
$fcontent = ' 0'; // avoid long content
}
// simulate other activity
usleep(rand(1000, 2000));
// start of file
rewind($fhandle);
// write
fwrite($fhandle, $fcontent) or exit('fwrite');
// truncate (in unexpected case file is shorter now)
ftruncate($fhandle, ftell($fhandle)) or exit('ftruncate');
// close
fclose($fhandle) or exit('fclose');
// echo
echo "instance:$instance c:$c<br />";
echo $timeStart ."<br />";
echo microtime(true) - $timeStart ."<br />";
echo $fcontent ."<br />";
} else {
echo 'File lock test<br />';
// iframes that will be requested in parallel, to check flock
for ($i = 0; $i < 14; $i++) {
echo '<iframe src="?instance='. $i .'&time='. date('H:i:s') .'"></iframe>'."\n";
}
}
There is a warning about flock() limitations in the PHP: flock - Manual, but it is about ISAPI (Windows) and FAT (Windows). My server configuration is:
PHP Version 7.2.5
System: Linux cluster026.gra.hosting.ovh.net
Server API: CGI/FastCGI
A way to do an atomic test and set instruction in PHP is to use mkdir(). It is a bit strange to use a directory for that instead of a file, but mkdir() will create a directory or return a false (and a suppressile warning) if it already exists. File commands like fopen(), fwrite(), file_put_contents() do not test and set in one instruction.
<?php
// lock
$fnLock = __FILE__ .'.lock'; // lock directory filename
$lockLooping = 0; // counter can be used for tuning depending on lock duration
do {
if (#mkdir($fnLock, 0777)) { // mkdir is a test and set command
$lockLooping = 0;
} else {
$lockLooping += 1;
$lockAge = time() - filemtime($fnLock);
if ($lockAge > 10) {
rmdir($fnLock); // robustness, in case a lock was not erased
} else {
// wait without consuming CPU before try again
usleep(rand(2500, 25000)); // random to avoid parallel process conflict again
}
}
} while ($lockLooping > 0);
// do stuff under atomic protection
// don't take too long, because parallel processes are waiting for the unlock (rmdir)
$content = file_get_contents($protected_file_name); // example read
$content = $modified_content; // example modify
file_put_contents($protected_file_name, $modified_content); // example write
// unlock
rmdir($fnLock);
Using files for data management coordinated only by PHP request handlers you are heading for a world of pain - you've only just dipped your toes in the water thus far.
Using LOCK_EX, your writer needs to wait for any (and every) instance of LOCK_SH to be released before it will acquire the lock. Here you are setting flock to block until the lock can be acquired. On a relatively busy system, the writer could be blocked indefinitely. There is no priority queuing of locks on most OS that would place any subsequent reader requesting the lock behind a process waiting for a write lock.
A further complication is that you can only use flock on an open file handle. Meaning that a opening the file and acquiring the lock is not atomic, further you need to flush the stat cache in order to determine the age of the file after acquiring the lock.
Any writes to the file (even using file_put_contents()) are not atomic. So in the absence of exclusive locking you can't be sure that nobody will read a partial file.
In the absence of additional components (e.g. a daemon providing a lock queuing mechanism, or a caching reverse proxy in front of the web server, or a relational database) then your only option is to assume that you cannot ensure exclusive access and use atomic operations to semaphore the file, something like:
$lock_age=time()-filectime(dirname(CACHE_FILE) . "/lock");
if (filemtime(CACHE_FILE)>time()-CACHE_TTL
&& $lock_age>MAX_LOCK_TIME) {
rmdir(dirname(CACHE_FILE) . "/lock");
mkdir(dirname(CACHE_FILE) . "/lock") || die "I give up";
}
$content=generate_content(); // might want to add specific timing checks around this
file_put_contents(CACHE_FILE, $content);
rmdir(dirname(CACHE_FILE) . "/lock");
} else if (is_dir(dirname(CACHE_FILE) . "/lock") {
$snooze=MAX_LOCK_TIME-$lock_age;
sleep($snooze);
$content=file_get_contents(CACHE_FILE);
} else {
$content=file_get_contents(CACHE_FILE);
}
(note that this is a really ugly hack)
There is one fopen() test and set mode: the x mode.
x Create and open for writing only; place the file pointer at the beginning of the file. If the file already exists, the fopen() call will fail by returning FALSE and generating an error of level E_WARNING. If the file does not exist, attempt to create it.
The fopen($filename ,'x') behaviour is the same as mkdir() and it can be used in the same way:
<?php
// lock
$fnLock = __FILE__ .'.lock'; // lock file filename
$lockLooping = 0; // counter can be used for tuning depending on lock duration
do {
if ($lockHandle = #fopen($fnLock, 'x')) { // test and set command
$lockLooping = 0;
} else {
$lockLooping += 1;
$lockAge = time() - filemtime($fnLock);
if ($lockAge > 10) {
rmdir($fnLock); // robustness, in case a lock was not erased
} else {
// wait without consuming CPU before try again
usleep(rand(2500, 25000)); // random to avoid parallel process conflict again
}
}
} while ($lockLooping > 0);
// do stuff under atomic protection
// don't take too long, because parallel processes are waiting for the unlock (rmdir)
$content = file_get_contents($protected_file_name); // example read
$content = $modified_content; // example modify
file_put_contents($protected_file_name, $modified_content); // example write
// unlock
fclose($lockHandle);
unlink($fnLock);
It is a good idea to test this, e.g. using the code in the question.
Many people rely on locking as documented, but surprises may appear during test or production under load (parallel requests from one browser may be enough).
I use the following code to copy/download files from an external server (any server via a URL) to my hosted web server(Dreamhost shared hosting at default settings).
<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body>
<form method="post" action="copy.php">
<input type="submit" value="click" name="submit">
</form>
</body>
</html>
<!-- copy.php file contents -->
<?php
function chunked_copy() {
# 1 meg at a time, adjustable.
$buffer_size = 1048576;
$ret = 0;
$fin = fopen("http://www.example.com/file.zip", "rb");
$fout = fopen("file.zip", "w");
while(!feof($fin)) {
$ret += fwrite($fout, fread($fin, $buffer_size));
}
fclose($fin);
fclose($fout);
return $ret; # return number of bytes written
}
if(isset($_POST['submit']))
{
chunked_copy();
}
?>
However the function stops running at about once 2.5GB (sometimes 2.3GB and sometimes 2.7GB, etc) of the file has downloaded. This happens every time I execute this function. Smaller files (<2GB) rarely exhibit this problem. I believe nothing is wrong with the source as I separately downloaded the file flawlessly onto my home PC.
Can someone please remedy and explain this problem to me? I am very new to programming.
Also,
file_put_contents("Tmpfile.zip", fopen("http://example.com/file.zip", 'r'));
exhibits similar symptoms as well.
I think the problem might be the 30 second time-out on many servers running PHP scripts.
PHP scripts running via cron or shell wont have that problem so perhaps you could find a way to do it that way.
Alternatively you could add set_time_limit([desired time]) to the start of your code.
Explain: perhaps. Remidy: probably not.
It may be caused by the limits of PHP: the manual on the filesize function mentions in the section on the return value:
Note: Because PHP's integer type is signed and many platforms use 32bit integers, some filesystem functions may return unexpected results for files which are larger than 2GB.
It seems that the fopen function may cause the issue, as two comments (1, 2) were added (although modded down) on the subject.
It appears as if you need to compile PHP from source (with the CFLAGS="-D_FILE_OFFSET_BITS=64" flag) to enable large files (>2GB), but it might break some other functionality.
Since you're using shared histing: I guess you're out of luck.
Sorry...
Since the problem occurs at an (as yet) unknown and undefined file-size, perhaps it is best to try a work-around. What if you just close and than re-open the output file after some number of bytes?
function chunked_copy() {
# 1 meg at a time, adjustable.
$buffer_size = 1048576;
# 1 GB write-chuncks
$write_chuncks = 1073741824;
$ret = 0;
$fin = fopen("http://www.example.com/file.zip", "rb");
$fout = fopen("file.zip", "w");
$bytes_written = 0;
while(!feof($fin)) {
$bytes = fwrite($fout, fread($fin, $buffer_size));
$ret += $bytes;
$bytes_written += $bytes;
if ($bytes_written >= $write_chunks) {
// (another) chunck of 1GB has been written, close and reopen the stream
fclose($fout);
$fout = fopen("file.zip", "a"); // "a" for "append"
$bytes_written = 0; // re-start counting
}
}
fclose($fin);
fclose($fout);
return $ret; # return number of bytes written
}
The re-opening should be with the append-mode, which will place the write-pointer (there is no read-pointer) at the end of the file, not overwriting bytes written earlier.
This will not solve any Operating System-level or File System-level issues, but it may solve any counting issue internal to PHP while writing to files.
Perhaps this trick can (or should) also be applied on the reading-end, but I'm not sure if you can perform seek-operations on a download...
Note that any integer overflows (beyond 2147483647 if you're on 32-bit) should be transparently solved by casting to float, so that should not be an issue.
Edit: count the actual number of bytes written, not the chunk size
Maybe you can try curl to download file.
function downloadUrlToFile($url, $outFileName)
{
//file_put_contents($xmlFileName, fopen($link, 'r'));
//copy($link, $xmlFileName); // download xml file
if(is_file($url)) {
copy($url, $outFileName); // download xml file
} else {
$options = array(
CURLOPT_FILE => fopen($outFileName, 'w'),
CURLOPT_TIMEOUT => 28800, // set this to 8 hours so we dont timeout on big files
CURLOPT_URL => $url
);
$ch = curl_init();
curl_setopt_array($ch, $options);
curl_exec($ch);
curl_close($ch);
}
}
You get a time-out after 30s, probably caused by PHP (with default max_execution_time = 30s). You could try setting it to a larger time:
ini_set('max_execution_time', '300');
However, there are some caveats:
If the script is running in safe mode, you cannot set max_execution_time with ini_set (I could not find whether Dreamhost has safe mode on or off in shared hosting, you need to ask them, or just try this).
The web server may have an execution limit as well. Apache has this default to 300s (IIS as well, but given that Dreamhost provides 'full unix shell', Apache is more likely then IIS). But with a file size of 5GB, this should help you out.
This is the best way I found for downloading very large files : fast and no need lot of memory.
public function download_large_file(string $url, string $dest)
{
ini_set('memory_limit', '3000M');
ini_set('max_execution_time', '0');
try {
$handle1 = fopen($url, 'r');
$handle2 = fopen($dest, 'w');
stream_copy_to_stream($handle1, $handle2);
fclose($handle1);
fclose($handle2);
return true;
}
catch(\Exception $e) {
return $e->getMessage();
}
return true;
}
I'm trying to export a lot of data trough a CSV export. The amount of data it's really big, around 100.000 records and counting.
My client usually uses two tabs to browse and check several stuff at the same time. So a requirement is that while the export is being made, he can continues browsing the system.
The issue is that when the CSV is being generated on the server, the session is blocked, you cannot load another page until the generation is completed.
This is what I'm doing:
Open the file
Loop trough the amount of data(One query per cycle, each cycle queries 5000 records) pd: I cannot change this, because of certain limitations.
write the data into the file
free memory
close the file
set headers to begin download
During the entire process, it's not possible to navigate the site in another tab.
The block of code:
$temp = 1;
$first = true;
$fileName = 'csv_data_' . date("Y-m-d") . '-' . time() . '.csv';
$filePath = CSV_EXPORT_PATH . $fileName;
// create CSV file
$fp = fopen($filePath, 'a');
// get data
for ($i = 1; $i <= $temp; $i++) {
// get lines
$data = $oPB->getData(ROWS_PER_CYCLE, $i); // ROWS_PER_CYCLE = 5000
// if something is empty, exit
if (empty($data)) {
break;
}
// write the data that will be exported into a file
fwrite($fp, $export->arrayToCsv($data, '', '', $first));
// count element
$temp = ceil($data[0]->foundRows / ROWS_PER_CYCLE); // foundRows is always the same value, doesn't change per query.
$first = false; // hide header for next rows
// free memory
unset($lines);
}
// close file
fclose($fp);
/**
* Begin Download
*/
$export->csvDownload($filePath); // set headers
Some considerations:
The count is being made in the same query, but it's not entering into an infinite loop, works as expected. It's contained into $data[0]->foundRows, and avoids an unnecesary query to count all the available records.
There're several memory limitations due to environment settings, that I cannot change.
Does anyone know How can I improve this? Or any other solution.
Thanks for reading.
I'm replying only because it can be helpful to someone else. A colleague came up with a solution for this problem.
Call the function session_write_close() before
$temp = 1;
Doing this, you're ending the current session and storing the session data, so I'm being able to download the file a continue navigating in other tabs.
I hope it helps some one.
Some considerations about this solution:
You must no require to use session data after session_write_close()
The export script is in another file. For ex: home.php calls trough a link export.php
I'm having the following problem with my VPS server.
I have a long-running PHP script that sends big files to the browser. It does something like this:
<?php
header("Content-type: application/octet-stream");
readfile("really-big-file.zip");
exit();
?>
This basically reads the file from the server's file system and sends it to the browser. I can't just use direct links(and let Apache serve the file) because there is business logic in the application that needs to be applied.
The problem is that while such download is running, the site doesn't respond to other requests.
The problem you are experiencing is related to the fact that you are using sessions. When a script has a running session, it locks the session file to prevent concurrent writes which may corrupt the session data. This means that multiple requests from the same client - using the same session ID - will not be executed concurrently, they will be queued and can only execute one at a time.
Multiple users will not experience this issue, as they will use different session IDs. This does not mean that you don't have a problem, because you may conceivably want to access the site whilst a file is downloading, or set multiple files downloading at once.
The solution is actually very simple: call session_write_close() before you start to output the file. This will close the session file, release the lock and allow further concurrent requests to execute.
Your server setup is probably not the only place you should be checking.
Try doing a request from your browser as usual and then do another from some other client.
Either wget from the same machine or another browser on a different machine.
In what way doesn't the server respond to other requests? Is it "Waiting for example.com..." or does it give an error of any kind?
I do something similar, but I serve the file chunked, which gives the file system a break while the client accepts and downloads a chunk, which is better than offering up the entire thing at once, which is pretty demanding on the file system and the entire server.
EDIT: While not the answer to this question, asker asked about reading a file chunked. Here's the function that I use. Supply it the full path to the file.
function readfile_chunked($file_path, $retbytes = true)
{
$buffer = '';
$cnt = 0;
$chunksize = 1 * (1024 * 1024); // 1 = 1MB chunk size
$handle = fopen($file_path, 'rb');
if ($handle === false) {
return false;
}
while (!feof($handle)) {
$buffer = fread($handle, $chunksize);
echo $buffer;
ob_flush();
flush();
if ($retbytes) {
$cnt += strlen($buffer);
}
}
$status = fclose($handle);
if ($retbytes && $status) {
return $cnt; // return num. bytes delivered like readfile() does.
}
return $status;
}
I have tried different approaches (reading and sending the files in small chunks [see comments on readfile in PHP doc], using PEARs HTTP_Download) but I always ran into performance problems when the files are getting big.
There is an Apache mod X-Sendfile where you can do your business logic and then delegate the download to Apache. The download will not be publicly available. I think, this is the most elegant solution for the problem.
More Info:
http://tn123.org/mod_xsendfile/
http://www.brighterlamp.com/2010/10/send-files-faster-better-with-php-mod_xsendfile/
The same happens go to me and i'm not using sessions.
session.auto_start is set to 0
My example script only runs "sleep(5)", and adding "session_write_close()" at the beginning doesn't solve the problem.
Check your httpd.conf file. Maybe you have "KeepAlive On" and that is why your second request hangs until the first is completed. In general your PHP script should not allow the visitors to wait for long time. If you need to download something big, do it in a separate internal request that user have no direct control of. Until its done, return some "executing" status to the end user and when its done, process the actual results.
How can I make it so when a user clicks on a link on my web page, it writes to a .txt file named "Count.txt", which contains only a number and adds 1 to that number? Thank you.
If you forego any validity checking you could do it with something as simple as:
file_put_contents($theCounterFile, file_get_contents($theCounterFile)+1);
Addition:
There's talk about concurrency in this thread and it should be noted that it is a good idea to use a database and transactions to deal with concurrency, I'd highly recommend against writing a bunch of plumbing code to do this in a file.
If you've ever had, or think you might ever have two requests for the resource in the same second you should look into PDO with mysql, or PDO with SQLite instead of a file, use transactions (and InnoDB or better if you're going for mysql).
But really, even if you get two requests in the same microsecond (highly unlikely), chances of locking the file are slim as it will not be kept open and the two requests will probably not be handled parallel enough to lock anyway. Reality check: how many hits on the same resource do you get on average in the same minute?...
If you decide to do anything more advanced, like say two numbers, you may want to consider using SQLite. It's about as about as fast and as simple as opening and closing a file, but is much more flexible.
Open the file, lock the file (VERY important), read the number currently in there, add 1 to the number, write number back to file, release the lock and close the file.
ie. something like :
$fp = fopen("count.txt", "r+");
if (flock($fp, LOCK_EX)) { // do an exclusive lock
$num = fread($fp, 10);
$num++;
fseek($fp, 0);
fwrite($fp, $num);
flock($fp, LOCK_UN); // release the lock
} else {
// handle error
}
fclose($fp);
should work (not tested).
Generally this is quite easy:
$count = (int)file_get_contents('/path/to/Count.txt');
file_put_contents('/path/to/Count.txt', $count++, LOCK_EX);
But you'll run into concurrency problems using this code. One way to generate a lock safe from any race condition is:
$countFile = '/path/to/Count.txt';
$countTemp = tempnam(dirname($countFile), basename($countFile));
$countLock = $countFile . '.lock';
$f_lock = fopen($countLock, 'w');
if(flock($f_lock, LOCK_EX)) {
$currentCount = (int)file_get_contents($countFile);
$f_temp = fopen($countTemp, 'w');
if(flock($f_temp, LOCK_EX)) {
fwrite($f_temp, $currentCount++);
flock($f_temp, LOCK_UN);
fclose($f_temp);
if(!rename($countTemp, $countFile)) {
unlink($countTemp);
}
}
flock($f_lock, LOCK_UN);
fclose($f_lock);
}