I have a function that send HTTP request via CURL to www.server.com
My task is to make sure that www.server.com gets no more than one request every 2 seconds.
Possible solution:
Create a function checktime() that will store current call time in database and check with database on every next call and make system pause for 2 seconds:
$oldTime = $this->getTimeFromDatabase();
if ($oldTime < (time() - 2) ) { // if its been 2 seconds
$this->setNewTimeInDatabase();
return true;
} else {
sleep(2);
return false;
}
The problem/question:
Lets say, the last request to www.server.com was on 1361951000. Then 10 other users attempt to do request on 1361951001 (1 seconds later). checktime() Function will be called.
As since it only has been 1 second, the function will return false. All 10 users will wait 2 seconds. Does it means that on 1361951003 there are 10 requests will be sent simultaneously? And is it possible that the time of last request will not be changed in database, because of the missed call of $this->setNewTimeInDatabase() in checktime()?
Thank you!
UPDATE:
I have just been told that using a loop might solve the problem:
for($i=0;$i<300;$i++)
{
$oldTime = $this->getTimeFromDatabase();
if ($oldTime < (time() - 2) ) { // if its been 2 seconds
$this->setNewTimeInDatabase();
return true;
} else {
sleep(2);
return false;
}
}
But i don't really see logic in it.
I believe you need some implementation of a semaphore. The database could work, as long as you can guarantee that only one thread gets to write to the db and then make the request.
For example, you might use an update request to the db and then check for the updated rows (in order to check whether the update actually happened). If the update was succesful you can assume you got the mutex lock and then make the request (assuming the time is right to make it). Something like this:
$oldTime = $this->getTimeFromDatabase();
if ($oldTime < (time() - 2) && $this->getLock()) { // if its been 2 seconds
$this->setNewTimeInDatabase();
$this->releaseLock();
return true;
} else {
sleep(2);
return false;
}
function getLock()
{
return $mysqli->query('UPDATE locktable set locked = 1 WHERE locked = 0');
}
function releaseLock()
{
$mysqli->query('UPDATE locktable set locked = 0');
}
I'm not sure about the mysql functions, but I believe it's ok to get the general idea.
Watch out with using a database. For example MySQL is not always 100% in sync with its sessions, and for that reason it is not safe to rely on that for locking purposes.
You could use a file-lock through the method flock, where you would save the access time in. Then you could be sure to lock the file, so no two or more processes would ever access it at the same time.
It would probably go something like this:
$filepath = "lockfile_for_source";
touch($filepath);
$fp = fopen("lockfile_for_resource", "r") or die("Could not open file.");
while(true){
while(!flock($fp, LOCK_EX)){
sleep(0.25); //wait to get file-lock.
}
$time = file_get_contents($filepath);
$diff = time() - $time;
if ($diff >= 2){
break;
}else{
flock($fp, LOCK_UN);
}
}
//Following code would never be executed simultaneously by two scripts.
//You should access and use your resource here.
fwrite($fp, time());
fflush($fp);
flock($fp, LOCK_UN); //remove lock on file.
fclose($fp);
Please be aware that I have not tested the code.
Related
I am writing a PHP script which, upon a request, will make a call to a SOAP service with various parameters, some of which are taken from the request.
However, the particular SOAP service I am using requires that each request includes a unique ID, which in this case needs to increment for each request. It must not be based on time, and must be unique for each request, however it does not matter if values are skipped.
Using a MySQL data base to store a single value seems massively overkill. I have thought about storing and loading it into a file, but the issue of race conditions springs to mind.
I do have complete access to the server, which will be some kind of Linux flavour dedicated to this task.
Is there a simple way this can be achieved?
Before any new request get incremental value using PHP's time() function, since time will be unique for each request.
$increment_id = time();
If your application is single server you can try to store incremental ID in APC using:
$key = 'soap_service_name';
if (!apc_exists($key)) {
apc_store($key, 0);
}
$id = apc_inc($key);
You need to check if a key exists in APC cache and set 0, otherwise apc_inc fails and returns false
If you have multiserver application you can store incremental id in Memcache/Redis (that needs to run additional service):
$key = 'soap_service_name';
$memcache = memcache_connect('memcache_host', 11211);
if (!empty(memcache_exists($memcache, $key))) {
memcache_set($memcache, 0);
}
$id = memcache_increment($memcache, $key);
Same situation as APC if you call memcache_increment it will fail if key doesn't exists yet.
If that incremental ID should be stored persistently Redis would be more usefull because it has disk write of all data. It's kind of Memcache with disk write.
This is how I achieved this in the end. After considering the various options, databases and the various caching options seemed a bit overkill. In addition, caching, cookies and sessions seem to be designed to be relatively temporary, whereas I was really looking for a non-volatile solution.
This is what I came up with - a simple file locking solution. I hadn't realised PHP could deal with file locks but on discovering this, it seems the best way to go.
This example acquires an exclusive lock on the file, before reading and updating the value. If it hits max int, it resets. Then it waits for 5 seconds. If the script is called a few times in quick succession, observe that each request will wait for the lock to be release from the previous before continuing.
What's nice is, as this is PHP, non-existent file, invalid contents etc, will just cause the value to default to zero.
<?php
$f = fopen('sequence_num.txt', 'r+');
echo "Acquiring lock<br />\n";
flock($f, LOCK_EX);
echo "Lock acquired, updating value<br />\n";
$num = intval(fread($f, strlen(PHP_INT_MAX)));
echo "Old val = " . $num;
if ($num >= PHP_INT_MAX) {
$num = 0;
} else {
$num++;
}
echo " New val = " . $num;
echo "<br />Waiting 5 seconds<br />\n";
rewind($f);
ftruncate($f, 0);
fwrite($f, $num);
sleep(5);
echo "Releasing lock<br />\n";
flock($f, LOCK_UN);
fclose($f);
If you're happy to use a float as a unique value use:
$unique_id = microtime(true);
If you wish to simply increment, you may do so using a session var:
/**
* Get session increment.
*
* #param string $id
* #param int $default
* #return int
*/
function get_increment($id, $default = 0)
{
if (array_key_exists($id, $_SESSION)) $_SESSION[$id] += 1;
else $_SESSION[$id] = $default;
return $_SESSION[$id];
}
var_dump(get_increment('unique_id'));
I have a PHP API front end running on a webserver. This specific PHP program is subject to distribution, thus it should be as portable as possible.
The feature I want to implement is an IP cooldown period, meaning that the same IP can only request the API a maximum of two times per second, meaning at least a 500ms delay.
The approach I had in mind is storing the IP in an MySQL database, along with the latest request timestamp. I get the IP by:
if (getenv('REMOTE_ADDR'))
$ipaddress = getenv('REMOTE_ADDR');
But some servers might not have a MySQL database or the user installling this has no access. Another issue is the cleanup of the database.
Is there a more portable way of temporarily storing the IPs (keeping IPv6 in mind)?
and
How can I provide an automatic cleanup of IPs that are older than 500ms, with the least possible performance impact?
Also: I have no interest at looking at stored IPs, it is just about the delay.
This is how I solved it for now, using a file.
Procedure
Get client IP and hash it (to prevent file readout).
Open IP file and scan each line
Compare the time of the current record to the current time
If difference is greater than set timeout goto 5., else 7.
If IP matches client, create updated record, else
drop record.
If IP matches client, provide failure message, else copy record.
Example code
<?php
$sIPHash = md5($_SERVER[REMOTE_ADDR]);
$iSecDelay = 10;
$sPath = "bucket.cache";
$bReqAllow = false;
$iWait = -1;
$sContent = "";
if ($nFileHandle = fopen($sPath, "c+")) {
flock($nFileHandle, LOCK_EX);
$iCurLine = 0;
while (($sCurLine = fgets($nFileHandle, 4096)) !== FALSE) {
$iCurLine++;
$bIsIPRec = strpos($sCurLine, $sIPHash);
$iLastReq = strtok($sCurLine, '|');
// this record expired anyway:
if ( (time() - $iLastReq) > $iSecDelay ) {
// is it also our IP?
if ($bIsIPRec !== FALSE) {
$sContent .= time()."|".$sIPHash.PHP_EOL;
$bReqAllow = true;
}
} else {
if ($bIsIPRec !== FALSE) $iWait = ($iSecDelay-(time()-$iLastReq));
$sContent .= $sCurLine.PHP_EOL;
}
}
}
if ($iWait == -1 && $bReqAllow == false) {
// no record yet, create one
$sContent .= time()."|".$sIPHash.PHP_EOL;
echo "Request from new user successful!";
} elseif ($bReqAllow == true) {
echo "Request from old user successful!";
} else {
echo "Request failed! Wait " . $iWait . " seconds!";
}
ftruncate($nFileHandle, 0);
rewind($nFileHandle);
fwrite($nFileHandle, $sContent);
flock($nFileHandle, LOCK_UN);
fclose($nFileHandle);
?>
Remarks
New users
If the IP hash doesn't match any record, a new record is created. Attention: Access might fail if you do not have rights to do that.
Memory
If you expect much traffic, switch to a database solution like this all together.
Redundant code
"But minxomat", you might say, "now each client loops through the whole file!". Yes, indeed, and that is how I want it for my solution. This way, every client is responsible for the cleanup of the whole file. Even so, the performance impact is held low, because if every client is cleaning, file size will be kept at the absolute minimum. Change this, if this way doesn't work for you.
I have written a script that gives you ability to download the file with my maximum file speed that I allow, however when I allow 'unlimited' speed like 10000kB/s then the ftell works strange, it behaves like it downloads with 10000kBps speed, which is not true and I can not make calculations in database like time remaining, current download speed and so on...
So browser downloads file after some time, but in database it is already like 'downloaded', how could I make some precision calculations even I set the unlimited speed so user can download a file at the speed of the network and the database values are also counted by his network speed not by the ftell(); which depends on $download_rate; ...?
Thanks in advance!
<?php
while(!feof($fopen)) {
//echo fread($fopen, 4096);
$this->get_allowed_speed_limit($download_rate);
//$download_rate = 350;
print fread($fopen, round($download_rate * 1024));
sleep(1); //needed for download speed limit
if(connection_status() != 0 || connection_aborted()) {
$bytes_transferred = ftell($fopen);
if($bytes_transferred < $bytes) {
//CANCELLED
$this->download_unsuccessfull($file_name);
} else {
//CANCELLED (but gets executed only on strange networks like eduroam in CZE)
$this->download_unsuccessfull($file_name);}
flush();
die;
} else {
$progress = ftell($fopen) / $bytes * 100;
if($progress >= 100) {
//DONE
$this->download_successfull($file_name);
flush();
} else {
//DOWNLOADING
if(ftell($fopen) != 0) {
$bytes_transferred = ftell($fopen);
$time_end = microtime(true);
$time = $time_end - $time_start;
$dl_speed = floor(($bytes_transferred / $time) / 1000);
///////HERE THE CALCULATIONS ARE TOTALLY WRONG, BECAUSE IT ALL DEPENDS ON THE INPUT OF $download_rate;
mysqli_query($con, "UPDATE `download_meter` SET `current_speed` = '".mysqli_real_escape_string($con, $bytes_transferred)."'");
$this->update_active_downloads($file_name, $bytes_transferred, $dl_speed);
}
flush();
}
}
//Activate this for delay download.
//flush();
//sleep(1);
}
?>
Limiting download speed is up to your webserver. PHP is too high level. It knows nothing of the outgoing data.
Apache: https://stackoverflow.com/a/13355834/247372
Nginx: http://www.nginxtips.com/how-to-limit-nginx-download-speed/
The same goes for measuring: the webserver will know and might tell you somehow. Logs, unix socket, after-the-fact, I don't know. Those links will know.
How about (re)adding that sleep(1); thing to the WHILE loop? From what I can see the script outputs the file almost all at once (as fast as it can) and there's nothing that pauses it so it can actually limit the download speed.
That way you will know that each second you send just 64kbytes (or whatever) and even though you can't be sure that the user can in fact recieve this much data/second (whoa, so fast!), it could be a bit more precise than what you have in there now.
Or am I getting this wrong?
I have a problem with a counter. I need to count two variables, separated with a |, but sometimes the counter doesn't increase a variable's value.
numeri.txt (the counter):
5240|593389
This is the PHP script:
$filename="numeri.txt";
$fp=fopen($filename,"r");
if(!flock($fp,LOCK_SH))
{
while(true)
{
usleep(100000);
if(flock($fp,LOCK_SH))
{
break;
}
}
}
$contents=fread($fp,filesize($filename));
flock($fp,LOCK_UN);
fclose($fp);
$fp=fopen($filename,'a');
if(!flock($fp,LOCK_EX))
{
while(true)
{
usleep(100000);
if(flock($fp,LOCK_EX))
{
break;
}
}
}
ftruncate($fp,0);
$contents=explode("|",$contents);
$clicks=$contents[0];
$impressions=$contents[1]+1;
fwrite($fp,$clicks."|".$impressions);
flock($fp,LOCK_UN);
fclose($fp);
I set the counter to the right value but after 3-4 days the counter hasn't counted about 50 impressions (the number after "|")
How to fix the code?
Two problems:
1) There is still the opportunity to do a read, another process to write the file, then this process write the file. Millisecond timing required, but it's possible.
2) You don't actually verify the "open" works - it can fail.
The solution is to firstly check for open failure and retry, and secondly do one lock - exclusive. Paraphrased code:
while (!$fh = fopen($file, 'c+')) { // Read write, do not truncate, place pointer at start.
usleep(100000);
}
while (!flock(LOCK_EX)) {
usleep(100000);
}
$content = fread();
// Process content...
ftruncate();
fseek(0); // Or frewind();
fwrite();
fclose(); // Also releases the lock.
i'm using Smarty with my php code and i like to cache some of website pages so i used the following code :
// TOP of script
ob_start(); // start the output buffer
$cachefile ="cache/cachefile.html";
// normal PHP script
$smarty->display('somefile.tpl.html') ;
$fp = fopen($cachefile, 'w'); // open the cache file for writing
fwrite($fp, ob_get_contents()); // save the contents of output buffer to the file
fclose($fp); // close the file
ob_end_flush(); // Send the output to the browser
but when i print ob_get_contents() at end of the php file it's empty ! and actually the created cache file is also empty ! so how could i cache the files in php when i using smarty i know i can use smarty cache but it isn't work for me for some reason .
in addition please give me information about APC cache . how to use it? is it worth using in my case , i thin it's just for caching database queries , i read the php manual about it but i can't get anything :)
tanks .
I've mashed up some of the code from the documentation (located here) for a more complete example of the smarty cache. Also, I'm not sure what you were using in your example, but you should be using smarty's methods to manipulate the cache.
require('Smarty.class.php');
$smarty = new Smarty;
// 1 Means use the cache time defined in this file,
// 2 means use cache time defined in the cache itself
$smarty->caching = 2;
// set the cache_lifetime for index.tpl to 5 minutes
$smarty->cache_lifetime = 300;
// Check if a cache exists for this file, if one doesn't exist assign variables etc
if(!$smarty->is_cached('index.tpl')) {
$contents = get_database_contents();
$smarty->assign($contents);
}
// Display the output of index.tpl, will be from cache if one exists
$smarty->display('index.tpl');
// set the cache_lifetime for home.tpl to 1 hour
$smarty->cache_lifetime = 3600;
// Check if a cache exists for this file, if one doesn't exist assign variables etc
if(!$smarty->is_cached('home.tpl')) {
$contents = get_database_contents();
$smarty->assign($contents);
}
// Display the output of index.tpl, will be from cache if one exists
$smarty->display('home.tpl');
As for APC cache, it will work the same way that smarty does. They both store the data in a file for a specific amount of time. Every time you wish to access the data, it checks if the cache is valid, and if so returns the cache value.
However, if not using smarty you can use APC as such:
This example goes through storing the result of a DB query in the cache, similarly, you can modify this to instead store the entire page output so you don't have to run expensive PHP functions frequently.
// A class to make APC management easier
class CacheManager
{
public function get($key)
{
return apc_fetch($key);
}
public function store($key, $data, $ttl)
{
return apc_store($key, $data, $ttl);
}
public function delete($key)
{
return apc_delete($key);
}
}
Combined with some logic,
function getNews()
{
$query_string = 'SELECT * FROM news ORDER BY date_created DESC limit 5';
// see if this is cached first...
if($data = CacheManager::get(md5($query_string)))
{
// It was stored, return the value
$result = $data;
}
else
{
// It wasn't stored, so run the query
$result = mysql_query($query_string, $link);
$resultsArray = array();
while($line = mysql_fetch_object($result))
{
$resultsArray[] = $line;
}
// Save the result inside the cache for 3600 seconds
CacheManager::set(md5($query_string), $resultsArray, 3600);
}
// Continue on with more functions if necessary
}
This example is slightly modified from here.
Do you mean you are calling ob_get_contents() again after you called ob_end_flush() ? If so, the stuff you wrote to the file will have been "deleted" from PHP's memory.
If you wish to still output the HTML, save ob_end_flush to a variable first, then pass that to fwrite. You can use the variable later down the page.