I would like to use exclusive/shared, both blocking and non-blocking locks like these found in flock(). How can this be achieved using semaphores?
It depends on the use case.
If you are on only one server, use a lock file.
function doSomething() {
$file = /temp/path/somthing.lock;
if (file_exists($file)) {
return false;
}
touch($file);
// Safely mess with things
unlink($file);
}
If you have multiple web servers, e.g. behind a load balancer, the same thing can be accomplished by using a table in mysql.
function doSomething() {
$query = "SELECT * FROM locks WHERE name='something'");
$res = mysqli_query($query);
if (mysql_num_rows($res) > 0) {
return false;
}
$query = "INSERT INTO locks (name) VALUES ('something')";
mysqli_query($query);
// Safely mess with things
$query = "DELETE FROM locks WHERE name='something'");
mysqli_query($query);
}
Memcache is another obvious candidate with multiple machine support.
You should not use ACP, because it is intended for caching only. This means you do not have control when ACP storage gets deleted, it could happen at any time.
You can also use semaphores, however the same caveats as with lock files apply if you use more than one server.
I would recommend creating lock($key), is_locked($key) and release($key) functions, and then religiously use them throughout your project. That way you can start with lock files (quick to implement), but then upgrade to something better later without editing the rest of your code. If you want to get really fancy, you can implement them as methods of a logger object that you put in a known location of your code.
Related
I would like to implement a quick and efficient serialization mechanism between PHP requests for virtual named resources that would unlock when the script is finished, either normally or due to error. I had eaccelerator_lock() and its corresponding eaccelerator_unlock() in the past, but eaccelerator doesn't implement that function anymore. What I want to do is something like:
lock_function("my-named-resource");
..
my_might_abort_abruptly_function();
..
unlock_function("my-named-resource");
Other PHP scripts calling lock_function() with the exact same parameter should block until this script calls unlock_function() or aborts. The resource name is unknown before the processing (it's a generated string) and can't be constrained to a small set (i.e., the locking mechanism should have good granularity). I would like to avoid try/catch code, because there are circunstances in which catch is not called. Also, any mechanism depending on manual usleep() spinning (instead of native OS blocking) should be avoided.
Mine is the only running application in the server. The system is a CentOS 6 Linux with PHP 5.3.3, Apache 2.2.15 and I have full control over it.
I explored the following alternatives:
semaphores: they are not well implemented in PHP; Linux allows arrays of thousands, while PHP only allocates one per id.
flock(): my resources are virtual, and flock() would only lock whole/real/existing files; I'd need to pre-create thousands of files and choose one to lock with a hash function. The granularity would depend on the number of files.
dio_fcntl(): I could attempt to reproduce the idea of flock() with a single file and fcntl(F_SETLK). This would have the advantage of a good granularity without the need of many files; the file could even be zero bytes long! (F_SETLK can lock beyond the end of the file). Alas! The problem is that nowhere in the documentation says that dio_fcntl() will release resources when the script terminates.
database lock: I could implement some key locking in a database with good key locking granularity, althought this is too database dependent. It would not be so quick either.
implement my own PHP extension: I'd really like to avoid that path.
The thing is, I think someone somewhere should have thought of this before me. What would be a good choice? Is there another solution I'm not seeing?
Thanks in advance. Guillermo.
You can always go old school and touch a file when your script starts and remove it when complete.
You could register_shutdown_function to remove the file.
The existence or absence of the file would indicate the locked state of the resource.
It turns out dio_open() does release the resources upon script termination. So I ended writing up the following functions:
$lockfile = $writable_dir."/serialized.lock";
function serialize_access($name)
{
$f = serialize_openfile();
if( !$f ) return false;
$h = serialize_gethash($name);
return dio_fcntl($f, F_SETLKW, array("whence"=>SEEK_SET,"start"=>$h, "length"=>1, "type"=>F_WRLCK)) >= 0;
}
function serialize_release($name)
{
$f = serialize_openfile();
if( !$f ) return false;
$h = serialize_gethash($name);
#dio_fcntl($f, F_SETLK, array("whence"=>SEEK_SET,"start"=>$h, "length"=>1, "type"=>F_UNLCK));
}
function serialize_gethash($name)
{
// Very good granularity (2^31)
return crc32($name) & 0x7fffffff;
}
function serialize_openfile()
{
global $lockfile, $serialize_file;
if( !isset($serialize_file) )
{
$serialize_file = false;
if( extension_loaded("dio") )
{
$serialize_file = #dio_open($lockfile,O_RDWR);
if( $serialize_file )
{
// Do not attempt to create the file with dio_open()
// because the file permissions get all mangled.
$prev = umask(0);
$temp = fopen($lockfile,"a");
if( $temp )
{
$serialize_file = #dio_open($lockfile,O_RDWR);
fclose($temp);
}
umask($prev);
}
}
}
return $serialize_file;
}
It seems to work very well.
implement my own PHP extension
You might want to check ninja-mutex library which does exactly what you want
We would like to implement a method that checks mysql load or total sessions on server and
if this number is bigger than a value then the next visitor of the website is redirected to a static webpage with a message Too many users try later.
One way I implemented it in my website is to handle the error message MySQL outputs when it denies a connection.
Sample PHP code:
function updateDefaultMessage($userid, $default_message, $dttimestamp, $db) {
$queryClients = "UPDATE users SET user_default_message = '$default_message', user_dtmodified = '$dttimestamp' WHERE user_id = $userid";
$resultClients = mysql_query($queryClients, $db);
if (!$resultClients) {
log_error("[MySQL] code:" . mysql_errno($db) . " | msg:" . mysql_error($db) . " | query:" . $queryClients , E_USER_WARNING);
$result = false;
} else {
$result = true;
}
}
In the JS:
function UpdateExistingMsg(JSONstring)
{
var MessageParam = "msgjsonupd=" + JSON.encode(JSONstring);
var myRequest = new Request({url:urlUpdateCodes, onSuccess: function(result) {if (!result) window.open(foo);} , onFailure: function(result) {bar}}).post(MessageParam);
}
I hope the above code makes sense. Good luck!
Here are some alternatives to user-lock-out that I have used in the past to decrease load:
APC Cache
PHP APC cache (speeds up access to your scripts via in memory caching of the scripts): http://www.google.com/search?gcx=c&ix=c2&sourceid=chrome&ie=UTF-8&q=php+apc+cache
I don't think that'll solve "too many mysql connections" for you, but it should really really help your website's speed in general, and that'll help mysql threads open and close more quickly, freeing resources. It's a pretty easy install on a debian system, and hopefully anything with package management (perhaps harder if you're using a if you're using a shared server).
Cache the results of common mysql queries, even if only within the same script execution. If you know that you're calling for certain data in multiple places (e.g. client_info() is one that I do a lot), cache it via a static caching variable and the info parameter (e.g.
static $client_info;
static $client_id;
if($incoming_client_id == $client_id){
return $client_info;
} else {
// do stuff to get new client info
}
You also talk about having too many sessions. It's hard to tell whether you're referring to $_SESSION sessions, or just browsing users, but too many $_SESSION sessions may be an indication that you need to move away from use of $_SESSION as a storage device, and too many browsing users, again, implies that you may want to selectively send caching headers for high use pages. For example, almost all of my php scripts return the default caching, which is no cache, except my homepage, which displays headers to allow browsers to cache for a short 1 hour period, to reduce overall load.
Overall, I would definitely look into your caching procedures in general in addition to setting a hard limit on usage that should ideally never be hit.
This should not be done in PHP. You should do it naturally by means of existing hard limits.
For example, if you configure Apache to a known maximal number of clients (MaxClients), once it reaches the limit it would reply with error code 503, which, in turn, you can catch on your nginx frontend and show a static webpage:
proxy_intercept_errors on;
error_page 503 /503.html;
location = /503.html {
root /var/www;
}
This isn't hard to do as it may sound.
PHP isn't the right tool for the job here because once you really hit the hard limit, you will be doomed.
The seemingly simplest answer would be to count the number of session files in ini_get("session.save_path"), but that is a security problem to have access to that directory from the web app.
The second method is to have a database that atomically counts the number of open sessions. For small numbers of sessions where performance really isn't an issue, but you want to be especially accurate to the # of open sessions, this will be a good choice.
The third option that I recommend would be to set up a chron job that counts the number of files in the ini_get('session.save_path') directory, then prints that number to a file in some public area on the filesystem (only if it has changed), visible to the web app. This job can be configured to run as frequently as you'd like -- say once per second if you want better resolution. Your bootstrap loader will open this file for reading, check the number, and give the static page if it is above X.
Of course, this third method won't create a hard limit. But if you're just looking for a general threshold, this seems like a good option.
I used the following code, but it is taking time. i want to cache without storing in a text file.
$file = 'cache_toppers.txt';
if (file_exists($file) &&
filemtime($file) > (time() - $expire)) {
$records = unserialize(file_get_contents($file));
} else {
include("kalvidbconnect.php");
$query = "SELECT * FROM vpfmsttoppers";
$result = mysql_query($query)
or die (mysql_error());
while ($record = mysql_fetch_array($result) ) {
$records[] = $record;
}
$OUTPUT = serialize($records);
$fp = fopen($file,"w");
fputs($fp, $OUTPUT);
fclose($fp);
}
Thanks,
Kamatchi.D
There are some ready to use PHP extensions providing cache functionality. Some of them:
memcache http://pecl.php.net/package/memcache
APC http://php.net/manual/en/book.apc.php
eAccelerator
XCache
these are the ones I know of, but surely there are many more.
Just a thought, not sure, but how about using CouchDB!?
Here is a good tutorial on IBM http://www.ibm.com/developerworks/opensource/library/os-php-couchdb/index.html?ca=drs-
If you don't want to use file-based caching, then one option is to build a wrapper and store it in shared memory,
http://se.php.net/manual/en/ref.sem.php
Maybe APC utilizes the same technique, I don't know, but if you don't want to install PECL-extensions then building your own cache-handling might be an option.
I would however consider caching rendered content to file, since that would put the least amount of load on the server.
Depending on a very long list of factors, I'd typically expect trying to unserialize the file to take longer than loading it fresh from the database.
Well, use cache then, for example APC - apc_store()/ apc_fetch()
Read some texts about locking in PHP.
They all, mainly, direct to http://php.net/manual/en/function.flock.php .
This page talks about opening a file on the hard-disk!!
Is it really so? I mean, this makes locking really expensive - it means each time I want to lock I'll have to access the hard-disk )=
Can anymore comfort me with a delightful news?
Edit:
Due to some replies I've got here, I want to ask this;
My script will run only by one thread, or several? Because if it's by one then I obviously don't need a mutex. Is there a concise answer?
What exactly I'm trying to do
Asked by ircmaxell.
This is the story:
I have two ftp servers. I want to be able to show at my website how many online users are online.
So, I thought that these ftp servers will "POST" their stats to a certain PHP script page. Let's assume that the URL of this page is "http://mydomain.com/update.php".
On the website's main page ("http://mydomain.com/index.php") I will display the cumulative statistics (online users).
That's it.
My problem is that I'm not sure if, when one ftp server updates his stats while another does it too, the info will get mixed.
Like when multi-threading; Two threads increase some "int" variable at the same time. It will not happen as expected unless you sync between them.
So, will I have a problem? Yes, no, maybe?
Possible solution
Thinking hard about it all day long, I have an idea here and I want you to give your opinion.
As said these ftp servers will post their stats, once every 60sec.
I'm thinking about having this file "stats.php".
It will be included at the updating script that the ftp servers go to ("update.php") and at the "index.php" page where visitors see how many users are online.
Now, when an ftp server updates, the script at "update.php" will modify "stats.php" with the new cumulative statistics.
First it will read the stats included at "stats.php", then accumulate, and then rewrite that file.
If I'm not mistaken PHP will detect that the file ("stats.php") is changed and load the new one. Correct?
Well, most of PHP runs in a different process space (there are few threading implementations). The easy one is flock. It's guaranteed to work on all platforms.
However, if you compile in support, you can use a few other things such as the Semaphore extension. (Compile PHP with --enable-sysvsem). Then, you can do something like (note, sem_acquire() should block. But if it can't for some reason, it will return false):
$sem = sem_get(1234, 1);
if (sem_acquire($sem)) {
//successful lock, go ahead
sem_release($sem);
} else {
//Something went wrong...
}
The other options that you have, are MySQL user level locks GET_LOCK('name', 'timeout'), or creating your own using something like APC or XCache (Note, this wouldn't be a true lock, since race conditions could be created where someone else gets a lock between your check and acceptance of the lock).
Edit: To answer your edited question:
It all depends on your server configuration. PHP May be run multi-threaded (where each request is served by a different thread), or it may be run multi-process (where each request is served by a different process). It all depends on your server configuration...
It's VERY rare that PHP will serve all requests serially, with only one process (and one thread) serving all requests. If you're using CGI, then it's multi-process by default. If you're using FastCGI, it's likely multi-process and multi-thread. If you're using mod_php with Apache, then it depends on the worker type:
mpm_worker will be both multi-process and multi-thread, with the number of processes dictated by the ServerLimit variable.
prefork will be multi-process
perchild will be multi-process as well
Edit: To answer your second edited question:
It's quite easy. Store it in a file:
function readStatus() {
$f = fopen('/path/to/myfile', 'r');
if (!$f) return false;
if (flock($f, LOCK_SH)) {
$ret = fread($f, 8192);
flock($f, LOCK_UN);
fclose($f);
return $ret;
}
fclose($f);
return false;
}
function updateStatus($new) {
$f = fopen('/path/to/myfile', 'w');
if (!$f) return false;
if (flock($f, LOCK_EX)) {
ftruncate($f, 0);
fwrite($f, $new);
flock($f, LOCK_UN);
fclose($f);
return true;
}
fclose($f);
return false;
}
function incrementStatus() {
$f = fopen('/path/to/myfile', 'rw');
if (!$f) return false;
if (flock($f, LOCK_EX)) {
$current = fread($f, 8192);
$current++;
ftruncate($f, 0);
fwrite($f, $current);
flock($f, LOCK_UN);
fclose($f);
return true;
}
fclose($f);
return false;
}
The question is: Where will you store the stats that the FTP servers are pushing with POST to your update.php file? If it's a local file, than ircmaxell in the second post has answered you. You can do this with a mutex as well - the semaphore functions. Another solution is to use MySQL MyISAM table to store the stats and use something like update info_table set value = value + 1. It should lock the table, and serialize your requests, and you will have no problems.
I recently created my own simple implementation of a mutex-like mechanism using the flock function of PHP. Of course the code below can be improved, but it is working for most use cases.
function mutex_lock($id, $wait=10)
{
$resource = fopen(storage_path("app/".$id.".lck"),"w");
$lock = false;
for($i = 0; $i < $wait && !($lock = flock($resource,LOCK_EX|LOCK_NB)); $i++)
{
sleep(1);
}
if(!$lock)
{
trigger_error("Not able to create a lock in $wait seconds");
}
return $resource;
}
function mutex_unlock($id, $resource)
{
$result = flock($resource,LOCK_UN);
fclose($resource);
#unlink(storage_path("app/".$id.".lck"));
return $result;
}
Yes that's true, as PHP is run by Apache, and Apache can organize the threads of execution as it deems the best (see the various worker model). So if you want to access a resource one at a time, you either lock to a file (which is good if you are dealing with cron jobs for example), or you rely on database transaction mechanism, ACID features, and database resources locking, if you are dealing with data.
PHP doesn't support multithreading, every request (and therefore every PHP script) will be executed in only one thread (or even process, depending on the way you run PHP).
I wanted to know if there is way to log the mysql queries in CakePHP being executed when we use the find method on the models, I know that rails database queries, so does Cake do the same, if so how can I enable it or use it?
Shiv
This page has instructions on how to get Cake to log queries the same way as rails.
A Very simple method to log all the queries being executed:
in your cake\libs\model\datasources\dbo\dbo_mysql.php
find the _execute function:
function _execute($sql) {
return mysql_query($sql, $this->connection);
}
add the line "$this->log($sql);" before "return mysql_query($sql, $this->connection);"
function _execute($sql) {
$this->log($sql);
return mysql_query($sql, $this->connection);
}
That's it!!!!! All your SQL Queries gets logged. Make sure the log file is configured properly and have sufficient permission. Enjoy
Assuming you are on a nix os, the best approach would actually to tail the mysql log itself.
You might learn some interesting things out of it.
log in Ubuntu when installing from repository
tail -f /var/log/mysql/mysql.log
As mentioned below, this is a huge performance killer (well, all logs have some performance impact). So, make sure you use it only on your dev/QA machines and only for short periods on your production machine.
CakePHP 1.3 uses the sql_dump element for that.
You can use the element directly when Configure::read('debug') is set to 2:
echo $this->element('sql_dump');
Or take it's code directly if you need to do something else with it (like echo it from a ShellTask)
$sources = ConnectionManager::sourceList();
$logs = array();
foreach ($sources as $source):
$db =& ConnectionManager::getDataSource($source);
if (!$db->isInterfaceSupported('getLog')):
continue;
endif;
$logs[$source] = $db->getLog();
endforeach;
Echo with e.g.:
print_r($logs)
This is what I use (put it in element folder then include in your layout)
<?php
ob_start();
echo $this->element('sql_dump');
$out = ob_get_contents();
ob_end_clean();
CakeLog::write('mysql' , $out);
?>
then you will find the mysql.log file in TMP.logs.DS.mysql.log