We are using the Recess framework for a web service. As part of this we're using Recess' caching mechanism, provided as an SQLite database.
We've been using this caching mechanism happily for about a year. However on 3 occasions now we've had issues with the SQLite database getting "locked" and causing issues. The message that we get is "exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 5 database is locked' in...".
I've had a search around and it seems to be a common issue, and there's a lot of discussion about ways to minimise its likelihood or prevent it (e.g. the drupal boards). However, my issue seems to be slightly different to this. The situations I've seen described seem to indicate that it's to do with concurrency - when two PHP processes try to access the SQLite database at exactly the same time, one of them gets a lock error. In this situation efforts to simply minimise the issue make sense. But for my application, when the issue starts happening (presumably because of concurrency), the SQLite database is from that point on permanently locked. Every single cache-accessing request from this point on gets a PDOException. Our solution has been to just delete the cache file, which is not the end of the world, but this requires manual intervention and plus means we lose the built-up cache data.
Why would this be happening? Is there other reasons why we might get the lock to start with? Why does the lock persist? Is there a way to programmatically clear it? Is there a way to prevent it in the first place?
The two "solutions" I'm considering so far are:
Put a try-catch around the cache accessing functions. If we get exceptions, just ignore the cache and notify tech support to manually clear the cache.
Use a mutex (using PHP flock) for the SQLite file to prevent the issue with concurrency (but again, I'm not even certain this is the root cause).
Any information or suggestions would be greatly appreciated!
Well...
Makes tech-support handle a non-existing problem.
Ignores the fact that SQLite does it's own locking. (SQLite3 has even optimized it!)
Leaving the geek in me stand at the door, I would use "solution" number 3 (which you didn't list) and simply put a try-catch around the cache accessing functions. If you get exceptions, do a short sleep and then call the exception-causing function again.
thisFunction(...)
{
....
try
{
....
}
catch(Exception $e)
{
sleep(rand(1,3))
thisFunction(...);
}
....
}
That way, you actually gain something from the error sqlite is trying to help you with. ;)
Related
I have php script running as a cron job, extensively using third party code. Script itself has a few thousands LOC. Basically it's the data import / treatment script. (JSON to MySQL, but it also makes a lot of HTTP calls and some SOAP).
Now, performance is downgrading with the time. When testing with a few records (around 100), performance is ok, it is done in a 10-20 minutes. When running whole import (about 1600 records), mean time of import of one record grows steadily, and whole thing takes more than 24 hours, so at least 5 times longer than expected.
Memory seems not to be a problem, usage growing as it should, without unexpected peaks.
So, I need to debug it to find the bottleneck. It can be some problem with the script, underlying code base, php itself, database, os or network part. I am suspecting for now some kind of caching somewhere which is not behaving well with a near 100 % miss ratio.
I cannot use XDebug, profile file grows too fast to be treatable.
So question is: how can I debug this kind of script?
PHP version: 5.4.41
OS: Debian 7.8
I can have root privileges if necessary, and install the tools. But it's the production server and ideally debugging should not be too disrupting.
Yes its possible and You can use Kint (PHP Debugging Script)
What is it?
Kint for PHP is a tool designed to present your debugging data in the absolutely best way possible.
In other words, it's var_dump() and debug_backtrace() on steroids. Easy to use, but powerful and customizable. An essential addition to your development toolbox.
Still lost? You use it to see what's inside variables.
Act as a debug_backtrace replacer, too
you can download here or Here
Total Documentations and Help is here
Plus, it also supports almost all php framework
CodeIgniter
Drupal
Symfony
Symfony 2
WordPress
Yii
framework
Zend Framework
All the Best.... :)
There are three things that come to mind:
Set up an IDE so you can debug the PHP script line by line
Add some logging to the script
Look for long running queries in MySQL
Debug option #2 is the easiest. Since this is running as a cron job, you add a bunch of echo's in your script:
<?php
function log_message($type, $message) {
echo "[{strtoupper($type)}, {date('d-m-Y H:i:s')}] $message";
}
log_message('info', 'Import script started');
// ... the rest of your script
log_message('info', 'Import script finished');
Then pipe stdout to a log file in the cron job command.
01 04 * * * php /path/to/script.php >> /path/to/script.log
Now you can add log_message('info|warn|debug|error', 'Message here') all over the script and at least get an idea of where the performance issue lies.
Debug option #3 is just straight investigation work in MySQL. One of your queries might be taking a long time, and it might show up in a long running query utility for MySQL.
Profiling tool:
There is a PHP profiling tool called Blackfire which is currently in public beta. There is specific documentation on how to profile CLI applications. Once you collected the profile you can analyze the application control flow with time measurements in a nice UI:
Memory consumption suspicious:
Memory seems not to be a problem, usage growing as it should, without unexpected peaks.
A growing memory usage actually sounds suspicious! If the current dataset does not depend on all previous datasets of the import, then a growing memory most probably means, that all imported datasets are kept in memory, which is bad. PHP may also frequently try to garbage collect, just to find out that there is nothing to remove from memory. Especially long running CLI tasks are affected, so be sure to read the blog post that discovered the behavior.
Use strace to see what the program is basically doing from the system perspective. Is it hanging in IO operations etc.? strace should be the first thing you try when encountering performance problems with whatever kind of Linux application. Nobody can hide from it! ;)
If you should find out that the program hangs in network related calls like connect, readfrom and friends, meaning the network communication does hang at some point while connecting or waiting for responses than you can use tcpdump to analyze this.
Using the above methods you should be able to find out most common performance problems. Note that you can even attach to a running task with strace using -p PID.
If the above methods doesn't help, I would profile the script using xdebug. You can analyse the profiler output using tools like KCachegrind
Although it is not stipulated, and if my guess is correct you seem to be dealing with records one at a time, but in one big cron.
i.e. Grab a record#1, munge it somehow, add value to it, reformat it then save it, then move to record#2
I would consider breaking the big cron down. ie
Cron#1: grab all the records, and cache all the salient data locally (to that server). Set a flag if this stage is achieved.
Cron #2: Now you have the data you need, munge and add value, cache that output. Set a flag if this stage is achieved.
Cron #3: Reformat that data and store it. Delete all the files.
This kind of "divide and conquer" will ease your debugging woes, and lead to a better understanding of what is actually going on, and as a bonus give you the opportunity to rerun say, cron 2.
I've had to do this many times, and for me logging is the key to identifying weaknesses in your code, identify poor assumptions about data quality, and can hint at where latency is causing a problem.
I've run into strange slowdowns when doing network heavy efforts in the past. Basically, what I found was that during manual testing the system was very fast but when left to run unattended it would not get as much done as I had hoped.
In my case the issue I found was that I had default network timeouts in place and many web requests would simply time out.
In general, though not an external tool, you can use the difference between two microtime(TRUE) requests to time sections of code. To keep the logging small set a flag limit and only test the time if the flag has not been decremented down to zero after reducing for each such event. You can have individual flags for individual code segments or even for different time limits within a code segment.
$flag['name'] = 10; // How many times to fire
$slow['name'] = 0.5; // How long in seconds before it's a problem?
$start = microtime(TRUE);
do_something($parameters);
$used = microtime(TRUE) - $start;
if ( $flag['name'] && used >= $slow['name'] )
{
logit($parameters);
$flag['name']--;
}
If you output what URL, or other data/event took to long to process, then you can dig into that particular item later to see if you can find out how it is causing trouble in your code.
Of course, this assumes that individual items are causing your problem and not simply a general slowdown over time.
EDIT:
I (now) see it's a production server. This makes editing the code less enjoyable. You'd probably want to make integrating with the code a minimal process having the testing logic and possibly supported tags/flags and quantities in an external file.
setStart('flagname');
// Do stuff to be checked for speed here
setStop('flagname',$moredata);
For maximum robustness the methods/functions would have to ensure they handled unknown tags, missing parameters, and so forth.
xdebug_print_function_stack is an option, but what you can also do is to create a "function trace".There are three output formats. One is meant as a human readable trace, another one is more suited for computer programs as it is easier to parse, and the last one uses HTML for formatting the trace
http://www.xdebug.org/docs/execution_trace
Okay, basically you have two possibilities - it's either the ineffective PHP code or ineffective MySQL code. Judging by what you say, it's probably inserting into indexed table a lot of records separately, which causes the insertion time to skyrocket. You should either disable indexes and rebuild them after insertion, or optimize the insertion code.
But, about the tools.
You can configure the system to automatically log slow MySQL queries:
https://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
You can also do the same with PHP scripts, but you need a PHP-FPM environment (and you probably have Apache).
https://rtcamp.com/tutorials/php/fpm-slow-log/
These tools are very powerful and versatile.
P.S. 10-20 minutes for 100 records seems like A LOT.
You can use https://github.com/jmartin82/phplapse to record the application activity for determinate time.
For example start recording after n iterations with:
phplapse_start();
And stop it in next iteration with:
phplapse_stop();
With this process you was created a snapshot of execution when all seems works slow.
(I'm the author of project, don't hesitate to contact with me to improve the functionality)
I have a similar thing running each night (a cron job to update my database). I have found the most reliable way to debug is to set up a log table in the database and regularly insert / update a json string containing a multi-dimensional array with info about each record and whatever useful info you want to know about each record. This way if your cron job does not finish you still have detailed information about where it got up to and what happened along the way. Then you can write a simple page to pull out the json string, turn it back into an array and print useful data onto the page including timing and passed tests etc. When you see something as an issue you can concentrate on putting more info from that area into the json string.
Regular "top" command can show you, if CPU usage by php or mysql is bottleneck. If not, then delays may be caused by http calls.
If CPU usage by mysqld is low, but constant, then it may be disk usage bottleneck.
Also, you can check your bandwidth usage by installing and using "speedometer", or other tools.
This may be better suited to server fault but I thought I'd ask here first.
We have a file that is prepended to every PHP file on our servers using auto-prepend that contains a class called Bootstrap that we use for autoloading, environment detection, etc. It's all working fine.
However, when there is an "OUT OF MEMORY" error directly preceding (i.e., less than a second or even at the same time) a request to another file on the same server, one of three things happens:
Our code for checking if(class_exists('Bootstrap'), which we used to wrap the class definition when we first got this error, returns true, meaning that class has already been declared despite this being the auto-prepend file.
We get a "cannot redeclare class Bootstrap" error from our auto-prepended file, meaning that class_exists('Bootstrap') returned false but it somehow was still declared.
The file is not prepended at all, leading to a one-time fatal error for files that depend on it.
We could, of course, try to fix the out of memory issues since those seem to be causing the other errors, but for various reasons, they are unfixable in our setup or very difficult to fix. But that's beside the point - it seems to me that this is a bug in PHP with some sort of memory leak causing issues with the auto-prepend directive.
This is more curiosity than anything since this rarely happens (maybe once a week on our high-traffic servers). But I'd like to know - why is this happening, and what can we do to fix it?
We're running FreeBSD 9.2 with PHP 5.4.19.
EDIT: A few things we've noticed while trying to fix this over the past few months:
It seems to only happen on our secure servers. The out of memory issues are predominantly on our secure servers (they're usually from our own employees trying to download too much data), so it could just be a coincidence, but it deserves pointing out
The dump of get_declared_classes when we have this issue contains classes that are not used on the page that is triggering the error. For example, the output of $_SERVER says the person is on xyz.com, but one of the declared classes is only used in abc.com, which is where the out of memory issues usually originate from.
All of this leads me to believe that PHP is not doing proper end-of-cycle garbage collection after getting an out of memory error, which causes the Bootstrap class to either be entirely or partly in memory on the next page request if it's soon enough after the error. I'm not familiar enough with PHP garbage collection to actually act on this, but I think this is most likely the issue.
You might not be able to "fix" the problem without fixing the out of memory issue. Without knowing the framework you're using, I'll just go down the list of areas that come to mind.
You stated "they're usually from our own employees trying to download too much data". I would start there, as it could be the biggest/loudest opportunity for optimizations, a few idea come to mind.
if the data being downloaded is files, perhaps you could use streams to chunk the reads, to a constant size, so the memory is not gobbled up on big downloads.
can you do download queueing, throttling.
if the data is coming from a database, besides optimizing your queries, you could rate limit them, reduce the result set sizes and ideally move such workloads to a dedicated environment, with mirrored data.
ensure your code is releasing file pointers and database connections responsibly, leaving it to PHP teardown, could result in delayed garbage collection and a sort of cascading effect, in high traffic situations.
Other low hanging fruit when it comes to memory limits
you are running php 5.4.19, if your software permits it, consider updating to more resent version "PHP 5.4 has not been patched since 2015" besides PHP 7 comes with a whole slew of performance improvements.
if you have a client side application involved monitor it's xhr and overall network activity, look for excessive polling and hanging connections.
as for your autoloader, based on your comment "The dump of get_declared_classes when we have this issue contains classes that are not used on the page that is triggering the error" you may want to check the implementation, to make sure it's not loading some sort of bundled class cache, if you are using composer, dump-autoload might be helpful.
sessions, I've seen some applications load files based on cookies and sessions, if you have such a setup, I would audit that logic and ensure there are no sticky sessions loading unneeded resources.
It's clear from your question you are running a multi-tenency server. Without proper stats it hard to be more specific, but I would think it's clear the issue is not a PHP issue, as it seems to be somewhat isolated, based on your description.
Proper Debugging and Profiling
I would suggest installing a PHP profiler, even for a short time, new relic is pretty good. You will be able to see exactly what is going on, and have the data to fix the right problem. I think they have a free trial, which should get you pointed in the right direction... There are others too, but their names escape me at the moment.
Even if class_exists returns false, it would never return true if an interface of the same name exists. However, you cannot declare an interface and class of the same name.
Try running class_exists('Bootstrap') && interface_exists('Bootstrap') to make sure you do not redeclare.
Did you have a look at __autoload function?
I believe that you could workaround this issue by creating some function like that in your code:
function __autoload($className)
{
if (\file_exists($className . '.php'))
include_once($className . '.php');
else
eval('class ' . $className . ' { function __call($method, $args) { return false; } }');
}
If you have a file called Bootstrap.php with class Bootstrap declared inside it, PHP will automatically load file, otherwise declare a ghost class that could handle any function call inside it, avoiding any error messages. Note that for ghost function I used __call magic method.
i have to write a very complex application for works.
All my previous software were software used in a local area and not application on cloud.
I notice that all the query that modify the database are subjected to concurrency problem.
I've just solved the problem using write lock and transaction in serializable isolation level.
But in only reading query i have to make a read lock? what happen if i read some records from a table while another query deleting some of these records? i have a dirty read.
So can you explain how do you usually resolve the query concurrency problem?. I use php.
Do you usually use write lock? as far as now i've never notice that problem. i work as Web developer in a small company and i don't now why i never occurred in this solution.
thank you, and sorry for my bad english.
Wrap your writes in a transaction, don't use locks.
In PHP it's far too easy for your application to crash out and leave a lock in place, rendering your entire database unusable until that's reset. Transactions will automatically release when committed, rolled back, or the connection is dropped.
I think most of what you're talking about is misplaced paranoia. If you're using a proper database engine like InnoDB you won't have problems.
Currently running an ecommerce store. In the last couple of months we have been experiencing lots of very odd / random issues that have been affecting the store. In almost all cases we have been unable to replicate the faults and it turned out to be a corrupt database.
What can cause this to happen?
How can it be prevented?
EDIT:
Sorry, this is pretty vague. Basically Im looking for things that could potentially cause a database corruption. Its a MySql4 database.
What essentially causes database corruption and how can you detect and prevent.
Just generally.
The question is very broad so I will try to answer with broad suggestions.
MySQL, while not what I would call an enterprise-level DBMS, should not have random corruption problems.
Check your hardware. I don't know your platform, but assuming it's linux you can use Try prime95 to test your processors and smartmontools to do some basic disk tests. Memtest86 can diagnose memory errors, but you probably don't want to reboot your server just for that.
Some specific kernel versions have, in my experience, possibly led to unexplained problems. Can you upgrade?
Test importing the data on a newer version of MySQL, or on a different system. This may not be useful because "no errors yet" doesn't mean the problem is resolved.
If you are using MyISAM tables, which should still work fine, you may try an ACID compliant engine. InnoDB was all the rage for a while, but I believe it's been replaced.
This is a fairly general question, so I will just give you what I have seen personally.
Most of the MySQL database failures that I have seen have been because of a hard drive being corrupted some how or power failures when a server is hard powered-off instead of shutdown properly.
The main thing you can do to prevent damage in these two cases is to make backups of your database often (and store them somewhere that will not be tampered with), this way you will always have something recent to revert back to. Also, storing your data in a RAID is helpful because if depending on your setup, you can survive if a disk or two crashes. Having backup power supplies for your disks in case power goes out is good too.
Also, try to use a robust tables that can recover if problems arise. I used to use MyISAM tables but if I ever ran into a problem I would lose the data and have to start from the latest backup. So I switched to InnoDB and then I was actually able to recover from most crashes. However, this was a while ago and InnoDB may not be the latest and greatest anymore.
Anyways, good luck with solving your issues and if you have anymore information, hopefully I can help more.
I've read previous answers here about caching in PHP, and the articles they link to. I've checked out the oft-recommended Pear Cache_Light, QuickCache, and WordPress Super Cache. (Sorry - apparently I'm allowed to hyperlink only once.)
Either none deal with concurrency issues, or none explicitly call out that they do in their documentation.
Can anyone point me in the direction of a PHP page cache that handles concurrency?
This is on a shared host, so memcache and opcode caches are unfortunately not an option. I don't use a templating engine and would like to avoid taking a dependency on one. WP Super Cache's approach is preferable - i.e. storing static files under wwwroot to let Apache serve them - but not a requirement.
Thanks!
P.S. Examples of things that should be handled automatically:
Apache / the PHP cache is in the middle of reading a cached file. The cached file becomes obsolete and deletion is attempted.
A cached file was deleted because it was obsolete. A request for that file comes in, and the file is in the process of being recreated. Another request for the file comes in during this.
It seems PEAR::Cache_Lite has some kind of security to deal with concurrency issues.
If you take a look at the manual of constructor Cache_Lite::Cache_Lite, you have those options :
fileLocking
enable / disable fileLocking. Can avoid cache corruption under bad
circumstances.
writeControl
enable / disable write control. Enable write control will lightly slow
the cache writing but not the cache
reading. Write control can detect some
corrupt cache files but maybe it's not
a perfect control.
readControl
enable / disable read control. If enabled, a control key is embeded in
cache file and this key is compared
with the one calculated after the
reading
readControlType
Type of read control (only if read control is enabled). Must be 'md5'
(for a md5 hash control (best but
slowest)), 'crc32' (for a crc32 hash
control (lightly less safe but
faster)) or 'strlen' (for a length
only test (fastest))
Which one to use is still up to you, and will depend on what kind of performance you are ready to sacrifice -- and the risk of concurrency access that probably exists in your application.
You might also want to take a look at Zend_Cache_Frontend_Output, to cache a page, using something like Zend_Cache_Backend_File as backend.
That one seems to support some kind of security as well -- the same kinf of stuff that Cache_Lite already gave you (so I won't copy-paste a second time)
As a sidenote, if your website runs on a shared host, I suppose it doesn't have that many users ? So the risks of concurrent access are probably not that high, are they ?
Anyway, I probably would not search any farther that what those tow Frameworks propose : it is already probably more than enough for the needs of your application :-)
(I've never seen any caching mecanism "more secure" than what those allow you to do... And i've never run into some catastrophic concurrency problem of that sort yet... In 3 years of PHP-development)
Anyway : have fun !
I would be tempted to modify one of the existing caches. Zend Framework's cache should be able to do the trick. If not, I would change it.
You could create a really primitive locking strategy. The database could be used to track all of the cached items, allow locking for update, allow people to wait for someone else's update to complete, ...
That would handle your ACID issues. You could set the lock for someone else's update to a very short period, or possibly have it just skip the cache altogether for that round trip depending on your server load/capacity and the cost of producing the cached content.
Jacob
Concurrent resource creation aka cache slamming / thread race can be a serious issue on busy websites. That's why I've created cache library that synchronize read/write processes/threads.
It has elegant and clear structure: interfaces -> adaptors -> classes for easy extension. At github page im explaining in details what's the problem with slamming and how The Library is resolving it.
Check it here:
https://github.com/tztztztz/php-no-slam-cache
Under Linux, generally, the file will remain "open" for read, even if it's "deleted" until the process closes the file. This is something built into the system, and can sometimes cause huge discrepancies in disk usage sizes (deleting a 3G file while it's still "open" would mean that is still allocated on the disk as in use until the process closes it) - I'm unsure as to whether the same is true under linux.
Assuming a Journalling Filesystem (most Linux Filesystems, and NTFS) - then the file should not be seen as "created" until the process closes the file. This should show up as a non-existant file!
Assuming a Journalling Filesystem (most Linux Filesystems, and NTFS) -
then the file should not be seen as "created" until the process
closes the file. This should show up as a non-existant file!
Nope, it is visible as soon as it is created, you have to lock it.
Rename is atomic though. So you could open(), write(), close(), rename(), but this will not prevent the same cache item being re-created twice at the same time.
A cached file was deleted because it was obsolete.
A request for that file comes in, and the file is in the process of being recreated. Another request for the file comes in during this.
If it is not locked, a half-complete file will be served, or two processes will try to regenerate the same file at the same time, giving "interesting" results.
You could cache pages in the database, just create a simple "name,value" table and store cached pages on it.