SEE THE LAST UPDATE AT THE BOTTOM FOR MY FINAL EVAL / SUGGESTIONS
This seems so basic that it should be a common problem.. but I've already searched for anything pertaining to this issue with no luck
-- Scenario --
I have a web application that, as one of it's functions, allows users to upload photos to the server. File size limits aren't the issue, but i can notice a visible difference in the speed of the server when i'm uploading a file vs not.
-- Testing --
I uploaded a 3MB file while signed in to another account (on another computer completely) to test the page load times in firebug. Caching has been disabled. The results are below:
Baseline page speed (without upload): 0.409, 0.449, 0.468
During 3MB file upload:1.28, 8.58, --upload complete -- 0.379
This problem obviously compounds if more than one user is uploading a photo at the same time. This seems insane considering all the power i have with the current setup.
-- Setup --
Mediatemple DV Level 3 Application server (4GB ram, 16 cores)
Mediatemple DV Dev level 1 database server (running mysiam tables)
Cloudflare CDN
Custom PHP application
Wordpress sales website (front end, same server - not connected in any way to the web app)
CentOS 6.5
Mysql 5.5
-- So Far --
I had the cloudtech team at MT tune the apache & nginx settings for me since i thought i had screwed this up, but the issue is still showing up
I am considering changing all the DB tables to innodb for concurrency, but this is not related to the question at hand
The server load averages do not seem to be significantly affected when uploading my test file
the output of "free -m" is below
total used free shared buffers cached
Mem: 4096 634 3461 0 0 215
-/+ buffers/cache: 419 3676
Swap: 1536 36 1499
EDIT 1
Is it possible to offload these types of things to an independent server? I realize the PHP used to upload the file would also have to be run from that server, but at least then only the upload / long process server would be affected and not the entire application.
Also, is there any other language or workflow that would work better for this? Perl? Python?
EDIT 2 (2014-08-28)
I forgot to mention two things
1) this issue isn't just with file uploads - it happens whenever a php script runs for an extended time. As a test, i ran a 3 minute php script on my end and sure enough, got a phone call from a client during the execution about the "slow" system.
2) I have high concurrent log in sessions running. Many of these users are likely on the same script at the same time.
Here is the output from htop. The "php-cgi" processes are the obvious offenders, but i don't know how to see which scripts are causing this load. Again, even if i did find the script, i feel like i should be able to run more than a handful of php scripts at a time.
EDIT 3 (2014-08-28)
Here is the htop at peak hours (right now). What's annoying is that the system is flying at the moment with 2x the traffic and processes.
EDIT 4 / UPDATE (2014-09-30)
After a month of staring at this, I've found some things to be true. I'm hoping some of this will help others get their high-growth applications in check before it turns into an issue of racing traffic with server upgrades (which is what happened here).
The Issue I was using MyISAM as the exclusive database engine. I had read through hundreds of docs and forum posts regarding whether InnoDB or MyISAM is the better engine to use, most sources giving vague evaluations or (at best) overly complicated benchmarking with vague settings claiming to increase (or even decrease..?) performance. Forget it all and USE InnoDB FOR ALL APPLICATIONS
Find a good resource to help you tune your MySQL server settings and run with it (see links below). Apparently the concurrent traffic on the server was overloading PHP while waiting for the table locks to release in MyISAM. This was causing excessive loads on the application server, while the DB server was just hanging out with hardly any CPU or MEM load. Transitioning to InnoDB allows high-concurrency at the cost of CPU and Memory (both GOOD things, buy a bigger DB server if you have to).
In the end, the load has transferred to the DB server, increasing concurrent traffic performance. To summarize DON'T USE MyISAM ON WEB APPLICATIONS. Period. I'm sure i'll get burned a bit by saying that, but the slight performance hit (yes, MyISAM is a BIT faster at low-concurrency, but who cares if your web app crashes) is well worth the increase in concurrency.
IMPORTANT POINTS
1) When you move your database over to InnoDB "ALTER TABLE my_table SET ENGINE="InnoDB" you will need to use the info found in the following links to set your innodb specific settings.
2) Be sure to code a loop wrapper in PHP (3 iterations sounds good) for your query calls to account for deadlocks (a situation where queries are basically competing for the same row(s) and one or both are stalled completely).
3) Write your PHP to look for ERRORS coming out of your new query wrapper.
EG: Send the query -> Error found -> Check for deadlock -> if deadlock, retry query after waiting 0.1 sec -> check again, if error found that isn't deadlock, return error to app, else continue until iteration limit is reached (3 in this example) -> if iteration limit is hit, return error to application
Do something now that your query has failed.
Helpful Links
http://www.percona.com/blog/2013/09/20/innodb-performance-optimization-basics-updated/
Handling innoDB deadlock
https://dba.stackexchange.com/questions/27328/how-large-should-be-mysql-innodb-buffer-pool-size
CAUTION
I crashed the MySQL server with the setting in the following link. You MUST completely stop mysqld (service mysqld stop) before renaming (NOT deleting) the original files (ib_logfile0, ib_logfile1), usually found here on RH/CentOS
/var/lib/mysql/ib_logfile0
http://www.percona.com/blog/2008/11/21/how-to-calculate-a-good-innodb-log-file-size/
Once they are renamed, start your mysql daemon service mysqld start
Hopefully this helps someone,
-B
Related
This is my first nervous question on SO because all of my questions in the last decade have already had excellent answers.
I have searched all the terms that I can think of with no hits that appear to address the problem - either on SO or Google generally...
For the last 15 years we have used phpMyAdmin to administer a linux MySQL manufacturing database of about 100 tables, some of which are now 50 to 300 million records each. Ongoing development is constant, and manual lookup of various tables to correct erroneous data, or to modify table indexes etc are frequent as the size of the data grows. All of this is internal to our fast network - i.e. accessed via our intranet. Most queries are short, and the database runs responsively at a low average loading.
As may be understood, DBA mistakes happen. For example to speed up a slow query, an additional index may be added to a large table without enough thought. At this point, the re-indexing may take 30 minutes, and the manufacturing applications (written in php for Apache2 also on a linux server) come to an immediate halt. This is not appreciated in the factory.
And here is the real problem. I cannot then from my development PC open a second instance of phpMyAdmin to kill the unwanted MySQL process while it is still busy. Which is the very time I need to most :-) The browser just goes into waiting for the phpMyAdmin page to load until after the long query is finished.
If I happen to have a second instance pf phpMyAdmin open already, I can look up the process and kill it satisfactorily. Normallly, my only resort is to restart Apache2 and/or MySQL on the server. This is too drastic and requires re-starting many client machines as well in order to re-establish necessary manufacturing connections to the database.
I have seen reference on SO that Apache will queue requests from the same IP address in the case of php programs using file-based session management, but it seems to me that I have no control over how phpMyAdmin uses its sessions.
I also read some time ago that if multiple CPU cores were brought into play on the database server, multiple simultaneous connections could be made despite one such query still being busy. I cannot now find any reference to this concept.
Does anyone please know how to permit or force a second phpMyAdmin connection from the same PC to the same database server using phpMyAdmin while the first instance of phpMyAdmin is still tied up with a previous slow query?
Many thanks, Jem Stanners
Try mySQL Workbench
https://dev.mysql.com/downloads/workbench/
Try upgrading servers RAMs an processors
Consider cleaning the tables and delete rows if possible
Consider shifting to Oracle (cost is to be considered)
I am running form submission tests on a website (WAMP) to create new users. Currently, my thread breaks when moving from 6-7 users to 8. The current configuration which breaks is:
1 Thread
8 Users
20 Loops
Each form submission contains approx. 25 input fields with no more than 25 characters in each.
I have removed all but one View Results In Tree module, increased the heap size to 4096 (which I come nowhere near), and am running it in non-GUI mode which made improvements but I am still running out of memory well before I should think. System monitor shows that Java is only using around 400MB.
Error says:
java.net.SocketException: No buffer space available (maximum connections reached?): connect
This results in an equal number of failed MySQL INSERTS (e.g. # Buffer Errors (in Results Tree) == # Failed Inserts).
I have looked into some of the Jmeter tuning tips, but none of them seem to apply beyond running in non-Gui. Is this Jmeter or is my application not handling the submissions correctly? Is this too many input fields per submission for this load? With only 5 users, everything works fine. Thanks in advance.
There is a known issue in Windows with leaky sockets. For those that fit the description, and if you are already running Jmeter in x64 mode but are still experiencing premature memory lapses during stress tests, this may help solve the problem: https://support.microsoft.com/kb/2577795
My Drupal 6 site has been running smoothly for years but recently has experienced intermittent periods of extreme slowness (10-60 sec page loads). Several hours of slowness followed by hours of normal (4-6 sec) page loads. The page always loads with no error, just sometimes takes forever.
My setup:
Windows Server 2003
Apache/2.2.15 (Win32) Jrun/4.0
PHP 5
MySql 5.1
Drupal 6
ColdFusion 9
Vmware virtual environment
DMZ behind a corporate firewall
Traffic: 1-3 hits/sec peak
Troubleshooting
No applicable errors in apache error log
No errors in drupal event log
Drupal devel module shows 242 queries in 366.23 milliseconds,page execution time 2069.62 ms. (So it looks like queries and php scripts are not the problem)
NO unusually high CPU, memory, or disk IO
Cold fusion apps, and other static pages outside of drupal also load slow
webpagetest.org test shows very high time-to-first-byte
The problem seems to be with Apache responding to requests, but previously I've only seen this behavior under 100% cpu load. Judging solely by resource monitoring, it looks as though very little is going on.
Here is the kicker - roughly half of the site's access comes from our LAN, but if I disable the firewall rule and block access from outside of our network, internal (LAN) access (1000+ devices) is speedy. But as soon as outside access is restored the site is crippled.
Apache config? Crawlers/bots? Attackers? I'm at the end of my rope, where should I be looking to determine where the problem lies?
------Edit:-----
Attached is a waterfall chart from webpagetest.org showing a 15 second load time. I've seen times as high as several minutes. And again, the server runs fine much of the time. The green areas indicate that the browser has sent a request and is waiting to recieve the first byte of data back from the server. This is certainly a back-end delay, but it is puzzling that the CPU is barely used during this slowness.
(Not enough rep to post an image, see https://webmasters.stackexchange.com/questions/54658/apache-very-high-page-load-time
------Edit------
On the Apache side of things - Is this possibly a ThreadsPerChild issue?
After much research, I may have found the solution. If I'm correct, it was an apache config problem. Specifically, the "ThreadsPerChild" directive. See... http://httpd.apache.org/docs/2.2/platform/windows.html
Because Apache for Windows is multithreaded, it does not use a
separate process for each request, as Apache can on Unix. Instead
there are usually only two Apache processes running: a parent process,
and a child which handles the requests. Within the child process each
request is handled by a separate thread.
ThreadsPerChild: This directive is new. It tells the server how many
threads it should use. This is the maximum number of connections the
server can handle at once, so be sure to set this number high enough
for your site if you get a lot of hits. The recommended default is
ThreadsPerChild 150, but this must be adjusted to reflect the greatest
anticipated number of simultaneous connections to accept.
Turns out, this directive was not set at all in my config and thus defaulted to 64. I confirmed this by viewing the number of threads for the second httpd.exe process in task manager. When the server was hitting more than 64 connections, the excess requests were simply having to wait for a thread to open up. I added ThreadsPerChild 150 in my httpd.conf.
Additionally, I enabled the apache status module
http://httpd.apache.org/docs/2.2/mod/mod_status.html
...which, among other things, allows one to see the total number of active request on the server at any given moment. Right away, I could see spikes of up to 80 active request. Time will tell, but I'm confident that this will resolve my issue. So far, 30 hours without a hiccup.
Apache is too bulk and clumsy for "1-3 hits/sec avg".
Once I have similar problem with much lighter (almost static-html, no DB) site, and similar hits/second.
No errors, no high network/CPU/memory/disk loads. Apache on WinXP.
I inserted nginx before Apache for static files and it started working like a charm.
Caching. The solution it caching.
Drupal (in common with most other large CMS platforms) has a tendency toward this kind of thing due to its nature -- every page is built on the fly, constructed from a whole stack of database tables and code modules. The more you've got in there, the slower it will be, but even fairly simple pages can become horribly slow if your site gets a bit of traffic.
Drupal has a page cache mechanism built-in which will cut your load dramatically. As long as your pages are static (ie no dynamic content) then you can simply switch on caching and watch the performance go right back up.
If you have dynamic content, you can still enable caching for the static parts of the page. It is a bit more complex (and beyond the scope of this answer), but it is worth the effort.
If that's still not enough, a server-based caching solution such as Varnish will definitely help.
I have a Lighttpd(1.4.28) web server running on Centos 5.3 and PHP 5.3.6 in fastcgi mode.
The server itself is a quad core with 1gb ram and is used to record viewing statistics for a video platform.
Each request consists of a very small bit of xml being posted and the receiving php script performs a simple INSERT or UPDATE mysql query. The php returns a very small response to acknowledge the request.
These requests are performed very frequently and i need the system to be able to handle as many concurrent connections as possible at a high rate of requests/second.
I have disabled keep alive as only single requests will be made and so I don't need to keep connections open.
One of the things that concern me is that in server-status I am seeing a lot of connections in the 'read' state. I take it this is controlled by server.max-read-idle which is set to 60 by default? Is it ok to change this to something like 5 as I am seeing the majority of connections being kept open for long periods of time.
Also what else can I do to optimise lighttpd to be able to server lots of small requests
This is my first experience setting up lighttpd as I thought it would be more suitable than apache in this case.
Thanks
Irfan
I believe the problem is not in the webserver, but in your PHP application, especially in MySQL part.
I would replace lighty with apache + mod_php, and mysql with some NoSQL such Redis, which will queue the INSERT requests to the database. Then I would write a daemon / crontab that insert the data in MySQL.
We had such thing before, but instead of Redis, we created TXT files in one directory.
I have a LAMP application running fine, however the number of users are increasing each day. I don't want to be hit with a surprise one morning and find that everything broke because of overload. Is there a way to get a rough estimate of what capacity of the LAMP it is at?
I know that a full detailed report is many books worth of study but can I get some quick litmus test to see if things are running fine.
So say for the mysql component, how can I tell how much more load can it take? Is it at 30% capacity, 50%? etc.
Same for my apache. Although I have a feeling the DB will die before apache.
Perhaps my original was not too good, as English is not my native language. What I am really asking is a way to measure the current load. And then have a way to estimate based on the that load, how much further can I go before it fails. (And this should be done seperately for each component, mysqld, httpd)
ab is a bit annoying if your site needs cookies, etc, ab is too simple.
Basically, from my experience in fixing several imploding PHP websites, it usually goes like this :
1) People use MySQL
You can totally use MySQL, facebook and flickr do it (mysql fanboys love those) IF YOU KNOW THE GOTCHAS which are :
If you have a non-read-only MyISAM table and any query longer than 100 us (even selects) you are dead
On one site I fixed, the guy had rented a double-quad-core server because "his site needs the power". I look at his site, I look at my previous site with > 100K members and a torrent tracker which ran on a Via C7 micro-half-pizzabox server, and I tell him, your site runs fine on the Celeron 300 that's in my basement, and that's even overkill, I can rent it to you for half the price of your Xeon, lol.
It turned out that the guy was a good developer and a real nice guy but he sucked at MySQL, so his site had the typical Search Query From Hell that can kill any website :
10 search queries from hell per second (he had like 300K members on his illegal warez site)
search query from hell takes about 0.1 - 0.2 seconds
a little stream of concurrent updates to the same MyISAM table to spice things up
=> total serialization (MyISAM write locks) of all queries. 1 core 100%, 7 cores idle, loadavg > 1000 (yes he was using apache), page times > 30 seconds, the works.
Fix was easy : optimize the search query from hell, fix point 2) below, switch to InnoDB, switch to lighttpd. loadavg dropped to 0.02
2) UPDATEs
Noone is interested in page counters.
Issue 1 UPDATE for every page view and you are dead.
Add some MyISAM for more effects. Also a killer on InnoDB, not about locking, rather about sync disk IO waits.
3) FULLTEXT
MyISAM not usable for read-write tables because of locking.
MyISAM is as reliable as a ramdisk (in fact, less : you need an OS crash to corrupt a ramdisk, corrupting MyISAM tables just needs a MySQL crash or just hitting it too much concurrently, you'll get "unknown table engine error", I saw this many times)
FULLTEXT not available on InnoDB
Any insertion in a FULLTEXT index triggers almost a full index rebuild (when I inserted a
forum post it was rebuilding 400 MB of index)
==> If you need full text indexing, performance, and reliability, use Sphinx or Xapian.
I've not tried Sphinx (people say good things about it), but Xapian happily searches through 4GB of text in a snap.
4) People use apache.
This nicely combines with the points above.
Unlike a proper server like lighttpd whose CPU usage is undetectable (the crummy Via C7 was serving 100 HTTP hits/s and lighttpd used less than 1% CPU), apache will kill your box.
When the MySQL starts to die (it dies easily), clients start to hit F5 hard, and soon you have about 1000 apache processes, each holding a PHP interpreter, and each PHP interpreter holds an idle MySQL connection, waiting on a MyISAM lock, except one, which is doing some trivial UPDATE of your page view counter, but that takes some time, because the server is gone to lunch swapping, because of the 1000 apache and 1000 php and 1000 mysql processes.
Lighttpd uses no cpu for static pages. The only way for lighttpd to saturate your CPU is if you hit it hard with apachebench at like 20K requests/s. Then Lighttpd talks to a few, like 10 php-fcgi backends (2-4 per core is good) which talk to a few MySQL connections. Everything is a lot faster as a result, and when overloaded, it degrades gracefully, not explosively.
To get to the original question, you definitely want to profile your SQL queries. Add a query log to your PHP application which displays (only to you), the list of queries and the time they take, and also the time from the start of the PHP script to its end (header/footer includes are a good place for this).
For a complex page (excluding search) you'd expect about 3 ms MySQL and 3 ms PHP, that's a good target. You need a PHP compiled code cache of course.
For the current load, there are a couple things your can do. The most expensive, yet most detailed answers will be provided through a enterprise application such as "Gomez".
However, if you're looking to do this yourself, see my previous answers below or use shell utilities such as: htop, top, w, and utilize Apache server-status
Previous answers before question revision:
What you are asking for is sometimes called application profiling.
You need to create a rough memory formula like:
httpd ram + php memory usage + mysql process usage = total request memory footprint
You will also need a CPU formula, but you can also eyeball top during a load test.
Apache has the command 'ab'.
"ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving." http://httpd.apache.org/docs/2.0/programs/ab.html
Here is a generic 'ab' benchmark command line:
ab -n 10 -c 1 http://www.yoursite.com/
# qty 10 total requests, 1 request at a time
The strategy is to test the per process (user) load on your application from the web page request through completion. If you can identify how much ram Apache, PHP, and MySQL uses for each request, then you can quickly identify your system capacity.
You'll probably have to use a mix of diagnostic tools like vmstat or top or iostat or ps, etc. to take a snapshot of what a number of requests will demand from your system.
Finally, you are going to want to install Xdebug. This tool will help you profile the php side of the application.
http://xdebug.org/
Here is IBM's tutorial on installing Xdebug:
http://www.ibm.com/developerworks/opensource/library/os-php-fastapps2/