Error
I have a web app with a mass uploader (Plupload) for photos and when I upload say twenty photos, about six (around 30 %) will fail with an Internal Server Error. I have checked the Apache error.log for this domain and it has nothing new (I know I'm looking at the right error.log since older errors did show here).
This only happens on my VPS on Dreamhost (my hosting provider) servers while on my development server it runs silky smooth.
Oh, and things used to work just fine a month ago and then just started to fail. Back then I was using Uploadify and since that used Flash, it was impossible for me to debug where the upload failed.
Files and script
Uploaded files are photos, all about 100 kB big, even though I've successfully uploaded (and still can) 3 MB photos. My .htaccess naturally doesn't change during uploads. On the server side is a PHP script that uses GD2 library to move and resize the photo.
Server state
I have recently upgraded my VPS from 300 to 400 MB of RAM. This thing used to work and I upgraded it just so that memory is ruled out as a reason. Also my memory limit for PHP is at 200 MB, so this should sufice.
I am getting mighty frustrated that Dreamhost does not want to help, stating that "we can not be responsible for an error your code causes" and "We still will not be able to assist you in debugging the issue unfortunately."
It has been a week of sparse "support" while my app doesn't work and my clients are frustrated.
Questions
Is this kind of "You're on your own" support a standard across the
industry, i.e. would your host handle this differently?
How exactly can I debug this?
I'm going to assume that you have a standard Apache + PHP setup. One possible configuration is the pre-forked setup; in this case Apache will adapt to system load by forking more children of itself.
With only 400 MB of RAM you're pretty tight, so if you're running 20 processes that each take 200MB (assuming every process handles pretty big files using GD) you're getting into some hot waters with the memory manager.
I would reduce the total number of instances to 2 first to see how this will go; also keep an eye on the memory usage by running top.
Regardless, it might be beneficial for you to run a separate task manager such as Gearman to perform resize tasks so that the upload only has to focus on moving the uploaded file and run the resize task; this way you can greatly reduce the memory required to run your PHP instances.
As to your Q1: the simple answer is that you get what you pay for. A 300Mb RAM Dreamhost VPS costs ~$360 per annum. For this you get the VPS service and responses on service failure relating to the provision of the virtual environment. The OS, the software stack and the applications are outside this service scope. Why? This sort of custom knowledge-base support could cost $50-300 per hour. You are being unreasonable and deluding yourself if you expect Dreamhost to provide such services pro-bono. That's what sites like this one do.
So my suggestion is that you suck up that anger and frustration and work out how to help yourself.
As to your Q2. (i) You need to understand where your Apache errors go to; (ii) Ditto any SQL errors if you are using a D/B. (iii) You need to ensure that PHP error logging is enable and verify where the PHP logs are going to. (iv) You need to inspect those logs, and verify that logging is working correctly, by using a small script which generates runtime errors.
You should also consider using enhanced facilities such as php_xdebug to enhance logging levels and introducing application logging.
In my experience systems and functions rarely die silently. However, applications programmers often ignore return statuses, etc. For example in the GD library, imagecopyresized() can fail and it returns a status code to tell the application when it has, but if the application doesn't test this status and act accordingly then it can end up going down bizarre execution paths silently, and just appear to the user (or developer) as "it just stopped working".
My last comment is that you should really consider setting up a private VPS within your development environment which mirrors your Dreamhost production config, and use this for integration, acceptance test and support. This is pretty easy to do and you can mess this and add debug / what if options and then roll back without polluting your production environment. Tools like VMare Appliances and VirtualBox make this easy. See this blog post for a description of how I did this for my hosted service.
trying to respond the question 2: if you checked all your code and you didn't see any bug, I thing that the best thing that you can do is to check the version of all the programs running on the server (apache, php, ...), e.g., I remember that I had a problem with a web service it was running on apache and php, the php version was 5.2.8, and after a lot of investigation I found out that that version had a problem parsing xml data.
Regarding the first part of the question: Dreamhost do offer a paid support service with "call back". We used this once to get the low down on something. They are very good with general support (better than many hosts IMO) but you can't expect dedicated service, and they must handle a lot of piddling questions. But pay for a call back and, in about 2 minutes on the phone, you can get the answer you want, plus they get their $10 (recurring) for the time. You both win. Just remember to cancel the recurring charges.
Regarding the second part of the question, we had this very same issue with them. Their response (as suggested by Linus in the comments) was that they keep a tally of the CPU use of all processes used by your "user". If that total exceeds a threshold, they will simply kill the process(es) to get the cycles down. No error messages, no warnings, no nothings. Processes can include MySQL, CGI (perl) or PHP. No way to monitor or predict, and we couldn't program round it. Solution... not DreamHost, unfortunately. (webhostingtalk.com will give you loads of host ideas). So we use for some sites, but not for others.
Related
By no means, NewRelic is taking the world by storm with many successful deployments.
But what are the cons of using it in production?
PHP monitoring agent works as a .so extension. If I understand correctly, it connects to another system aggregation service, which filters data out and pushes them into the NewRelic cloud.
This simply means that it works transparently under the hood. However, is this actually true?
Any monitoring, profiling or api service adds some overhead to the entire stack.
The extension itself is 0.6 MB, which adds up to each php process, this isn't much so my concern is rather CPU and IO.
The image shows CPU Utilization on a production EC2 t1.micro instances with NewRelic agent (top blue one) and w/o the agent (other lines)
What does NewRelic really do what cause the additional overhead?
What are other negative sides when using it?
Your mileage may vary based on the settings, your particular site's code base, etc...
The additional overhead you're seeing is less the memory used, but the tracing and profiling of your php code and gathering analytic data on it as well as DB request profiling. Basically some additional overhead hooked into every php function call. You see similar overhead if you left Xdebug or ZendDebugger running on a machine or profiling. Any module will use some resources, ones that hook deep in for profiling can be the costliest, but I've seen new relic has config settings to dial back how intensively it profiles, so you might be able to lighten it's hit more than say Xdebug.
All that being said, with the newrelic shared PHP module loaded with the default setup and config from their site my company's website overall server response latency went up about 15-20% across the board when we turned it on for all our production machines. I'm only talking about the time it takes for php-fpm to generate an initial response. Our site is http://www.nara.me. The newrelic-daemon and newrelic-sysmon services running as well, but I doubt they have any impact on response time.
Don't get me wrong, I love new relic, but the perfomance hit in my specific situation hit doesn't make me want to keep the PHP module running on all our live load balanced machines. We'll probably keep it running on one machine all the time. We do plan to keep the sysmon stuff going 100% and keep the module disabled in case we need it for troubleshooting.
My advice is this:
Wrap any calls to new relic functions in if(function_exists ( $function_name )) blocks so your code can run without error if the new relic module isn't loaded
If you've multiple identical servers behind a loadbalancer sharing the same code, only enable the php module on one image to save performance. You can keep the sysmon stuff running if you use new relic for this.
If you've just one server, only enable the shared php module when you need it--when you're actually profiling your code or mysql unless a 10-20% performance hit isn't a problem.
One other thing to remember if your main source of info is the new relic website: they get paid by the number of machines you're monitoring, so don't expect them to convince you to not use it on anything less than 100% of your machines even if it not needed. I think one of their FAQ's or blogs state basically you should expect some performance impact, but if you use it as intended and fix the issues you see from it, you should recoup the latency lost. I agree, but I think once you fix the issues, limit the exposure to the smallest needed number of servers.
The agent shouldn't be adding much overhead the way it is designed. Because of the level of detail required to adequately troubleshoot the problem, this seems like a good question to ask at https://support.newrelic.com
We're running a magento web store on Knownhost (VPS).
Most of the time the site works fine. Occasionally (every few hours?) the site will get very slow and unresponsive, and will throw '500 Internal Server Errors'. There doesn't seem to be anything relevant in the webserver or Magento system/exception logs.
Also, it seems that we're seeing high CPU usage on this account.
I have increased the memory limit to 512MB, and tried everything else I could find. No dice.
We have a managed VPS, so we can change pretty much everything. We had our hosted provider install ImageMagick after reading a suggestion online - didn't help.
Any ideas?
(website is available at myerstownsheds.com if anyone would like a look)
TL;DR; You have an under resourced server. Any code or configuration steps you take to reduce load are only going to postpone the inevitable.
It's impossible to provide a concrete answer to your question with the information given. If you could look at your server logs and see the full error message being generated it would be a big help. "Server logs" probably mean "Apache Logs" in this situation, since the error text you provided is a standard Apache/PHP error, and not a Magento error.
All that said, the most likely culprit is a PHP out of memory error. Magento's performance profile is different than most LAMP stack applications, and most generic VPS hosts are unable/unwilling to make the tweaks needed to run it. If you want to solve this problem long term you need a web host that specializes in Magento. I recommend Nexcess (affiliate link) these days, but Magento has a list of recommended hosting partners, and the Magento Speed Test site offers a nice breakdown of the top Magento hosts.
Take a look at your host's plans
The highest level plan tops out at 4GB of RAM (4096 MB).
Take a look at the starting Nexcess plans
The entry level plan provides 16GB. Four times as much RAM as your current host. Magento is a RAM hungry application. Your current host isn't equipped to handle Magento. Any code or configuration steps you take to reduce load are only going to postpone the inevitable.
I followed the instructions posted by allendar:
Backup and delete app/etc/local/local.xml
Go to site in browser, and follow the configuration process
So far, everything seems to be working fine! It's a little hard to say this soon since the issue was so intermittent, but our site has remained responsive for nearly two hours.
I'm going to mark this as Answered in a couple days. I'll look into getting a better hosting plan.
Thanks everyone!
I am designing a file download network.
The ultimate goal is to have an API that lets you directly upload a file to a storage server (no gateway or something). The file is then stored and referenced in a database.
When the file is requsted a server that currently holds the file is selected from the database and a http redirect is done (or an API gives the currently valid direct URL).
Background jobs take care of desired replication of the file for durability/scaling purposes.
Background jobs also move files around to ensure even workload on the servers regarding disk and bandwidth usage.
There is no Raid or something at any point. Every drive ist just hung into the server as JBOD. All the replication is at application level. If one server breaks down it is just marked as broken in the database and the background jobs take care of replication from healthy sources until the desired redundancy is reached again.
The system also needs accurate stats for monitoring / balancing and maby later billing.
So I thought about the following setup.
The environment is a classic Ubuntu, Apache2, PHP, MySql LAMP stack.
An url that hits the currently storage server is generated by the API (thats no problem far. Just a classic PHP website and MySQL Database)
Now it gets interesting...
The Storage server runs Apache2 and a PHP script catches the request. URL parameters (secure token hash) are validated. IP, Timestamp and filename are validated so the request is authorized. (No database connection required, just a PHP script that knows a secret token).
The PHP script sets the file hader to use apache2 mod_xsendfile
Apache delivers the file passed by mod_xsendfile and is configured to have the access log piped to another PHP script
Apache runs mod_logio and an access log is in Combined I/O log format but additionally estended with the %D variable (The time taken to serve the request, in microseconds.) to calculate the transfer speed spot bottlenecks int he network and stuff.
The piped access log then goes to a PHP script that parses the url (first folder is a "bucked" just as google storage or amazon s3 that is assigned one client. So the client is known) counts input/output traffic and increases database fields. For performance reasons i thought about having daily fields, and updating them like traffic = traffic+X and if no row has been updated create it.
I have to mention that the server will be low budget servers with massive strage.
The can have a close look at the intended setup in this thread on serverfault.
The key data is that the systems will have Gigabit throughput (maxed out 24/7) and the fiel requests will be rather large (so no images or loads of small files that produce high load by lots of log lines and requests). Maby on average 500MB or something!
The currently planned setup runs on a cheap consumer mainboard (asus), 2 GB DDR3 RAM and a AMD Athlon II X2 220, 2x 2.80GHz tray cpu.
Of course download managers and range requests will be an issue, but I think the average size of an access will be around at least 50 megs or so.
So my questions are:
Do I have any sever bottleneck in this flow? Can you spot any problems?
Am I right in assuming that mysql_affected_rows() can be directly read from the last request and does not do another request to the mysql server?
Do you think the system with the specs given above can handle this? If not, how could I improve? I think the first bottleneck would be the CPU wouldnt it?
What do you think about it? Do you have any suggestions for improvement? Maby something completely different? I thought about using Lighttpd and the mod_secdownload module. Unfortunately it cant check IP adress and I am not so flexible. It would have the advantage that the download validation would not need a php process to fire. But as it only runs short and doesnt read and output the data itself i think this is ok. Do you? I once did download using lighttpd on old throwaway pcs and the performance was awesome. I also thought about using nginx, but I have no experience with that. But
What do you think ab out the piped logging to a script that directly updates the database? Should I rather write requests to a job queue and update them in the database in a 2nd process that can handle delays? Or not do it at all but parse the log files at night? My thought that i would like to have it as real time as possible and dont have accumulated data somehwere else than in the central database. I also don't want to keep track on jobs running on all the servers. This could be a mess to maintain. There should be a simple unit test that generates a secured link, downlads it and checks whether everything worked and the logging has taken place.
Any further suggestions? I am happy for any input you may have!
I am also planning to open soure all of this. I just think there needs to be an open source alternative to the expensive storage services as amazon s3 that is oriented on file downloads.
I really searched a lot but didnt find anything like this out there that. Of course I would re use an existing solution. Preferrably open source. Do you know of anything like that?
MogileFS, http://code.google.com/p/mogilefs/ -- this is almost exactly thing, that you want.
I currently have more than 1000 visits / day and hosting at 1and1 with PHP memory 30MB limit
I have a dynamic shopping guide with more than 5000 items and users enter to browse / search for items. I started getting "Internal Error 500" every now and then. Which are show more on days, and I don't notice them on others. 1and1 support say that I have outgrown the 30MB PHP limit.
What do you think? Is that true? or they just want to sell me a more expensive hosting? I currently can't afford more than a shared host :(
I am using PHP / MySQL Javascript / My BB Forum / PHP thumbail (which I am now trying to switch with static thumbnails to ease the load a bit)
Advice is appreciated
It's very unlikely that disk space constraints are causing 500s, even with 5000 items over 30mb, that's roughly 6k per item - only probably if each item has an image.
When you reach your disk quota, most PHP frameworks will report an error themselves, rather than sending a bog standard 500 response, so the 30Mb limit is almost certainly not the problem. Your hosting company may have other limits they're imposing, but even 1000 visits a day shouldn't break the most draconian hosting thresholds. It's more likely that your hosting company are looking for an easy up-sell.
Give us a URL and maybe we can get a clearer picture, but it's more likely something is simply broken in your site's code. Of course, without any reference material, this is strictly a theory.
If you are bumping up against 30 megabytes of PHP memory (if you are in fact talking about the php memory_limit setting, which it seems you are), then most likely some optimization could solve it. I usually only hit that kind of number if I'm processing large arrays and/or converting to PDF files, etc. And then I usually figure out a way to drop the array size by using memcached or writing the PDF some other way or something like that.
Sometimes the easiest thing to do is to buy their upgrade. Throwing hardware at the problem can at least be a quick fix. If you are making money from your web site, any time you get a 500, you could be losing money so it might be wise financially for you. But, you didn't ask that question. :)
To determine how much memory your scripts are using, sprinkle http://us2.php.net/memory_get_usage around at strategic spots to see (assuming you don't have a profiler that can handle this better).
I have moved to another hosting "Blue Host" and the errors stopped ever since. It seems they handle the resources in a different way: "CPU throttling". And they told me I can use their services till I feel the website is slow and I then should switch to VPS or dedicated server. 1and1 handle the overload on shared limits with error 500. This is not good!
In the next few weeks I'll be taking my site from the localhost (WAMP) and puting it on a new server. This will be the first site, on my first server, so basically...i'm a noob!
This must be an important moment for any independent web developer / small business so i'd love to hear about some experiences, mistakes and system default security holes that one should fix straight away...
I'm using php, mysql, cpanel and WHM, and looking for tips like "Turn off error reporting in PHP"
First and foremost if you are worried about security then you should use LAMP. As long as the Linux platform is using AppArmor or SELinux (Ubuntu and fedora respectively), then you are much better off than any version of Windows. I know this from first hand experience of developing exploit code for the two platforms.
Before you lock your system down, test your code for vulnerablites using Wapiti. Acunetix is also good, but expensive. This type of testing, especially sql injection testing must be done with dispaly_errors=On set in your php.ini
There is a lot that can go wrong with PHP Configuration that makes your system less secure. You should run PHPSecInfo and remove all red. dispaly_errors=Off is what you want, and phpsecinfo tests for it.
You should also use a web application firewall like Mod_secuirty.
It's actually quite a huge undertaking, but well worth the experience. Here are just one or two suggestions...
Site security also means being heavily involved in managing your sometimes scarce resources. Just as important is obeying any limits your host has, and guessing all possible ways your site users can push you over those limits, leaving you responsible to pay a hefty bill. IE downloading or uploading large files over and over, spamming email lists, repeatedly requesting pages using too many database connections and queries, etc. Get overusage limits and fees in writing from your host before you begin, and have response plans ready. Really, this part is like buying a cellphone service.
A lot would also depend on what features you'll have on your site. File uploads? Forum? Logins? Email? Etc? For example - If you're running a file-sharing site: along with upload/download rate limiting, I suggest you first check available disk space before permitting any file to be uploaded, or do regular audits so you're prepared to archive or delete old and unused files. It's a quick check just to make sure you're not caught by surprise a year down the road when you suddenly start getting disk full errors or get shafted by your host with a large bill.
There are literally a hundred more issues to consider. Gather up a complete overview - an itemized list - of all features and functions of your site. Google each one to get more ideas on handling security. Your host should also publish their own security considerations and have a handy manual for operating with all of their services. If they don't, well, I wouldn't personally feel comfortable with them.