How can I configure Magento so that:
1) it can handle 10000 customers at a time
2) it can tolerate sudden increase in load
I searched Google but most of articles are explaining to improve Magento load time.
Where should I set the cookies and session expiration time?
I want to achieve this by modifying .htaccess ,php.ini and Magento admin panel setting.
Other ways are also welcome.
I have done extensive testing on this and have found the single biggest thing to do to improve performance is the following.
(all the following links can be found at http://www.magentocommerce.com/)
Make sure Magento Caching is enabled (easy to overlook when you have
it turned off while developing)
Use some sort of full page caching such as
magento-connect/zoom-full-page-cache-1742.html
Use a CDN such as MaxCDN or AWS Cloudfront (using
magento-connect/6274.html makes this pretty easy)
The above make the big improvements. If you need more improvements installing varnish really does the biggest but can be a pain to use since it is normally setup to take over port 80. This makes managing and developing your site later a bit of a pain as you will generally need to disable it or have it bypass varnish to do any major development work.
Install varnish - magento-connect/pagecache-powered-by-varnish.html
or magento-connect/2984.html
Make sure you have APC and Memcached installed
Make sure you have gzip compression turned on.
Advance performance method (these are helpful if using AWS and you want to use multiple servers in different zones)
Install varnish on it's own server and direct the web-server port to
your Magento server. This reduces the number of hits your web-server
sees.
Install your magento database on it's own server. Magento has funny resource requirements memory and cpu wise. What may be good for Magento might not be as good for the web server etc or database. Splitting your database off should be fine if you are one the same local net (i.e. AWS same region). This method allows you to use multiple web servers.
Use a AWS with elastic IP and place webservers in different zones and multiple web servers with a single database server. Use multiple varnish servers infront of the web servers.
Some additional notes:
APC, Memcache, php, using ngix will only provide about a 10% improvement vs just using Magento Cache, full page cache and Varnish. Also before testing make sure you test your server setup using a stock magento install with dummy data. This will help you set a baseline performance to see if hardware changes or need or to identify if a module or other plugin might be causing big performance hits. Sites like loadimpact.com can be helpful.
If you have access to php.ini then I am going to take a wild guess and assume you also have access to your database configuration files (my.conf).
In case you are using MySQL adjusting the query_cache_size parameter can have a tremendous positive effect on Magento performance because its constantly polling a large amount of the same data that gets reused. The exact amount of memory you'll use depends entirely on your needs so you will have to adjust it accordingly.
FPC is the most important point that should be applied.This will reduce your Mysql load considerably.
Solr for search is the second thing that should be done.
Make sure your code is to Magento standards so that it effectively uses magento cache and FPC.
Above three will result in Max optimization.
Look here : Tweaking magento for performance for lots of additional points.
There are so many ways to optimize Magento. Some of the configuration are from Magento admin panel it self.
Go to: System->Configuration->CATALOG/Catalog->Frontend
Use Flat Catalog Category: Yes
Use Flat Catalog Product : Yes
Go to: System->Configuration->ADVANCED/Developer: Merge javascript and CSS
Install memcache on you server
Some of the Helping links:
http://magento2x.com/speed-up-magento/
http://support.metacdn.com/entries/25027521-Slow-Magento-Speed-up-Magento-with-our-Magento-Optimization-guide
Also you can go for Google and find some more terrific configuration for server.
one simple way is to install and config APC
APC
# in php.ini:
extension=apc.so
[apc]
apc.enabled = 1
apc.cache_by_default = On
apc.shm_segments = 1
apc.shm_size = 128M ; memory size when using as nosql
apc.ttl = 60
apc.user_ttl = 7200
apc.gc_ttl = 600
apc.num_files_hint = 0
apc.write_lock = On
;apc.stat = 0 ; enable detecting file stat, reload if changed
I recommend to red this "White Paper: Optimizing Magento for Peak Performance"
This white paper documents the exceptional performance that can be achieved by properly optimizing and configuring Magento Enterprise Edition. The resulting optimization not only can contribute to higher conversion rates and support for greater numbers of customers and orders per day, it can also lead to improved hardware efficiency and overall cost savings.
https://info.magento.com/Optimizing_Magento_for_Peak_Performance.html
Related
I am in the middle of migrating 7 Wordpress sites to one single server which is running Ubuntu and has 4 core CPUs & 15gb ram)
So far I have moved one website and installed APC, but I am unsure if this method of caching supports multiple websites on one server.
Can anyone give me some advice on configuration? Can I just install it and use it 'Out of the box' or do I need to make some tweaks that will increase performance for all 7 websites.
It looks like it currently only uses 30mb ram which doesnt seem like a lot to me.
Cheers,
Rich
You will have to use some apc cache plugin, the most common one in use is http://wordpress.org/extend/plugins/w3-total-cache/.
It's easy to set up and provides a lot more.
W3 Total Cache would be a good choice, and it would pose no problems for APC. For a basic install it is sufficient to just configure the main configuration page.
Typically you would set the options there as follows:
page cache: enable, disk enhanced
minify: disable (you could experiment with this, but maybe causes errors on your site. If you try it, set it to disk)
database cache: enable, apc
object cache: enable, apc
browser cache: enable
CDN: disable
Varnish: disable
Of course there are a lot of other options, with which you can experiment a little to suit your needs. But these settings should give you a solid performance boost.
I would also take a look at the settings in your apc.ini. By default, the amount of memory allocated for the service is not very large. Maybe increase that to 512mb or 1024mb.
I looking for advice on how to set up the default configuration of php.ini and my.cnf for a small site (100 pages) with very little traffic (300 visitors per day). All pages have a bit of text, some images, no video, no audio, no flash/silverlight, very little javascript and jquery. For tracking I'm using GA and Piwik. The main site database is around 50MB.
The site is hosted on a virtual server with 20GB RAM and 6 vCPUs so there's hopefully a lot of muscle to make it run very fast.
I don't know much about tweaking php and mysql settings and would appreciate it if your answers can be as detailed as possible.
Thanks
You don't need any special configuration. Your server is so severely oversized for the task that it really hurts. Any cheap webhosting offer with some PHP and database would suffice, given your access numbers are correct.
If you really grow into areas where your server shows signs of overload, your problems will be so special that any general advice on configuration given today is wrong.
Just follow the recommended default settings for production servers for PHP 5.4 and MySQL, unless you use software that needs them different and states so in its documentation.
I don't think you can tune php.ini and my.cnf that much. You can run mysql tuning primer script: How can I optimize my MySQL server? and https://stackoverflow.com/questions/10820933/ive-run-mysql-tuning-primer-but-i-cant-understand-it but it's difficult to understand. I would suggest to enable slow_query log and examine slow queries. I also suggest to install nginx or lighttpd and fastcgi (php-cgi) with eaccelerator. It's much faster and easier to configure. There are some interesting lighttpd parameter. When you can get a kvm virtualization to get access to the kernel parameters. I also suggest to compile php yourself and configure php to your needs. When you use php from repository I don't think you can get every php configure killswitch as module. Also enable http compression and http cache headers. When you have 20 GB ram install a ramdisk and move temporary folders to the ramdisk.
I need to speed up my magento installation, so I'm planning to put the content of 'var/' (or only var/cache and var/sessions) on a tmpfs.
I'm also buying a reserved instance on Amazon, so I would like to keep a sufficent amount of RAM. I want to enable memcached, PHP Apc, MySQL caching and HTTP caching.
I'm thinking of a Medium Reserved Instance with the following specs:
3.75 GB memory
2 EC2 Compute Unit (1 virtual core with 2 EC2 Compute Unit)
410 GB instance storage
32-bit or 64-bit platform
I/O Performance: Moderate
EBS-Optimized Available: No
API name: m1.medium
Will the RAM be enough to appy a good caching system?
Looking now (after 3 months) the var directory is 14gb, but I think cleaning it up each 5/7 days would be good too.
Do you have any suggestion for me?
P.S. the store will contain an average of 100/150 products.
I think moving /var to a tmpfs is probably not your biggest bottleneck and would likely be more trouble than its worth. Make sure Magento caching is enabled and you have APC enabled.
This post covers some general tips on increasing Magento performance:
Why is Magento so slow?
I would suggest looking into setting up a reverse proxy like Varnish.
Getting Varnish To Work on Magento
If you do plan on just using a tmpfs in memory I would suggest looking into Colin's improved over Zend_Cache_Backend_File
https://github.com/colinmollenhour/Cm_Cache_Backend_File
Also I would suggest looking into mytop to keep tabs of if you have any places you can optimize queries in the application itself or in my.cnf to help ease any DB bottlenecks.
http://jeremy.zawodny.com/mysql/mytop/
Session Digital has a good white paper (although somewhat dated) on optimizing Magento enterprise and the same can be applied to Community. Out of everything I've tried, Varnish, as mentioned in the White paper offered the most significant increase in response time.
http://www.sessiondigital.com/resources/tech/060611/Mag-Perf-WP-final.pdf
Hope this helps!
Firstly, +1 to all of the answers here.
if you're thinking about running /var/ out of tmpfs it's probably because you've heard of the lousy file IO on AWS or you have experienced issues with it yourself. However, the /var/ directory is the least of your concern - Zend / Magento's autoloaders are more taxing to IO. To mitigate that you want to run APC and the compilation (assuming you're not using persistent shopping cart).
As echoed by other commenters, anything that runs from cache or memory will circumvent PHP and thus the need to touch the disk and incur IO issues. Varnish is a bit of a brute-force approach and is a wonderful tool on massive sites that scale to millions of views; but I believe that Varnish's limitations with SSL and the lack of real documentation and support from our Magento community make it a better intellectual choice than an actual alternative.
When running Magento Community I prefer to run Tinybrick's Lightspeed on AWS on a Medium instance - which gives me the most bang-for-buck and is itself a full-page-cache. I get 200+ concurrent pages/second in this setup and I'm not running memcached or using compilation.
http://www.tinybrick.com/improve-magentos-slow-performance.html/
Be careful with running memcached in your AWS instance as well - I find that it can be impeded by a power-hungry Apache gone wild in the rare instance you haven't got a primed cache which causes Apache maxclients issues while it waits for cache response. If you could afford it I would rather run two micro Apache instances with a shared memcached session store and a load balancer in front of them - give some horsepower to the db on a separate box for them to share, though. But all setups are unique and your traffic/usage will dictate what you need.
I have run Magento in the AWS cloud for 3 years with great success - and I wish the same to you. Cheers.
I have made site on drupal
My site has 7500 users and approx (20 to 50 without logged in)(2 to 10 logged in) users are online (and this is not heavy traffic I think)
The site is on dedicated server. I have enabled setting in performance from drupal admin and also installed memcache and eaccelerator
I looked in query logs from using devel module. it is firing total 600 to 900 queries on each page
When I have installed patch of path.inc to reduce the queries of drupal_look_path(). It has reduced queries to around 400
I have also made some positive changes in mysql (my.cnf) file, but still there are many same queries run form user_load() function again and again
I have 60 to 70 modules enabled and all are use full. I can't remove the modules
Still the site is running slow it is taking approx 10 to 15 sec
Now I don't know why the site is running so slow
Is it because the drupal has the large php code ?
Is it because it is firing so many queries on each page?
Does the InnoDB engine improve the performance?
Please, any kind of suggestions are welcome
400 queries for each requests is a sucidie (but even 50+).
You should implement some html cacher. My website generally doens't even make the db connection. It just fires the html cached in a file.
Some additional things to look into:
Install a tool like Yslow/PageSpeed to see how much of those 10-15s are client and server time.
Instal XhProf (on a development site, not live) together with Devel to see which are the functions that use the most time. Look into these first. Edit, now with link: http://groups.drupal.org/node/82889
Using pressflow might help a bit, but since you are alrady using the path.inc patch, probably not so much.
You mentioned that you installed memcache. Did you also install the memcache module and configure the cache plugin to use memcache?
EDIT: Yes, switching to InnoDB can help. One of the main performance advantages of InnoDB is row-level locking (as opposed to table-level locking of MyISAM), which means that multiple INSERT/UPDATE queries against the same table won't block each other unless really necessary. However, InnoDB does not perform well at all out of the box, you really need to fine-tune your mysql configuration for your specific site. So this is a step that you should only take carefully and after testing and optimizing on a development site. There are various questions already on this site and elsewhere about InnoDB tuning...
Anything else than that is then site specific and depends on the modules you are using. But especially things like complex node_access setups and multiple languages (i18n!) tend to either cause slow queries and/or a lot of them.
Not all modules make use of the caching mechanisms you can switch on in the performance settings area. It would be worth trying to identify which ones are doing the most/slowest queries and attempting to get the developer(s) to improve them.
Alternatively, examine whether you could achieve things with fewer modules. Some modules do overlap somewhat in functionality, so you may be able to reorganise the way the site functions a bit.
Additionally, you need to look at whether your settings MySQL are allowing enough memory for these queries to be carried out. Most MySQL distributions come with different versions of my.ini labelled 'small', 'medium', 'huge' etc. Copy the 'huge' one to my.ini (back up the old one first) and restart the DB to see if maxing out all the cache sizes makes a difference. You may well have a bottle neck there, but it can be hard to work out what setting is causing it.
Same goes for PHP. Set memory_limit in php.ini to 500MB or something and see if it helps. Of course, you may not be able to do this, depending on your hosting arrangements, but it will eliminate one possible cause (or not) if you can.
Performance of your Drupal website also depends on how well your hosting platform is tuned for Drupal. Drupal requires special optimization of LAMP stack components. You can try Drupal-specific hosting companies http://www.drupalspecific.com to make your website run faster.
facing the drupal slowness issue myself. But have a very different issue than the others mentioned.
I disabled all the content also the drupal header for a drupal page of a specific content type.
Still the time taken by this page to load is above 20 secs!
I took help of YSlow and NET firebug panels.
Upon looking at them, noticed:
JS and CSS files inclusion individually takes 3 to 2 secs, and there are fair bit of inclusions happening, as a result it takes like 20 secs.
But i am not able to figure out, why the js and css inclusions are taking so much time. (this includes normal drupal core js and css files as well)
I have an installation of 1.3.2.4 running two Store Views and 2,734 products. The site sees around 15,000 visits a month.
Apache and MySQL (mostly Apache) hovers at around 1.5 GB RAM usage most of the time and peaks over 3 GB. My questions is, considering the stats, is this normal? Seems like a lot.
If that memory usage is in fact abnormal, would an upgrade to 1.4.1.1 help?
If you consider your stores, then you are doing just fine. But regarding the traffic you're getting, it seems that you need to provide some extra features to Magento to let it fire up. For this, you can have some of the following:-
Install APC (Alternative PHP Cache) or XCache (or any other alternative) and configure the use of it in your Magento back-end. It dramatically increases the speed of Magento.
You can have Magento's cache stored in memory (tmpfs in Linux).
You can also tell Magento to save sessions into Memcache so that your sessions are in memory & distributed.
Check your Magento's Index Management section for any requirement of indexes, every month or bi-monthly. If you do find any indexing required, then do it immediately & clear the cache from your Cache Management.
Check your database every week or bi-monthly for any overhead in any of your database's tables. If you do find any overhead, then "optimize" those tables immediately.
Try reading some of these articles, to know more about these.
Also, upgrading to 1.4.1.1 will help you out in terms of features provided by Magento. But for performance, I think it will be best to wait for some more time, until Magento releases its version 2 in the market, in which some performance issues may be taken care of by Magento.
Hope it helps.
1.3.2.4 is a good stable release, upgrading to 1.4.0.1 is very painless and will give you the added benefit of split index management and much faster administration area (mass attribute update is fixed).
Don't be overly concerned about memory usage, depending on the number of Apache modules you have loaded, you should expect to see about 30MB per child. As long as your not swapping or encroaching your limits, you shouldn't have any real concerns over how much is being consumed. Disabling unused modules will help cut down memory - but to be honest, not by any noticeable margin.
You could always throw Nginx in front as a reverse proxy to serve static content requests and pipe PHP/dynamic reqs. back to Apache. That way you can keep the modular Apache build with .htaccess support and cut down your memory overheads significantly.
However, this could do with more information, such as the output from
free -m
To see how some of the memory is being allocated.
I'd probably suggest downloading tuning-primer.sh to run on your MySQL config. It will give a good (entry-level) indication of how efficient your memory allocation is.
Those stats look quite typical for Magento, if you consider a single hit/page load can use upwards of 64MB RAM.
Your Apache settings can also drastically effect the amount of RAM your system uses. Upgrading your Magento installation may give some small performance boost, but don't expect it to do much for memory consumption etc.
If your memory consumption is a real issue for you then you have several possible routes to reduce resource usage, such as:
Install Nginx as a reverse caching proxy to apache (apache is a hog and is poor serving static content).
Use Nginx + PHP Fast CGI and remove apache
Try using worker MPM module for apache, or Fast CGI.
Install caching proxy such as Varnish/Squid.
If you are stuck with apache you can tweek KeepAlive and other settings to allow you to reduce memory usage
Tweek MySQL settings, such as query caching to imporove resource usage / performance
I have found 1. to work very well in reducing cpu/memory usage as it will allow Nginx to serve static images etc without requiring apache to hog RAM trying to serve them.