I wonder how my optimal MySQL server configuration would look like when I am using an Amazon Cloud Server which is being adapted from 8GB RAM to 144GB Ram according to the traffic peaks. I don't want to change the MySQL configuration every time I change the hardware resources of my server. Is there a suitable configuration for things like this?
Further I wonder which configurations are especially important for my case:
It is a Forum with 400-2500 live users online (Analytics real time analysis)
We got an authentication and session-check (two php files, which do 1-2 small SQL queries) which is being called 40,000 user x 4times per hour (I mentioned this because I worry about cached results = session check for this user may fail?)
In the past our database server went offline at peak times before all resources (CPU/RAM/Disk I/O, Bandwidth) has been used completely. Thus I think that we use a bad configuration currently. What is about the property max_connections , this is currently 500. Would it help to set it up to something like 2000 for servers of my size?
Should I use a perssistant PDO connection for the session-check / authentication?
Related
My client currently has only one server with both MySql and Apache running on it, and at busy times of the year they're occasionally seeing Apache fall over as it has so many connections.
They run two applications; their busy public ecommerce PHP based website and their (busy during working hours only) internal order processing type application with 15-20 concurrent users.
I've managed to get them to increase their budget enough to get two servers. I'm considering either:
A) one server running Apache/PHP and the other as a dedicated MySQL server, or
B) one running their public website only, and the other running MySQL and the internal application.
The benefit I see of A) is that Mysql my.cnf can be tuned to use all of the resources of that server, but it has the drawback of only having one Apache instance running.
B) would spread the load on Apache across both servers, but would limit MySQL's resources on that server, even out of working hours when the internal application won't be used.
I just can't decide which way to go with this and would be grateful of any feedback you may have.
Both approaches are wrong.
You have 2 goals here; availability and performance (I'm considering capacity to be an aspect of performance in this context).
To improve availability, you should be ensuring that there is no single point of failure in your architecture. But with the models you propose, you're actually creating multiple single points of failure - hence your 2 server models are less available than your single server.
From a performance point of view, you want to spread the workload across the available resources. You can't move CPU and memory between the servers but you can move the traffic.
Hence the optimal solution is to run both applications on both servers. Setting up MySQL clustering is a bit more complex, but probably the out-of-the-box asynch replication will be adequate - with the nodes configured as master-master (but writes from the 2 applications targeted sensibly).
There's probably a lot of scope for increasing the capacity of the system further but without a lot more detail (more than is appropriate in this forum, and possibly more than your client is comfortable payng for) it is hard to advise.
I have 9 databases on an MYSQL server right now. I configured them using the command line, adding a user and giving that user only full permissions to a specific database. The mysql server has 512MB ram right now, and i don't know if i should be worried security wise of any problems that might arise with 9 databases on one server. Should I split it up into two servers each with about 4 to 5 at most? I have 2 other app servers running to handle the load of the websites, but those 2 servers hit the database server for everything. So far, no problems. I have all 3 servers set with IP restrictions (iptables and other firewall), so hacking from elsewhere isn't possible, but only from the apps themselves.
Since I created the users each with a restriction to a specific database, a hacker who hacks one can't get to the rest, i assume?
Thanks!
The mysql server has 512MB ram right now, and i don't know if i should
be worried security wise of any problems that might arise with 9
databases on one server.
Is the server cup running at 100% all the time?
Are there a lot of slow queries?
This could indicate that the server needs more resources.
You can also check the size of the InnoDB Buffer Usage. Increasing this is often a good way to relieve some pressure.
I have 2 other app servers running to handle the load of the websites,
but those 2 servers hit the database server for everything. So far, no
problems. I have all 3 servers set with IP restrictions (iptables and
other firewall), so hacking from elsewhere isn't possible, but only
from the apps themselves
This is good. That way no one can access your db server directly only through the app servers.
Since I created the users each with a restriction to a specific
database, a hacker who hacks one can't get to the rest, i assume?
Correct.
I am currently using an AWS micro instance as a web server for a website that allows users to upload photos. Two questions:
1) When looking at my CloudWatch metrics, I have recently noticed CPU spikes, the website receives very little traffic at the moment, but becomes utterly unusable during these spikes. These spikes can last several hours and resetting the server does not eliminate the spikes.
2) Although seemingly unrelated, whenever I post a link of my website on Twitter, the server crashes (i.e.,Error Establishing a Database Connection). Once restarting Apache and MySQL, the website returns to normal functionality.
My only guess would be that the issue is somehow the result of deficiencies with the micro instance. Unfortunately, when I upgraded to the small instance, the site was actually slower due to fact that the micro instances can have two EC2 compute units.
Any suggestions?
If you want to stay in the free tier of AWS (micro instance), you should off load as much as possible away from your EC2 instance.
I would suggest you to upload the images directly to S3 instead of going through your web server (see some example for it here: http://aws.amazon.com/articles/1434).
S3 can also be used to serve most of your web pages (images, js, css...), instead of your weak web server. You can also add these files in S3 as origin to Amazon CloudFront (CDN) distribution to improve your application performance.
Another service that can help you in off loading the work is SQS (Simple Queue Service). Instead of working with online requests from users, you can send some requests (upload done, for example) as a message to SQS and have your reader process these messages on its own pace. This is good way to handel momentary load cause by several users working simultaneously with your service.
Another service is DynamoDB (managed NoSQL DB service). You can put on dynamoDB most of your current MySQL data and queries. Amazon DynamoDB also has a free tier that you can enjoy.
With the combination of the above, you can have your micro instance handling the few remaining dynamic pages until you need to scale your service with your growing success.
Wait… I'm sorry. Did you say you were running both Apache and MySQL Server on a micro instance?
First of all, that's never a good idea. Secondly, as documented, micros have low I/O and can only burst to 2 ECUs.
If you want to continue using a resource-constrained micro instance, you need to (a) put MySQL somewhere else, and (b) use something like Nginx instead of Apache as it requires far fewer resources to run. Otherwise, you should seriously consider sizing up to something larger.
I had the same issue: As far as I understand the problem is that AWS will slow you down when you reach a predefined usage. This means that they allow for a small burst but after that things will become horribly slow.
You can test that by logging in and doing something. If you use the CPU for a couple of seconds then the whole box will become extremely slow. After that you'll have to wait without doing anything at all to get things back to "normal".
That was the main reason I went for VPS instead of AWS.
I'm working on a PHP/CodeIgniter web app that will be the backend for a non-realtime game. We want the ability to record game activity for later analysis. In my performance tests using either codeigniter's own logging system or log4php, file logging seems slow, reducing the number of requests per second the server can handle by 50%. I've tried it on both a WAMP machine and an Apache/Ubuntu server. If I change logging to use MongoDB, the performance only drops by a few percent, even if I'm logging the same amount of information.
Is file logging going to be inherently slow for php scripts because they are all waiting on locks on the same file or is it likely a configuration issue?
You can try logging in RAM drive file(s).
Also consider naming of the logs by a date stamp like YYYY-mm-dd-HH.log so you can periodically take older logs and process (archive) them and the drive will stay clean.
I am designing a file download network.
The ultimate goal is to have an API that lets you directly upload a file to a storage server (no gateway or something). The file is then stored and referenced in a database.
When the file is requsted a server that currently holds the file is selected from the database and a http redirect is done (or an API gives the currently valid direct URL).
Background jobs take care of desired replication of the file for durability/scaling purposes.
Background jobs also move files around to ensure even workload on the servers regarding disk and bandwidth usage.
There is no Raid or something at any point. Every drive ist just hung into the server as JBOD. All the replication is at application level. If one server breaks down it is just marked as broken in the database and the background jobs take care of replication from healthy sources until the desired redundancy is reached again.
The system also needs accurate stats for monitoring / balancing and maby later billing.
So I thought about the following setup.
The environment is a classic Ubuntu, Apache2, PHP, MySql LAMP stack.
An url that hits the currently storage server is generated by the API (thats no problem far. Just a classic PHP website and MySQL Database)
Now it gets interesting...
The Storage server runs Apache2 and a PHP script catches the request. URL parameters (secure token hash) are validated. IP, Timestamp and filename are validated so the request is authorized. (No database connection required, just a PHP script that knows a secret token).
The PHP script sets the file hader to use apache2 mod_xsendfile
Apache delivers the file passed by mod_xsendfile and is configured to have the access log piped to another PHP script
Apache runs mod_logio and an access log is in Combined I/O log format but additionally estended with the %D variable (The time taken to serve the request, in microseconds.) to calculate the transfer speed spot bottlenecks int he network and stuff.
The piped access log then goes to a PHP script that parses the url (first folder is a "bucked" just as google storage or amazon s3 that is assigned one client. So the client is known) counts input/output traffic and increases database fields. For performance reasons i thought about having daily fields, and updating them like traffic = traffic+X and if no row has been updated create it.
I have to mention that the server will be low budget servers with massive strage.
The can have a close look at the intended setup in this thread on serverfault.
The key data is that the systems will have Gigabit throughput (maxed out 24/7) and the fiel requests will be rather large (so no images or loads of small files that produce high load by lots of log lines and requests). Maby on average 500MB or something!
The currently planned setup runs on a cheap consumer mainboard (asus), 2 GB DDR3 RAM and a AMD Athlon II X2 220, 2x 2.80GHz tray cpu.
Of course download managers and range requests will be an issue, but I think the average size of an access will be around at least 50 megs or so.
So my questions are:
Do I have any sever bottleneck in this flow? Can you spot any problems?
Am I right in assuming that mysql_affected_rows() can be directly read from the last request and does not do another request to the mysql server?
Do you think the system with the specs given above can handle this? If not, how could I improve? I think the first bottleneck would be the CPU wouldnt it?
What do you think about it? Do you have any suggestions for improvement? Maby something completely different? I thought about using Lighttpd and the mod_secdownload module. Unfortunately it cant check IP adress and I am not so flexible. It would have the advantage that the download validation would not need a php process to fire. But as it only runs short and doesnt read and output the data itself i think this is ok. Do you? I once did download using lighttpd on old throwaway pcs and the performance was awesome. I also thought about using nginx, but I have no experience with that. But
What do you think ab out the piped logging to a script that directly updates the database? Should I rather write requests to a job queue and update them in the database in a 2nd process that can handle delays? Or not do it at all but parse the log files at night? My thought that i would like to have it as real time as possible and dont have accumulated data somehwere else than in the central database. I also don't want to keep track on jobs running on all the servers. This could be a mess to maintain. There should be a simple unit test that generates a secured link, downlads it and checks whether everything worked and the logging has taken place.
Any further suggestions? I am happy for any input you may have!
I am also planning to open soure all of this. I just think there needs to be an open source alternative to the expensive storage services as amazon s3 that is oriented on file downloads.
I really searched a lot but didnt find anything like this out there that. Of course I would re use an existing solution. Preferrably open source. Do you know of anything like that?
MogileFS, http://code.google.com/p/mogilefs/ -- this is almost exactly thing, that you want.