Here Google offers different tiers of their google-cloud-sql
I dont understand when someone will need to upgrade the very basic d0 tier.
My questions are:
1) If you are connecting GAE to cloud-sql, will the sql concurrent connections limit the scalability of your GAE app to 250 concurrent requests? I mean, will GAE create a new connection to cloud-sql on every request?
1bis) Can a very requested GAE app use only one sql connection?
2) Could you give some case-scenarios when Dx may be recomendable?
what i dont understand is when someone will need to upgrade the very
basic d0 tier.
When its performance proves insufficient for your workload (number and size of queries) resulting in too-slow responses to user queries (or back-end tasks). https://cloud.google.com/sql/docs/instance-info explains how to view all the info about a given Cloud SQL instance.
1) if you are connecting GAE to cloud-sql, will the sql concurrent
connections limit the scalability of your GAE app to 250 concurrent
requests ? i mean, will GAE create a new connection to cloud-sql on
every request ?
Actually, your PHP code will do that, e.g with a call such as
$sql = new mysqli( ... etc, etc
if and when it needs a Cloud SQL connection to serve a request. I do not believe there can be any way to share a single connection among different servers (and multiple concurrent requests are typically served by different servers -- although if your code is threadsafe a single server might be responding to a few requests concurrently, and I guess you could try to share a single connection among threads with locking, though that might impact latency and would only give you a small amount of connection-reuse anyway).
1bis) can a very requested GAE app use only one sql connection ?
A "very requested GAE app" is no doubt going to be using multiple servers at once, and there is no way separate servers can share 1 mySql connection.
2) could you give some case-scenarios when Dx may be recomendable ?
You'll just want larger instances in proportion to how big/demanding your workload is -- larger databases and indices, big/heavy requests including ones processing or returning lots of data, many concurrent requests, heavy background "data mining" going on at the same time, and so forth.
I would recommend using the calculator at https://cloud.google.com/products/calculator/ -- click on the Cloud SQL icon if that's specifically what you want to explore -- to determine expected monthly costs for an instance.
As for the performance you can expect in return, that's so totally dependent on your data, indices, workloads, &c, that there's really no shortcut for it: rather, I recommend building a minimal meaningful sample of your app's needs and a stress-load test for it, tune it first on a local MySQL installation, then deploy experimentally to Cloud SQL in different configurations to measure the effects.
Once you've gone to the trouble of building and calibrating such benchmarks, you may of course also want to try out other competing providers of "mysql in the cloud" services, to know for sure exactly what performance you're getting for your money -- I'm unfortunately not very knowledgeable about what all is available on the market, but my key message is to use your own benchmarks, built to be meaningful for your app, rather than relying on "canned" benchmarks...
Related
I have this scenario: I have a PhP 5.6 application that connects to a MariaDB (edit: I stated MySQL before but is not correct) instance.
In this instance I have a "master" DB and a DB for each firm using the application. Each new firm -> a new DB.
The number of DBs is growing exponentially and we are considering a way to split these client's DBs between multiple MariaDB servers.
The problem is that there are many queries that join between client's DB and master DB, so we cannot blindly just connect to another Host.
Is it possible to setup a MariaDB instance that have databases in other hosts, but still "virtually sees them" as on the same instance (so that cross DB queries still work) ?
I tried to google this without success.
Thank you in advance!
EDIT: I've found out about Federated Tables that might be the solution.
Question is, we like to split DBs between servers because we might have a number of DBs in the range of 50.000-100.000 and we fear about performances.
If I create these DBs locally all with federated tables, will this solve my issue or are we still facing performance issues?
Sounds like you need FederatedX to get to the common database from the client dbs.
And "sharding" is putting several clients on each of several servers. Not one client on each of 50K servers.
Or is there some requirement you have not provided yet?
How big are these "clients"? How big is the largest? Are there security concerns if multiple clients live on the same server? In the same MariaDB instance? In the same table?
Have you already set up some form of Proxy to route a client request to the server for that client? Or are you expecting the client to own his server?
Etc, etc.
Keep in mind that reaching into a 'federated' table is slow, in some cases orders of magnitude slower. So work hard on minimizing the need for federation.
Another approach would be less burden on the individual queries, but more admin effort -- duplicate the "common" database across all of the shards. This could probably be done via replication by having the "common" server be a Master and all the shards by Slaves.
You can use Firebase it's fast easy more then MySQL v
But if you need to this
You can make a simple API on the other server that has the instance for MySQL this API will run your query on the server and return back the results
I am building an ecommerce site hosted on godaddy shared hosting and I saw that only 200 database connections are allowed.
I am using Codeigniter framework for the site and I have 2 databases for the project.
Database 1 for storing sessions only with a user with read, write, update and delete privileges
Database 2 rest of tables needed for site, with a read only user.
Since 1 website visitor will be connecting to 2 databases does this mean that I can only have 100 visitoras at a time? Since each one will be using 2 connections.
Or can someone explain the 200 connections limit please.
As #Drew said, it depends. Limits exist everywhere (hardware, software, bandwidth etc). GoDaddy has it's own limitations (not only to database connections). Optimizing your code can help you take the maximum of the web-server and database servers.
For example if your code uses the database connection for 1 second to each database, you can serve 100 visitors per second. If you use it for 0.2 of a sec then you can serve 500 visitors every second.
Optimization is necessary especially on heavy web applications. Maybe you could organize your app so it does not need connecting to both databases for every request (this would double the available connectios per time fraction). Optimizing the SQL queries and minimizing JOINing tables will help your app too (also it will make it run faster).
Finally you can use caching, so you will not have your server constructing the same content again and again. This is not a full list of all optimizations you can do, but a start point to do your research and planning. Hope you find it helpful.
My client currently has only one server with both MySql and Apache running on it, and at busy times of the year they're occasionally seeing Apache fall over as it has so many connections.
They run two applications; their busy public ecommerce PHP based website and their (busy during working hours only) internal order processing type application with 15-20 concurrent users.
I've managed to get them to increase their budget enough to get two servers. I'm considering either:
A) one server running Apache/PHP and the other as a dedicated MySQL server, or
B) one running their public website only, and the other running MySQL and the internal application.
The benefit I see of A) is that Mysql my.cnf can be tuned to use all of the resources of that server, but it has the drawback of only having one Apache instance running.
B) would spread the load on Apache across both servers, but would limit MySQL's resources on that server, even out of working hours when the internal application won't be used.
I just can't decide which way to go with this and would be grateful of any feedback you may have.
Both approaches are wrong.
You have 2 goals here; availability and performance (I'm considering capacity to be an aspect of performance in this context).
To improve availability, you should be ensuring that there is no single point of failure in your architecture. But with the models you propose, you're actually creating multiple single points of failure - hence your 2 server models are less available than your single server.
From a performance point of view, you want to spread the workload across the available resources. You can't move CPU and memory between the servers but you can move the traffic.
Hence the optimal solution is to run both applications on both servers. Setting up MySQL clustering is a bit more complex, but probably the out-of-the-box asynch replication will be adequate - with the nodes configured as master-master (but writes from the 2 applications targeted sensibly).
There's probably a lot of scope for increasing the capacity of the system further but without a lot more detail (more than is appropriate in this forum, and possibly more than your client is comfortable payng for) it is hard to advise.
I am in the final stages of configuring a service that is accessible from four global locations (with plans to add more later). I will be running the servers on an Ubuntu 12.04 box with MariaDB. My initial thought was to create servers that run independently of each other with 4 distinct databases and live with the constraint that users would only be able to login to the server where they were initially registered.
However, I have just run into this article that has got me thinking... .
From my reading of things if I set up a Galera cluster with master-master replication as suggested in the article I can move have the luxury of one large database that is consistently available across all four servers. I have gathered (and am hoping) that with the cluster setup correctly and functioning well I need do pretty much nothing in my PHP code (the four MariaDB instances will have the same user to access the database) - not even alter the PDO connection string.
However, this sounds almost too good to be true. My questions are:
are there other issues involved here that make for complications?
Do the PHP PDO connection strings need to be altered in anway?
Does the fact that my application is already structured to ensure that there is absolutely zero chance of two servers attempting to simultaneously write the same row help?
And finally, reading from the MariaDB docs, that this will not work with the TokuDB storage engine?
Is there a way to specifically stop the replication of a selected table? Could I in fact exploit the "only InnoDB/XtraDB" constraint and use another storage engine on the table I do not want to have replicated?
are there other issues involved here that make for complications?
There are some Known Limitations that you should be aware of. Generally, with clusters, you should ideally have an odd number of nodes to prevent split brain conditions, but an even number will usually work just as well.
Do the PHP PDO connection strings need to be altered in anway?
No. Your existing connection strings should work.
Does the fact that my application is already structured to ensure that there is absolutely zero chance of two servers attempting to simultaneously write the same row help?
Look at the known limitations and make sure your application will still do that. If you're using named locks, you'll need to change your application.
And finally, reading from the MariaDB docs, that this will not work with the TokuDB storage engine?
TokuDB support was added in the recent galera cluster distribution. I have used some and it does replicate just like InnoDB but I wouldn't rely on it since it's new in the galera cluster build.
Is there a way to specifically stop the replication of a selected table? Could I in fact exploit the "only InnoDB/XtraDB" constraint and use another storage engine on the table I do not want to have replicated?
I've heard a lot of people ask if they can omit tables or databases from replication but I still haven't heard a good reason why. Galera replication provides HA and is cheap and easy so even if some tables aren't important I can't find any realistic reason to not replicate the data. That being said, you could have data not replicated by using MyISAM/Aria.
I've been using MariaDB with galera in multiple moderately sized projects and it is the best solution I've found for HA and it also provides performance benefits. Other solutions are generally expensive or not mature. One thing you should consider is setting up a proxy for connecting to the database servers like HA Proxy, mysql-proxy, or glbd (which I use) to provide better redundancy and connection balancing for performance.
In response to DroidOS's comment below:
Every write in the cluster needs to be agreed upon by every node so any latency between nodes is added to every write. So, basically, every write will have the greatest round trip time between the writing server and the other nodes added to it.
No. Galera replication is all or nothing across the entire cluster. If any node has a problem writing the data, which can happen if a table doesn't have a primary key, the node will gracefully kill itself since it can't guarantee its data is consistent with the rest of the cluster. If that happens, the rest of the cluster will continue to operate normally. If there is a network issue, if one of the segments has quorum, it will continue to operate normally. Any segments without quorum will wait for more nodes to get quorum but will not accept queries. With this behavior, you can be sure that any node that you are able to query is consistent with the rest of the cluster.
Given that this has turned out to be such a popular question I thought I should add an extra answer by way of comment for anyone who runs into it.
The big issue with synchronous replication is the latency that introduced by the process. There will certainly be times when synchronous replication is required and latency has to be managed and then lived with. However, you might on reflection -as I did - realize that you can live with lazy replication. There are commercial solutions that deliver this albeit at a hefty fee. You also have the possibility of spinning your own solution - easier than you might think.
Consider a web app in which a call to the app consists of PHP script running several MySQL queries, some of them memcached.
The PHP does not do very complex job. It is mainly serving the MySQL data with some formatting.
In the past it used to be recommended to put MySQL and the app engine (PHP/Apache) on separate boxes.
However, when the data can be divided horizontally (for example when there are ten different customers using the service and it is possible to divide the data per customer) and when Nginx +FastCGI is used instead of heavier Apache, doesn't it make sense to put Nginx Memcache and MySQL on the same box? Then when more customers come, add similar boxes?
Background: We are moving to Amazon Ec2. And a separate box for MySQL and app server means double EBS volumes (needed on app servers to keep the code persistent as it changes often). Also if something happens to the database box, more customers will fail.
Clarification: Currently the app is running with LAMP on a single server (before moving to EC2).
If your application architecture is already designed to support Nginx and MySQL on separate instances, you may want to host all your services on the same instance until you receive enough traffic that justifies the separation.
In general, creating new identical instances with the full stack (Nginx + Your Application + MySQL) will make your setup much more difficult to maintain. Think about taking backups, releasing application updates, patching the database engine, updating the database schema, generating reports on all your clients, etc. If you opt for this method, you would really need to find some big advantages in order to offset all the disadvantages.
You need to measure carefully how much memory overhead everything has - I can't see enginex vs Apache making much difference, it's PHP which will use all the RAM (this in turn depends on how many processes the web server chooses to run, but that's more of a tuning issue).
Personally I'd stay away from enginex on the grounds that it is too risky to run such a weird server in production.
Databases always need lots of ram, and the only way you can sensibly tune the memory buffers is to have them on dedicated servers. This is assuming you have big data.
If you have very small data, you could keep it on the same box.
Likewise, memcached makes almost no sense if you're not running it on dedicated boxes. Taking memory from MySQL to give to memcached is really robbing Peter to pay Paul. MySQL can cache stuff in its innodb_buffer_pool quite efficiently (This saves IO, but may end up using more CPU as you won't cache presentation logic etc, which may be possible with memcached).
Memcached is only sensible if you're running it on dedicated boxes with lots of ram; it is also only sensible if you don't have enough grunt in your db servers to serve the read-workload of your app. Think about this before deploying it.
If your application is able to work with PHP and MySQL on different servers (I don't see why this wouldn't work, actually), then, it'll also work with PHP and MySQL on the same server.
The real question is : will your servers be able to handle the load of both Apache/nginx/PHP, MySQL, and memcached ?
And there is only one way to answer that question : you have to test in a "real" "production" configuration, to determine own loaded your servers are -- or use some tool like ab, siege, or OpenSTA to "simulate" that load.
If there is not too much load with everything on the same server... Well, go with it, if it makes the hosting of your application cheapier ;-)