We are moving from one server architecture to two servers, web and DB.
The two new servers are VMs on one physical box. The VMs have a private virtual network setup so they can talk to each other directly with local IP without going through any other switches in the data center but their virtual switch. Pinging one VM from the other is less < 1ms.
We have two applications running on these servers, one is built in c#.NET and the other one is in PHP.
.NET application has no issues with the two VMs setup.
PHP, however, has a latency of about 400ms more compared to the application running on a server with application and db residing on that server.
To make sure the queries in the db are not a problem, we installed the application on the db VM to run it locally, and it was running at the same speed as on our previous server. So queries are not a problem.
Any ideas what could be causing the issue???
Related
Overview
I have a Laravel 9 application which is hosted with Digital Ocean. I use Laravel Forge to handle provisioning of the servers, management, etc. I've created two separate servers for my production environment. One to host my Laravel application code and another for the database which runs MySQL 8. These two servers are networked together and communicate over their VPC assigned private IP addresses.
Problem
I initially provisioned one server to host my application. This single server hosted both the Laravel application code and database. I have an endpoint that I hit to measure the response time for my application.
With one server that hosts the codebase and database the average response time was: ~70ms
When I hit the same endpoint again but with my two dedicated servers the average response time was: ~135ms
Other endpoints in my application also have a significant increase in response time when the database lives on a dedicated server vs a single server that houses everything.
Things I have done
All database queries have been optimized. (n+1, etc.)
Both networked servers are in the same region.
Both networked server's resources (CPU, RAM) are low and are not capping out.'
I've turned on Laravel's database config "sticky" option with no noticeable improvements.
I've enabled PHP OPcache for PHP 8.1.
Questions
How can I achieve a faster response time when my database is on a separate server than my codebase?
Am I sacrificing performance for scalability with dedicated servers?
TLDR
I'm experiencing slower response times in my Laravel application when the codebase and database run on separate dedicated servers vs hosting everything on one server.
Are your servers in the same data center and on the same VLAN?
Are you sure that you are connecting with your private VLAN IP address?
Some latency is expected if you need to connect to a database on another server. Have you tried to ping between the servers to see what the latency is?
Do you really need to have the web server and the database on separate servers? If so, I would probably try Digital Oceans managed database. I have used that for several projects and it works great.
Q: How can I achieve a faster response time when my database is on a separate server than my codebase?
A: If hosted in the same data center, the connection latency should be 30ms or less. Tested between AWS RDS and EC2 instances. Your mileage could vary depending on host.
Q: Am I sacrificing performance for scalability with dedicated servers?
A: It's standard practice to host databases separately to your application. It would be unrealistic to do otherwise for bigger projects. You can soften the impact by selectively caching data that doesn't change regularly on the main server. Unfortunately, PHP is not particularly good at this kind of fine tuning so you might be out of luck.
I can tell you that I currently run a central MySQL RDS instance that many ubuntu EC2 instances communicate with. While the queries take around 30ms, smart use of caching gives the majority of my web requests a 30ms response time in their own right. I do have the advantage of using NodeJS which is always doing things in the background without needing a request before performing work.
You may unfortunately find that you're running into one of the limitations of PHP.
I'm doing a group project and we're creating an online game. We're about half way done and now it's time to implement a database to store our records/data and make the website go live on the internet.
I'm just confused on how PSQL works exactly. My understanding is that PSQL needs to be running on some server in order to access it. For previous assignments, I downloaded Postgres for my Mac and ran it on localhost. The PHP code was something along the lines of:
$dbconn = pg_connect("host=localhost port=5432 dbname=mydbname");
So, if we intend to use PSQL, where would the server be? Do one of us have to host the server? Can we use some sort of free online server? How do we connect to that server with PHP?
In summary, I have two main questions:
How do we make our code go live on the internet for free? (It's just a temporary website and will only be up for a few weeks at most)
How can we all access a shared PSQL database?
Sorry for the noob questions, I just got started with web development and am still learning.
So, if we intend to use PSQL, where would the server be? Do one of us have to host the server? Can we use some sort of free online server? How do we connect to that server with PHP?
PostGreSQL is going to have to run on some machine visible to anyone who needs to access it. If only your web server (i.e., the machine running PHP and your website) needs to talk to the PGSQL, then PGSQL can be installed on your web server. This is a very common configuration.
The server might also run on the LAN where your web server is running or it might be running on an entirely different network on a different continent. The most important thing is that any machine which must connect directly to the database can actually connect to it. If you're building a website, this means you have a web server. Your web server will need to connect to the PGSQL server. The second most important thing is that your web server and the PGSQL server should share a very fast connection for the sake of performance and efficiency.
It's probably most common for your web server to also host the database. On an ubuntu machine, installing a PostGreSQL server is as easy as running a few commands. A quick search yields many examples like this one.
How do we make our code go live on the internet for free? (It's just a temporary website and will only be up for a few weeks at most)
I don't know anyone who is in the habit of offering free web hosting or DBMS services. You could ask a friend. Or put an ad on craigslist or something. Or if you are tech-savvy (it doesn't sound like you are) then you could configure a high-end router at your home to use Dynamic DNS to point some domain at a machine running at your house.
How can we all access a shared PSQL database?
I have no experience with Heroku, but you might sniff around there. PostGreSQL's website also maintains a list of hosting companies. Amazon offers RDS instances running PGSQL. Digital Ocean has a variety of tutorials and how-tos on dealing with PostGres. You could probably fire up a 'droplet' server for super cheap and install it yourself without too much effort.
Amazon offer a free tier database solution for Postgres. Something like 300 hours (don't quote me on it) for a low level set up.
They have tutorials on doing this here:
https://aws.amazon.com/rds/?nc2=h_m1
Once set up you get the end point and your connection string becomes something like
db_connect ("host=[URLENDPOING] user=postgres dbname=postres")
I am experimenting with using Redis for a Drupal website, hosted on Ubuntu 14.04.
I have installed the redis drupal module and am using the Predis library. I have also installed the 'redis-server' Ubuntu package and left the default configuration.
Configuring the Drupal site to use Redis for its cache backend works fine and the pages are lightning fast.
The problem arrives when I tried to spark up an m3.medium AWS instance and hosting the redis server there. The reason behind this is so that we can use one redis server and connect to it from multiple servers (live website hosted on multiple instances behind a load balancer, so each instance should connect to the same redis server).
I have set up the redis server on the instance, modified the redis.conf file to bind the correct IP address so it can be accessed from the outside, opened up the 6379 port, then tried connecting to it from my local computer
redis-cli -h IP
It worked fine so I decided to flip my local site's configuration to point to the new redis server.
The moment I did that the site became painfully slow, and at first I thought it might not even load at all. After almost a minute it finally loaded the home page. Clicking around the site was almost as slow, but the time reduced to maybe 10-15 seconds. That it still unacceptable and doesn't even compare to the lightning fast page load when using the redis server.
My question is: is there some specific configuration I need to do to make the remote connection faster? Is there something preventing it from performing well? some bottleneck somewhere?
Let me know if you want me to add the drupal settings.php configuration, although I am using a pretty standard config.
Although I ran the same configuration for a php application as you are trying, I had no issues hosting redis on either a small or medium instance and handling large amounts of traffic. There must be a config issue somewhere. Another option to debug it would be to try switching to Elasticcache (AWS' redis offering) it requires that all clients be within the same region, but could make finding your problem very easy.
We have our database servers separate from our webserver. The database servers are replicated (we know there is overhead here). Even with replication turned off however, performance for large number of queries in a PHP script is 4 times slower than our staging server that has the db and apache on the same machine. I realize that network latency and other issues with a network mean that there is no way they will be equal, but our productions servers are exponentially more powerful and our production network is all on gigabit switches. We have tuned MYSQL as best as we can but the performance marker is still at 4x slower. We are running over nginx with Apache proxies and replicated MYSQL dbs. UCarp is also running. What are some suggestions for areas to look for improving the performance? I would be happy with twice as slow on production.
It's difficult to do much more than stab in the dark given your description, but here's some starting points to try independently, which will hopefully narrow down the cause:
Move your staging DB to another host
Add your staging host to the production pool and remove the others
Profile your PHP script to ensure it's the queries causing the delay
Use an individual MySQL server rather than via your load balancer
Measure a single query to the production pool and the staging server from the MySQL client
Run netperf between your web server and your DB cluster
Profile the web server with [gb]prof
Profile a MySQL server receiving the query with [gb]prof
If none of these illuminate anything other than the expected degradation due to the remote host, then please provide a reproducible test case and your full MySQL config (with sensitive data redacted.) That will help someone more skilled in MySQL assist you ;)
Not every web request on a web site will (if properly designed) need a mysql connection. Most likely, if you are requiring a connection on every http request, your application will not scale and will start having issues very quickly.
Do more caching at app. server to request mysql less often. E.g. use
memcache.
Try to use persistent connections from application to your mysql servers.
Use mysql data compression.
Minify data (limit your selects, use column names instead of "*" in select statements)
Shamanic tuning:
Make sure, that nothing slows down network at mysql servers: big firewall rulesets, network filters, etc.
Add another (client inaccesible) network interface for app. server
and mysql server.
Tune network connection between app. server and mysql. Sometimes you
can win several ms by creating hardcoded network routes.
Don't think any of above would help - if network connection is slow, nothing of above will significantly speed it up.
I have a local intranet application which runs off a basic WAMP server in our offices. Every morning, one of our team members manually syncs our internal mysql db with our external mysql db (where our online enrollments occur). If a change is made during the day on the intranet application, it is not reflected on the external db until the following day.
I am wondering if it is possible to (essentially) tunnel to an external mysql connection from say a wamp or xampp server from within our offices and work in 'real-time'.
Anybody had any luck or advice?
Yes
Replication enables data from one MySQL database server (the master) to be replicated to one or more MySQL database servers (the slaves). Replication is asynchronous - slaves need not to connected permanently to receive updates from the master. This means that updates can occur over long-distance connections and even over temporary or intermittent connections such as a dial-up service. Depending on the configuration, you can replicate all databases, selected databases, or even selected tables within a database.
If you use the external server directly, performance is likely to suffer. A Gigabit LAN might be a thousand times faster than your Internet connection - particularly the upload speed of an ADSL connection.
Just make your internal application use the database from the external one. You may need to add permission to the external server to allow connections from your internal server IP, but otherwise this is just like having a webserver and sperate db server that need to access each other.
Can't really tell you how to do this here - it all depends on your specific configuration, something that I would thing is a little complicated (and too specialized) to figure out on SO.