I'm using Gearman in a custom Joomla application and using Gearman UI to track active workers and jobs number.
I'm facing an issue with MYSQL load and number of connections, I'm unable to track the issue, But I've few questions that might help me.
1- Does Gearman Workers launch a new database connection for each job or do they share the same connection?
2- If Gearman launches a new connection everytime a job runs how can I change that to make all jobs share same connection?
3- How can I balance the load between more than one server?
4- Is there is something like "Pay-as-you-go" package for MYSQL hosting? if yes, Please mention them.
Thanks a lot!
This is often an overlooked issue when using any kind of a job queue with workers. 100 workers will open a separate database connection each (they are separate PHP processes). If MySQL is configured allow 50 connections, workers will start failing. To answer your questions:
1) Each worker runs inside one PHP process each, and that process will open 1 database connection. Workers do not share database connections.
2) If only one worker is processing jobs, then only one database connection will be opened. If you have 50 workers running, expect 50 database connections. Since these are not web requests, persistent connections will not work, sharing will not work.
3) You can balance the load by adding READ slaves, and using a MySQL proxy to distribute the load.
4) I've never seen a pay-as-you-go MySQL hosting solution. Ask your provider to increase your number of connections. If they won't, it might be time to run your own server.
Also, the gearman server process itself will only use one database connection to maintain the queue (if you have enabled mysql storage).
Strategies you can use to try and make your worker code play nicely with the database:
After each job, terminate the worker and start it up again. Don't open a new database connection until a new job is received. Use supervisor to keep your workers running all the time.
Close database connections after every query. if you see a lot of connections open in a 'sleep' state, this will help clean them up and keep database connections low. Try $pdo = null; after each query (if you use PDO).
Cache frequently used queries where the result doesn't change, to keep database connections low.
Ensure your tables are properly indexed so queries run as fast as possible.
Ensure database exceptions are caught in a try/catch block. Add retry logic (while loop), where the worker will fail gracefully after say, 10 attempts. Make sure the job is put back on the queue after a failure.
I think the most important thing to look at, before anything else, is the MySQL load. It might be that you have some really heavy queries that are causing this mess. Have you checked the MySQL slow query log? If yes, what did you find? Note that any query that takes more than a second to execute is a slow query.
Related
I need to write a socket server that will be handling at least something about 1000 (much more in future) low-traffic permanent connections. I have made a draft version on PHP for testing purposes (we are developing a monitoring hardware, so we needed to develop and test a conversation protocol and hardware capabilities), which suited me very well when i had just a couple of clients connected. But when the amount of connections had grown to ten, some critical issues appeared. Here some info about server architecture:
I have a master process, which waits for socket connections and on connecting creates a child process (that serves this connection from now on) using pcntl_fork(). Also i am setting up a PDO connection to MySQL in master process. All the child processes are sharing the same single PDO object. At first i was afraid of getting some collisions during simultaneous queries, but i haven't encountered them, even through stress-test (10 children were making queries in the loop without stopping). But there is usleep(500000) in each child, so it could be luck, though i had this testing running for a couple of hours. But such load should not be present even at 1k clients connected, due to rare conversations between them and server.
So here is my first question: is it safe to use single PDO object for a big amount of child processes (ideally there would be around 1000)? I can use single connection for each child, but MySQL doesn't support nearly as much connections.
The second issue is in getting parasite MySQL connections. As i mentioned before, i have only one PDO object. But when i have more than one clients connected, and after they had run some queries, i see in mytop that there is more than one DB connection, and i could not find any correlation between the amount of connections and amount of child processes i have. For example i have 3 childs, and 5 DB connections. I tried to establish persistent connections, and it didn't changed anything.
Second question: Is it PDO who makes those additional connecitons to MySQL, or it is the MySQL driver? And is there a way to force them to use one connection? I don't think it could be my fault, my code prints an alert to console every time i call method which creates PDO object, and that happens only once, at the script start, before forking. After that i only run querys out of children, using parent's PDO object. Once again i can not afford to have so many connections due to MySQL limitations.
Third question: Will be one thousand of socket connections a problem by itself? Aside of the CPU and database load, i mean. Or i should do some amount of lesser servers (128 connections for example), that will tell the clients to connect to other one if max number of connects is exceeded?
Thanks in advance for your time and possible answers.
Currently your primary concern should be your socket server architecture. Forking a process for each client is super heavy. AFAIK an average PC can tolerate around 2000 threads and it's not going to work fast. Switching between processes means that CPU should save its state in memory, and if you have enormous amount of processes, CPU will be busy with memory IO and will have little time for actually doing stuff.
You may want to look at Apache for inspiration. In Apache they use a fixed amount of worker processes/threads, each process/thread working with multiple clients via select function and sockets in a non-blocking mode. This is a far more robust approach.
Regarding database IO, I would spawn a process/thread that would be the sole owner of database connections. Worker processes would communicate with the DB IO process using IPC (in case of processes) or lock-free queues (in case of threads). This approach makes you independent of PDO implementation details (if it is thread safe or does it spawn connections etc).
P. S. I suspect that you actually spawn new PDO objects with forking (forking merely means making a copy of a process with its memory and everything inside it) and PDO objects create and shut down connections on demand. It may explain why you're not seeing correlation between low traffic clients and DB connections.
I am using gearman workers in my symfony app. The workers use the symfony services like doctrine and others. I can run multiple workers simultaneously. I want to know how doctrine handles db queries when it gets more than one query at a time(each worker is doing one query at the same time). Also is it possible to have different connections to db so that my gearman workers can use each connection for certain purpose like one connection to read from db and one connection to write to db ?
thanks
Each of your workers is its own process, with its own EntityManager, which each have their own connections. So, by default, you'll have one-connection-per-worker.
As far as "more than one query at a time" this is just like what happens with web-bound processes. From the database's perspective, it's exactly the same. Multiple simultaneous connections executing queries.
The tricky bits with workers is that they tend to live a lot longer than a web-bound process (which is reinitialized for each HTTP request that comes in). You need to be particularly careful that workers that sit idle may have their connections time out, so when they eventually pick a up a job, they explode.
I'm running a web service which runs algorithms that serve millions of calls daily and run some background processing as well.
Every now and than I see "Too many connections" error in attempts to connect to the MySQL box" for a few seconds. However this is not necessarily attributed to high traffic times or anything I can put my finger on.
I want to find the bottleneck causing it. Other than in the specific times this happens the server isn't too loaded in terms of CPU and Memory, and has 2-3 connections (threads) open and everything works smoothly. (I use Zabbix for monitoring)
Any creative ideas on how to trace it?
try to have an open mysql console when this happens and issue a SHOW PROCESSLIST; to see what queries are being executed.
Alternatively you could enable logging slow queries (in my.cnf insert this line:
log-slow-queries=/var/log/mysql-log-slow-queries.log
in the [mysqld] section and use set-variable=long_query_time=1 to define what's the minimum time a query should take in order to be considered slow. (remember to restart mysql in order for changes to take effect)
What MySQL table type are you using? MyISAM or InnoDB (or another one)? MyISAM will use table level locking, so you could run into a scenario where you have a heavy select running, followed by an update on the same table and numerous select queries. The last select queries will then have to wait until the update is finished (which in turn has to wait until the first - heavy - select is finished).
For InnoDB a tool like innotop could be useful to find the cause of the deadlock (see http://www.xaprb.com/blog/2006/07/31/how-to-analyze-innodb-mysql-locks/).
BTW The query that is causing the lock to occur should be one of those not in locked state.
The SHOW OPEN TABLES command will display the lock status of all the tables in MySQL. If one or more of your queries is causing the connection backlock, combining SHOW PROCESSLIST and the open tables should narrow it down as to exactly which query is holding up the works.
Old topic. However, I just had this issue and it was because I had a mysqldump script scheduled for 3 times per day. At these times, if my web application was also getting a fair amount of usage, all of the web application queries just queued themselves up on top of each other while the mysqldump was locking all of the tables in the database. The best option is to setup a replication slave on a separate machine, and take your backups from the slave rather than from the production server.
May be related to this bug in MySQL for FULLTEXT search:
http://bugs.mysql.com/bug.php?id=37067
In this case, the FULLTEXT initialization actually hangs MySQL. Unfortunately there doesn't seem to be a solution.
Without knowing too much of your implementation, and PHP in general, but are you sure that you do not have any problems with lingering DB connections? E.g connections that stay open even after the request has been processed?
In PHP a connection is usually closed automatically when the script ends or when calling mysql_close($conn); but if you use any sort of homegrown connection pooling, that could introduce problems.
I'm trying to debug an error I got on a production server. Sometimes MySQL gives up and my web app can't connect to the database (I'm getting the "too many connections" error). The server has a few thousand visitors a day and on the night I'm running a few cron jobs which sometimes does some heavy mysql work (Looping through 50 000 rows, inserting and deletes duplicates etc)
The server runs both apache and mysql on the same machine
MySQL has a pretty standard based configuration (max connections)
The web app is using PHP
How do I debug this issue? Which log files should I read? How do I find the "evil" script? The strange this is that if I restart the MySQL server it starts working again.
Edit:
Different apps/scripts is using different connectors to its database (mostly mysqli but also Zend_Db)
First, use innotop (Google for it) to monitor your connections. It's mostly geared to InnoDB statistics, but it can bet set to show all connections including those not in a transaction.
Otherwise, the following are helpful: Use persistent connections / connection pools in your web apps. Increase your max connections.
It's not necessarily a long-running SQL query.
If you open a connection at the start of a page, it won't be released until the PHP script terminates - even if there is no query running.
You should add some stats to your pages to find out the slowest ones, and the most-hit ones. Closing the connection early would help, if possible.
Try using persistent connections (mysql_pconnect), it will help reduce the server load caused by constantly opening and closing MySQL connections.
The starting point is probably to use mysqladmin processlist to get a list of the processes on the mysql server. The next step depends on what you find.
Trying to separate out my LAMP application into two servers, one for php and one for mysql. So far the application connects locally through a file socket and works fine.
I'm worried about the number connections I can establish if it is over the network. I have been testing tcp connections on unix for benchmark purposes and I know that you cannot exceed a certain amount of connections per second otherwise it halts due to the lack of resources (be it sockets, or file handles or whatever). I also understand that php does not implement connection pooling so for each page load a new connection over the network must be made. I also looked into pconnect for php and it seems to bring more problems.
I know this is a very very common setup (php+mysql), can anyone provide some typical usage and statistics they get out of their servers? Thanks!
The problem is not related to running out of connections allowed my MySQL. The main problem is that unix cannot very quickly create and tear down tcp connections. Sockets end up in TIME_WAIT and you have to wait for a period before you free up more sockets to connect again. These two screenshots clearly shows this pattern. MySQL does work up to a certain point and then pauses because the web server ran out of sockets. After certain amount of time passed, the web server was able to make new connections.
alt text http://img35.imageshack.us/img35/3809/picture4k.png
alt text http://img35.imageshack.us/img35/4580/picture2uyw.png
I think the limit is at 65535. So you'd have to have 65535 connections at the same time to hit that limit since a regular mysql connection closes automatically.
mysql_connect()
Note: The link to the server will be closed as soon as the execution of the script ends, unless it's closed earlier by explicitly calling mysql_close().
But if you're using a persistent mysql connection, then you can run into trouble.
Using persistent connections can require a bit of tuning of your Apache and MySQL configurations to ensure that you do not exceed the number of connections allowed by MySQL.
Each MySQL connection actually uses several meg of ram for various buffers, and takes a while to set up, which is why MySQL is limited to 100 concurrent open connections by default. You can up that limit, but it's better to spend your time trying to limit concurrent connections, via various methods.
Beware of raising the connection limit too high, as you can run out of memory (which, I believe, crashes mysql), or you may push important things out of memory. e.g. MySQL's performance is highly dependent on the OS automatically caching the data it reads from disk in memory; if you set your connection limit too high, you'll be contending for memory with the cache.
If you don't up your connection limit, you'll run out of connections long before your run out of sockets/file handles/etc. If you do increase your connection limit, you'll run out of RAM long before you run out of sockets/file handles/etc.
Regarding limiting concurrent connections:
Use a connection pooling solution. You're right, there isn't one built in to PHP, but there are plenty of standalone ones out there to choose from. This saves expensive connection setup/tear down time.
Only open database connections when you absolutely need them. In my current project, we automatically open a database connection when the first query is issued, and not a moment before; we also release the connection after we've done all our database work, but before the page's HTML is actually generated. The shorter the period of time you hold connections open, the fewer connections will be open simultaneously.
Cache what you can in a lighter-weight solution like memcached. My current project temporarily caches pages displayed to anonymous users (since every anonymous user gets the same HTML, in the end -- why bother running the same database queries all over again a few scant milliseconds later?), meaning no database connection is necessary at all. This is especially useful for bursts of anonymous traffic, like a front-page digg.