MongoDB limiting the amount of concurrency - php

Question

I am creating an application that has several servers running at the same time and several process on each server all of those are processing data making query/updates and inserts. So a total of 35+ concurrent connections are being made at all times. These servers are all processing data that is being sent to a single mongodb server (mongod). I am not sharding my database at the moment. The problem is that I am being limited by my mongodb server. Whenever I add more servers the queries/updates/inserts are running slower (they take more time). I was running this mongohq.com, then I just recently created my own amazon server for mongod but I am still getting nearly the same result. List below is my db.serverStatus({}). I am somewhat new to mongodb but basically I need to know how to speed up the process for the amount of concurrent operations going on with my mongo server. I need it to be able to handle a lot of requests. I know sharding is a possible way around this but if it is at all possible can you list some other solutions available. Thanks.
> db.serverStatus({})
{
"host" : "ip-10-108-245-21:28282",
"version" : "2.0.1",
"process" : "mongod",
"uptime" : 11380,
"uptimeEstimate" : 11403,
"localTime" : ISODate("2011-12-13T22:27:56.865Z"),
"globalLock" : {
"totalTime" : 11380429167,
"lockTime" : 86138670,
"ratio" : 0.007569017717695356,
"currentQueue" : {
"total" : 0,
"readers" : 0,
"writers" : 0
},
"activeClients" : {
"total" : 35,
"readers" : 35,
"writers" : 0
}
},
"mem" : {
"bits" : 64,
"resident" : 731,
"virtual" : 6326,
"supported" : true,
"mapped" : 976,
"mappedWithJournal" : 1952
},
"connections" : {
"current" : 105,
"available" : 714
},
"extra_info" : {
"note" : "fields vary by platform",
"heap_usage_bytes" : 398656,
"page_faults" : 1
},
"indexCounters" : {
"btree" : {
"accesses" : 798,
"hits" : 798,
"misses" : 0,
"resets" : 0,
"missRatio" : 0
}
},
"backgroundFlushing" : {
"flushes" : 189,
"total_ms" : 29775,
"average_ms" : 157.53968253968253,
"last_ms" : 185,
"last_finished" : ISODate("2011-12-13T22:27:16.651Z")
},
"cursors" : {
"totalOpen" : 34,
"clientCursors_size" : 34,
"timedOut" : 0,
"totalNoTimeout" : 34
},
"network" : {
"bytesIn" : 89743967,
"bytesOut" : 59379407,
"numRequests" : 840133
},
"opcounters" : {
"insert" : 5437,
"query" : 8957,
"update" : 4312,
"delete" : 0,
"getmore" : 76,
"command" : 821388
},
"asserts" : {
"regular" : 0,
"warning" : 0,
"msg" : 0,
"user" : 0,
"rollovers" : 0
},
"writeBacksQueued" : false,
"dur" : {
"commits" : 29,
"journaledMB" : 0.147456,
"writeToDataFilesMB" : 0.230233,
"compression" : 0.9999932183619632,
"commitsInWriteLock" : 0,
"earlyCommits" : 0,
"timeMs" : {
"dt" : 3031,
"prepLogBuffer" : 0,
"writeToJournal" : 29,
"writeToDataFiles" : 2,
"remapPrivateView" : 0
}
},
"ok" : 1
}

Answer 1

What is surprising about more load generating higher response times from mongod? There are a few possible reasons for degradation of performance.
For example, every write to mongod uses a process wide write lock. So the more servers you add the more updates will be attempted (assuming update load is about stable per server) and thus the longer the process will spend in write lock. You can keep an eye on this through mongostat's "locked %" field.
Additionally if you use JS powered functionality (m/r, db.eval(), etc.) these operations cannot be executed concurrently by mongod due to the fact that each mongod has a single JavaScript context (which is single threaded).
If you want a more specific analysis then you might want to consider posting exact numbers. How many reads and writes per second, what are the query plans for the queries you execute, what effect does adding an additional app server have on your overall database performance, etc.