How to setup PHP MongoDB automatic failover - php

Question

When the primary server of my replication set fails, currently open connections also fail immediately (!) throwing MongoConnectionException (No candidate servers found) or with MongoCursorException (mongoUbuntu:8004: Remote server has closed the connection) when I use GridFS.
Is this a bug or do I have to change my setup in order to get automatic failover working?
Installation
Linux ubuntu kernel 3.16.0-31-generic
PHP 5.5.12-2ubuntu4.3
pecl/mongo 1.6.6
mongo 2.6.3
PHP via cli
Server's hostname is mongoUbuntu, mongodb processes were started on one single computer with the following command line
Mongod options
I run 4 servers on port 8001, 8002, 8003 and 8004 plus one arbiter on 8010. Command line for 8001 is like following:
mongod --replSet rs1 --dbpath /var/lib/mongodb1 --port 8001 --smallfiles --oplogSize 200 --httpinterface --rest
Replication Set rs.status()
{
"set" : "rs1",
"date" : ISODate("2015-04-08T14:48:57Z"),
"myState" : 3,
"members" : [
{
"_id" : 0,
"name" : "mongoUbuntu:8004",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 467,
"optime" : Timestamp(1428501340, 1),
"optimeDate" : ISODate("2015-04-08T13:55:40Z"),
"lastHeartbeat" : ISODate("2015-04-08T14:48:56Z"),
"lastHeartbeatRecv" : ISODate("2015-04-08T14:48:55Z"),
"pingMs" : 0,
"syncingTo" : "mongoUbuntu:8001"
},
{
"_id" : 1,
"name" : "mongoUbuntu:8003",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 987,
"optime" : Timestamp(1428501340, 1),
"optimeDate" : ISODate("2015-04-08T13:55:40Z"),
"lastHeartbeat" : ISODate("2015-04-08T14:48:56Z"),
"lastHeartbeatRecv" : ISODate("2015-04-08T14:48:56Z"),
"pingMs" : 0,
"syncingTo" : "mongoUbuntu:8001"
},
{
"_id" : 2,
"name" : "mongoUbuntu:8002",
"health" : 1,
"state" : 3,
"stateStr" : "RECOVERING",
"uptime" : 3142,
"optime" : Timestamp(1428498901, 1),
"optimeDate" : ISODate("2015-04-08T13:15:01Z"),
"infoMessage" : "still syncing, not yet to minValid optime 55252e9b:37",
"self" : true
},
{
"_id" : 3,
"name" : "mongoUbuntu:8001",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 3139,
"optime" : Timestamp(1428501340, 1),
"optimeDate" : ISODate("2015-04-08T13:55:40Z"),
"lastHeartbeat" : ISODate("2015-04-08T14:48:56Z"),
"lastHeartbeatRecv" : ISODate("2015-04-08T14:48:56Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1428503596, 1),
"electionDate" : ISODate("2015-04-08T14:33:16Z")
},
{
"_id" : 4,
"name" : "mongoUbuntu:8010",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 3139,
"lastHeartbeat" : ISODate("2015-04-08T14:48:56Z"),
"lastHeartbeatRecv" : ISODate("2015-04-08T14:48:55Z"),
"pingMs" : 0
}
],
"ok" : 1
}
PHP Script
The following script is running when I terminate the current primary (Example without real usage of GridFS but directly executing a query.)
<?php
$conn = new \MongoClient(
'mongodb://mongoUbuntu:8001,mongoUbuntu:8002,mongoUbuntu:8003',
array('replicaSet' => 'rs1', 'readPreference' => \MongoClient::RP_PRIMARY_PREFERRED)
);
$db = $conn->bat; //db name: bat
$gridfs = $this->_db->getGridFS();
while(true) {
$documents = $db->execute('db.getCollection(\'fs.files\').count()');
echo $documents['retval']."\n";
sleep(1);
}
Issue
Until I terminate the current primary, the script is printing the count of documents to command line. When I terminate the current primary (pressing Ctrl+C on respectiv mongod command line) the php script immediately because an exception, in this case MongoConnectionException No candidate servers found.
I used the script of github to create full debug log, the and saved it as Gist mongo-php-log-automaticfailover-fails.
Do I have to add another option to the creation of the connection or the config of the replication set? If this is described in mongodb documentation or documentation of mongodb php driver, where can I find it?

Answer 1

Yes, the driver does throw a exception. Technically it is the right thing to do, especially if you wish to know when the set fails over. What you need to do is catch the exception and do a retry.
Your biggest problem here is that for some reason you are evaling. Eval requires to be run ONLY on the master, as such when a set fails over it has to wait for a elected master which can take upto 10 seconds.
However, it seems this facet of eval is not actually documented, though it is recognised: https://dba.stackexchange.com/questions/75852/mongodb-replicaset-reconfig-when-primary-and-majority-dont-exist since the answerer of that question is in fact 10gen (MongoDB Inc) and he seems to not deny that eval does not work on secondaries. I am fairly certain that this used to be in the documentation though, that is where I first saw it.
This is quite well explained within the MongoDB 101 course for Python. The same applies for most languages. I personally just disallow the connection until a new primary is found (but then I don't eval stuff), but if you are solely reading you can remove the eval and do this through PHP. That should allow you to read without hinderance.