Is GridFS via PHP useable if primary is unavailable? - php

Konfiguration
Ubuntu 14.04
PHP 5.5.9
MongoDB 3.0.2
MongoDB PHP driver 1.6.6
Setup
I have a globally distributed MongoDB replica set (for testing purpose) with 3 servers. One has priority = 1, others have priority = 0 so they will never become primary.
This setup is used to distribute files to the replicated servers by adding them directly on the primary server using GridFS. Distribution works fine.
I have created a simple php watcher script which is executed on the secondary servers using read preference \MongoClient::RP_NEAREST. I wanted to determine the timestamp when the replicated received all files from the primary.
I wanted to make sure that the php script on the secondary servers are using the mongodb instance on their server (and not the primary), and therefore I stopped the primary mongodb server. After doing this, the two servers are keeping their secondary role.
Issue
If the primary server is unavailable, I was still able to execute queries like count() and find() on regular collections (also fs.files collection).
But calls that use GridFS will throw an MongoConnectionException: No candidate servers found exception.
Script
I have created a little script with which you should be able to reproduce the error.
$serverList = '...';
$conn = new \MongoClient(
'mongodb://'.$serverList,
array(
'replicaSet' => 'r0',
'readPreference' => \MongoClient::RP_NEAREST,
'username' => 'bat',
'password' => '',
'db' => 'bat'
)
);
$db = $conn->selectDB('bat');
echo 'works fine...';
$files = $db->selectCollection('fs.files');
$documentsCount = $files->count();
$documents = $files->find();
foreach($documents as $document) {
echo $document['filename'] . ', ';
}
echo 'throws exception...';
$gridfs = $db->getGridFS();
$documentsCount = $gridfs->find()->count();
$documents = $gridfs->find();
foreach($documents as $document) {
echo $document->getFilename();
}
If the primary server is unavailable, the lines after echo 'works fine...'; will work fine, while the line after echo 'throws exception...'; will throw an exception.
Maybe this is related to an issue in the java driver JAVA-401, there was a similar problem with usage of secondary servers and gridfs. Maybe the gridfs ist trying to ensuring indices if the fs.files collection contains less than 1000 files which is not possible on secondary.

I figured out the problem, it was simply the incorrect line $gridfs->find()->count(). If you execute $gridfs->count() or $gridfs->find() it works fine.
So you'll have to use $gridfs->count() instead of $gridfs->find()->count().
I dont't know why $gridfs->find()->count() works correctly (and only correctly) if a primary server is available.

Related

How to get 200+ hundred of Pages in Silverstripe in a single query?

One of our Silverstripe sites is on shared hosting and having major performance issues. The issues seem to be caused by the shared SQL server throttling the number of queries that can be made.
The pages that are running the slowest get 200+ hundred pages to place on a Google Map:
$DirectoryItems = DirectoryItem::get()->where("\"Latitude\" IS NOT NULL AND \"Longitude\" IS NOT NULL ")->sort('Title ASC');
$MapItems = new ArrayList();
foreach ($DirectoryItems as $DirectoryItem) {
$MapItems->push(new ArrayData(array(
"Latitude" => $DirectoryItem->Latitude,
"Longitude" => $DirectoryItem->Longitude,
"MapMarkerURL" => $DirectoryItem->MapMarkerURL,
"Title" => addslashes($DirectoryItem->Title),
"Link" => $DirectoryItem->Link()
)));
}
Each of the 200+ MapItems generate it's own SQL Query which is overloading the shared SQL server.
I started off trying to get the same information with a single query:
$DirectoryItems = DB::query('SELECT `DirectoryItem`.`Latitude`, `DirectoryItem`.`Longitude`, `DirectoryItem`.`MapMarkerURL`, `SiteTree_Live`.`Title`
FROM `DirectoryItem`, `SiteTree_Live`
WHERE `DirectoryItem`.`ID` = `SiteTree_Live`.`ID`
AND `DirectoryItem`.`Latitude` IS NOT NULL AND `DirectoryItem`.`Longitude` IS NOT NULL
ORDER BY `SiteTree_Live`.`Title`');
$MapItems = new ArrayList();
foreach ($DirectoryItems as $DirectoryItem) {
$MapItems->push(new ArrayData(array(
"Latitude" => $DirectoryItem['Latitude'],
"Longitude" => $DirectoryItem['Longitude'],
"MapMarkerURL" => $DirectoryItem['MapMarkerURL'],
"Title" => addslashes($DirectoryItem['Title']),
"Link" => ??????
)));
}
But this falls over when it comes to getting the link to the DirectoryItem.
I thought about adding the Link as a DB field in DirectoryItem but that's beginning to feel needless complicated for what should be a straightforward operation.
What is the best way of getting the information for 200+ DirectoryItems in a single query?
Did you have a look into caching? If you show the same items on the map all the time you don't need to hit the db on every request.
See
Silverstripe Docs for caching
partial caching of elements of your site
Static publisher module for real fast, static pages managed with SilverStripe
Static publish queue module, another approach for generating static pages
It takes a huge load off your server if you cache properly.
If you still have problems when caching you should think about a better server.
The SiteTree class has static function, which is used in the CMS to get the link for a certain SiteTreeID. So you just need to extend your SQL Query to get the ID and you can get the link to any page via ID by calling:
$link = SiteTree::link_shortcode_handler(array('id' => $id), false);
Edit: wmk suggested a different and probably more future-proof way of using:
$page = SiteTree::get()->byID($id);
if ($page instanceof SiteTree) $link = $page->Link();
Untested; src: http://api.silverstripe.org/master/class-SiteTree.html

Rackspace - php-opencloud filters - Documentation for valid ObjectList filters?

Anyone know if/where there is documentation for valid ObjectList filter arrays?
The project's entry on github has a tiny blurb on it directing me to the API documentation, but that also fails to have a comprehensive list, and a search on 'filters' talks about containers only, not the object themselves.
I have a list of videos, each in four different formats named the same thing (sans filetype). Using the php-opencloud API, I want to GET only one of those video formats (to grab the unique filename rather than all its different formats).
I figured using a filter is the way to go, but I can't find any solid documentation.
Someone has got to have done this before. Help a noob out?
Most of the links on this page are dead now. Here's a current link to the php-opencloud documentation, which includes an example of using a prefix to filter the objectList results:
http://docs.php-opencloud.com/en/latest/services/object-store/objects.html#list-objects-in-a-container
I didn't find documentation of this, but apparently when the Rackspace Cloud Files documentation mentions arguments in a query string, those translate to arguments in an objectList method call like this:
GET /v1/MossoCloudFS_0672d7fa-9f85-4a81-a3ab-adb66a880123/AppleType?limit=2&marker=grannysmith
equals
$container->objectList(array('limit'=>'2', 'marker'=>'grannysmith'));
As Glen's pointed out, there isn't support (at the moment) for the service to apply filters on objects. The only thing which you might be interested in is supplying a prefix, which allows you to refine the objects returned based on how the filenames start. So if you sent 'bobcatscuddling' as the prefix, you'd get back all associated video formats for that one recording.
Your only option, it seems, is to get back all objects and iterate over the collection:
use OpenCloud\Rackspace;
$connection = new Rackspace(RACKSPACE_US, array(
'username' => 'foo',
'apiKey' => 'bar'
));
$service = $connection->objectStore('cloudFiles', 'DFW', 'publicURL');
$container = $service->container('CONTAINER_NAME');
$processedObjects = array();
$marker = '';
while ($marker !== null) {
$objects = $container->objectList('marker' => $marker);
$total = $objects->count();
$count = 0;
while ($object = $objects->next()) {
// Extract the filename
$filename = pathinfo($object->name, PATHINFO_FILENAME);
// Make sure you only deal with the filename once (i.e. to ignore different extensions)
if (!in_array($processedObjects, $filename)) {
// You can do your DB check here...
// Stock the array
$processedObjects[] = $filename;
}
$count++;
$marker = ($count == $total) ? $object->name : null;
}
}
What you'll notice is that you're incrementing the marker and making a new request for each 10,000 objects. I haven't tested this, but it'll probably lead you in the right direction.
Unfortunately, the underlying API doesn't support filtering for objects in Swift/Cloud Files containers (cf. http://docs.rackspace.com/files/api/v1/cf-devguide/content/List_Objects-d1e1284.html). The $filter parameter is supported as part of the shared code, but it doesn't actually do anything with Cloud Files here.
I'll see if I can get the docs updated to reflect that.

Anyway to do MongoDB Sharding in PHP?

I have started using MongoDB for one of my PHP project. In that database, i am trying to use MongoDB sharding concept. I got the below link and tried,
MongoDB Sharding Example
It is working well. But the problem is, in the above example, everything is done in command prompt. But i am trying to do everything in PHP. I am not able to get any example in PHP.
So far, i have started with these piece of codes,
$connection = new Mongo();
$db = $connection->selectDB('TestDB');
$db = $connection->TestDB;
$connection->selectDB('admin')->command(array('addshard'=>'host:port'));
$connection->selectDB('admin')->command(array('enablesharding'=>'TestDB'));
$connection->selectDB('admin')->command(array('shardcollection'=>'TestDB.large', 'key' => '_id'));
It is not working. Also, i dont know how to set shard servers and config database in PHP.
Is there any other way to do MongoDB sharding in PHP?
MongoDB's database commands are case-sensitive. You were passing in lowercase command names, while the real commands are camelCased (see: sharding commands). Additionally, the key paremeter of shardCollection needs to be an object.
A corrected version of your above code would look like:
$connection->selectDB('admin')->command(array('addShard'=>'host:port'));
$connection->selectDB('admin')->command(array('enableSharding'=>'TestDB'));
$connection->selectDB('admin')->command(array('shardCollection'=>'TestDB.large', 'key' => array('_id' => 1)));
Additionally, you should have been able to examine the command result of to determine why this wasn't working. The ok field would be zero, and an errmsg field would explain the error (e.g. "no such cmd: addshard").
Lastly, if you're attempting to mimic the shell's functionality to configure sharding in PHP, you may realize that some of the shell methods are more than just database commands. For any of those methods, you can get the JS source by omitting the parentheses. For example:
> sh.addTagRange
function ( ns, min, max, tag ) {
var config = db.getSisterDB( "config" );
config.tags.update( {_id: { ns : ns , min : min } } ,
{_id: { ns : ns , min : min }, ns : ns , min : min , max : max , tag : tag } ,
true );
sh._checkLastError( config );
}
That would be a useful hint for what an addTagRange() function in PHP would need to do.

PHP 5.2.13 MongoCursorException 'duplicate key error index'

I'm getting duplicate _ids when inserting documents into our mongo database. This is an intermittent problem that only happens under some load (is reproducable with some test scripts).
Here's some test code so you don't think I'm trying to double-insert the same object (I know that the PHP mongo driver adds the _id field):
// Insert a job
$job = array(
'type' => 'cleanup',
'meta' => 'cleaning the data',
'user_id' => new MongoId($user_id),
'created' => time(),
'status' => 'pending'
);
$this->db->job->insert($job, array('safe' => true)); // <-- failz here
I went on a frenzy and installed the latest stable (1.1.4) mongo driver to no avail. This isn't under heavy load. We're doing maybe 5 req/s on one server, so the 16M rec/s limit for the inc value probably isn't the issue.
Any ideas would be greatly appreciated. I'm hoping someone somewhere has used mongo with PHP and inserted more than 5 docs/s and had this issue ;).
-EDIT-
On CentOS 5.4 x86_64, linux 2.6.18-164.el5xen, Apache worker 2.2.15, PHP 5.2.13, MongoDB 1.8.1
-EDIT2-
As noted in the comments, I'm using the latest version of the PECL driver as of now (1.2.0) and the problem is still happening.
-EDIT3-
Forgot to post exact error:
Uncaught exception 'MongoCursorException' with message 'E11000 duplicate key error index: hannibal.job.$_id_ dup key
There is a different solution for this (the preform/worker MPM didn't help in my case, we were running as prefork which is default anyway).
The issue is that the insert array is passed by reference, and modified by the PHP MongoDB library to include the ID. You need to clear the ID.
So imagine the following code:
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert); << This will have '_id' added
$aToInsert['field'] = $val2
$collection->insert($aToInsert); << This will fail with the above error
Why? What happens with the library is:
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
// $aToInsert has '_id' added by PHP MongoDB library
// Therefore $aToInsert = array('field'=>$val1, '_id'=>MongoID() );
$aToInsert['field'] = $val2
// Therefore $aToInsert = array('field'=>$val2, '_id'=>MongoID() );
$collection->insert($aToInsert);
// This will not add '_id' as it already exists. But will now fail.
Solution is to reinitialise the array
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
$aToInsert = array('field'=>$val2);
$collection->insert($aToInsert);
or to unset the id
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
unset($aToInsert['_id']);
$aToInsert['field'] = $val2
$collection->insert($aToInsert); << This will now work
Looks like it had to do with the Apache version installed (worker). After installing apache prefork, we've seen no more duplicate _id errors on the server.
My guess is this has something to do with the global counter the Mongo driver uses. I'm thinking the lack of communication between the threads may be the cause...maybe one pool has instance counters per-thread, but since the PID is the same, you get conflicts.
I don't know the internals, but this seems to be the most likely explanation. Don't use Apache Worker MPM with the PHP MongoDB driver. Please comment and correct me if this is not the case, or if you know of a fix.

PHP AMI Connection - saving incoming and outgoing calls to a database is not working

Using the AMI (API connection with an Asteriskserver so I can use a PHP Socket connection) I'm trying to catch the recieving data using PHP in a way that I can record outgoing and incomming calls for the CRM system (webbased) used at the company I work for.
But I'm not getting the result I am hoping for...
The full code can be found on PasteBin http://pastebin.com/AwRNBW2G
I catch the outgoing calls this way, and that works:
if($givenkey = array_search("Context: from-internal", $content)){
$calleridKey = $givenkey + 1;
$idSIP = $givenkey - 1;
$dialNumber = str_replace("Extension: 0","31",$content[$calleridKey]);
$dialNumber = str_replace("Extension: ", "", $dialNumber);
$fromSIP = str_replace("Channel: SIP/", "", $content[$idSIP]);
$fromSIP = substr($fromSIP, 0, 2);
$dialTime = date('r');
$uitgaand = array(
"Phonenumber" => $dialNumber,
"Type" => "Uitgaand",
"datetime" => $dialTime,
"SIP" => $fromSIP
);
The incomming calls are being catched this way, but that's not working properly:
if($givenkey = array_search("AppData: Using CallerID ", $content)){
if(array_search("Channel: SIP/31000000000", $content)+5 == $InCallKey = array_search("AppData: Using CallerID", $content)){
$calleridNum = explode('"',str_replace('AppData: Using CallerID "',"",$content[$InCallKey]));
$pickupSource = array_search("Source: SIP/31000000000", $content);
if($pickupSource+1 == $pickupKey = array_search("Destination: SIP/", $content)){
$pickupBy = str_replace("Destination: SIP/","",$content[$pickupkey]);
$pickupBy = substr($pickupBy, 0, 2);
$dialTime = date('r');
$inkomend = array(
"Phonenumber" => $calleridNum[0],
"Type" => "Binnenkomend",
"datetime" => $dialTime,
"SIP" => $pickupBy
);
I have the array that I make not available right now but, if necessary, I can save the array and post it here with personal data filtered.
I know that the code I'm using right now is not neat, but I wrote it with the goal: quick result. If I have a working code I will optimize it and clean it up. Tips about this are also very welcome. It's too bad that I cannot find any good documentation about this so I have to start from the beginning and could only find the working class that I'm using right now, however it's not very complete.
I had to write this part without any knowledge about VOIP or AMI or Asterisk.
To be short, here are my questions:
- How can I record incoming and outgoing calls to eventually save them in a database by using the AMI?
- How can I keep alive the connection with the server the best way? The method I'm using now is not optimal as the connection fails atleast once within 48 hours.
- Do you have tips or suggestions about optimizing the code and neater code-writing? And do you maybe know any functions that I could use instead of a function that I am using?
With regards,
Dempsey
Since recently I get this error which
I cannot solve properly. This error
creates itself after about 15 minutes
running. It would run atleast 24 hours
before:
PHP Notice: fwrite(): send of 16 bytes failed with errno=32 Broken pipe in /var/www/html/phpami/AMILoader.php on line 147
Net_AsteriskManagerException: Authorisation failed in /var/www/html/phpami/AMILoader.php on line 173
#0 /var/www/html/phpami/AMILoader.php(173): Net_AsteriskManager-login('GEBRUIKERSNAAM','WACHTWOORD')
#1 /var/www/html/phpami/AMILoader.php(306): Net_AsteriskManager-_sendCommand('Action: Ping???...')
#2 /var/www/html/phpami/AMILoader.php(543): Net_AsteriskManager->ping()
#3 {main}
Can anyone help me with this too? The
authorisation data is correct (it is
using the same data in the whole
script and it does get a connection).
Also I don't get the response 'Action:
Ping???...' which it says is being
send by the script as command, but
where do the three questionmarks and
periods come from?
this framework should be handy:
https://github.com/marcelog/PAMI
otherwise you can check vTiger sources and how it handles ami integration:
http://www.vtiger.com/index.php?Itemid=57&id=30&option=com_content&task=view
If your using php, one of the easiest way to connect to the AMI is using the php-astmanager class. It supports callback on certain "events" so that you can catch the data you need. The best (only?) maintained copy is part of FreePBX and can be pulled right out of the latest version (2.9 as of this writing).

Categories