PHP 5.2.13 MongoCursorException 'duplicate key error index' - php

I'm getting duplicate _ids when inserting documents into our mongo database. This is an intermittent problem that only happens under some load (is reproducable with some test scripts).
Here's some test code so you don't think I'm trying to double-insert the same object (I know that the PHP mongo driver adds the _id field):
// Insert a job
$job = array(
'type' => 'cleanup',
'meta' => 'cleaning the data',
'user_id' => new MongoId($user_id),
'created' => time(),
'status' => 'pending'
);
$this->db->job->insert($job, array('safe' => true)); // <-- failz here
I went on a frenzy and installed the latest stable (1.1.4) mongo driver to no avail. This isn't under heavy load. We're doing maybe 5 req/s on one server, so the 16M rec/s limit for the inc value probably isn't the issue.
Any ideas would be greatly appreciated. I'm hoping someone somewhere has used mongo with PHP and inserted more than 5 docs/s and had this issue ;).
-EDIT-
On CentOS 5.4 x86_64, linux 2.6.18-164.el5xen, Apache worker 2.2.15, PHP 5.2.13, MongoDB 1.8.1
-EDIT2-
As noted in the comments, I'm using the latest version of the PECL driver as of now (1.2.0) and the problem is still happening.
-EDIT3-
Forgot to post exact error:
Uncaught exception 'MongoCursorException' with message 'E11000 duplicate key error index: hannibal.job.$_id_ dup key

There is a different solution for this (the preform/worker MPM didn't help in my case, we were running as prefork which is default anyway).
The issue is that the insert array is passed by reference, and modified by the PHP MongoDB library to include the ID. You need to clear the ID.
So imagine the following code:
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert); << This will have '_id' added
$aToInsert['field'] = $val2
$collection->insert($aToInsert); << This will fail with the above error
Why? What happens with the library is:
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
// $aToInsert has '_id' added by PHP MongoDB library
// Therefore $aToInsert = array('field'=>$val1, '_id'=>MongoID() );
$aToInsert['field'] = $val2
// Therefore $aToInsert = array('field'=>$val2, '_id'=>MongoID() );
$collection->insert($aToInsert);
// This will not add '_id' as it already exists. But will now fail.
Solution is to reinitialise the array
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
$aToInsert = array('field'=>$val2);
$collection->insert($aToInsert);
or to unset the id
$aToInsert = array('field'=>$val1);
$collection->insert($aToInsert);
unset($aToInsert['_id']);
$aToInsert['field'] = $val2
$collection->insert($aToInsert); << This will now work

Looks like it had to do with the Apache version installed (worker). After installing apache prefork, we've seen no more duplicate _id errors on the server.
My guess is this has something to do with the global counter the Mongo driver uses. I'm thinking the lack of communication between the threads may be the cause...maybe one pool has instance counters per-thread, but since the PID is the same, you get conflicts.
I don't know the internals, but this seems to be the most likely explanation. Don't use Apache Worker MPM with the PHP MongoDB driver. Please comment and correct me if this is not the case, or if you know of a fix.

Related

PHP 7.4 laravel data saves garbage value

We are noticing a weird issue in Laravel 8 with PHP 7.4 version.
When we save data to the Mysql db, it is storing garbage value.
An example data:
$product = Product::create([
'product_id' => 6791044858042,
'inventory_id' => 42309695242426
]);
The same is happening using $product = new Product();
The value in the DB is
"product_id":"701563066","inventory_id":"-27590470"
When we log the $product in the file just after saving, it has the correct value. The data is not re-saved and the data modified is notified either.
Inspected the max integer value for the column in db schema. It has bigint(20), I did change it to biging(100) and even varchar(255), yet it is the same result.
Further the garbage value is random and not a fixed value.
I had changed it back to 7.3 in the server and it seems fine.
Any one has faced the issue before?

Is GridFS via PHP useable if primary is unavailable?

Konfiguration
Ubuntu 14.04
PHP 5.5.9
MongoDB 3.0.2
MongoDB PHP driver 1.6.6
Setup
I have a globally distributed MongoDB replica set (for testing purpose) with 3 servers. One has priority = 1, others have priority = 0 so they will never become primary.
This setup is used to distribute files to the replicated servers by adding them directly on the primary server using GridFS. Distribution works fine.
I have created a simple php watcher script which is executed on the secondary servers using read preference \MongoClient::RP_NEAREST. I wanted to determine the timestamp when the replicated received all files from the primary.
I wanted to make sure that the php script on the secondary servers are using the mongodb instance on their server (and not the primary), and therefore I stopped the primary mongodb server. After doing this, the two servers are keeping their secondary role.
Issue
If the primary server is unavailable, I was still able to execute queries like count() and find() on regular collections (also fs.files collection).
But calls that use GridFS will throw an MongoConnectionException: No candidate servers found exception.
Script
I have created a little script with which you should be able to reproduce the error.
$serverList = '...';
$conn = new \MongoClient(
'mongodb://'.$serverList,
array(
'replicaSet' => 'r0',
'readPreference' => \MongoClient::RP_NEAREST,
'username' => 'bat',
'password' => '',
'db' => 'bat'
)
);
$db = $conn->selectDB('bat');
echo 'works fine...';
$files = $db->selectCollection('fs.files');
$documentsCount = $files->count();
$documents = $files->find();
foreach($documents as $document) {
echo $document['filename'] . ', ';
}
echo 'throws exception...';
$gridfs = $db->getGridFS();
$documentsCount = $gridfs->find()->count();
$documents = $gridfs->find();
foreach($documents as $document) {
echo $document->getFilename();
}
If the primary server is unavailable, the lines after echo 'works fine...'; will work fine, while the line after echo 'throws exception...'; will throw an exception.
Maybe this is related to an issue in the java driver JAVA-401, there was a similar problem with usage of secondary servers and gridfs. Maybe the gridfs ist trying to ensuring indices if the fs.files collection contains less than 1000 files which is not possible on secondary.
I figured out the problem, it was simply the incorrect line $gridfs->find()->count(). If you execute $gridfs->count() or $gridfs->find() it works fine.
So you'll have to use $gridfs->count() instead of $gridfs->find()->count().
I dont't know why $gridfs->find()->count() works correctly (and only correctly) if a primary server is available.

Anyway to do MongoDB Sharding in PHP?

I have started using MongoDB for one of my PHP project. In that database, i am trying to use MongoDB sharding concept. I got the below link and tried,
MongoDB Sharding Example
It is working well. But the problem is, in the above example, everything is done in command prompt. But i am trying to do everything in PHP. I am not able to get any example in PHP.
So far, i have started with these piece of codes,
$connection = new Mongo();
$db = $connection->selectDB('TestDB');
$db = $connection->TestDB;
$connection->selectDB('admin')->command(array('addshard'=>'host:port'));
$connection->selectDB('admin')->command(array('enablesharding'=>'TestDB'));
$connection->selectDB('admin')->command(array('shardcollection'=>'TestDB.large', 'key' => '_id'));
It is not working. Also, i dont know how to set shard servers and config database in PHP.
Is there any other way to do MongoDB sharding in PHP?
MongoDB's database commands are case-sensitive. You were passing in lowercase command names, while the real commands are camelCased (see: sharding commands). Additionally, the key paremeter of shardCollection needs to be an object.
A corrected version of your above code would look like:
$connection->selectDB('admin')->command(array('addShard'=>'host:port'));
$connection->selectDB('admin')->command(array('enableSharding'=>'TestDB'));
$connection->selectDB('admin')->command(array('shardCollection'=>'TestDB.large', 'key' => array('_id' => 1)));
Additionally, you should have been able to examine the command result of to determine why this wasn't working. The ok field would be zero, and an errmsg field would explain the error (e.g. "no such cmd: addshard").
Lastly, if you're attempting to mimic the shell's functionality to configure sharding in PHP, you may realize that some of the shell methods are more than just database commands. For any of those methods, you can get the JS source by omitting the parentheses. For example:
> sh.addTagRange
function ( ns, min, max, tag ) {
var config = db.getSisterDB( "config" );
config.tags.update( {_id: { ns : ns , min : min } } ,
{_id: { ns : ns , min : min }, ns : ns , min : min , max : max , tag : tag } ,
true );
sh._checkLastError( config );
}
That would be a useful hint for what an addTagRange() function in PHP would need to do.

PHP AMI Connection - saving incoming and outgoing calls to a database is not working

Using the AMI (API connection with an Asteriskserver so I can use a PHP Socket connection) I'm trying to catch the recieving data using PHP in a way that I can record outgoing and incomming calls for the CRM system (webbased) used at the company I work for.
But I'm not getting the result I am hoping for...
The full code can be found on PasteBin http://pastebin.com/AwRNBW2G
I catch the outgoing calls this way, and that works:
if($givenkey = array_search("Context: from-internal", $content)){
$calleridKey = $givenkey + 1;
$idSIP = $givenkey - 1;
$dialNumber = str_replace("Extension: 0","31",$content[$calleridKey]);
$dialNumber = str_replace("Extension: ", "", $dialNumber);
$fromSIP = str_replace("Channel: SIP/", "", $content[$idSIP]);
$fromSIP = substr($fromSIP, 0, 2);
$dialTime = date('r');
$uitgaand = array(
"Phonenumber" => $dialNumber,
"Type" => "Uitgaand",
"datetime" => $dialTime,
"SIP" => $fromSIP
);
The incomming calls are being catched this way, but that's not working properly:
if($givenkey = array_search("AppData: Using CallerID ", $content)){
if(array_search("Channel: SIP/31000000000", $content)+5 == $InCallKey = array_search("AppData: Using CallerID", $content)){
$calleridNum = explode('"',str_replace('AppData: Using CallerID "',"",$content[$InCallKey]));
$pickupSource = array_search("Source: SIP/31000000000", $content);
if($pickupSource+1 == $pickupKey = array_search("Destination: SIP/", $content)){
$pickupBy = str_replace("Destination: SIP/","",$content[$pickupkey]);
$pickupBy = substr($pickupBy, 0, 2);
$dialTime = date('r');
$inkomend = array(
"Phonenumber" => $calleridNum[0],
"Type" => "Binnenkomend",
"datetime" => $dialTime,
"SIP" => $pickupBy
);
I have the array that I make not available right now but, if necessary, I can save the array and post it here with personal data filtered.
I know that the code I'm using right now is not neat, but I wrote it with the goal: quick result. If I have a working code I will optimize it and clean it up. Tips about this are also very welcome. It's too bad that I cannot find any good documentation about this so I have to start from the beginning and could only find the working class that I'm using right now, however it's not very complete.
I had to write this part without any knowledge about VOIP or AMI or Asterisk.
To be short, here are my questions:
- How can I record incoming and outgoing calls to eventually save them in a database by using the AMI?
- How can I keep alive the connection with the server the best way? The method I'm using now is not optimal as the connection fails atleast once within 48 hours.
- Do you have tips or suggestions about optimizing the code and neater code-writing? And do you maybe know any functions that I could use instead of a function that I am using?
With regards,
Dempsey
Since recently I get this error which
I cannot solve properly. This error
creates itself after about 15 minutes
running. It would run atleast 24 hours
before:
PHP Notice: fwrite(): send of 16 bytes failed with errno=32 Broken pipe in /var/www/html/phpami/AMILoader.php on line 147
Net_AsteriskManagerException: Authorisation failed in /var/www/html/phpami/AMILoader.php on line 173
#0 /var/www/html/phpami/AMILoader.php(173): Net_AsteriskManager-login('GEBRUIKERSNAAM','WACHTWOORD')
#1 /var/www/html/phpami/AMILoader.php(306): Net_AsteriskManager-_sendCommand('Action: Ping???...')
#2 /var/www/html/phpami/AMILoader.php(543): Net_AsteriskManager->ping()
#3 {main}
Can anyone help me with this too? The
authorisation data is correct (it is
using the same data in the whole
script and it does get a connection).
Also I don't get the response 'Action:
Ping???...' which it says is being
send by the script as command, but
where do the three questionmarks and
periods come from?
this framework should be handy:
https://github.com/marcelog/PAMI
otherwise you can check vTiger sources and how it handles ami integration:
http://www.vtiger.com/index.php?Itemid=57&id=30&option=com_content&task=view
If your using php, one of the easiest way to connect to the AMI is using the php-astmanager class. It supports callback on certain "events" so that you can catch the data you need. The best (only?) maintained copy is part of FreePBX and can be pulled right out of the latest version (2.9 as of this writing).

PHP Memcached CAS (check and set) issue

Hey all, here are the version of my current setup
Memcached (1.2.2)
Pecl Memcached Client 1.0.2 (using libmemcached 0.43)
Issue: I cant get a cas token returned during a get request
Here is the code in question!
27 public function action_test() {
28 //phpinfo();
29 $m = Model_Mem::getSingleton();
30 $found = $m->get('navigation');
31 echo (int)count($found).'<br />'; // Returns an array of navigation objects
32
33 $cas = 0;
34 $found = $m->get('navigation', null, &$cas);
35 echo (int)count($found); // Returns nothing!
36
37 exit;
38 }
The output from the first echo is 7, and the second echo is 1. Also, the $cas variable as well as the $found variable from the second group of code are both empty. Im not 100% sure if I am doing this right but the cas token just doesnt seem to be working for me at all. Ive went through the php Memcached documentation with no mention on any kind of CAS enable flag that i could easily spot. Ive also tried to look at the memcached.org site for some info but im lost!
Ive never had any problems with it, its just everytime i try and use the cas functionality on a get request i do something wrong. Thanks for anyone helping me out!
*EDIT
Here is what the Model_Mem::getSingleton() function returns
static function getSingleton() {
if (self::$m)
return self::$m;
self::$m = new Memcached();
$servers = array(
array('127.0.0.1', 11211, 25),
array('127.0.0.1', 11212, 25),
array('127.0.0.1', 11213, 25),
array('127.0.0.1', 11214, 25)
);
// Sets up some options for the memcache server
self::$m->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
self::$m->setOption(Memcached::OPT_PREFIX_KEY, Kohana::config('globals.prefix'));
self::$m->addServers($servers);
return self::$m;
}
Arg, debian's latest [secure] memcached release didn't have this feature yet. Upgraded to the latest by installing the memcached server's source and all is well.
Just FYI, CAS means "compare-and-swap": https://en.wikipedia.org/wiki/Compare-and-swap

Categories