I am using MongoDB, with PHP YII. I have used YiiMongoDbSuite for setting up the criteria for mongoDB Queries.
Currently, I am using Group by and Limit together. But due to some reason queries are returning less number of rows than that are expected.
$criteria=new EMongoCriteria();
$criteria->group('col_1');
$criteria->limit(10);
$result = TableName::model()->findAll($criteria);
Can somebody guide me as I am quite new to MongoDB and YiiMongoDbSuite.
Thanks in advance,
Well to do it using MongoYii (which I maintain):
$result = MongoModel::model()->aggregate(
array(
'$group' => array('_id' => 'col_1'),
'$limit' => 10
)
)
I am unsure how to do it with YiiMongoDbSuite, in fact there is no group command in its EMongoCriteria from what I see.
Related
I'm retrieving some traffic data of a website using "scan" option in Dynamodb. I have used filterExpression to filter those out.
I will be doing scanning against a large table which will have more than 20GB of data.
I found that DynamoDB scans throguh the entire table and filter the results out. The document says it only returns 1MB of data and then i have to loop through again to get the rest. It seems to be bad way to make this work.
got the reference from here: Dynamodb filter expression not returning all results
For a small table that should be fine.
MySQL dose the same I guess. I'm not sure.
Which is faster to read is it MySQL select or DynamoDB scan on a large set of data. ?
Is there any other alternative? what are your thoughts and suggestions?
I'm trying to migrate those traffic data into Dynamodb table and then query it out. It seems like a bad idea to me now.
$params = [
'TableName' => $tableName,
'FilterExpression' => $this->filter.'=:'.$this->filter.' AND #dy > :since AND #dy < :now',
'ExpressionAttributeNames'=> [ '#dy' => 'day' ],
'ExpressionAttributeValues'=> $eav
];
var_dump($params);
try {
$result = $dynamodb->scan($params);
After considering the suggestion this is what worked for me
$params = [
'TableName' => $tableName,
'IndexName' => self::GLOBAL_SECONDARY_INDEX_NAME,
'ProjectionExpression' => '#dy, t_counter , traffic_type_id', 'KeyConditionExpression' => 'country=:country AND #dy between :since AND :to',
'FilterExpression' => 'traffic_type_id=:traffic_type_id' 'ExpressionAttributeNames' => ['#dy' => 'day'],
'ExpressionAttributeValues' => $eav
];
If your data is like Key-Value pair and you have fixed fields on which you want to index, use DynamoDB - you can create indexes on all fields you want to query and it will work great
If you require complex querying on multiple indexes, then any RDBMS is good.
If you can query on just about anything, think about Elastic search
If your queries are very simple, but you have large data to be retrieved in each query. Think about S3. Maybe you can index metadata in DynamoDb and actual data can be in S3
I am fairly new to PHP and working with a MongoDB, so please be patient :)
This is an example of what I am trying to do, not the actual data, so hopefully this helps!
I have a MongoDB Collection that has nested objects in it and I want to sort the results of a query based on one of the nested fields.
Sample row from the DB via MongoDB Compass:
_id:"12345"<br>
houses:Object<br>
> 124 Happy St.:Object<br>
> name:"124 Happy St."<br>
> price:"$123,000"<br>
> 123 Happy St.:Object<br>
> name:"123 Happy St."<br>
> price:"$124,000"<br>
> 125 Happy St.:Object<br>
> name:"125 Happy St."<br>
> price: "$200,000"<br>
I want to be able to sort my query based on the price field (highest to lowest) in the houses object.
I've tried the following but I can't see to get the results to return sorted as expected - remember, I'm not that strong yet with PHP and MongoDB so go easy on me :P
$houses_query = new MongoDB\Driver\Query([ "_id" => "12345" ], [ 'sort' => [ 'houses.price' => 1 ]])
Is there a way to filter the query or do I have to read the rows into an array and sort that way?
Absolutely, mongoDB has a robust sort algorithm and tools like sort, order_by etc. You can do with php arrays but mongoDB has a lot of blessings :)
I did similar solution nowadays, you could do like this (your mongo class can be diffrent)
$videos = $this->mongo_db->where(
array('$text' => array('$search' => $q),
'status' => 1))
->order_by(array('video_sales' => 'desc'))
->get('videos');
Mongo documentation says to us;
db.collection.find().sort( { age: -1 } )
Deprecated in the mongo Shell since v3.2
Starting in v3.2, the $orderby operator is deprecated in the mongo shell. In the mongo shell, use cursor.sort() instead.
I have a query like this
$results = $collection->find([
'status' => "pending",
'short_code' => intval($shortCode),
'create_time' => ['$lte' => time()],
])
->limit(self::BATCH_NUMBER)
->sort(["priority" => -1, "create_time" => 1]);
Where BATCH_SIZE is 70.
and i use the result of query like below :
foreach ($results as $mongoId => $result) {
}
or trying to convert in to array like :
iterator_to_array($results);
mongo fetch data and traveling on iterate timing is :
FetchTime: 0.003173828125 ms
IteratorTime: 4065.1459960938 ms
As you can see, fetching data by mongo is too fast, but iterating (in both case of using iterator_to_array or using foreach) is slow.
It is a queue for sending messages to another server. Destination server accept less than 70 documents per each request. So i forced to fetch 70 document. anyway. I want to fetch 70 documents from 1,300,000 documents and we have problem here.
query try to fetch first 70 documents which have query conditions, send them and finally delete them from collection.
can anybody help? why it takes long? or is there any config for accelerating for php or mongo?
Another thing, when total number of data is like 100,000 (isntead of 1,300,000) the traveling is fast. traveling time will increase by increasing number of total documents.
That was because of sorting.
The problem :
Fetching data from mongo was fast, but traveling in iterator was slow using foreach.
solution :
there was a sort which we used. sorting by priority DESC and create_time ASC. These fileds was index ASC seperatly. Indexing priority DESC and create_time ASC together fixed problem.
db.queue_1.createIndex( { "priority" : -1, "create_time" : 1 } )
order of fileds on indexing is important. means you should use priority at first. then use create_time.
because when you try to sort your query, you sort them like below :
.sort({priority : -1, create_time : 1});
So I have a MongoDB document that tracks logons to our app. Basic structure appears thusly:
[_id] => MongoId Object
(
[$id] => 50f6da28686ba94b49000003
)
[userId] => 50ef542a686ba95971000004
[action] => login
[time] => 1358354984
Now- the challenge is this: there are about 20,000 of these entries. I have been challenged to look at the number of times each user logged in (as defined by userId)...so I am looking for a good way to do this. There are a couple of possible approaches that I've seen (in SQL, for example, I might pull down number of logins by grouping by UserID and doing a count on it- something like SELECT userID, count(*) from....group by UserId...and then sub-selecting on that (CASE WHEN or something in the top select).
Anyways- wondering if anyone has any suggestions on the best way to do this. Worst case scenario I can limit the result set and do the grouping in memory- but ideally would like to get the full answer directly from Mongo.
The other limitation (even after I get past the first set) is that I am looking to do a unique count by date...which will be even tougher!
Now- the challenge is this: there are about 20,000 of these entries.
At 20,000 you will probably be better off with the aggregation framework ( http://docs.mongodb.org/manual/applications/aggregation/ ):
$db->user->aggregate(array(
array( '$group' => array( '_id' => '$userId', 'num_logins' => array( '$sum' => 1 ) ) )
));
That will group ( http://docs.mongodb.org/manual/reference/aggregation/#_S_group ) by userId and count (sum: http://docs.mongodb.org/manual/reference/aggregation/sum/#_S_sum ) the amount of grouped login there are.
Note: As stated in the comments, the aggregate helper is in version 1.3+ of the PHP driver. Before version 1.3 you must use the command function directly.
You can use MapReduce to group the results by user ID
http://docs.mongodb.org/manual/applications/map-reduce/#map-reduce-examples
Or you can use the Group method:
db.logins.aggregate(
{ $group : {
_id : "$userId",
loginsPerUser : { $sum : 1 }
}}
);
For MongoDB 20K or even more won't be a problem to walk and combine them so no worries about performance.
http://docs.mongodb.org/manual/reference/command/group/
db.user.group({key: {userId: 1}, $reduce: function ( curr, result ) { result.total++ }, initial: {total: 0}});
I ran this on 191000 rows in just a couple seconds but group is limited to 20,000 unique entries so it really isn't a solution for you.
New to Mongo here. I'm using the PHP lib and trying to work out how I can find in a collection from multiple queries. I could do this by repeating the query with a different query, but I wondered if it can be done in one.
I.e.
$idsToLookFor = array(2124,4241,5553);
$query = $db->thisCollection->find(array('id' => $idsToLookFor));
That's what I'd like to do. However it doesn't work. What I'm trying to do is find a set of results for all the id's at one time.
Possible or just do a findOne on each with a foreach/for?
Use $in Operator
$idsToLookFor = array(2124,4241,5553);
$query = $db->thisCollection->find(array(
'id' => array('$in' => $idsToLookFor)
));