I used to group on mongoDB via PHP to get the max date of my items.
As i have too many items (more than 10 000), i read i must use MapReduce.
Here's my past group function :
$keys = array('ItemDate'=> true);
$initial = array('myLastDate' => 0);
$reduce = "function(obj, prev) {
if (myLastDate < obj.ItemDate) {
myLastDate = ItemDate;
}
}";
$conds = array( 'ItemID' => (int) $id );
$results = $db->items->group($keys, $initial, $reduce,
array('condition'=> $conds ) );
I've tried something but seems not to work ...
$map = new MongoCode("function() {
emit(this.ItemID,this.ItemDate);
}");
$reduce = new MongoCode("function(obj, prev) {
if(prev.myLastDate < obj.ItemDate) {
prev.myLastDate = obj.ItemDate;
}
}");
$items = $db->command(array(
"mapreduce" => "items",
"map" => $map,
"reduce" => $reduce,
"query" => array("ItemID" => $id);
$results = $db->selectCollection($items['result'])->find();
Can you please help ?
Solution
You don't need to use map/reduce for that. Provided your date field contains an ISODate, a simple query does the trick:
db.yourColl.find({},{_id:0,ItemDate:1}).sort({ItemDate:-1}).limit(1)
In order to have this query done efficiently, you need to set an index on ItemDate
db.yourColl.createIndex({ItemDate:-1})
Explanation
The query
Let us dissect the query. db.yourColl...
.find({} The default query
,{_id:0,ItemDate:1} We want only ItemDate to be returned. This is called a projection.
.sort({ItemDate:-1}) The documents returned should be sorted in descending order on ItemDate, making the document with the newest date the first to be returned.
.limit(1) And since we only want the newest, we limit the result set to it.
The index
We create the index in descending order, since this is the way you are going to use it. However, if you need to change the default query to something else, the index you create should include all fields you inspect in the query, in the exact order.
Related
I have this following Yii 2 query
$find = People::find()->where(['c_id' => $c_id])->orderBy('totals DESC, id DESC')->all();
So imagine this query was an array. Everything found by this query has an "id" attribute.
Since it's sorted by "totals", I essentially want to return the position in the array where I can find this specific id.
Currently, I'm using this code.
foreach ($find as $t) {
$arr[] = $t->id;
if ($t->id == $id) {
break;
}
}
$key = count($arr);
return $key;
However, this code is vany wayow on a 100k+ result query.
Is there anyway to speed this up?
You could get the result as an array (instead of object) as
$find = People::find()->where(['c_id' => $c_id])
->orderBy('totals DESC, id DESC')
->asArray()
->all();
then you could find your value using array_search()
$my_index = array_search($id,$find);
but for 100k+ you should find using a direct select in db...instead tha looping on php or load all in php and scan with array_search()
To get array from query in YII, you can use queryAll();
$find = People::find()->where(['c_id' => $c_id])->orderBy('totals DESC, id DESC')->queryAll();
OR, another way to convert the object into an array is:
$find = json_decode(json_encode($find), true); // to convert all data into array.
And once you get results in array, you can implement the actual code for your requirement as given below.
You can use array_search() function to get index of your value.
$a=array("a"=>"red","b"=>"green","c"=>"blue");
echo array_search("red",$a);
The array_search() function search an array for a value and returns the key.
Maybe I didn't understand you correctly but I assume that you are trying to detect the index or key for your desired id inside an array returned from an SQL query that is sorted by some other column like total.
So let us fetch records from the database with your query with a little change asArray() like this
$find = People::find()
->where(['c_id' => $c_id])
->orderBy('totals DESC, id DESC')
->asArray()
->all();
in the result, let us assume the People table returns you an array with the following dataset ordered by columns total and id DESC.
[
0 => [
'id' => 2 ,
'c_id'=>2,
'name' => 'John' ,
'age'=>18,
'totals'=>100,
],
1=>[
'id'=>1,
'c_id'=>55,
'name'=>'Bob',
'age'=>20,
'totals'=>80,
],
2=>[
'id'=>3,
'c_id'=>85,
'name'=>'Peter',
'age'=>15,
'totals'=>75,
]
];
Now if you look into \yii\helpers\ArrayHelper you will find ArrayHelper::getColumn().
Let us use this on the array we received from the query, I assume that you are searching $id inside the column id so we will first filter out the id column like below.
$idsArray = ArrayHelper::getColumn($find, 'id');
this will give us the ids in the following sequence which is in the same order as the initial result set.
[2,1,3]
then lets use the built-in php function array_search()
$key=array_search($yourId,$idsArray);
Hope this is what you are looking for.
I use this code in the MongoDB PHP driver to get all documents in the database
$result = $collection->find();
foreach ($result as $doc) {
print_r($doc);
}
However, when adding a limit to it, it doesn't work anymore: no documents get printed anymore:
$result = $collection->find()->limit(10);
foreach ($result as $doc) {
print_r($doc);
}
There are certainly enough documents in the database. I cannot figure out what the problem with this is.
I have fixed the problem by taking a look at the source of the beta version. The documentation only appeared to be for the legacy mongo extension and not the newer mongodb extension.
The error logs showed this: Call to undefined method MongoDB\\Driver\\Cursor::addOption(). I checked out the documentation and concluded the function should have worked because it said (PECL mongo >=0.9.0). Note the missing db after mongo.
I fixed it by doing:
$collection->find([], [ 'limit' => 2 ]);, providing an empty filters array and adding my options in another array afterwards.
I am trying to describe with example for new php mongodb driver. showing in example skip,limit,Fields slection
require 'vendor/autoload.php'; // include Composer's autoloader
$client = new MongoDB\Client("mongodb://localhost:27017");
// SELECT * FROM YOUR_TABLE_NAME ;
// db.YOUR_COLLECTION_NAME.find({});
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array());
//SELECT * from YOUR_TABLE_NAME WHERE YOUR_COLUMN = "A"
// db.YOUR_COLLECTION_NAME.find({{ YOUR_FIELD: "A" }});
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array('YOUR_FIELD'=>'A'));
//Return the Specified Fields and the _id Field Only
//SELECT _id, item,status YOUR_TABLE_NAME from inventory WHERE status = "A"
//db.YOUR_COLLECTION_NAME.find( { status: "A" }, { item: 1, status: 1 } )
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array('status'=>'A'),array('projection' =>array('item'=>TRUE,'status' => TRUE)));
//Suppress _id Field
//SELECT item, status from YOUR_TABLE_NAME WHERE status = "A"
//db.YOUR_COLLECTION_NAME.find( { status: "A" }, { item: 1, status: 1, _id: 0 } )
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array('status'=>'A'),array('projection' =>array('item'=>TRUE,'status' => TRUE,'_id'=>FALSE)));
//SELECT * FROM YOUR_TABLE_NAME LIMIT 10
//db.YOUR_COLLECTION_NAME.find({}).limit(10);
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array(),array('limit'=>10));
//SELECT * FROM YOUR_TABLE_NAME LIMIT 5,10
//db.YOUR_COLLECTION_NAME.find({}).skip(5).limit(10)
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array(),array('skip'=>5,'limit'=>10));
//Suppress _id Field
//SELECT item, status from YOUR_TABLE_NAME WHERE status = "A" LIMIT 5,10;
//db.YOUR_COLLECTION_NAME.find( { status: "A" }, { item: 1, status: 1, _id: 0 } ).skip(5).limit(10);
$result = $clinet->YOUR_DB_NAME->YOUR_COLLECTION_NAME->find(array('status'=>'A'),array('projection' =>array('item'=>TRUE,'status' => TRUE,'_id'=>FALSE),'skip'=>5,'limit'=>10));
foreach ($result as $entry){
echo "<pre>";
print_r($entry);
echo "</pre>";
}
As a solution to your above mentioned problem please try executing following code snippet.
$result = $collection->find();
$result->limit(10);
foreach ($result as $doc) {
print_r($doc);
}
I had the same issue. There are a lot of code examples using $result = $collection->find()->limit(10);
It turns out, that while this was totally valid for the original version of the MongoDB PHP driver, there is a new version of that very same driver. The original driver is now considered "The Legacy Driver".
Here is one example, how the "old" driver was supposed to be used:
<?php
$m = new MongoClient;
// Select 'demo' database and 'example' collection
$collection = $m->demo->example;
// Create the cursor
$cursor = $collection->find();
At this moment, although a cursor object had been created, the query had not yet executed (i.e. it was not sent to the server). The query would only be executed by starting iteration with foreach ( $cursor as $result ) or calling $cursor->rewind(). This gives you the chance to configure the cursor's query with sort(), limit(), and skip() before it is executed by the server:
// Add sort, and limit
$cursor->sort( [ 'name' => 1 ] )->limit( 40 )
In the new driver, as soon as you have a \MongoDB\Driver\Cursor object, it has already been processed by the server. Because sort (and limit and skip) parameters need to be sent to the server before the query is executed, you can not retroactively call them on an existing Cursor object.
This is the reason, why there is no limit() method anymore, as there used to be. Also, the accepted answer is correct. I want to give a more elaborate example:
$filter = [
'author' => 'rambo',
'views' => [
'$gte' => 100,
],
];
$options = [
/* Return the documents in descending order of searchPage */
'sort' => [
'searchPage' => -1
],
/* Limit to 2 */
'limit' => 2,
/* close the cursor after the first batch */
'singleBatch' => true,
];
$cursor = $collection->find($filter, $options);
This is the first time i create my own webservice (someone always did it for me before), so please bear with me.
I post this array :
$data = array(
'user_id' => $this->post('user_id'),
'group_id' => $this->post('group_id'),
'child_id' => $this->post('child_id'), //will be nested array
'custom' => $this->post('custom'),
'time' => $this->post('time'),
'date' => $this->post('date')
);
I tried to create a nested array with this : $this->post('child_id'), because user can post multiple child_id at once.
Then i tried to iterate through the child_id, because i need to insert them to the mysql :
for($i = 0; $i < sizeof($data['child_id']); $i++)
{
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data, $result_id[0]['id']);
}
What should i do, so i can have an array of child_id in my $data array? (nested array)
And how to iterate through it?
UPDATE :
I have updated the codes above.
I use advanced rest client for testing, and i tried to post something like this in the form content type :
child_id=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id=2
Notice that theres two child_id (left most and right most), but only the last one (right most) is inserted.
And this is the add_trans in the model :
function add_trans($table, $data, $schedule_id) {
$query = $this->db->insert($table, array('child_id' => $data['child_id'], 'schedule_id' => $schedule_id));
return $query;
}
Thanks a lot for your time.
Even thought you set the name attribute as child[] on the markup,
You still need to call it as:
'child_id' => $this->post('child_id')
It will still return an array.
for($i = 0; $i < sizeof($data['child_id']); $i++) {
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data, $result_id[0]['id']);
}
EDIT:
Looking upon you query string, that seems to be the culprit:
child_id=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id=2
^ same index , same index, same index, it will overwrite and you will get only `2`
If you want to get them all into an array format, you need to set them like this
child_id[]=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id[]=2
^ it needs to be set like this
UPDATE:
And in your model, if you want each id per row, well you can also loop in this case:
function add_trans($table, $data, $schedule_id) {
foreach($data['child_id'] as $child_id) {
$query = $this->db->insert($table, array('child_id' => $child_id, 'schedule_id' => $schedule_id));
}
// return $this->db->insert_id();
return $query;
}
ofcourse that won't work, it has to be
for($i = 0; $i < sizeof($data['child_id']); $i++)
{
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data['child_id'][$i], $result_id[0]['id']);
}
because you've not set $data['child_id[]'] so it doesn't exist, the key is just a string or number, it does not validate or parse anything
you don't need to give child[] in post method. just give only child, it will get complete array what are you sending from views
replace
'child_id' => $this->post('child_id[]')
with
'child_id' => $this->post('child_id')
I have reached very strange problem using Zend Framework 1 and binding the resultset to BvbGrid.
We have a method which queries an external API. You need to specify the following params: methodName, params.
So the controller recieves a resultset as an array from the external API.
The resultset (in the database) is 4 columns - 1 string, 2 DateTime, 3 int, 4 int. And is with about 3000 rows.
When I recieve the resultset in the controller I bind it to Bvb Grid, which should evaluate in the view as columns with search boxes.
So far, so good, when I search in the first or in the last column (string, int), it searches in the whole resultset.
But when I search 2, 3 (DateTime, int) it searches only in the current page.
I don't understand why is that. There is no difference in data binding, or something.
The code is as follows
public function indexAction() {
$pagesize = $this->_request->getPost('rows');
$pagenum = $this->_request->getPost('page');
$pagesize = ($pagesize) ? $pagesize : 30;
$pagenum = ($pagenum) ? $pagenum : 1;
$params = array();
$params['page_num'] = $pagenum;
$params['page_limit'] = $pagesize;
if ($this->_request->isXmlHttpRequest()) {
// additional search filter
if ($this->_request->getPost('username')) {
$params['username'] = $this->_request->getPost('username');
}
if ($this->_request->getPost('lastlogin')) {
$params['lastlogin'] = $this->_request->getPost('lastlogin');
}
if ($this->_request->getPost('type')) {
$params['type'] = $this->_request->getPost('type');
}
if ($this->_request->getPost('level')) {
$params['level'] = $this->_request->getPost('level');
}
}
if (($order_by = $this->_request->getPost('sidx')) && ($order_type = $this->_request->getPost('sord'))) {
$params['order_by'] = $order_by;
$params['order_by_type'] = $order_type;
}
$resultset = $this->web_service->_apiRequest('SearchUsers', $params);
/**
* The resultset is like
* array(
'found_rows' => 3000,
'body' => array(
'username' => 'blabla',
'lastlogin' => '2014-02-25 13:33:38.1234',
'type' => 1,
'level' => 199
)
);
*/
$this->view->count = $resultset['found_rows'];
if ($resultset['found_rows'] > 0 || $this->_request->isXmlHttpRequest()) {
$grid = new Bvb_Grid_Deploy_JqGrid($this->_gridconfig);
$grid->setSource(new Bvb_Grid_Source_Array($resultset['body']));
$grid->getSource()->setTotalRecords($resultset['found_rows']);
$grid->updateColumn('username', array('width' => '110',));
$grid->updateColumn('lastlogin', array('width' => '110',));
$grid->updateColumn('type', array('width' => '110',));
$grid->updateColumn('level', array('width' => '110',));
$grid->setJqgParams(array('caption' => 'resultset'
, 'forceFit' => true
, 'viewrecords' => true
, 'rowList' => array(15, 30, 50)
, 'altRows' => false
)
);
$grid->ajax(get_class($grid));
$grid->setNumberRecordsPerPage(30);
$grid->setExport(array());
$this->view->grid = $grid->deploy();
}
I have tried to examine the BvbGrid's code, but found nothing strange.
In another project the same code is used, and it works fine, without any problems on searching in any column. I have dump'd the API response to be sure it gives me 3000 records, and it actually is and they are in the way as in the comment. I really cannot find the reason why i.e. the lastlogin column searches only on the current page.
If someone can help me, or give me directions where to look at to fix the issue (maybe somewhere in the BvbGrid's code?) I will appreciate it.
Thank you in advance.
I have a problem that I need some help on but I feel I'm close. It involves Lithium and MongoDB Code looks like this:
http://pastium.org/view/0403d3e4f560e3f790b32053c71d0f2b
$db = PopularTags::connection();
$map = new \MongoCode("function() {
if (!this.saved_terms) {
return;
}
for (index in this.saved_terms) {
emit(this.saved_terms[index], 1);
}
}");
$reduce = new \MongoCode("function(previous, current) {
var count = 0;
for (index in current) {
count += current[index];
}
return count;
}");
$metrics = $db->connection->command(array(
'mapreduce' => 'users',
'map' => $map,
'reduce' => $reduce,
'out' => 'terms'
));
$cursor = $db->connection->selectCollection($metrics['result'])->find()->limit(1);
print_r($cursor);
/**
User Data In Mongo
{
"_id" : ObjectId("4e789f954c734cc95b000012"),
"email" : "example#bob.com",
"saved_terms" : [
null,
[
"technology",
" apple",
" iphone"
],
[
"apple",
" water",
" beryy"
]
] }
**/
I am having a user savings terms they search on and then I am try to get the most populars terms
but I keep getting errors like :Uncaught exception 'Exception' with message 'MongoDB::__construct( invalid name '. does anyone have any idea how to do this or some direction?
First off I would not store this in the user object. MongoDb objects have an upper limit of 4/16MB (depending on version). Now this limit is normally not a problem, but when logging inline in one object you might be able to reach it. However a more real problem is that every time you need to act on these objects you need to load them into RAM and it becomes consuming. I dont think you want that on your user objects.
Secondly arrays in objects are not sortable and have other limitations that might come back to bite you later.
But, if you want to have it like this (low volume of searches should not be a problem really) you can solve this most easy by using a group query.
A group query is pretty much like a group query in sql, so its a slight trick as you need to group on something most objects share. (An active field on users maybe).
So, heres a working group example that will sum words used based on your structure.
Just put this method in your model and do MyModel::searchTermUsage() to get a Document object back.
public static function searchTermUsage() {
$reduce = 'function(obj, prev) {
obj.terms.forEach(function(terms) {
terms.forEach(function(term) {
if (!(term in prev)) prev[term] = 0;
prev[term]++;
});
});
}';
return static::all(array(
'initial' => new \stdclass,
'reduce' => $reduce,
'group' => 'common-value-key' // Change this
));
}
There is no protection against non-array types in the terms field (you had a null value in your example). I removed it for simplicity, its better to probably strip this before it ends up in the database.