Count too slow in MongoDB with PHP - php

I'm trying to check my code, with count lines. But this code works very slow. how can i optimize this code? is there anyway to count?
$find = $conn_stok->distinct("isbn");
for($i=0;$i<=25; $i++) {
$isbn = $find[$i];
$countit= $conn_kit->find(array('isbn'=>$isbn))->count();
if($countit> 0){
echo "ok<br>";
} else {
echo "error<br>";
}
}

Looks like you are trying to do a simple count(*) group by in the old SQL speak. In MongoDB you would use the aggregation framework to have the database do the work for you instead of doing it in your code.
Here is what the aggregation framework pipeline would look like:
db.collection.aggregate({$group:{_id:"$isbn", count:{$sum:1}}}
I will let you translate that to PHP if you need help there are plenty of examples available.

It looks like you're trying to count the number of 25 top most ISBNs used, and count how often they have been used. In PHP, you would run the following queries. The first one to find all ISBNs, and the second is an aggregation command to do the grouping.
$find = $conn_stok->distinct( 'isbn' );
$aggr = $conn_kit->aggregate(
// find all ISBNs
array( '$match' => array( 'isbn' => array( '$in' => $find ) ) ),
// group those
array( '$group' => array( '_id' => '$isbn', count => array( '$sum' => 1 ) ) ),
// sort by the count
array( '$sort' => array( 'count' => 1 ) ),
// limit to the first 25 items (ie, the 25 most used ISBNs)
array( '$limit' => 25 ),
)
(You're a bit vague as to what $conn_stok and $conn_kit contain and what you want as answer. If you can update your question with that, I can update the answer).

Related

data escaping remove for specific filed in cakephp

I am using subquery for id field.
$db = $this->AccountRequest->getDataSource();
$subQuery = $db->buildStatement(
array(
'fields' => array('MAX(id)'),
'table' => $db->fullTableName($this->AccountRequest),
'alias' => 'MaxRecord',
'limit' => null,
'offset' => null,
'order' => null,
'group' => array("user_id")
),
$this->AccountRequest
);
$searching_parameters = array(
#"AccountRequest.id IN " => "(SELECT MAX( id ) FROM `account_requests` GROUP BY user_id)"
"AccountRequest.id IN " => "(".$subQuery.")"
);
$this->Paginator->settings = array(
#'fields' => array('AccountRequest.*'),
'conditions' => $searching_parameters,
'limit' => $limit,
'page' => $page_number,
#'group' => array("AccountRequest.user_id"),
'order' => array(
'AccountRequest.id' => 'DESC'
)
);
$data = $this->Paginator->paginate('AccountRequest');
This structure is producing a query is:
SELECT
`AccountRequest`.`id`,
`AccountRequest`.`user_id`,
`AccountRequest`.`email`,
`AccountRequest`.`emailchange`,
`AccountRequest`.`email_previously_changed`,
`AccountRequest`.`first_name`,
`AccountRequest`.`first_namechange`,
`AccountRequest`.`f_name_previously_changed`,
`AccountRequest`.`last_name`,
`AccountRequest`.`last_namechange`,
`AccountRequest`.`l_name_previously_changed`,
`AccountRequest`.`reason`,
`AccountRequest`.`status`,
`AccountRequest`.`created`,
`AccountRequest`.`modified`
FROM
`syonserv_meetauto`.`account_requests` AS `AccountRequest`
WHERE
`AccountRequest`.`id` IN '(SELECT MAX(id) FROM `syonserv_meetauto`.`account_requests` AS `MaxRecord` WHERE 1 = 1 GROUP BY user_id)'
ORDER BY
`AccountRequest`.`id` DESC
LIMIT 25
In the subquery, its add an extra single quote so it's producing an error.
So, How can I remove these single quotes from this subquery?
Thanks
What are you trying to achieve with the sub query?
The MAX(id) just means it will pull the id with the largest value AKA the most recent insert. The sub query is completely redundant when you can just ORDER BY id DESC.
using MAX() will return only one record, if this is what you want to achieve you can replicate by adding LIMIT 1
If the sub query is just an example and is meant to be from another table I would just run the query that gets the most recent id before running the main query. Getting the last inserted id in a separate query is very quick and I cant see much of a performance loss. I think it will result in cleaner code that`s easier to follow to.
edit 1: From the comments it sounds like all your trying to get is a particular users latest account_requests.
You dont need the sub query at all. My query below will get the most recent account record for the user id you choose.
$this->Paginator->settings = array(
'fields' => array('AccountRequest.*'),
'conditions' => array(
'AccountRequest.user_id' => $userID // you need to set the $userID
)
'page' => $page_number,
'order' => array(
'AccountRequest.id DESC' //shows most recent first
),
'limit' => 1 // set however many you want the maximum to be
);
The other thing you cold be meaning is to get multiple entries from multiple users and display them in order of user first and then the order of recent to old for that user. MYSQL lets you order by more than one field, in that case try:
$this->Paginator->settings = array(
'conditions' => array(
'AccountRequest.user_id' => $userID // you need to set the $userID
)
'page' => $page_number,
'order' => array(
'AccountRequest.user_id', //order by the users first
'AccountRequest.id DESC' //then order there requests by recent to old
)
);
If the example data you have added into the question is irrelevant and you are only concerned about how to do nested subqueries it has already been answered here
CakePHP nesting two select queries
However I still think based on the data in the question you can avoid using a nested query.

Cakephp get all months records with a single query i.e with single find

I am working on a ecommerce with platform cakephp and using google charts for reports.My requirement is to get all records as per all 12 months, so I have used following code for a single month
Query
SELECT COUNT(*) AS count FROM orderproductmasters AS Orderproductmaster
LEFT JOIN ordermasters AS Ordermaster ON
(Orderproductmaster.ordermaster_id = Ordermaster.id) LEFT JOIN productmasters AS Productmaster ON
(Orderproductmaster.productmaster_id = Productmaster.id)
WHERE Ordermaster.orderstatusmaster_id = 1 AND Month(Orderproductmaster.created) = 8
Code
$this->Orderproductmaster->find('count',
array('conditions'=>array('Ordermaster.orderstatusmaster_id'=>1,'
Month(Orderproductmaster.created)'=>8)));
Since, I need records as per Jan, feb,march and all 12 months...,so for 12 months I am using following code
for($i=1;$i<13;$i++)
{
$orderproductmasters[$i] = $this->Orderproductmaster->find('count',
array('conditions'=>array('Ordermaster.orderstatusmaster_id'=>1,
'Month(Orderproductmaster.created)'=>$i)));
}
So question might be silly, but is it possible to get all months record without using for loop i.e, within a single query.
Thanks in advance
I think , your need can be fulfilled by using cursors in stored procedure. And then using stored procedure to cake-php.
Example on db side is here
$options = array();
$options['fields'] = array('COUNT(Orderproductmaster.id)');
$options['conditions'] = array('Ordermaster.orderstatusmaster_id = 1',
'Month(Orderproductmaster.created) BETWEEN 1 AND 12');
$options['joins'] = array(
array(
'table' => 'ordermasters',
'alias' => 'Ordermaster',
'type' => 'left',
'conditions' => array(
'Orderproductmaster.ordermaster_id = Ordermaster.id'
)
),
array(
'table' => 'productmasters',
'alias' => 'Productmaster',
'type' => 'left',
'conditions' => array(
'Orderproductmaster.productmaster_id = Productmaster.id'
)
)
);
$options['group'] => array('Month(Orderproductmaster.created)');
$this->Orderproductmaster->find('all',$options);
What About something like:
$this->Orderproductmaster->find('count',
array(
'fields'=>'DISTINCT(Month(Orderproductmaster.created)),
'conditions'=>array('Ordermaster.orderstatusmaster_id'=>1,'
Month(Orderproductmaster.created)'=>8)));

MongoDB find query with $and $or and $nearSphere in php

After quite some trying out and web research I go crazy with this query. I want to build a query for 'Clubs' around a geo point (distance max 500 meters) in php on MongoDB.
But when I run query it ignores the distance limit and shows all clubs in database BUT sorted by distance.
Here is my dataset (2dsphere index geoLoc):
{"_id":ObjectId("547c649e30afe32c23000048"),"name":"Club Ritzz","category":"Club","category_list":[{"id":"191478144212980","name":"Night Club"}],"location":{"city":"Mannheim"},"geoLoc":{"type":"Point","coordinates":[8.473665839156,49.484065272756]}}
{"_id":ObjectId("547c649f30afe32c2300004a"),"name":"Das Zimmer Mannheim","category":"Club","category_list":[{"id":"191478144212980","name":"Night Club"}],"geoLoc":{"type":"Point","coordinates":[8.4709362941178,49.487260552592]}}
{"_id":ObjectId("547c64ab30afe32c23000063"),"name":"Nationaltheater Mannheim","category":"Arts/entertainment/nightlife","category_list":[{"id":"173883042668223","name":"Theatre"}],"geoLoc":{"type":"Point","coordinates":[8.4776534992592,49.48782606969]}}
{"_id":ObjectId("547c64a130afe32c2300004f"),"name":"SOHO Bar Club Lounge","category":"Club","category_list":[{"id":"191478144212980","name":"Night Club"},{"id":"164049010316507","name":"Gastropub"}],"geoLoc":{"type":"Point","coordinates":[8.4630844501277,49.49385193591]}}
{"_id":ObjectId("547c64a730afe32c2300005a"),"name":"Loft Club","category":"Club","category_list":[{"id":"191478144212980","name":"Night Club"},{"id":"176139629103647","name":"Dance Club"}],"geoLoc":{"type":"Point","coordinates":[8.4296300196465,49.484211928258]}}
And here my php code (updated Dec-2):
$qry = $pub->find(
array( '$and' =>
array(
array( 'geoLoc' =>
array('$nearSphere' =>
array('$geometry' =>
array('type'=>'Point',
'coordinates'=>
array(
floatval($sLon), floatval($sLat)
)
),
'maxDistance' => 500
)
)
),
array( '$or' =>
array(
array( 'name' => new MongoRegex("/.*club/i")),
array( 'name' => new MongoRegex("/.*zimm/i"))
)
),
array('$or' =>
array(
array('category_list.name' => 'Night Club'),
array('category_list.name' => 'Dance Club'),
array('category' => 'Club')
)
)
)
),
array('id' => 1, 'name' => 1, '_id' => 0)
);
Anyone know why the results are not limited to the specified maxDistance?
I found a similar issue on StackOverflow which outlines that one has to use radians for the maxDistance parameter.
See https://dba.stackexchange.com/questions/23869/nearsphere-returns-too-many-data-what-am-i-missing-am-i-wrong-is-it-a-bug-d
Also it is probably helpful if you'd test the query in mongo shell without using the PHP APIs first (just to see if the query is generally working and append '.explain()' to it to see what generally happens inside DB).

Compound Indexes on MongoDB

Sorry for my english, I need help on mongodb indexes. I have a capped collection (size: 10GB) with some fields for my application logs.
Example structure: Logs[_id, userId, sum, type, time, response, request]. I have created compound index: [userId,time,type]. I get two arrays are grouped records by userId for today, where 'type' is "null" and "1". And my two query example:
$group = array(
array(
'$match' => array(
'userId' => $userId,
'time' => array(
'$gt' => date("Y-m-d")
),
'type' => array('$ne' => null)
)
),
array(
'$group' => array(
"_id" => '$userId',
"total" => array('$sum' => '$sum'),
"count" => array('$sum' => 1)
),
)
);
$results = $collections->aggregate($group);
$group = array(
array(
'$match' => array(
'userId' => $userId,
'time' => array(
'$gt' => date("Y-m-d")
),
'type' => 1
)
),
array(
'$group' => array(
"_id" => '$userId',
"count" => array('$sum' => 1)
),
)
);
$results2 = $collections->aggregate($group);
If current user has more 100000 documents on collection for today - the speed of my query is very slow (more 10 sec). Give me some advices on creating the right index, please :) Thanks.
Based on the explain that you posted, the correct index is being used (BtreeCursor), it is using only the index (i.e. it is a covered index query - indexOnly is true) and nothing is being matched (n = 0) in this case. So, that all checks out generally, though $ne as a clause in the first example is not going to be very efficient.
However the main issue based on the explain is likely the fact that the index does not appear to be fully in memory. There are 13 yields listed and the most common reason for a query like this to yield is when it has to fault to disk to page something in. Since, as mentioned previously, it is only using the index, those yields imply faults to disk for the index and hence indicate that the whole index is not in memory.
If you re-run the query immediately after this it should be faster (assuming the index can actually fit into available memory) because the index will have been paged in by the first run. If it is still slow on the second run and showing yields, then you either don't have enough memory to hold the index in memory or something else is evicting it from memory and you essentially have memory contention causing performance problems.

Paginate do not sort DESC

I can't display Agreements with agreement_number less than 7 and order it by agreement_number DESC.
I have read Pagination CakePHP Cook Book and can't find where my code is wrong. It display only less than 7, but always ASC. I have found similar question here, [that works],(CakePHP paginate and order by) and do not know why. Agreement.agreement_number is int(4).
$this->Agreement->recursive = 0;
$agreements = $this->Paginator->paginate('Agreement', array(
'Agreement.agreement_number <' => '7'
), array(
'Agreement.agreement_number' => 'desc'
)
);
$this->set('agreements', $agreements);
}
Exact cake version is 2.5.2.
... Where did you read that that was the correct syntax?
The paginate function's third parameter is for sorting (and I mean, within the table... with those down and up arrows).
List of allowed fields for ordering. This allows you to prevent
ordering on non-indexed, or undesirable columns.
You have the exact link used for documentation of the API, but you don't seem to be following it (like, from here and here)
$this->Paginator->settings = array(
'Agreement' => array(
'order' => array('Agreement.agreement_number' => 'desc')
)
);
$agreements = $this->Paginator->paginate('Agreement', array(
'Agreement.agreement_number <' => '7'));

Categories