mongo1:27017: exception: aggregation result exceeds maximum document size (16MB) - php

I'm trying to run a mongo query in my laravel app but receiving the error listed in the title of this post. I think I need to set useCursor to true but am having trouble getting it to work. Here's my code:
$email_duplicates = User::raw(function ($collection) {
return $collection->aggregate(
[
[
'$group' => [
'_id' => [
'email' => '$email',
],
'uniqueIds' => [
'$addToSet' => '$_id',
],
'count' => [
'$sum' => 1,
],
],
],
[
'$match' => [
'count' => [
'$gt' => 1,
],
],
],
],
[
'allowDiskUse' => true,
],
[
'useCursor' => true,
]
);
});
anyone know what I'm doing wrong?
Thanks!

Related

How to use MDword to generate multi-level nested Office Word?

effect picture:
https://i.stack.imgur.com/f2r3O.png
github address:
https://github.com/mkdreams/MDword
data:
$arr = [
[
"title1" => "title1",
"meeting_content"=>[
[
"title11" => "title11,title11",
"content11" => "content,content,content,content,content,content,content,content,content,"
],
[
"title22" => "title22,title22",
"content22" => "content,content,content,content,content,content,content,content,content,"
],
],
"children" => []
],
[
"title" => "title",
"meeting_content"=>[
],
"children"=>[
[
"title1" => "title1",
"meeting_content"=>[
[
"title11" => "title11,title11",
"content11" => "content,content,content,content,content,content,content,content,content,"
],
[
"title22" => "title22,title22",
"content22" => "content,content,content,content,content,content,content,content,content,"
],
],
],
[
"title2" => "title2",
"meeting_content"=>[
[
"title11" => "title11,title11",
"content11" => "content,content,content,content,content,content,content,content,content,"
],
[
"title22" => "title22,title22",
"content22" => "content,content,content,content,content,content,content,content,content,"
],
],
],
],
],
];
How to use MDword to generate multi-level nested Office Word?
Now I need to, using a MDword extension of PHP, write this multidimensional data into a Word document, I don't know what to do, it has the effect picture, and Github address, thank you
You can use pstyle.You can see the demo.
Details as follows(You must update to the latest version):
data
$numDatas = [
[
'title'=>'title-1',
'content'=>'content-1'
],
[
'title'=>'title-2',
'sub'=>[
[
'title'=>'subTitle-2-1',
'content'=>'content-2-1',
],
[
'title'=>'subTitle-2-2',
'content'=>'content-2-2',
],
]
],
[
'title'=>'title-3',
'sub'=>[
[
'title'=>'subTitle-3-1',
'content'=>'content-3-1',
],
[
'title'=>'subTitle-3-2',
'content'=>'content-3-2',
],
]
],
];
temple IMG:
https://i.stack.imgur.com/dS1U1.png
code
$TemplateProcessor->cloneP('num',count($numDatas));
foreach($numDatas as $idx => $numData) {
$TemplateProcessor->cloneP('num'.'#'.$idx,3);
$TemplateProcessor->setValue('num'.'#'.$idx.'#0',[['text' => $numData['title'], 'pstyle' => 'numstyle-level-1', 'type' => MDWORD_TEXT]]);
if(isset($numData['content'])) {
$TemplateProcessor->setValue('num'.'#'.$idx.'#1',[['text' => $numData['content'], 'pstyle' => 'numstyle-level-3', 'type' => MDWORD_TEXT]]);
}else{
$TemplateProcessor->deleteP('num'.'#'.$idx.'#1');
}
$subName = 'num'.'#'.$idx.'#2';
if(isset($numData['sub'])) {
$TemplateProcessor->cloneP($subName,count($numData['sub']));
foreach($numData['sub'] as $subIdx => $subData) {
$TemplateProcessor->cloneP($subName.'#'.$subIdx,2);
$TemplateProcessor->setValue($subName.'#'.$subIdx.'#0',[['text' => $subData['title'], 'pstyle' => 'numstyle-level-2', 'type' => MDWORD_TEXT]]);
$TemplateProcessor->setValue($subName.'#'.$subIdx.'#1',[['text' => $subData['content'], 'pstyle' => 'numstyle-level-3', 'type' => MDWORD_TEXT]]);
}
}else{
$TemplateProcessor->deleteP($subName);
}
}
$TemplateProcessor->deleteP('numstyle');
result IMG:
https://i.stack.imgur.com/sb0MB.png

How to filter mongodb field from linked collection

I want to know how can I propery filter field from another collection that is linked from main collection.
Here is my filter values
if(!empty($filterDataBuyer['price'])){
$findArray['price'] = intval($filterDataBuyer['price']);
}
if(!empty($filterDataBuyer['search'])){
$findArray['full_name'] = $filterDataBuyer['search'];
}
}
$findArray['type'] = 'buyer';
The price and type works fine but the full_name does not work
And here is for getting the data
$getData = [
[
'$match' => $findArray
],
[
'$project' => [
'_id' => ['$toString' => '$_id'],
'type' => '$type',
'created_date' => '$created_date',
'offer_id' => '$offer_id',
'buyer_id' => '$buyer_id',
'traveler_id' => '$raveler_id',
'order_id' => '$order_id',
'price' => '$price'
]
],
[
'$lookup' => [
'from' => 'users',
'let' => [
'admin_id' => ['$toObjectId' => '$buyer_id'],
],
'pipeline' => [
[
'$match' => [
'$expr' => [
'$eq' => [
'$_id',
'$$admin_id'
]
],
],
],
[
'$project' => [
'_id' => ['$toString' => '$_id'],
'profile_image' => '$profile_image',
'full_name' => '$full_name'
]
],
],
'as' => 'profileData'
]
],
What I want to do is if full_name field is available, it could also filter the result by it, price and type can filter successfully but full_name is not. Result returns only if full_name has value on it.
My PHP version is 7.4.27 and mongodb extension is 1.12.0
This is just two collections, payment_details and users where payment_details has buyer_id field that is reference to users id field
From mongocompass:
payment_details:
users:
Full Code update:
if(!empty($filterDataBuyer['price'])){
/* FLIGHT-31 fix */
$findArray['price'] = intval($filterDataBuyer['price']);
}
if(!empty($filterDataBuyer['search'])){
$findArray['full_name'] = $filterDataBuyer['search'];
}
$findArray['type'] = 'buyer';
$buyer_trasections = $db->payment_details->find($findArray);
$buyerRes_trasections = iterator_to_array($buyer_trasections);
$getData = [
[
'$match' => $findArray
],
[
'$project' => [
'_id' => ['$toString' => '$_id'],
'type' => '$type',
'created_date' => '$created_date',
'offer_id' => '$offer_id',
'buyer_id' => '$buyer_id',
'traveler_id' => '$raveler_id',
'order_id' => '$order_id',
'price' => '$price'
]
],
[
'$lookup' => [
'from' => 'users',
'let' => [
'admin_id' => ['$toObjectId' => '$buyer_id'],
],
'pipeline' => [
[
'$match' => [
'$expr' => [
'$eq' => [
'$_id',
'$$admin_id'
]
],
],
],
[
'$project' => [
'_id' => ['$toString' => '$_id'],
'profile_image' => '$profile_image',
'full_name' => '$full_name'
]
],
],
'as' => 'profileData'
]
],
[
'$skip' => $page
],
[
'$limit' => $config['per_page'],
],
[
'$sort' => [ 'created_date'=> -1]
]
];
$buyer_trasec = $db->payment_details->aggregate($getData);
$buyerRes_trasec = iterator_to_array($buyer_trasec);
echo "<pre>";print_r($buyerRes_trasec);exit;
And this is the sample, first result for echo "<pre>";print_r($buyerRes_trasec);exit;

How to optimize elastic search query

I have been reading through elastic search docs over the last few months and have continued to optimize my query, but I can't seem to get a search query below 500-600ms. Locally with less data I can get responses in ~80-200ms.
To outline what I am trying to accomplish:
I have 12 different models in Laravel that are searchable from a single search bar. As someone types it is searched and returned in a list of results.
Currently, I have this for my search query. Are there any references for how I can improve this? I looked into multi_match, but I was having issues with partial matches and specifying all fields.
$results = $this->elastic->search([
'index' => config('scout.elasticsearch.index'),
'type' => $type ?? implode(',', array_keys($this->permissions, true, true)),
'body' => [
'query' => [
'bool' => [
'must' => [
[
'query_string' => [
'query' => "$searchQuery*",
],
],
],
'filter' => [
[
'term' => [
'account_id' => $accountId,
],
],
],
'should' => [
[
'term' => [
'_type' => [
'value' => 'customers',
'boost' => 1.3,
],
],
],
[
'term' => [
'_type' => [
'value' => 'contacts',
'boost' => 1.3,
],
],
],
[
'term' => [
'_type' => [
'value' => 'users',
'boost' => 1.3,
],
],
],
[
'term' => [
'_type' => [
'value' => 'chart_accounts',
'boost' => 1.2,
],
],
],
],
],
],
'from' => $from,
'size' => $size,
],
]);

Update by query (updateByQuery) Elasticsearch-PHP

I am Using Elasticsearch 2.3 along with the official php driver. The updateByQuery is giving me troubles to use in php. A little help on how to use it will be appreciated.
$client = \Elasticsearch\ClientBuilder::create()->setHosts(['127.0.0.1:9200'])->build();
# Request
$updateRequest = [
'index' => 'gorocket',
'type' => 'logs',
'body' => [
'query' => [
'filtered' => [
'filter' => [
'bool' => [
'must' =>
[
[
'match' => [ 'enabled' => 1 ],
],
]
]
]
]
]
]
]
];
# Update
$results = $client->updateByQuery($updateRequest);
Basically i want to update a couple of document fields(name,price) that matches a certain query
Thank you.
So, with the help of how the CURL api works i managed to come up with a way.
First you need to edit your elasticsearch.yml to allow Scripting. Append the following lines at the end.
script.engine.groovy.inline.search: on
script.engine.groovy.inline.aggs: on
script.engine.groovy.inline.update: on
There after you can start doing batch updates using queries as the example bellow.
$client = \Elasticsearch\ClientBuilder::create()->setHosts(['127.0.0.1:9200'])->build();
# Request
$updateRequest = [
'index' => 'testindex',
'type' => 'logs',
'conflicts' => 'proceed',
'body' => [
'query' => [
'filtered' => [
'filter' => [
'bool' => [
'must' =>
[
[
'match' => [ 'enabled' => 1 ],
],
]
]
]
]
],
'script' => [
'inline' => 'ctx._source.enabled = value',
'params' => [
'value' => 0
]
]
]
]
];
# Update
$results = $client->updateByQuery($updateRequest);
That's it. It is not easily and well documented as of now so.
I want to add a small addition to the previous answer
You may not add the following params in elasticsearch.yml
script.engine.groovy.inline.search: on
script.engine.groovy.inline.aggs: on
script.engine.groovy.inline.update: on
And your query will be:
$client = \Elasticsearch\ClientBuilder::create()->setHosts(['127.0.0.1:9200'])->build();
# Request
$updateRequest = [
'index' => 'testindex',
'type' => 'logs',
'conflicts' => 'proceed',
'body' => [
'query' => [
'filtered' => [
'filter' => [
'bool' => [
'must' => [
[
'match' => [ 'enabled' => 1 ],
],
]
]
]
]
],
'script' => [
'lang' => 'painless',
'source' => 'ctx._source.enabled = params.value',
'params' => [
'value' => 0
]
]
]
];
# Update
$results = $client->updateByQuery($updateRequest);

Invalid Input Resource Mongodb Command PHP

I am trying to get around the 30 second cursor time limit and avoid a MongoCursorTimeoutException by using DB->command method as seen here: MongoCursorTimeoutException for aggregate function
When I try to aggregate, command instantly returns:
array(3) { ["ok"]=> float(0) ["errmsg"]=> string(42) "Invalid input resource, " ["code"]=> int(17138) } There is pretty much no documentation on this and the only think I could find is the source code where the error takes place https://github.com/mongodb/mongo/blob/master/src/mongo/db/pipeline/pipeline.cpp#L327
Can someone please help? The pipeline works fine if I run collection->aggregate(pipeline, options) so I don't think it is the pipeline. My code is below:
$connection = new MongoClient( 'mongodb://user:pass#mydb.biz' );
$summaryDB = $connection->selectDB('Summary');
$summaryCollection = $summaryDB->selectCollection('hitSummary');
//agg pipeline
$pipeline =
[
[
'$match' => [
'date' => [
'$gte' => $weekAgo,
'$lt' => $today,
]
]
], [
'$group' => [
'_id' => [
'client' => '$client',
'date' => '$date',
],
//conversions
'z1is' => [ '$sum' => '$actions.sales.import.z1.count' ],
'z2is' => [ '$sum' => '$actions.sales.import.z2.count' ],
'z3is' => [ '$sum' => '$actions.sales.import.z3.count' ],
'z1ps' => [ '$sum' => '$actions.sales.pixel.z1.count' ],
'z2ps' => [ '$sum' => '$actions.sales.pixel.z2.count' ],
'z3ps' => [ '$sum' => '$actions.sales.pixel.z3.count' ],
'z1as' => [ '$sum' => '$actions.sales.apiOnly.z1.count' ],
'z2as' => [ '$sum' => '$actions.sales.apiOnly.z2.count' ],
'z3as' => [ '$sum' => '$actions.sales.apiOnly.z3.count' ],
'z1ss' => [ '$sum' => '$actions.sales.s2s.z1.count' ],
'z2ss' => [ '$sum' => '$actions.sales.s2s.z2.count' ],
'z3ss' => [ '$sum' => '$actions.sales.s2s.z3.count' ],
//clicks
'z1ic' => [ '$sum' => '$actions.click.import.z1.count' ],
'z2ic' => [ '$sum' => '$actions.click.import.z2.count' ],
'z3ic' => [ '$sum' => '$actions.click.import.z3.count' ],
'z1pc' => [ '$sum' => '$actions.click.pixel.z1.count' ],
'z2pc' => [ '$sum' => '$actions.click.pixel.z2.count' ],
'z3pc' => [ '$sum' => '$actions.click.pixel.z3.count' ],
'z1ac' => [ '$sum' => '$actions.click.apiOnly.z1.count' ],
'z2ac' => [ '$sum' => '$actions.click.apiOnly.z2.count' ],
'z3ac' => [ '$sum' => '$actions.click.apiOnly.z3.count' ],
'z1sc' => [ '$sum' => '$actions.click.s2s.z1.count' ],
'z2sc' => [ '$sum' => '$actions.click.s2s.z2.count' ],
'z3sc' => [ '$sum' => '$actions.click.s2s.z3.count' ],
//impressions
'z1ii' => [ '$sum' => '$actions.display.import.z1.count' ],
'z2ii' => [ '$sum' => '$actions.display.import.z2.count' ],
'z3ii' => [ '$sum' => '$actions.display.import.z3.count' ],
'z1pi' => [ '$sum' => '$actions.display.pixel.z1.count' ],
'z2pi' => [ '$sum' => '$actions.display.pixel.z2.count' ],
'z3pi' => [ '$sum' => '$actions.display.pixel.z3.count' ],
'z1ai' => [ '$sum' => '$actions.display.apiOnly.z1.count' ],
'z2ai' => [ '$sum' => '$actions.display.apiOnly.z2.count' ],
'z3ai' => [ '$sum' => '$actions.display.apiOnly.z3.count' ],
'z1si' => [ '$sum' => '$actions.display.s2s.z1.count' ],
'z2si' => [ '$sum' => '$actions.display.s2s.z2.count' ],
'z3si' => [ '$sum' => '$actions.display.s2s.z3.count' ],
]
], [
'$project' => [
'impressions' => [ '$add' => ['$z1ii', '$z2ii', '$z3ii',
'$z1pi', '$z2pi', '$z3pi',
'$z1ai', '$z2ai', '$z3ai',
'$z1si', '$z2si', '$z3si'] ],
'clicks' => [ '$add' => ['$z1ic', '$z2ic', '$z3ic',
'$z1pc', '$z2pc', '$z3pc',
'$z1ac', '$z2ac', '$z3ac',
'$z1sc', '$z2sc', '$z3sc'] ],
'importSales' => [ '$add' => ['$z1is', '$z2is', '$z3is'] ],
'pixelSales' => [ '$add' => ['$z1ps', '$z2ps', '$z3ps'] ],
'apiSales' => [ '$add' => ['$z1as', '$z2as', '$z3as'] ],
's2sSales' => [ '$add' => ['$z1ss', '$z2ss', '$z3ss'] ],
]
]
];
//do something with this
$options = [ 'timeout' => -1 ];
$result = $summaryDB->command(
[
'aggregate' => $summaryCollection,
'pipeline' => $pipeline,
],
$options
);
var_dump($result);
When you run DB command, the first argument is the name of the command as the key and the value is the name of the collection.
In your case you should be passing "hitSummary" as the value of 'aggregate' and not the collection object.
See simple command example here: http://php.net/manual/en/mongodb.command.php

Categories