MongoDB 3.4
I have a project and project_permission collections. The project_permission collection contains permissions to the projects for some users. A single user can have multiple different permissions to the project.
[
'$lookup' => [
'from' => ProjectPermission::collectionName(),
'localField' => '_id',
'foreignField' => 'project_id',
'as' => 'project_permissions'
]
],
[
'$project' => [
// ... irrelevant fields here
'permissions' => '$project_permissions'
]
],
this is how the project query results looks like without filtering:
// other project results
// ... other fields
'permissions' => [
0 => [
'_id' => '5d2873aafa873b2b7c000fad'
'project_id' => '56a9e5c5d18cacc72a485839'
'user_id' => '562f6bfc05dfe9570fb6e427'
'permission' => 'read'
'created_at' => 1562932138
'updated_at' => 1562932139
]
1 => [
'_id' => '5d2879fdfa873b2b7c000fbd'
'project_id' => '56a9e5c5d18cacc72a485839'
'user_id' => '562f6bfc05dfe9570fb6e427'
'permission' => 'write'
'created_at' => 1562932139
'updated_at' => 1562932140
]
2 => [
'_id' => '5db960b5fa873b1604005e8e'
'project_id' => '56a9e5c5d18cacc72a485839'
'user_id' => '582b30dd1e634e6362e1b504'
'permission' => 'write'
'created_at' => 1572430005
'updated_at' => 1572430005
]
]
What I would like to achieve is to return with only those projects where the client - who requested the query - has a specific permission to the project, for example write.
The way I tried it:
pipeline: [
0 => [
'$match' => [
// not related to the problem
]
]
1 => [
'$match' => [
'$and' => [
0 => [
'shared_permissions' => [
'$eq' => true
]
]
1 => [
'$or' => [
0 => [
'project_permissions' => [
'$exists' => true
'$ne' => []
]
]
1 => [
'owner_id' => [
'$ne' => MongoDB\BSON\ObjectId#1
(
[oid] => '582b30dd1e634e6362e1b504'
)
]
]
]
]
]
]
]
2 => [
'$lookup' => [
'from' => 'project_permission'
'localField' => '_id'
'foreignField' => 'project_id'
'as' => 'project_permissions'
]
]
3 => [
'$project' => [
// more not important fields here
'shared_permissions' => 1
'permissions' => [
'$map' => [
'input' => [
'$filter' => [
'input' => '$project_permissions'
'as' => 'project_permission'
'cond' => [
'$and' => [
0 => [
'$eq' => ['$$project_permission.user_id', MongoDB\BSON\ObjectId#1
(
[oid] => '582b30dd1e634e6362e1b504'
)
]
1 => [
'$eq' => ['$$project_permission.permission', 'write']
]
]
]
]
]
'as' => 'project_permission'
'in' => [
'user_id' => '$$project_permission.user_id'
'permission' => '$$project_permission.permission'
]
]
]
]
]
]
For this I almost get the correct response:
[
0 => [
'_id' => '56a9e5c5d18cacc72a485839'
'short_id' => 3
'title' => 'Modified title'
'owner_id' => '562f692a05dfe9560fb6e428'
'updated_at' => 1572435428
'owner_name' => 'Borat Sagdiyev'
'shared_permissions' => true
'permissions' => [
0 => [
'user_id' => '582b30dd1e634e6362e1b504'
'permission' => 'write'
]
]
]
1 => []
]
The problem with this is that empty array, where the result was filtered out - and it wouldn't be a problem if the empty array wouldn't be in the result, because if I use the pagination, then it says two results, instead of one. And we know that in the worst case we would get back an array of empty arrays only.
So what I would like to achieve is this last example results without empty arrays in a way where the pagination will be fine with it too.
ps.: unwind is not an option, because of some structural conventions.
Any ideas?
To perform an equality match between a field from the input documents with a field from the documents of the “joined” collection, the $lookup stage has the following syntax:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
I'm struggling to update a document and add new elements (fields) into it's existing array without losing the array elements on update.
This is my code which inserts a new document into the collection if it doesn't already exist (upsert):
$updateResult = $collection->findOneAndUpdate(
[
'recording-id' => $out['recording']->id
],
['$set' => [
'release' => [
'id' => $out['release']->id,
'title' => $out['release']->title,
'date' => $out['release']->date,
'country' => $out['release']->country
],
'artist' => [
'id' => $out['artist']->id,
'name' => $out['artist']->name,
],
'recording' => [
'id' => $out['recording']->id,
'title' => $out['recording']->title,
'score' => $out['recording']->score,
'length' => $out['recording']->length,
'release-count' => count($out['recording']->releases),
],
'release-group' => [
'id' => $out['release-group']['id'],
'title' => $out['release-group']['title'],
'first-release-date'=>$out['release-group']['first-release-date'],
'primary-type' => $out['release-group']['primary-type'],
'musicbrainz' => $out['release-group']['musicbrainz'],
'url-rels' => $out['release-group']['url-rels'],
'coverart' => $out['release-group']['coverart']
],
'execution' => [
'firstfind' => $out['execution']->time
]
]
],
['upsert' => true,
'projection' =>
[
'_id' => 0,
'release' => 1,
'artist' => 1,
'recording' => 1,
'release-group' => 1,
'execution' => 1
],
'returnDocument' => MongoDB\Operation\FindOneAndUpdate::RETURN_DOCUMENT_AFTER,
]
);
So now I have an existing document in a collection:
{
"_id" : ObjectId("5d1a6aaf5ecc8001ee858f6c"),
"recording-id" : "d0d439f9-5324-4728-8706-2da39adb89c5",
"artist" : {
"id" : "9d97b077-b28d-4ba8-a3d9-c71926e3b2b6",
"name" : "Gordon Lightfoot"
},
"recording" : {
"id" : "d0d439f9-5324-4728-8706-2da39adb89c5",
"title" : "Sundown",
"score" : 100,
"length" : 184000,
"release-count" : 2
},
"release" : {
"id" : "0c008d76-2bc9-44a3-854b-0a08cde89337",
"title" : "All Live",
"date" : "2012-04-24",
"country" : "CA"
},
"release-group" : {
"id" : "0a5d5f33-8e9d-4fa4-b622-a95e4218a3c4",
"title" : "All Live",
"first-release-date" : "2012-04-24",
"primary-type" : "Album",
"musicbrainz" : "https://musicbrainz.org/release-group/0a5d5f33-8e9d-4fa4-b622-a95e4218a3c4",
"url-rels" : "https://musicbrainz.org/ws/2/release-group/0a5d5f33-8e9d-4fa4-b622-a95e4218a3c4?inc=url-rels&fmt=json",
"coverart" : null
}
}
Now, I would like to update this document, and add new fields into the arrays. The new fields are to be added to certain fields.
Here is the code doing that:
$collection = (new MongoDB\Client)->stream->musicbrainz;
$updateResult = $collection->updateOne(
[
'recording-id' => $out['recording']['id']
],
['$addToSet' => [
'artist' => [
'wikiQiD' => $out['artist']['qid'],
'wiki-extract' => $out['artist']['wiki-extract'],
'wiki-pageid' => $out['artist']['pageid'],
],
'release-group' => [
'wikiQiD' => $out['release-group']['qid'],
'wiki-extract' => $out['release-group']['wiki-extract']
]
]
],
[
'upsert' => true,
'returnDocument' => MongoDB\Operation\FindOneAndUpdate::RETURN_DOCUMENT_AFTER,
]
);
I've noticed there's "$addToSet" and "$push" commands, and could use assistance with what the difference is between these two commands.
If the field is absent in the document to update, $push adds the array
field with the value as its element.
The $addToSet operator adds a value to an array unless the value is
already present, in which case $addToSet does nothing to that array.
I did some googling, and reading of the MongoDB/Client UpdateOne function, but can't seem to find a way to append these fields to the existing arrays.
The error I'm getting is:
Fatal error: Uncaught MongoDB\Driver\Exception\BulkWriteException: The field 'artist' must be an array but is of type object in document {_id: ObjectId('5d1a6aaf5ecc8001ee858f6c')} in ...
I know the following:
It could be my document, as it's not a proper array that Fatal error is complaining about.
It could be my `findOneAndUpdate' formatting, and I'm not doing that correctly.
It could be both and I have it all wrong from the very start.
Any insight or constructive criticism is appreciated, just refrain from flames, pls.
Here's the working code I finally use to get it do what I want it to.
It's a simple matter of just setting your field names and values and mongo will update the found record replacing it with the array sent to it. No need to push or anything.
function firstFindMongoUpdate($out) {
$collection = (new MongoDB\Client)->stream->musicbrainz;
$updateResult = $collection->findOneAndUpdate(
[
'recording-id' => $out['recording']->id
],
['$set' => [
'query' => $out['query'],
'release' => [
'id' => $out['release']->id,
'title' => $out['release']->title,
'date' => $out['release']->date,
'country' => $out['release']->country,
'label' => $out['release']->label,
],
'artist' => [
'id' => $out['artist']->id,
'name' => $out['artist']->name,
'wiki' => $out['artist']->wiki,
],
'recording' => [
'id' => $out['recording']->id,
'title' => $out['recording']->title, // sometimes contains apostophe ie; Bill‘s Love (option ] key)
'score' => $out['recording']->score,
'length' => $out['recording']->length,
'release-count' => count($out['recording']->releases)
],
'release-group' => [
'id' => $out['release-group']['id'],
'title' => $out['release-group']['title'],
'first-release-date'=>$out['release-group']['first-release-date'],
'primary-type' => $out['release-group']['primary-type'],
'musicbrainz' => $out['release-group']['musicbrainz'],
'url-rels' => $out['release-group']['url-rels'],
'coverart' => $out['release-group']['coverart'],
'wiki' => $out['release-group']['wiki']
],
'execution' => [
'artistQuery' => $out['execution']->artistQuery,
'recordingQuery'=> $out['execution']->recordingQuery,
'time' => $out['execution']->time
]
]
],
['upsert' => true,
'projection' =>
[
'_id' => 1,
'query' => 1,
'release' => 1,
'artist' => 1,
'recording' => 1,
'release-group' => 1,
'execution' => 1
],
'returnDocument' => MongoDB\Operation\FindOneAndUpdate::RETURN_DOCUMENT_AFTER,
]
);
I am trying to insert records from mysql to elastic. I am inserting two records only. 'prod_id' is primary key in mysql DB.
below is my insert query.
$elastic->setIndex("tbl_product");
$elastic->setType("products");
//$elastic->setConnection(new MySQLiConnection());
//Where 1st param is the database connection and 2nd param is tableName
$elastic->setSqlQuery("SELECT *
FROM tbl_product
LEFT JOIN tbl_product_details ON tbl_product.prod_id = tbl_product_details.prod_id
LEFT JOIN tbl_product_seo ON tbl_product.prod_id = tbl_product_seo.prod_id
LEFT JOIN tbl_product_keywords ON tbl_product.prod_id = tbl_product_keywords.prod_id
LEFT JOIN tbl_category ON tbl_category.cat_id = tbl_product.cat_id
LEFT JOIN tbl_manufacturer ON tbl_manufacturer.manuf_id = tbl_product.manuf_id LIMIT 2000 OFFSET 18000");
$elastic->insertAllData($connection);
/***** Below is elastic array after inserting *****/
Search Function:
$params = [
'index' => $this->client->getIndex(),
'type' => $this->client->getType(),
"from" => 0, "size" => 1000,
//"scroll" => "1m",
'body' => [
"query" => [
"bool" => [
"must" => [
[
"multi_match" => [
"fields" => ["prod_name", "prod_seo_name"],
"type" => "phrase_prefix",
"query" => $query
]
],
[
"term"=> ["cat_type_id"=>1]
]
]
]
]
],
];
$query = $client->search($params);
//return array($this->extractResult($query),$this->extractResult($query1));
return $this->extractResult($query);
Above function returns below array. For example I have shown only two fields in array.
[0] => Array
(
[prod_id] => 16579
[lang_id] => 1
)
[1] => Array
(
[prod_id] =>
[lang_id] => 1
)
In second 'prod_id' is null. Any idea why this is happening?
Below is my mapping.
'mappings' => [
'article' => [
'properties' => [
'prod_id' => [
'type' => 'integer'
],
'lang_id' => [
'type' => 'string'
],
]
]
]
I have a data of stores, report_type is every type of data
- type 3 stored how much money that clients buy and how many transaction.
- type 5 stored how many people and average age of them.
I need an array like this:
"report_type3"
(
[_id] => 2441
[amountclient] => 3749078452,
[trans] => 1000
),
(
[_id] => 2442
[amountclient] => 15533754,
[trans] => 900
)
,
"report_type5"
(
[_id] => 2441
[people] => 120,
[avgage] => 23,
),
(
[_id] => 2441
[people] => 120,
[avgage] => 23,
)
My query looks like this, but it's not working
$cursor = $collection->aggregate([
['$match' => [ 'id_station' => ['$in' => $store ], 'report_type' => 3,
'date' => ['$gte'=> new MongoDB\BSON\UTCDateTime(strtotime("2017-07-01")*1000), '$lte'=> new MongoDB\BSON\UTCDateTime(strtotime("2017-08-05")*1000)]
]
],
['$group' => [
'_id' => [ 'id_station' => '$id_station', 'receiptid' => '$receiptid' ],
'amount' => [ '$sum' => '$amount' ]
]], .... *I group by receiptid because there are many items in one clients.*
[ '$group' => [
'_id' => '$_id.id_station',
'amount' => [ '$sum' => '$amount' ],
'trans' => [ '$sum' => '$trans' ]
]],
[
'$project' => [
'report_type3' => ['_id','$amountclient','$trans']
]
],
['$match' => [ 'id_station' => ['$in' => $store ], 'report_type' => 5,
'date' => ['$gte'=> new MongoDB\BSON\UTCDateTime(strtotime("2017-07-01")*1000), '$lte'=> new MongoDB\BSON\UTCDateTime(strtotime("2017-08-05")*1000)]
]
],
[ '$group' => [
'_id' => ['$_id.id_station', 'area_type' => '$area_type'],
'people' => [ '$sum' => '$people' ],
'avgage' => [ '$avg' => '$age' ]
]],
[
'$project' => [
'report_type5' => [ '_id', 'people', 'avgage' ]
]
]
]);
Could you please help me on this?
I'm using Elastic search in Laravel.
So I have query that returns correct result:
$results = Es::search(array(
'index' => 'testindex',
'type' => $type,
'body' => [
'query' => [
'filtered' => [
'query' => [
'match' => [
'_all' => '2015-02'
]
],
'filter' => [
'bool' => [
'must' => [
'term' => [
'type' => 11
]
],
]
]
]
],
'size' => 5,
'from' => 0
]
));
What it does is:
searches for $query in all fields AND
field "type" must be 11.
Here is output: http://pastebin.com/icWniix4
Total 9 results which is correct.
But when I add another must term then it returns invalid results
$results = Es::search(array(
'index' => 'testindex',
'type' => $type,
'body' => [
'query' => [
'filtered' => [
'query' => [
'match' => [
'_all' => '2015-02'
]
],
'filter' => [
'bool' => [
'must' => [
'term' => [
'type' => 11
],
'term' => [
'public' => 1
]
],
]
]
]
],
'size' => 5,
'from' => 0
]
));
So this only add term for "public".
What it does is:
searches for $query in all fields AND
field "type" must be 11 AND
field "public" must be 1
So now results are in total 429. It ignores "type" term and returns everything with "public" = 1. But as per docs if I use MUST then it should match all of them. Search result http://pastebin.com/cVcatcyi
So how can I write query that I need? $query + type + public
Official doc's fail to answer my questions.
Any advice?
This is your trouble spot:
'must' => [
'term' => [
'type' => 11
],
'term' => [
'public' => 1
]
]
Here, you're assigning the value of must as an associative array that has only one distinct key - term actually gets assigned twice, so presumably only one of the assignments will "survive" (presumably public survives because it appears last in the definition). The end result is that must ends up pointing at an associative array with only one key-value pair.
What I suspect you have to do is this:
'must' => [
[
'term' => [
'type' => 11
]
],
[
'term' => [
'public' => 1
]
]
]
Now must is actually pointing at an array with two items.