Is it possible to do this within the $project array using PHP's built in round function?
I try to enclose my output value within the round function to 2 decimal places:
"Energy" => round(array('$multiply' => array("$energy", 10)), 2),
The output error I get is this:
Type: MongoDB\Driver\Exception\RuntimeException
Code: 16406
Message: The top-level _id field is the only field currently supported for exclusion
File: C:\wamp\www\DRM\vendor\mongodb\mongodb\src\Operation\Aggregate.php
Line: 168
Currently I have a separate parsing method which takes care of all the rounding, but what I'd like is to do it within the aggregate function in PHP.
Is this possible? I know MongoDB doesn't have round, but there is an external library for that.
there is no runding capabilities for mongo yet.
as per this answer, you could add extra steps to aggregation pipeline to get it rounded - below mongo shell code:
> db.a.save({x:1.23456789})
> db.a.save({x:9.87654321})
> db.a.aggregate([{$project:{ _id:0,
y:{$divide:[
{$subtract:[
{$multiply:['$x',100]},
{$mod:[{$multiply:['$x',100]}, 1]}
]},
100]}
}}])
{ "y" : 1.23 }
{ "y" : 9.87 }
jira ticket
Related
I have an API endpoint that points to /api/invoices and it returns a list of invoices.
Some invoices return something like this:
{
"id": 2555,
"entity_id": 88,
"net_total": 7.5,
"total_vat": 1.725000000000000088817841970012523233890533447265625,
"grand_total": 9.2249999999999996447286321199499070644378662109375,
}
As you can see, there's too many decimal places. My money columns are all defined as double(20,5), because the software needs to handle up to 5 decimal places (but mostly is only 2) and 20 units.
Is there any way to force either through MySQL or Laravel, to always return 5 decimal places? I can't use:
->selectRaw('ROUND(total_vat, 5) as total_vat')
Because I have around 100 columns and I get them all without using ->select().
I would suggest using API Resources for transforming data.
If you just return Invoices::all() in your API, then yes, it will return "as is".
But a typical way to transform any data would be API Resources:
php artisan make:resource InvoiceResource
Then, inside of that resource, you return only what you need and transformed however you want, including the 5 digits:
app/Http/Resources/InvoiceResource.php:
public function toArray($request)
{
return [
'id' => $this->id,
'entity_id' => $this->entity_id,
'net_total' => $this->net_total,
'total_vat' => number_format($this->total_vat, 5),
'grand_total' => number_format($this->grand_total, 5),
];
}
Then, in Controller:
public function index() {
return InvoiceResource::collection(Invoice::all());
}
Also, API Resources would add "data" wrapper layer by default when returned, so you may want to avoid that, then add this to AppServiceProvider:
app/Providers/AppServiceProvider.php:
use Illuminate\Http\Resources\Json\JsonResource;
class AppServiceProvider extends ServiceProvider
{
public function boot()
{
JsonResource::withoutWrapping();
}
}
The solution is to convert the columns into decimals and maintain the same structure, like said in this post https://stackoverflow.com/a/24135191/3355243.
I did a few tests and the column value kept the original data.
However, this will lead to another issue, which is laravel casting decimal as string. By using postman, we can see that currency values comes as string, like "15.00". To solve this we can use cast Eloquent casts decimal as string:
protected $casts =
[
'net_total' => 'float',
];
I wasn't able to find / check if using the cast creates any problem with the original value.
Be careful while converting DOUBLE to DECIMAL, as, like in my case, this may trigger severe problems in your project. For example, my mobile App stopped working after the conversion, because is not ready to handle the currency values as string.
I am trying to use the following functionality:
[quote_elements.product.factor;ope=mul:quote_elements.qty]
But all I get is always 0.
if I use:
[quote_elements.product.factor;ope=mul:4]
it works fine and I get 4 times the factor number.
But this is not what I need. I need to multiply dynamically the factor with the quantity. this can be for each row different.
any tips what I am missing here?
Embedded TBS fields does not work in parameter ope.
That why the string « quote_elements.qty » is always converted to 0.
Parameter ope=mul can work only with fixed values.
In order to solve you problem, you can use a custom ondata function. It will enable you to add a calculated column in you record before to merge it.
PHP side:
function f_my_ondata($BlockName, &$CurrRec, $RecNum) {
$CurrRec['my_result'] = $CurrRec['product']['factore'] * $CurrRec['qty'];
}
Template side :
[quote_elements;block=...;ondata=f_my_ondata] // block definition
...
[quote_elements.my_result]
I have a vary large dataset in MongoDB, in which there are documents with numeric fields. Due to some issue in the data import, some of these fields ended up in int32 datatype with some are in int64 datatype.
I need to convert all of them to int32. Since many of the fields are nested documents/array I cannot use MongoChef or RoboMongo to edit the field and do a collection wide replace.
What is my next best option? Would I need to write a script that loop through each document/field and explicitly typecast them to NumberInt(). I could do this in PHP or Python, but I was wondering if there is a way to do this without writing extra code.
Is there any mongoshell magic that can be done? I would appreciate if any Mongo Masters can give me any insights.
To anyone looking to do this and coming here. You can run
db.foo.find().forEach(doc => {
const newBar = bar.valueOf()
db.foo.update({
"_id" : doc._id
}, {
"$set" : {
"bar" : newBar
}
})
})
in the mongo shell. This might not be doable in large collections. The key is to use .valueOf() on the Int64. You might want to check that this doesn't overflow
I have the following structure within a mongoDB collection:
{
"_id" : ObjectId("5301d337fa46346a048b4567"),
"delivery_attempts" : {
"0" : {
"live_feed_id" : 107,
"remaining_attempts" : 2,
"delivered" : false,
"determined_status" : null,
"date" : 1392628536
}
}
}
// > db.lead.find({}, {delivery_attempts:1}).pretty();
I'm trying to select any data from that collection where remaining_attempts are greater than 0 and a live_feed_id is equal to 107. Note that the "delivery_attempts" field is of a type hash.
I've tried using an addAnd within an elemMatch (not sure if this is the correct way to achieve this).
$qb = $this->dm->createQueryBuilder($this->getDocumentName());
$qb->expr()->field('delivery_attempts')
->elemMatch(
$qb->expr()
->field('remaining_attempts')->gt(0)
->addAnd($qb->expr()->field('live_feed_id')->equals(107))
);
I do appear to be getting the record detailed above. However, changing the greater than
test to 3
->field('remaining_attempts')->gt(3)
still returns the record (which is incorrect). Is there a way to achieve this?
EDIT: I've updated the delivery_attempts field type from a "Hash" to a "Collection". This shows the data being stored as an array rather than an object:
"delivery_attempts" : [
{
"live_feed_id" : 107,
"remaining_attempts" : 2,
"delivered" : false,
"determined_status" : null,
"date" : 1392648433
}
]
However, the original issue still applies.
You can use a dot notation to reference elements within a collection.
$qb->field('delivery_attempts.remaining_attempts')->gt(0)
->field('delivery_attempts.live_feed_id')->equals(107);
It works fine for me if I run the query on mongo.
db.testQ.find({"delivery_attempts.remaining_attempts" : {"$gt" : 0}, "delivery_attempts.live_feed_id" : 107}).pretty()
so it seems something wrong with your PHP query, I suggest running profiler to see which query is actually run against mongo
db.setProfilingLevel(2)
This will log all operation since you enable profiling. Then you can query the log to see which the actual queries
db.system.profile.find().pretty()
This might help you to find the culprit.
It sounds like your solved your first problem, which was using the Hash type mapping (instead for storing BSON objects, or associative arrays in PHP) instead of the Collection mapping (intended for real arrays); however, the query criteria in the answer you submitted still seems incorrect.
$qb->field('delivery_attempts.remaining_attempts')->gt(0)
->field('delivery_attempts.live_feed_id')->equals(107);
You said in your original question:
I'm trying to select any data from that collection where remaining_attempts are greater than 0 and a live_feed_id is equal to 107.
I assume you'd like that criteria to be satisfied by a single element within the delivery_attempts array. If that's correct, the criteria you specified above may match more than you expect, since delivery_attempts.remaining_attempts can refer to any element in the array, as can the live_feed_id criteria. You'll want to use $elemMatch to restrict the field criteria to a single array element.
I see you were using elemMatch() in your original question, but the syntax looked a bit odd. There should be no need to use addAnd() (i.e. an $and operator) unless you were attempting to apply two query operators to the same field name. Simply add extra field() calls to the same query expression you're using for the elemMatch() method. One example of this from ODM's test suite is QueryTest::testElemMatch(). You can also use the debug() method on the query to see the raw MongoDB query object created by ODM's query builder.
First of all it's my first time in Mongo...
Concept:
A user is able to describe an image in natural language.
Divide the user input and store the words he described in a Collection called
words.
Users must be able to go through the most used words and add those words to their description.
The system will use the most used words (for all users) and use
those words to describe the image.
My words document (currently) is as follows (example)
{
"date": "date it was inserted"
"reported": 0,
"image_id": "image id"
"image_name": "image name"
"user": "user _id"
"word": "awesome"
}
The words will be duplicated so that each word can be associated to a user...
Problem: I need to perform a Mongo query to allow me to know the most used words (to describe an image) that were not created by a given user. (to meet point 3. above)
I've seen MapReduce algorithm, but from what I read there are a couple of issues with it:
Can't sort results (I can order from the most used to less used)
In millions of documents it can have a large processing time.
Can't limit the number of the results returned
I've thought about running a task at a given time each day to store on a document (in a different collection) the list the rank of words that a given user hasn't used to describe the given image. I would have to limit this to 300 results or something (any idea on a proper limit??) Something like:
{
user_id: "the user id"
[
{word: test, count: 1000},
{word: test2, count: 980},
{word: etc, count: 300}
]
}
Problems I see with this solution are:
Results would have quite a delay which is not desirable.
Server loads while generating this documents for all users can spike (I actually know very little about this in Mongo so this is just an assumption)
Maybe my approach doesn't make any sense... And maybe my lack of experience in Mongo is pointing me at the wrong "schema design".
Any idea of what could be a good approach for this kind of problem?
Sorry for the big post and thanks for your time and help!
João
As already mentioned you could use the group command which is easy to use, but you will need to sort the result on the client side. Also the result is returned as a single BSON object and for this reason must be fairly small – less than 10,000 keys, else you will get an exception.
Code example based on your data structure:
db.words.group({
key : {"word" : true},
initial: {count : 0},
reduce: function(obj, prev) { prev.count++},
cond: {"user" :{ $ne : "USERNAME_TO_IGNORE"}}
})
Another option is to use the new Aggregation framework, which will be released in the 2.2 version. Something like that should work.
db.words.aggregate({
$match : { "user" : { "$ne" : "USERNAME_TO_IGNORE"} },
$group : {
_id : "$word",
count: { $sum : 1}
}
})
Or you can still use MapReduce. Actually you can limit and sort the output, because the result is
an collection. Just use .sort() and .limit() on the output. Also you can use the incremental
map-reduce output option, which will help you solve your performance issues. Have a look at the out parameter in MapReduce.
Bellow it's an example, which use the incremental feature to merge the existing collection with new data in a words_usage collection:
m = function() {
emit(this.word, {count: 1});
};
r = function( key , values ){
var sum = 0;
values.forEach(function(doc) {
sum += doc.count;
});
return {count: sum};
};
db.runCommand({
mapreduce : "words",
map : m,
reduce : r,
out : { reduce: "words_usage"},
query : <query filter object>
})
# retrieve the top 10 words
db.words_usage.find().sort({"value.count" : -1}).sort({"value.count" : -1}).limit(10)
I guess you can run the above MapReduce command in a cron every few minutes/hours, depends how accurate results you want. For the update query criteria you can use the words documents creation date.
Once you have the system top words collection you can build per user top words or just compute them in real time (depends on the system size).
The group function is supposed to be a simpler version of MapReduce. You could use it like this to get a sum for each word:
db.coll.group(
{key: { a:true, b:true },
cond: { active:1 },
reduce: function(obj,prev) { prev.csum += obj.c; },
initial: { csum: 0 }
});