Issue with Dates - trying to plot MongoDB data in Highcharts via PHP - php

Question

I have this data in MongoDB:
{
"_id": ObjectId("542bxxxxxxxxxxxx"),
"TEMP_C": 13,
"time": ISODate("2014-08-21T05:30:00Z")
}
I want to group it by day and feed it to Highcharts, to display Temp average per day.
Like this: http://jsfiddle.net/tw7n6wxb/3/
Using MongoDB Aggregation Pipeline, I was able to do the grouping, based on some other examples and this great post: http://www.kamsky.org/stupid-tricks-with-mongodb/stupid-date-tricks-with-aggregation-framework
Side question: Why is it so complicated to group by date in MongoDB??? The most annoying part is having to re-compose the date object after splitting it into '$dayOfMonth', '$month', and '$year'.
Is there any simpler way of doing this?
In any case, I got this part working (I think). This is the result:
{
"_id" : {"sec":1409346800,"usec":0},
"avg" : 12
},
{
"_id" : {"sec":1409356800,"usec":0},
"avg" : 15
},
But, Highcharts series take arrays of value pairs as input:
Example: data: [[5, 2], [6, 3], [8, 2]].
The first value on each pair is the X value, and this value has to be a number (when X axis is configured as datetime, X values are in milliseconds).
The PROBLEM I'm having is that MongoDB returns the date as a MongoDate Object with two values inside, 'sec' and 'usec', while Highcharts is expecting one number.
Is there anyway to convert a MongoDate Object to integer in the pipeline? using a $project for example?
I'm using PHP, but I would like to avoid post-pocessing in the application (like PHP date formatting).
Or, any other ideas on how to solve this?
Thanks,

Answer 1

You seem to just want the timestamp values returned from the result. There is indeed a simple way to do this in the aggregation framework without using the date aggregation operators. You can use basic "date math" instead, and with a trick that can be used to extract the "timestamp" value from the date object and manipulate it:
db.collection.aggregate([
{ "$group": {
"_id": {
"$subtract": [
{ "$subtract": [ "$time", new Date("1970-01-01") ] },
{ "$mod": [
{ "$subtract": [ "$time", new Date("1970-01-01") ] },
1000 * 60 * 60 * 24
]}
]
},
"avg": { "$avg": "$TEMP_C" }
}}
])
So the basic "trick" there is that when subtract one date object from another ( or similar operation ) the result returned is a number for "milliseconds" of the time difference between the two. So by using the "epoch" date of "1970-01-01", you get the "epoch timestamp" value for the date object as a number.
Then the basic date math is applied by subtracting from this value the modulo ( or remainder ) from the milliseconds in a day. This "rounds" the value to represent the "day" on which the entry is recorded.
I like posting the JSON because it parses everywhere, but in a more PHP way, then like this:
$collection->aggregate(array(
array( '$group' => array(
'_id' => array(
'$subtract' => array(
array( '$subtract' => array(
'$time', new MongoDate(strtotime("1970-01-01 00:00:00"))
) ),
array( '$mod' => array(
array( '$subtract' => array(
'$time', new MongoDate(strtotime("1970-01-01 00:00:00"))
) ),
1000 * 60 * 60 * 24
))
)
),
"avg" => array( '$avg' => '$TEMP_C' )
))
))
So that is a little cleaner than using the date aggregation operators to get to your intended result. Of course this is still not "all the way" to how you want the data to be presented where you can use it in the client.
The real thing to do here is manipulate the result so that you get the output format you want. This is probably better suited to your server code doing the manipulation before you return the response, but if you have MongoDB 2.6 or greater then it is "possible" to do this within the aggregation pipeline itself:
db.collection.aggregate([
{ "$group": {
"_id": {
"$subtract": [
{ "$subtract": [ "$time", new Date("1970-01-01") ] },
{ "$mod": [
{ "$subtract": [ "$time", new Date("1970-01-01") ] },
1000 * 60 * 60 * 24
]}
]
},
"avg": { "$avg": "$TEMP_C" }
}},
{ "$group": {
"_id": null,
"data": {
"$push": {
"$map": {
"input": { "$literal": [ 1,2 ] },
"as": "el",
"in": {
"$cond": [
{ "$eq": [ "$$el", 1 ] },
"$$_id",
"$avg"
]
}
}
}
}
}}
])
So this is pretty sneaky really. After the initial "grouping" is done to determine the averages for each day you get two fields in your result documents per day for _id and avg. What the $map operator does here is takes and array as input ( in this case, just a numbered template with a pair of values to identify position ) and processes each element to return an array equal to the elements present in the original.
The $cond operator here allows you to look at the value of the current element of that array and "swap it" with another value present in the current document. So for each document, the results contain something that is a paired array like:
[ 1409346800000, 12 ]
Then all that happens is all results are pushed into a single document with a "data" array that appears as follows:
{ "_id": null, "data": [ [..,..], [..,..], (...) ] }
Now your data element in that one result is an array of array pairs representing the points you want.
Of course though, operators like $map are only available from MongoDB 2.6 and onwards, so if you have that available then you can use them but otherwise just process the results in code with a similar "map" operation:
function my_combine($v) {
return array($v["_id"],$v["avg"])
}
$newresult = array_map( "my_combine", $result )
So this really comes down to array manipulation from whichever way you approach it, but the date manipulation trick should also save you some work in obtaining the results as the expected timestamp values as well.