Displaying all aggregated results from Elasticsearch query in PHP - php

I have a field called "arrivalDate" and this field is a string. Each document has an arrivalDate in string format (ex: 20110128). I want my output to be something like this (date and the number of records that have that date):
Date : how many records have that date
20110105 : 5 records
20120501 : 2 records
20120602 : 15 records
I already have the query to get these results.
I am trying to display aggregated results in PHP from Elasticsearch. I want my output to be something like this:
Date : how many records have that date
20110105 : 5 records
20120501 : 2 records
20120602 : 15 records
This is what I have so far:
$json = '{"aggs": { "group_by_date": { "terms": { "field": "arrivalDate" } } } }';
$params = [
'index' => 'pickups',
'type' => 'external',
'body' => $json
];
$results = $es->search($params);
However, I don't know how to display the results in PHP. For example, if I wanted to display the total number of documents I would do echo $results['hits']['total'] How could I display all the dates with the number of records they have in PHP?

I'd suggest using aggregations in the same way you construct the query, from my experience it seems to work quicker. Please see the below code:
'aggs' => [
'group_by_date' => [
'terms' => [
'field' => 'arrivalDate',
'size' => 500
]
]
]
Following that, instead of using the typical results['hits']['hits'] you would switch out the hits parts to results['aggregations']. Then access the returning data by accessing the buckets in the response.
For accessing the data from the aggregation shown above, it would likely be something along the lines of:
foreach ($results as $result){
foreach($result['buckets'] as $record){
echo($record['key']);
}
}
There will be a better way of accessing the array within the array, however, the above loop system works well for me. If you have any issues with accessing the data, let me know.

Related

A csv file has a state, county, and data on each line. How can I use PHP associative arrays to convert to states=>counties=>county=>data

My data looks like:
countyFIPS,County Name,State,stateFIPS,1/22/20,1/23/20,1/24/20,1/25/20,....
1001,Autauga County,AL,1,0,0,0,0,0,0,0,0,....
...
I've been able to retrieve it using an Ajax call and collect it into a simple PHP array, then convert it to json to use in my javascript application. While it appears that the data is all counties of a state, followed by the same configuration for the next state, there is no guarantee that it won't be mixed up in some later set of data.
I'm an old Fortran programmer, and would tend to build a hash table for the "states", then check if the state exists in the hash table. If not create a new hash table and add this empty hash table as the value for the key with the name of the state to the "state" hash table. Then check the state hash table to see if it has a key for the county. Again, if it doesn't, then add an empty array as the value for the key with the county name and add that to the state hash table, then proceed to put the values for that row into the county array. I know this will work, but thought maybe there was some clever way to use associative arrays in PHP to accomplish the same thing.
I look at array_filter, but can't seem to figure out how to adapt it to this case. Are there other functions that might work here?
Then, once I have this structure of
$nested_object = { state1=>{county1,county2,county3...},state2=>{counties}},
and those counties have:
county=>[values],
how can I easily convert this to a json structure? Should it have other keys like "states", and within a state "counties". From looking at Haroldo's question "Convert a PHP object to an associative array" of Dec 3, 2010, it appears like I would use:
$array = json_decode(json_encode($nested_object), true);
Will this give me the structure I am looking for?
I want to end up with a structure that I can ask for the states as a set of keys, then for a selected state ask for the counties in that state as a set of keys, and upon selecting one, get the array of values for that state/county. This has to run on a server with potentially a large amount of data and a moderate amount of hits per unit time so I wanted as reasonably efficient way as possible.
I want to end up with a structure that I can ask for the states as a set of keys, then for a selected state ask for the counties in that state as a set of keys, and upon selecting one, get the array of values for that state/county
Okay, so you need something like:
$structure = [
'AL' => [
'counties' => [
'FIPS1' => 'County1',
'FIPS2' => 'County2',
],
'data' => [
'FIPS1' => [
[ 'date1' => value1, 'date2' => value2, 'date3' => value3... ]
],
],
],
'AK' => [ ... ]
];
You can do that using array_map() and a lambda function writing to $structure, but... in my experience, it is not worth it.
Best to do like you said:
while ($row = get_another_row()) {
$countyFIPS = array_unshift($row);
$countyName = array_unshift($row);
$stateName = array_unshift($row);
$stateFIPS = array_unshift($row);
if (!array_key_exists($stateName, $structure)) {
$structure[$stateName] = [
'counties' => [ ],
'data' => [ ],
];
}
if (!array_key_exists($countyFIPS, $structure[$stateName]['counties'])) {
$structure[$stateName]['counties'][$countyFIPS] = $countyName;
$structure[$stateName]['data'][$countyFIPS] = [ ];
}
// Now here you will have $headers, obtained from the header row unshifting
// the first four fields.
foreach ($headers as $i => $key) {
$structure[$stateName]['data'][$countyFIPS][$key] = $row[$i];
}
}
This way if you add two CSVs with different dates, the code will still work properly. Dates will not be sorted though, but you can do that with a nested array_map and the aksort function.
To output this in JSON, just use json_encode on $structure.

How to aproach this: Sort results under same date

I am trying to achieve something like this:
I am using Laravel, and for example to display the results I would do a foreach on the array and display the title, the body etc.
Example
#foreach($resArray as $res)
{{$res->Title}}
#endforeach
And that will display the title of each result in the array, I also have the date in $res->startDate but I am not sure how to list each result with same date under their specific date.
You probably didn't understood..
So for example if I have two notifications from 10/07/2017 and one from 11/07/2017 they will display as this:
10/07/2017
- notification 1
- notification 2
11/07/2017
- notification
I was thinking at an if statement but what statement
if($res->startDate) what so this won't work either, I was thinking to store them in arrays, for example array of date 11/07/2017 and display those arrays but would that even work...
Couldn't find too much from google as I am not too sure how to google this in a good maneer.
EDIT 1
I tried doing this:
$notifResult = notifications::where('userID', '=', $userID)->select('*')->groupBy(['date'])->get();
dd($notifResult);
but it didn't work, first of all, I had 3 results in database, it only got 2 of them and it didn't even group them by date, the two results that are listed before are from different days...
EDIT 2
I added toArray to it and this is what I've got:
still, it only picks two results...
One naive but simple solution is to sort the results by start date and then in the foreach check if the previous date (store it in temp var) is different from the current one, and if true display the new date.
$prevDate = '';
#foreach($resArray as $res)
#if($prevDate != '' && $prevDate != $res->startDate) {
//do sth to break the html and display the new date
#endif
{{$res->Title}}
{{$prevDate = $res->startDate;}}
#endforeach
if i understand your problem you must use group by
$collection = collect([
['date' => '10/07/2017', 'title' => 'notification 1'],
['date' => '10/07/2017', 'title' => 'notification 2'],
['date' => '11/07/2017', 'title' => 'notification'],
]);
$grouped = $collection->groupBy('date');
$grouped->toArray();
dd($grouped);

How to KeyBy where multiple items have the same key

I am using Laravel Collections methods and am trying to key my query results (which are a collection) by the id. The problem is I have multiple entries with the same id, but point to different countries and I want to have all of the values, not just the last one.
Here is my code that i am using so far:
$allCountries = new Collection($allCountries);
$offerCountries = $allCountries->keyBy('id');
dd($offerCountries);
foreach ($offer as $o) {
$o->countries = $allCountries->get($o->id);
}
To explain, my query puts the results in $allCountries which contains ids and countries and those results looks something like this
id=>225, country=>US
id=>225, country=>IT
id=>3304, country=>NZ
Just to give you a quick idea. I want to key this by the id which results in $offerCountries. I then loop thru a previous Collection that contains offers which have a certain ID that relates to the country result by id. So for the offer 225, the countries it contains are US and IT. I loop thru each offer and set the countries object equal to all the $allCountries id that it equals. The problem I have here is keyBy overwrites the value and only takes the last one. I am hoping to get some results like this:
[
225 => countries: {'id' => 225, 'country' => 'US'}, {'id' =>
'225', 'country' => 'IT'}
3304 => ['id' => 3304, 'country' => 'NZ'],
]
Is there a laravel method to do this, or do I need to write my own keyBy so it does not overwrite. If so, how can I get started to write this method?
Thanks
Instead of using keyBy, use groupBy:
$countriesById = collect($allCountries)->groupBy('id');
You could use filter and create a custom filter
$filtered = $allCountries->filter(function ($item) use ($id) {
return $item->id == $id;
});
$filtered->all();

DynamoDB Count Group By

We are trying to search a dynamodb, and need to get count of objects within a grouping, how can this be done?
I have tried this, but when adding the second number, this doesn't work:
$search = array(
'TableName' => 'dev_adsite_rating',
'Select' => 'COUNT',
'KeyConditions' => array(
'ad_id' => array(
'ComparisonOperator' => 'EQ',
'AttributeValueList' => array(
array('N' => 1039722, 'N' => 1480)
)
)
)
);
$response = $client->query($search);
The sql version would look something like this:
select ad_id, count(*)
from dev_adsite_rating
where ad_id in(1039722, 1480)
group by ad_id;
So, is there a way for us to achieve this? I can not find anything on it.
Trying to perform a query like this on DynamoDB is slightly trickier than in an SQL world. To perform something like this, you'll need to consider a few things
EQ ONLY Hash Key: To perform this kind of query, you'll need to make two queries (i.e. ad_id EQ 1039722 / ad_id EQ 1480)
Paginate through query: Because dynamodb returns your result set in increments, you'll need to paginate through your results. Learn more here.
Running "Count": You can take the "Count" property from the response and add it to the running total as you're paginating through the results of both queries. Query API
You could add a Lambda function triggered by the DynamoDBStream, to aggregate your data on the fly, in your case add +1 to the relevant counters. Your search function would then simply retrieve the aggregated data directly.
Example: if you have a weekly online voting system where you need to store each vote (also to check that no user votes twice), you could aggregate the votes on the fly using something like this:
export const handler: DynamoDBStreamHandler = async (event: DynamoDBStreamEvent) => {
await Promise.all(event.Records.map(async record => {
if (record.dynamodb?.NewImage?.vote?.S && record.dynamodb?.NewImage?.week?.S) {
await addVoteToResults(record.dynamodb.NewImage.vote.S, record.dynamodb.NewImage.week.S)
}
}))
}
where addVoteToResults is something like:
export const addVoteToResults = async (vote: string, week: string) => {
await dynamoDbClient.update({
TableName: 'table_name',
Key: { week: week },
UpdateExpression: 'add #vote :inc',
ExpressionAttributeNames: {
'#vote': vote
},
ExpressionAttributeValues: {
':inc': 1
}
}).promise();
}
Afterwards, when the voting is closed, you can retrieve the aggregated votes per week with a single get statement. This solution also helps spreading the write/read load rather than having a huge increase when executing your search function.

Comparing 2 fields within a Mongo Aggregation Query

I have a collection in my Mongo Database called WorkOrder with 2 fields DateComplete and DateDue. Using those 2 fields I'd like to use the aggregation framework to count the number of 'Late' Work Orders by comparing the two fields. However the research I've found hasn't had any useful ways to format the query so that the 'Late' Work Orders will be filtered through. Does anyone know of a way to format a Mongo DB Aggregation Query (preferably in PHP) that can compare 2 fields in the collection?
EDIT:
An example entry in WorkOrder might look like
_id
some mongo id
DateDue
2014-10-10
DateCompleted
2014-10-12
This entry would want to be filtered through since DateCompleted is greater than DateDue. I didn't know about the $cond operator so I haven't tried anything for that yet.
EDIT:
After trying #BatScream's suggestion with the following query in my PHP script
array(
'$cond' => array(
'if' => array(
'dateDue' => array(
'$lt' => 'dateComplete
)
)
)
)
However the MongoCollection::Aggregate function told me that $cond wasn't a recognized operator.
EDIT: #BatScream's answer seems to work but I wasn't aware of the fact that the group operator doesn't work properly after a $project is applied. I was hoping to be able to group these document on another field cID, is that possible?
The below aggregation pipeline would give you the result, considering your fields are of ISODate type. If not i suggest you to store them as ISODate type and not Strings.
db.collection.aggregate([
{$project:{"isLateWorkOrder":{$cond:[{$lt:["$dateDue","$dateCompleted"]},
true,false]}}},
{$match:{"isLateWorkOrder":true}},
{$group:{"_id":null,"lateWorkOrders":{$sum:1}}},
{$project:{"_id":0,"lateWorkOrders":1}}
])
The PHP syntax should look similar to,
$projA = array("isLateWorkOrder" =>
array("$cond" =>
array(array("$lt" =>
array("$dateDue","$dateCompleted")),
true,false)))
$matchA = array("isLateWorkOrder" => true)
$grp = array("_id" => null,"lateWorkOrders" => array("$sum" => 1))
$projB = array("_id" => 0,"lateWorkOrders" => 1)
$pipeline = array($projA,$matchA,$grp,$projB);
$someCol -> aggregate($pipeline)
or, simply using the count function:
db.collection.count({$where:"this.dateDue < this.dateCompleted"})

Categories