Matching across Multiple documents with ElasticSearch - php

I am relatively new to ElasticSearch. I am using it as a search platform for pdf documents. I break the PDFs into text-pages and enter each one as an elasticSearch record with it's corresponding page ID, parent info, etc.
What I'm finding difficult is matching a given query not only to a single document in ES, but making it match any document with the same parent ID. So if two terms are searched, if the terms existed on page 1 and 7 of the actual PDF document (2 separate entries into ES), I want to match this result.
Essentially my goal is to be able to search through the multiple pages of a single PDF, matching happening on any of the document-pages in the PDF, and to return a list of matching PDF documents for the search result, instead of matching "pages"

You will need to use the "has_child" query on pages. I'm assumed that you're already defined the mapping for parent/child relationship of documents and pages. Then you can write a "has_child" query that search on pages (child type) but return PDF documents (parent type):
{
"query": {
"has_child": {
"type": "your_pages_type",
"score_type": "max", // read document for more
"query": {
"query_string": {
"query": "some text to search",
"fields": [
"your_pages_body"
],
"default_operator": "and" // "and" if you want to search all words, "or" if you want to search any of words in query
}
}
}
}
}

It's somewhat tricky. First of all, you will have to split your query into terms yourself. Having a list of terms (let's say foo, bar and baz, you can create a bool query against type representing PDFs (parent type) that would look like this:
{
"bool" : {
"must" : [{
"has_child" : {
"type": "page",
"query": {
"match": {
"page_body": "foo"
}
}
}
}, {
"has_child" : {
"type": "page",
"query": {
"match": {
"page_body": "bar"
}
}
}
}, {
"has_child" : {
"type": "page",
"query": {
"match": {
"page_body": "baz"
}
}
}
}]
}
}
This query will find you all PDFs that contain at least one page with each term.

Related

Elasticsearch "Join" tables

I need to do "Join" between 2 indexes (tables) and preform a check on specific field on documents that exists in both indexes.
I want to add condition like "dateExpiry" below, but I get an error. Is it possible to join 2 or more indexes?
GET cache-*/_search
{
"query": {
"bool": {
"must_not": [
{
"query": {
"terms": {
"TagId": {
"index": "domain_block-2016.06",
"type": "cBlock",
"id": "57692ef6ae8c50f67e8b45",
"path": "TagId",
"range" : {
"dateExpiry" : {
"gte" : "20160705T12:00:00"
}
}
}
}
}
]
}
}
}
Filters within a Terms Query Lookup are currently not supported. However, Elasticsearch has some great documentation on joins / relationships here.
Your best bet may be to run two queries against Elasticsearch - one to fetch the list of TagIds, then another that includes the list as an exclusion clause.

elasticsearch: applying an additional boost on a given field for a given value

I have a Symfony 2.7.6 application using the FOSElasticaBundle.
I have 2 types of search:
One without keyword, in this case only filters are applied and all documents scores are 1 (sometimes with a random order), in this case the main query is:
$query = new Elastica\Query\MatchAll();
One with keyword, same filters are applied and the match is run again a list of fields, (one with a different boost). And the results are stored by score. The main query is now:
$match = new Elastica\Query\MultiMatch();
$match->setQuery($keyword);
$match->setOperator('AND');
$match->setFields([
'field1^30',
'field2',
'field3',
'field4',
'_all'
]);
Those 2 search are working well.
Now for both search I want a dynamic boost to be applied for a given field value. Let's say: if field5 == 'value' then add boost 15, (15 is just an example, we will make tests to see what additional boost value has to be chosen) the value used here is not the keyword, it is another parameter.
I tried with a FunctionScore and with Boosting queries but without success. Any hint with a very simple elasticsearch query would be appreciated.
How about this:
{
"query": {
"function_score": {
"query": {
"multi_match": {
"query": "blabla",
"operator": "AND",
"fields": [
"field1^30",
"field2",
"field3",
"field4",
"_all"
]
}
},
"functions": [
{
"filter": {
"term": {
"field5": "some_value"
}
},
"boost_factor": 15
}
]
}
}
}

Elasticsearch either or match query

I am trying to write a query to search for a products on two columns called category1 and category2. I am working using elastic search php client and tried with match should query but this giving me wrong results because of match of substring.
But i am looking for exact match with OR operation on two columns. I am new to this please guide me.
$params['index'] = 'furnit';
$params['type'] = 'products';
$params['body']['query']['bool']['should'] = array(
array('match' => array('category1' => $category->name)),
array('match' => array('category2' => $category->name)),
);
$results = $this->elasticsearch->search($params);
If you are not searching then using a bool query in this scenario is not the right way to do it in elasticsearch. Queries are used when you are searching something and relevancy of your search keyword and score of matching documents matters.
Here you can apply a bool filter of elasticsearch to filter out the desired results. Using filters with queries (filtered query) is right way to do it as it excludes all non-matching documents and then you can search for desired documents by using match queries.
here's an example of a bool filter
{
"from": 0,
"size": 50,
"sort": [
{
"name" : {
"order": "asc"
}
}
],
"query": {
"filtered": {
"query": {
"match_all" : {}
},
"filter": {
"bool": {
"should": [
{
"term": {
"category1" : "category1"
}
},
{
"term": {
"category2" : "category2"
}
}
]
}
}
}
}
}
you can refer to docs as well (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-filter.html)
Maybe your problem is you have used default analyzer (which is standard analyzer).
could you give me your mapping ?
I suggest you to change to use not_analyzer when indexing and use term filter/query.
You could use put mapping here to setting for your analyzer: Put Mapping
Edit: I have created a gist for you, check it here:
Mappings & Terms Filter

multiple group by in elasticsearch including missing values

I'm trying to do a group by in elasticsearch, by multiple fields. I know that nested aggregation exists, but what I want is including in a certain bucket the record for which the field I'm grouping by is empty.
Say that we have this kind of data structure:
SONG_ID | SONG_GENRE | SONG_ARTIST
and i want to group by genere, artists.
I would like to have a group for each possibile combination, i.e
group by genre gives me 5 buckets (if genres are 5) plus the bucket in which there are the songs without a genre. grouping then by artist gives me, for each genre, bucket by artists plus the one with songs without an artist.
Basically, I'd like to have the same results that I have using a group by. Is that even possible?
You can approach in different ways to solve your need.
The simplest way would be to index a fix value say "notmentioned" against the genre field of songs if genre is not present. you can do it while indexing or by defining "null_value" in your field mapping.
"SONG_GENRE": {"type": "string", "null_value": "notmentioned"},
"SONG_ARTIST": {"type": "string", "null_value": "notmentioned"},
So during aggregation (nested) you will automatically find the count against "notmentioned" for songs not having genre.
Another approach would be to use the missing filter as another aggregation along with normal aggregation. Something like below.
{
"aggs": {
"SONG_GENRE": {
"terms": {
"field": "SONG_GENRE"
},
"aggs": {
"SONG_ARTIST": {
"terms": {
"field": "SONG_ARTIST"
}
},
"MISSING_SONG_ARTIST": {
"filter": {
"missing": {
"field": "SONG_ARTIST"
}
}
}
}
},
"MISSING_SONG_GENRE": {
"filter": {
"missing": {
"field": "SONG_GENRE"
}
},
"aggs": {
"MISSING_SONG_GENRE_SONG_ARTIST": {
"terms": {
"field": "SONG_ARTIST"
}
},
"MISSING_SONG_GENRE_MISSING_SONG_ARTIST": {
"filter": {
"missing": {
"field": "SONG_ARTIST"
}
}
}
}
}
}
}
I haven't verified the syntax. It is just to give you an idea
Another hacking way could be to treat the missing count (total hits - all aggregation count) as the count against no genre.

matching whole string with dashes in elasticsearch

I have an elasticsearch query which I am trying to match properly, the field data itself contains -(dashes), the string data are GUIDS
It was not matching properly because it was splitting the term up into separate words split by the -
I have since changed the query to use a match_phrase query like this:
"query": {
"filtered": {
"query": {
"match_phrase":{
"guid":{"operator" : "or","query":"bd2acb42-cf01-11e2-ba92-12313916f4be"}
}
}
}
}
When I am trying to match just one GUIDS, this works just fine.
However I am trying to match multiple GUIDS
So it currently looks like
"query": {
"filtered": {
"query": {
"match_phrase":{
"guid":{"operator" : "or","query":"bd2acb42-cf01-11e2-ba92-12313916f4be d1091f08-ceff-11e2-ba92-12313916f4be"}
}
}
}
}
I assume its not working because its trying to match the whole string, and not each GUID separately.
I tried added "analyzer" : "whitespace", to the query, but this broke the query entirely.
So what is the best method to ensure the query is looking for the whole GUID string and allows matching of multiple GUIDS?
I have been setting the field mapping to not_analyzed for similar purposes.
"guid" : {
"type" : "string",
"index" : "not_analyzed"
}
Building the query manually then works.
{
"bool" : {
"should" : [
{
"term" : { "guid" : "bd2acb42-cf01-11e2-ba92-12313916f4be" }
},
{
"term" : { "guid" : "d1091f08-ceff-11e2-ba92-12313916f4be" }
}
],
"minimum_number_should_match" : 1
}
}

Categories