My range filter keeps failing on my windows elastic 1.4 instance. It gives no error just returns no records.
PUT reseller_api_logging
{
"mappings": {
"record": {
"properties": {
"PastelAccountNumber": { "type": "string" },
"CreatedAt": { "type": "date" ,"format":"yyyy-MM-dd HH:mm:ss" },
"HTTPmethod": { "type": "string" },
"ContentTypeRequest": { "type": "string" },
"Endpoint": { "type": "string" },
"RequestParams": { "type": "string" , "index": "not_analyzed" },
"RawResponse": { "type": "string" , "index": "not_analyzed" },
"Response": { "type": "string" , "index": "not_analyzed" }
}
}
}
}
Here is the range query:
GET reseller_api_logging/_search
{
"query": {
"range" : {
"CreatedAt" : {
"gte" : "2017-01-01 01:01:49",
"lte" : "2017-01-30 09:01:49"
}
}
}
}
Here is a piece of one of the documents return on a match all search:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 50,
"max_score": 1,
"hits": [
{
"_index": "reseller_api_logging",
"_type": "logging",
"_id": "2142183",
"_score": 1,
"_source": {
"PastelAccountNumber": "103206",
"CreatedAt": "2017-01-23 09:01:49",
"HTTPmethod": "GET",
"Endpoint": "getServiceSessionDetailsById",
"RequestParams": "
Related
I've been trying to build an index in ES and add the initial items to it (around 350k), using PHP.
I tried all kinds of batch sizes (from 10 items to 1k), check the count, check the threshold, but for some reason it doesn't index every item.
It just skips over some random items, without any errors in the batch result response. I feel like I tried everything and I have to idea what to do next
I'm using Amazon OpenSearch with the latest supported ES (7.10).
The index looks like this:
{
"wonder-search": {
"aliases": {},
"mappings": {
"properties": {
"address": {
"type": "text"
},
"city": {
"type": "text"
},
"city_id": {
"type": "integer"
},
"duration": {
"type": "integer"
},
"filename": {
"type": "text"
},
"geo_point": {
"type": "geo_point"
},
"icon": {
"type": "keyword"
},
"is_sandbox": {
"type": "integer"
},
"item_id": {
"type": "integer"
},
"item_label": {
"type": "keyword"
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
},
"search_text_caption_json": {
"type": "text",
"index_phrases": true
},
"search_text_city_json": {
"type": "text",
"index_phrases": true
},
"search_text_completion": {
"type": "completion",
"analyzer": "simple",
"preserve_separators": true,
"preserve_position_increments": true,
"max_input_length": 50,
"contexts": [
{
"name": "type",
"type": "CATEGORY"
}
]
},
"search_text_country_json": {
"type": "text",
"index_phrases": true
},
"search_text_cuisine_name_json": {
"type": "text",
"index_phrases": true
},
"search_text_location_name_json": {
"type": "text",
"index_phrases": true
},
"search_text_state_json": {
"type": "text",
"index_phrases": true
},
"search_text_tag_name_json": {
"type": "text",
"index_phrases": true
},
"search_text_username_json": {
"type": "text",
"index_phrases": true
},
"sort": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"sort_score": {
"type": "double"
},
"type": {
"type": "text"
},
"user_icon": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user_id": {
"type": "integer"
},
"username": {
"type": "keyword"
},
"vanity_url": {
"type": "keyword"
},
"video_count": {
"type": "integer"
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"mapping": {
"ignore_malformed": "true"
},
"number_of_shards": "1",
"provided_name": "wonder-search",
"creation_date": "1671003076106",
"number_of_replicas": "1",
"uuid": "YQh1q40WTneLE4MWDWhArw",
"version": {
"created": "7100199"
}
}
}
}
}
and one item looks like this:
{
"_index": "wonder-search",
"_type": "_doc",
"_id": "wq2LD4UBUAuy7FQPhtZh",
"_version": 1,
"_seq_no": 2003,
"_primary_term": 1,
"found": true,
"_source": {
"sort": "4004",
"item_id": "4934",
"user_id": "434",
"user_icon": "/site-content/avatars/Sp8AXjTJvMbRao2oZbuiUuSVH042-1597776099045.jpeg",
"username": "chuurros",
"item_label": "Kyoto Katsugyu【京都勝牛】",
"search_text_username_json": [
"chuurros"
],
"search_text_caption_json": [
"Absolutely love their gyukatsu (beef katsu) here! Delicious and will keep you wanting more! 🥰"
],
"search_text_city_json": [
"Toronto"
],
"search_text_state_json": [
"Ontario"
],
"search_text_country_json": [
"Canada"
],
"search_text_location_name_json": [
"Kyoto Katsugyu【京都勝牛】"
],
"search_text_tag_name_json": [
"japanese",
"restaurant",
"asian",
"dining",
"topcollection-4934"
],
"search_text_cuisine_name_json": [],
"type": "video",
"vanity_url": "",
"icon": "",
"city": "Toronto",
"city_id": "439",
"latitude": "43.65682410",
"longitude": "-79.37617410",
"address": "134 Dundas St E",
"duration": "9.57",
"video_count": "0",
"sort_score": "43",
"filename": "373d75fd-4292-4e5b-a239-4b1c39ffc86c.MOV",
"is_sandbox": "0",
"geo_point": {
"lat": "43.65682410",
"lon": "-79.37617410"
},
"search_text_completion": {
"input": [
"Kyoto Katsugyu【京都勝牛】"
],
"contexts": {
"type": [
"video"
]
}
}
}
}
Any ideas why does it work like this?
JSON spec:
"responses": {
"200": {
"description": "Успешный ответ сервиса",
"schema": {
"$ref": "#/definitions/BaseResponse"
},
"examples": {
"application/json": {
"status": true,
"response": {
"$ref": "#/definitions/Product"
},
"errors": null
}
}
}
}
Result:
But I need:
{
"status": true,
"response": {
"ProductNumber": "number",
"Barcode": "number",
"Length": 12,
"Width": 34,
"Height": 423,
"Volume": 1232
}
},
"errors": null
}
How I can using $refs into example array for custom format response?
It's a typical case, but I cannot found documentation for it. Thank you for you feedback.
Inline examples do not support $ref - the example must be a complete example:
"responses": {
"200": {
"description": "Успешный ответ сервиса",
"schema": {
"$ref": "#/definitions/BaseResponse"
},
"examples": {
"application/json": {
"status": true,
"response": {
"ProductNumber": "number",
"Barcode": "number",
"Length": 12,
"Width": 34,
"Height": 423,
"Volume": 1232
},
"errors": null
}
}
}
}
Instead of using responses.<code>.examples, you can specify the example values in your BaseResponse schema, and Swagger UI will use those instead.
For example, you can add a complete example to your BaseResponse schema:
"definitions": {
"BaseResponse": {
"type": "object",
"properties": {
"status": {
"type": "boolean"
},
...
},
"example": { // <------ schema-level example
"status": true,
"response": {
"ProductNumber": "number",
"Barcode": "number",
"Length": 12,
"Width": 34,
"Height": 423,
"Volume": 1232
},
"errors": null
}
}
}
or use property-level examples:
"definitions": {
"BaseResponse": {
"type": "object",
"properties": {
"status": {
"type": "boolean",
"example": true // <------
},
"response": {
"$ref": "#/definitions/Product"
},
"errors": {
"example": null // <------
}
}
},
"Product": {
"type": "object",
"properties": {
"ProductNumber": {
"type": "string",
"example": "number" // <------
},
"Length": {
"type": "integer",
"example": 12 // <------
},
...
}
}
}
I'd like to note that "errors": null and "example": null are not actually valid in OpenAPI 2.0 (fka Swagger), because it does not support nullable types. Nullable types are supported in OpenAPI 3.0 only.
I'v tried many solutions, the only one that seems to work is using 'examples' key and not 'example'.
then put a single example (you should name it) and put the $ref under it.
just like this:
"responses": {
"200": {
"description": "Успешный ответ сервиса",
"schema":
"$ref": "#/definitions/BaseResponse"
"examples":
"example_name": # You can give as an example name any name you want
$ref: "#/examples/200_res"
examples:
200_res:
description: example for 200_res
value:
{
"status": true,
"response": {
"ProductNumber": "number",
"Barcode": "number",
"Length": 12,
"Width": 34,
"Height": 423,
"Volume": 1232
},
"errors": null
}
also notice you have problems with your { } at the example you added
I have created an index named filebeat in elasticsearch.The log data is being sent in elasticsearch by filebeat agent.
I want to filter the results based on a particular column/field named value_of_type. Using PHP API :
$json =
'{
"query" : {
"bool" : {
"filter": [
{
"term" :
{
"value_of_type" : "sound"
}
}
]
}
}
}';
But its returning 0 results.
{"took":4,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
The result of hitting myurl:9200/filebeat in browser looks like this :
{
"filebeat": {
"aliases": {},
"mappings": {
"doc": {
"properties": {
"#timestamp": {
"type": "date"
},
"beat": {
"properties": {
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"fields": {
"properties": {
"node": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value_of_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"input_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"offset": {
"type": "long"
},
"source": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
},
"settings": {
"index": {
"creation_date": "1494116541083",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "IdhWgIqiQ-GNrZK3AvCP9g",
"version": {
"created": "5020199"
},
"provided_name": "filebeat"
}
}
}
}
Your query is correct, but it doesn't match anything in the index: the documents you have indexed don't have the correct structure.
For you query to match a document, documents in your index should have a value_of_type field with value "sound" (like returned in the sample response below.)
A simple GET /filebeat/_search query (without any filter) should give a result like this:
{
"took": 28,
"timed_out": false,
"hits": {
"total": N, // the number of documents in your index
"max_score": 1,
"hits": [
...
{
"_index": "filebeat",
"_type": "some_doc_type",
"_id": "some_id",
"_score": 1,
"_source": {
...
"value_of_type": "sound", // that's what you query will match
...
}
]
}
I have a mapping like this
{
"settings": {
"analysis": {
"filter": {
"nGramFilter": {
"type": "nGram",
"min_gram": 3,
"max_gram": 20,
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
},
"email" : {
"type" : "pattern_capture",
"preserve_original" : 1,
"patterns" : [
"([^#]+)",
"(\\p{L}+)",
"(\\d+)",
"#(.+)"
]
},
"number" : {
"type" : "pattern_capture",
"preserve_original" : 1,
"patterns" : [
"([^+-]+)",
"(\\d+)"
]
},
"edgeNGramFilter": {
"type": "nGram",
"min_gram": 1,
"max_gram": 10,
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
}
},
"analyzer": {
"nGramAnalyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"nGramFilter"
]
},
"whitespaceAnalyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase"
]
},
"email" : {
"tokenizer" : "uax_url_email",
"filter" : [
"email",
"lowercase",
"unique"
]
},
"number" : {
"tokenizer" : "whitespace",
"filter" : [ "number", "unique" ]
},
"edgeNGramAnalyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"edgeNGramFilter"
]
}
}
}
},
"users": {
"mappings": {
"user_profiles": {
"properties": {
"firstName": {
"type": "string",
"analyzer": "nGramAnalyzer",
"search_analyzer": "whitespaceAnalyzer"
},
"lastName": {
"type": "string",
"analyzer": "nGramAnalyzer",
"search_analyzer": "whitespaceAnalyzer"
},
"email": {
"type": "string",
"analyzer": "email",
"search_analyzer": "whitespaceAnalyzer"
},
"score" : {
"type": "string"
},
"homeLandline": {
"type": "string",
"analyzer": "number",
"search_analyzer": "whitespaceAnalyzer"
},
"dob": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"mobile": {
"type": "integer"
},
"residenceCity": {
"type": "string",
"analyzer": "edgeNGramAnalyzer",
"search_analyzer": "whitespaceAnalyzer"
},
"created_at": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
}
}
}
}
}
I can get the score as integer as well as "NA" so I mapped the type as string but while posting data to the index i am getting Number Format Exception.
For Example:
if I post first data as integer and followed by "NA". I am getting these exception.
while checking my log file I am getting this errors:
[2016-08-29 15:19:01] elasticlog.WARNING: Response ["{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed
to parse
[score]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"failed
to parse
[score]\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For
input string: \"NH\"\"}},\"status\":400}"] []
Your mapping is incorrect. It should be, assuming, users is the index name and user_profiles is the type:
{
"users": {
"mappings": {
"user_profiles": {
"properties": {
"score": {
"type": "string"
}
}
}
}
}
}
You have a missing mappings before user_profiles.
I have a JSON Schema for new orders, that consists of order list and address.
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "array",
"properties": {
"order": {
"type": "array",
"items": {
"type": "array",
"properties": {
"product_id": {
"type": "integer"
},
"quantity": {
"type": "integer"
}
},
"required": [
"product_id",
"quantity"
]
}
},
"address": {
"type": "array",
"properties": {
"name": {
"type": "string"
},
"phone": {
"type": "integer"
},
"address1": {
"type": "string"
},
"address2": {
"type": "string"
},
"city": {
"type": "string"
},
"state_or_region": {
"type": "string"
},
"country": {
"type": "string"
}
},
"required": [
"name",
"phone",
"address1",
"city",
"state_or_region",
"country"
]
}
},
"required": [
"order",
"address"
]
}
But it doesn't seem to actually validate the items at all (I'm using Laravel 5.2 with "justinrainbow/json-schema": "~2.0" ):
$refResolver = new \JsonSchema\RefResolver(new \JsonSchema\Uri\UriRetriever(), new \JsonSchema\Uri\UriResolver());
$schema = $refResolver->resolve(storage_path('schemas\orders.post.json'));
$errors = [];
$input = Request::input();
// Validate
$validator = new \JsonSchema\Validator();
$validator->check($input, $schema);
$msg = [];
if ($validator->isValid()) {
return Response::json(['valid'], 200, [], $this->pritify);
} else {
$msg['success'] = false;
$msg['message'] = "JSON does not validate";
foreach ($validator->getErrors() as $error) {
$msg['errors'][] = [
'error' => ($error['property'] = ' ') ? 'wrong_data' : $error['property'],
'message' => $error['message']
];
}
return Response::json($msg, 422, [], $this->pritify);
}
A request like this always comes valid:
{
"order": [
{
"product_id": 100,
"quantity": 1
},
{
"product_id": 0,
"quantity": 2
}
],
"address": []
}
Any ideas what am I doing wrong?
You have messed array and object types. The only array value in your scheme must be order. Fixed scheme:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"order": {
"type": "array",
"items": {
"type": "object",
"properties": {
"product_id": {
"type": "integer"
},
"quantity": {
"type": "integer"
}
},
"required": [
"product_id",
"quantity"
]
}
},
"address": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"phone": {
"type": "integer"
},
"address1": {
"type": "string"
},
"address2": {
"type": "string"
},
"city": {
"type": "string"
},
"state_or_region": {
"type": "string"
},
"country": {
"type": "string"
}
},
"required": [
"name",
"phone",
"address1",
"city",
"state_or_region",
"country"
]
}
},
"required": [
"order",
"address"
]
}
And validation errors I got with you test data:
JSON does not validate. Violations:
[address.name] The property name is required
[address.phone] The property phone is required
[address.address1] The property address1 is required
[address.city] The property city is required
[address.state_or_region] The property state_or_region is required
[address.country] The property country is required