I am parsing an external JSON datasource which gives me list of Bus Stop codes and corresponding street names.
Data is presented to me in this format:
BusStop01 Street01 Description22 Latitude31 Longitude44
BusStop12 Street05 Description72 Latitude13 Longitude32
BusStop22 Street34 Description28 Latitude43 Longitude21
...
I am able to parse the JSON and use the AWS PHP SDK and store this in DynamoDB in the same format with BusStop a the primary key. Here's the relevant section of my code:
$putResponse = $client->putItem([
'TableName' => 'TABLEDUMMY01',
'Item' => [
'BusStopCode' => ['N' => $BusStopCode ], // Primary Key
'RoadName' => ['S' => $RoadName ],
'Description' => ['S' => $Description ],
'Latitude' => ['S' => strval($Latitude) ],
'Longitude' => ['S' => strval($Longitude) ]
]
]);
What I really want to achieve is to store the same data in this format using StreetName as the Primary Key:
StreetName01, BusStop01, BusStop33, BusStop43...
StreetName02, BusStop23, BusStop29, BusStop67...
NOTE: The external datasource gives me only 50 values in one API call, hence I need to be able to append/push new BusStop IDs in to existing Street Names on DynamoDB
Related
I am trying to push data to a firestore DB using PHP and the Google apis.
Inside the documentation and examples I have seen around the web, I am able to use mapValue and arrayValue when sending data.
The example I am using is as follows:-
[
"orderName" => [
"stringValue" => "Gbeila Aliu Wahab"
],
"orderLocationName" => [
"stringValue" => "Accra Mall Limited"
],
"orderTotalAmount" => [
"doubleValue" => 150.5
],
"orderDescription" => [
"stringValue" => "Lorem Ipsum is simply dummy text of the printing and typesetting industry"
],
"orderLocationGeoPoints" => [
"geoPointValue" => (object) [
"latitude" => 5.5557,
"longitude" => -0.1963
]
],
"orderStatus" => [
"stringValue" => "NotAssigned"
],
]
This works perfectly fine, but when I attempt to send an object or an array I get the following error returned to me:-
"message": "Invalid JSON payload received. Unknown name \"map_value\" at 'document.fields[0].value': Proto field is not repeating, cannot start list.",
when attempting to map the value using the following code:-
"orderName" => [
"mapValue" => ["Gbeila Aliu Wahab", 123]
]
// or
"orderName" => [
"arrayValue" => [
"first" => [
"stringValue" => "test"
],
"second" => [
"stringValue" => "test123"
]
]
]
I have tried many variations to try to get this to work.
How am I supposed to be using the mapValue and arrayValue I can see a lot of mentions regarding the value option but I cannot see any examples on how to use the.
Any help would be greatly appreciated.
Payload to your array or map you're generating is incorrect as per the documentation. You need to wrap your actual data (to store) under values key, your final array should be:
["orderName" => ["arrayValue" => ["values" => [["stringValue" => "test"], ["stringValue" => "test123"]]]]]
Similarly your mapValue should be
["orderName" => ["mapValue" => ["fields" => ["field1" => ["stringValue" => "Gbeila Aliu Wahab"]]]]]
Also, you can play with other data mapper via this package.
In the table called template, I get the id of staffmembers by using session. However, I really don't know how I can to it for the table module. I am coding using the Laravel framework.
I also get this error message :
Invalid datetime format: 1366 Incorrect integer value: 'templates_id' for column 'templates_id' at row 1 (SQL: insert into modules (moduleName, moduleCode, year, templates_id)
template::create([
'paperTitle' => $request->paperTitle,
'markedElementOne' => $request->markedElementOne,
'scoreOne' => $request-> scoreOne,
'markedElementTwo' => $request->markedElementTwo,
'scoreTwo' => $request-> scoreTwo,
'staffmembers_id'=>\Session::get('staffmembers_id'),
]);
module::create([
'moduleName' => $request->moduleName,
'moduleCode' => $request->moduleCode,
'year' => $request->year,
'templates_id'=>('templates_id'),
]);
Get the template back to a variable when creating.
$tempalte = template::create([
'paperTitle' => $request->paperTitle,
'markedElementOne' => $request->markedElementOne,
'scoreOne' => $request-> scoreOne,
'markedElementTwo' => $request->markedElementTwo,
'scoreTwo' => $request-> scoreTwo,
'staffmembers_id' => \Session::get('staffmembers_id'),
]);
get template id from that variable to use to create 'module'
module::create([
'moduleName' => $request->moduleName,
'moduleCode' => $request->moduleCode,
'year' => $request->year,
'templates_id' => $template->id
]);
Could someone please provide a simple example of the usage for dealing with Odoo's one2many, many2many and selection fields when using Laradoo (or ripcord)?
Specifically how one would use them with create() and update(). In Python, it seems as if these are dealt with using special tuple commands however for PHP documentation seems very hard to find for these types of things and it would be extremely helpful.
For illustrative purposes in my particular project, I haven't been able to figure out how to relate a CRM lead tag to a lead during the creation process using Laradoo:
$id = $odoo->create('crm.lead', [
'type' => 'lead',
'priority' => 0, <-- what do we pass here for this selection field?
'name' => 'Example',
'contact_name' => 'John Doe',
'phone' => '555-555-5555',
'email_from' => 'example#domain.com',
'description' => 'Just some text.',
'tag_ids' => [1], <-- What do we pass here for this one2many field?
]);
In the example above when trying to set the priority selection field to an int other than 0 fails and when trying to pass an array of tag_ids (1 is valid tag id in my project), the lead remains untagged.
First of all selection field values are just string values that need to be part of the field defined selection values.
The values for relational fields like Onetomany and Many2many are ruled by the command formated values that you could read at:
https://github.com/odoo/odoo/blob/11.0/odoo/models.py#L3020-L3055
For the php api usage with ripcord you could set the tag_ids field value like:
$id = $odoo->create('crm.lead', [
'type' => 'lead',
'priority' => '0',
'name' => 'Example',
'contact_name' => 'John Doe',
'phone' => '555-555-5555',
'email_from' => 'example#domain.com',
'description' => 'Just some text.',
'tag_ids' => array(array(4,1)),
]);
This translate as that 1 is the id of a known and already existing crm.lead.tag that you could link to the m2m tag_ids field using the command 4. This could also be expressed using command 6 to link multiple ids on the same command value:
'tag_ids' => array(array(6,0,array(1,2,3))),
where using command 4 it will be:
'tag_ids' => array(array(4,1), array(4,2), array(4,3)),
I want to update data using 'typeId', 'type_id' is not a primary key.
While this code is work, if we use other primary key.
Unable to update record.
getting following error :
{"__type":"com.amazon.coral.validate#ValidationException","message":"The provided key element does not match the schema"}
$response = $this->dbo->updateItem([
'TableName' => $this->tableName,
'Key' => [
'typeId' => ['S' => "qtwr234"]
],
'ExpressionAttributeValues' => [
':val1' => ['N' => '1']
],
'UpdateExpression' => 'set count = :val1',
'ReturnValues' => 'ALL_NEW'
]);
According the error message you're receiving
This error occurs when your key (hash / primary key) doesn't not match in parameters your passing to update the data in table.
Solution :
Run listTables commands and check for key element you created while creating elements.
Now, replace that key element in query parameters.
Thanks
In my example code I am using the php client library, but it should be understood by anyone familiar with elasticsearch.
I'm using elasticsearch to create an index where each document contains an array of nGram indexed authors. Initially, the document will have a single author, but as time progresses, more authors will be appended to the array. Ideally, a search could be executed by an author's name, and if any of the authors in the array get matched, the document will be found.
I have been trying to use the documentation here for appending to the array and here for using the array type - but I have not had success getting this working.
First, I want to create an index for documents, with a title, array of authors, and an array of comments.
$client = new Client();
$params = [
'index' => 'document',
'body' => [
'settings' => [
// Simple settings for now, single shard
'number_of_shards' => 1,
'number_of_replicas' => 0,
'analysis' => [
'filter' => [
'shingle' => [
'type' => 'shingle'
]
],
'analyzer' => [
'my_ngram_analyzer' => [
'tokenizer' => 'my_ngram_tokenizer',
'filter' => 'lowercase',
]
],
// Allow searching for partial names with nGram
'tokenizer' => [
'my_ngram_tokenizer' => [
'type' => 'nGram',
'min_gram' => 1,
'max_gram' => 15,
'token_chars' => ['letter', 'digit']
]
]
]
],
'mappings' => [
'_default_' => [
'properties' => [
'document_id' => [
'type' => 'string',
'index' => 'not_analyzed',
],
// The name, email, or other info related to the person
'title' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'authors' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'comments' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
]
],
]
]
];
// Create index `person` with ngram indexing
$client->indices()->create($params);
Off the get go, I can't even create the index due to this error:
{"error":"MapperParsingException[mapping [_default_]]; nested: MapperParsingException[No handler for type [list] declared on field [authors]]; ","status":400}
HAD this gone successfully though, I would plan to create an index, starting with empty arrays for authors and title, something like this:
$client = new Client();
$params = array();
$params['body'] = array('document_id' => 'id_here', 'title' => 'my_title', 'authors' => [], 'comments' => []);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$ret = $client->index($params);
return $ret;
This seems like it should work if I had the desired index to add this structure of information to, but what concerns me would be appending something to the array using update. For example,
$client = new Client();
$params = array();
//$params['body'] = array('person_id' => $person_id, 'emails' => [$email]);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$params['script'] = 'NO IDEA WHAT THIS SCRIPT SHOULD BE TO APPEND TO THE ARRAY';
$ret = $client->update($params);
return $ret;
}
I am not sure how I would go about actually appending a thing to the array and making sure it's indexed.
Finally, another thing that confuses me is how I could search based on any author in the array. Ideally I could do something like this:
But I'm not 100% whether it will work. Maybe there is something fundemental about elasticsearch that I am not understanding. I am completely new to so any resources that will get me to a point where these little details don't hang me up would be appreciated.
Also, any direct advice on how to use elasticsearch to solve these problems would be appreciated.
Sorry for the big wall of text, to recap, I am looking for advice on how to
Create an index that supports nGram analysis on all elements of an array
Updating that index to append to the array
Searching for the now-updated index.
Thanks for any help
EDIT: thanks to #astax, I am now able to create the index and append to the value as a string. HOWEVER, there are two problems with this:
the array is stored as a string value, so a script like
$params['script'] = 'ctx._source.authors += [\'hello\']';
actually appends a STRING with [] rather than an array containing a value.
the value inputted does not appear to be ngram analyzed, so a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hello';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // SUCCESS
will find the new value but a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hel';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // NO RESULTS
does not
There is no type "list" in elasticsearch. But you can use "string" field type and store array of values.
....
'comments' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
....
And index a document this way:
....
$params['body'] = array(
'document_id' => 'id_here',
'title' => 'my_title',
'authors' => [],
'comments' => ['comment1', 'comment2']);
....
As for the script for apending an element to array, this answer may help you - Elasticsearch upserting and appending to array
However, do you really need to update the document? It might be easier to just reindex it as this is exactly what Elasticsearch does internally. It reads the "_source" property, does the required modification and reindexes it. BTW, this means that "_source" must be enabled and all properties of the document should be included into it.
You also may consider storing comments and authors (as I understand these are authors of comments, not the document authors) as child document in ES and using "has_child" filter.
I can't really give you specific solution, but strongly recommend installing Marvel plugin for ElasticSearch and use its "sense" tool to check how your overall process works step by step.
So check if your tokenizer is properly configured by running tests as described at http://www.elastic.co/guide/en/elasticsearch/reference/1.4/indices-analyze.html.
Then check if your update script is doing what you expect by retrieving the document by running GET /document/example_type/some_existing_id
The authors and comments should be arrays, but not strings.
Finally perform the search:
GET /document/_search
{
'query' : {
'match': { '_all': 'hel' }
}
}
If you're building the query yourself rather than getting it from the user, you may use query_string with placeholders:
GET /document/_search
{
'query' : {
'query_string': {
'fields': '_all',
'query': 'hel*'
}
}
}