Related
Stored a new index by the following code.
$data = array(
array("Name"=>"Norman","Email"=>"in.dolor#vulputatemauris.ca","Created"=>"2019-12-29 10:28:03","Modified"=>"2020-11-07 01:45:23"),
array("Name"=>"Drake","Email"=>"posuere#sedorcilobortis.co.uk","Created"=>"2020-11-08 14:37:00","Modified"=>"2019-08-10 06:42:07"),
array("Name"=>"Wynne","Email"=>"ligula.Donec#adipiscingenim.net","Created"=>"2019-05-19 23:30:42","Modified"=>"2019-06-09 08:13:58"),
array("Name"=>"Kirsten","Email"=>"lobortis#Suspendisseeleifend.net","Created"=>"2020-01-09 23:34:19","Modified"=>"2020-04-16 10:23:07"),
array("Name"=>"Ainsley","Email"=>"elit.dictum.eu#Quisquelibero.org","Created"=>"2019-01-22 18:14:39","Modified"=>"2019-09-02 18:44:30"),
array("Name"=>"Walker","Email"=>"ullamcorper#luctussitamet.org","Created"=>"2020-11-05 23:04:46","Modified"=>"2020-01-03 09:29:36"),
array("Name"=>"Evelyn","Email"=>"amet.metus.Aliquam#dui.org","Created"=>"2020-06-28 13:23:09","Modified"=>"2019-04-02 05:41:33"),
array("Name"=>"James","Email"=>"amet.risus#nullaCras.net","Created"=>"2020-04-20 10:15:54","Modified"=>"2020-07-22 12:04:49"),
array("Name"=>"Melvin","Email"=>"nec.eleifend.non#elit.edu","Created"=>"2020-03-07 05:19:53","Modified"=>"2018-12-30 19:33:29"),
);
$hosts = ['http://localhost:9200'];
$client = ClientBuilder::create()->setHosts($hosts)->build();
$params = [
'index' => 'dummy_data',
'id' => 'my_id',
'body' => ['data' => $data]
];
$response = $client->index($params);
Trying to search the first record by the following:
$params = [
'index' => 'dummy_data',
'body' => [
'query' => [
'bool' => [
'must' => [
[ 'match' => [ 'data.Name' => 'Norman' ] ],
],
]
]
]
];
$results = $client->search($params);
The issue is it's returning all the records where the query is matching with only the first one.
Please help with this.
Im guessing you are using the official Elasticsearch-PHP library. In that case it looks like you are indexing all provided data as exactly 1 document with the id being 'my_id'. That is why you get the the entire dataset back when your search matches.
If you want to index multiple documents at the same time, you should look at the bulk_index endpoint.
Here is the official example for reference:
for($i = 0; $i < 100; $i++) {
$params['body'][] = [
'index' => [
'_index' => 'my_index',
]
];
$params['body'][] = [
'my_field' => 'my_value',
'second_field' => 'some more values'
];
}
$responses = $client->bulk($params);
In my MongoDB database one name field is available, in this name field data is stored in the format Aaa, Bbb, Ccc, Ddd. The first letter of the word is uppercase and remaining letter is lowercase. When I apply MongoDB sort query on this data. The sorting is not working properly
My code is like below one:-
$manager = new MongoDB\Driver\Manager("mongodb://localhost:27017");
$pipeline = [
[ '$match' => ['listingStatus' => 'Active'] ],
[ '$sort' => ['listingParticipants.firstName' => -1]],
[ '$group' => ['_id' => '$listingParticipants.email'] ],
[ '$limit' => 10],
[ '$skip' => 0],
];
$aggregate = new \MongoDB\Driver\Command([
'aggregate' => 'test_collection',
'pipeline' => $pipeline,
'cursor' => new stdClass
]);
$cursor = $manager->executeCommand('test_database', $aggregate);
I seen multiple solution on google, they use duplicate field with lower case and apply filter on that newly created field.
But in my project I don't create new field so please help me to solve this issue.
I have created elasticsearch index:
$es = Elasticsearch\ClientBuilder::create()->build();
$params = [
'index'=>'articles',
'type' => 'article'
];
for ($i=0; $i<30; $i++) {
$params['body'] = [ 'title'=>'title '.$i, 'body'=>'text '.$i ];
$response = $es->index($params);
}
So, 30 documents are added, now I need get total count of records. This works
$search_params= [
'index'=>'articles',
'type' => 'article',
];
$query = $es->search($search_params);
echo $query['hith']['total'];
But as I've read, more efficient is using straight counting _count method.
My problem is, that I not understood how to implement _count in php API?
tried:
$search_params= [
'index'=>'articles',
'type' => 'article',
'body' => [
'query' => ['_count'=>[] ]
]
];
$query = $es->search($search_params);
and several another variants, but don't get right one syntax.
Help?
Have you tried
$query = $es->count($search_params);
https://www.elastic.co/guide/en/elasticsearch/client/php-api/2.0/ElasticsearchPHP_Endpoints.html#Elasticsearch_Clientcount_count
In my example code I am using the php client library, but it should be understood by anyone familiar with elasticsearch.
I'm using elasticsearch to create an index where each document contains an array of nGram indexed authors. Initially, the document will have a single author, but as time progresses, more authors will be appended to the array. Ideally, a search could be executed by an author's name, and if any of the authors in the array get matched, the document will be found.
I have been trying to use the documentation here for appending to the array and here for using the array type - but I have not had success getting this working.
First, I want to create an index for documents, with a title, array of authors, and an array of comments.
$client = new Client();
$params = [
'index' => 'document',
'body' => [
'settings' => [
// Simple settings for now, single shard
'number_of_shards' => 1,
'number_of_replicas' => 0,
'analysis' => [
'filter' => [
'shingle' => [
'type' => 'shingle'
]
],
'analyzer' => [
'my_ngram_analyzer' => [
'tokenizer' => 'my_ngram_tokenizer',
'filter' => 'lowercase',
]
],
// Allow searching for partial names with nGram
'tokenizer' => [
'my_ngram_tokenizer' => [
'type' => 'nGram',
'min_gram' => 1,
'max_gram' => 15,
'token_chars' => ['letter', 'digit']
]
]
]
],
'mappings' => [
'_default_' => [
'properties' => [
'document_id' => [
'type' => 'string',
'index' => 'not_analyzed',
],
// The name, email, or other info related to the person
'title' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'authors' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'comments' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
]
],
]
]
];
// Create index `person` with ngram indexing
$client->indices()->create($params);
Off the get go, I can't even create the index due to this error:
{"error":"MapperParsingException[mapping [_default_]]; nested: MapperParsingException[No handler for type [list] declared on field [authors]]; ","status":400}
HAD this gone successfully though, I would plan to create an index, starting with empty arrays for authors and title, something like this:
$client = new Client();
$params = array();
$params['body'] = array('document_id' => 'id_here', 'title' => 'my_title', 'authors' => [], 'comments' => []);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$ret = $client->index($params);
return $ret;
This seems like it should work if I had the desired index to add this structure of information to, but what concerns me would be appending something to the array using update. For example,
$client = new Client();
$params = array();
//$params['body'] = array('person_id' => $person_id, 'emails' => [$email]);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$params['script'] = 'NO IDEA WHAT THIS SCRIPT SHOULD BE TO APPEND TO THE ARRAY';
$ret = $client->update($params);
return $ret;
}
I am not sure how I would go about actually appending a thing to the array and making sure it's indexed.
Finally, another thing that confuses me is how I could search based on any author in the array. Ideally I could do something like this:
But I'm not 100% whether it will work. Maybe there is something fundemental about elasticsearch that I am not understanding. I am completely new to so any resources that will get me to a point where these little details don't hang me up would be appreciated.
Also, any direct advice on how to use elasticsearch to solve these problems would be appreciated.
Sorry for the big wall of text, to recap, I am looking for advice on how to
Create an index that supports nGram analysis on all elements of an array
Updating that index to append to the array
Searching for the now-updated index.
Thanks for any help
EDIT: thanks to #astax, I am now able to create the index and append to the value as a string. HOWEVER, there are two problems with this:
the array is stored as a string value, so a script like
$params['script'] = 'ctx._source.authors += [\'hello\']';
actually appends a STRING with [] rather than an array containing a value.
the value inputted does not appear to be ngram analyzed, so a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hello';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // SUCCESS
will find the new value but a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hel';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // NO RESULTS
does not
There is no type "list" in elasticsearch. But you can use "string" field type and store array of values.
....
'comments' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
....
And index a document this way:
....
$params['body'] = array(
'document_id' => 'id_here',
'title' => 'my_title',
'authors' => [],
'comments' => ['comment1', 'comment2']);
....
As for the script for apending an element to array, this answer may help you - Elasticsearch upserting and appending to array
However, do you really need to update the document? It might be easier to just reindex it as this is exactly what Elasticsearch does internally. It reads the "_source" property, does the required modification and reindexes it. BTW, this means that "_source" must be enabled and all properties of the document should be included into it.
You also may consider storing comments and authors (as I understand these are authors of comments, not the document authors) as child document in ES and using "has_child" filter.
I can't really give you specific solution, but strongly recommend installing Marvel plugin for ElasticSearch and use its "sense" tool to check how your overall process works step by step.
So check if your tokenizer is properly configured by running tests as described at http://www.elastic.co/guide/en/elasticsearch/reference/1.4/indices-analyze.html.
Then check if your update script is doing what you expect by retrieving the document by running GET /document/example_type/some_existing_id
The authors and comments should be arrays, but not strings.
Finally perform the search:
GET /document/_search
{
'query' : {
'match': { '_all': 'hel' }
}
}
If you're building the query yourself rather than getting it from the user, you may use query_string with placeholders:
GET /document/_search
{
'query' : {
'query_string': {
'fields': '_all',
'query': 'hel*'
}
}
}
I've got a form for a job application, where administrator needs to pick from the list of certain user ids. The list contains only user ids of type "employer", however I want to validate the administrator input, so if he manually inserts an id that doesn't exist or is for user of different type than "employer", the validation should fail. I thought that code to do this would be:
new Zend_Validate_Db_RecordExists(
array(
'table' => 'users',
'field' => 'id',
'exclude' => "mode != 'employer'"
)
)
so, I'm searching for all the records in table users, excluding those where mode != 'employer' - if such record exists, where id is equal to the one picked from input, it passes the validation. However, the code above doesn't work - I have to do 'exclude' => "mode = 'employer'", so exclude actually equals where statement. My understanding of the logic here is wrong - can somebody tell me why?
PHP: 5.2.17, Zend: 1.10.4
EDIT: (the comment to #ro ko enquiries, as it probably clears things out)
Please find the table and sample code here: http://pastebin.com/C7AXMNTZ . In my understanding this should return valid for Joker (is employer), but false for Kingpin (not employer) and Poison Ivy (not in the db) - as you can see the results are not what I'm expecting.
A) 'exclude' => "mode != 'employer'"
$id = new Zend_Form_Element_Select("id");
$id->setRegisterInArrayValidator(false);
$id->addValidator(new Zend_Validate_Db_RecordExists(array(
'table' => 'villains',
'field' => 'id',
'exclude' => "mode != 'employer'"
)));
Produces the following query:
SELECT `villains`.`id`
FROM `villains`
WHERE (`id` = :value) AND (mode != 'employer')
LIMIT 1
B) 'exclude' => "mode = 'employer'"
$id = new Zend_Form_Element_Select("id");
$id->setRegisterInArrayValidator(false);
$id->addValidator(new Zend_Validate_Db_RecordExists(array(
'table' => 'villains',
'field' => 'id',
'exclude' => "mode = 'employer'"
)));
Produces the following query:
SELECT `villains`.`id`
FROM `villains`
WHERE (`id` = :value) AND (mode = 'employer')
LIMIT 1
C) 'exclude' => array("field" => "mode", "value" => "employer")
$id = new Zend_Form_Element_Select("id");
$id->setRegisterInArrayValidator(false);
$id->addValidator(new Zend_Validate_Db_RecordExists(array(
'table' => 'villains',
'field' => 'id',
'exclude' => array(
"field" => "mode",
"value" => "employer"
)
)));
Produces the following query:
SELECT `villains`.`id`
FROM `villains`
WHERE (`id` = :value) AND (`mode` != 'employer')
LIMIT 1
Outcomes
You want B. It is confusing and arguably the logic and behaviour of the component is backwards. Nonetheless, the behaviour you want is from example B.
Appendix
We can hack a test (and I really mean hack together) to check that the above works as expected.
Both test1 and test2 pass, but as you can see from the providers, they both produce different results.
class SO14706653Test extends PHPUnit_Framework_TestCase
{
/**
* #var Zend_Test_PHPUnit_Db_Connection
*/
public $dbConnection;
public function getRowCount($tableName) {
$query = "SELECT COUNT(*) FROM ".$this->dbConnection->quoteSchemaObject($tableName);
return (int) $this->dbConnection->getConnection()->query($query)->fetchColumn();
}
// hack a very quick setup for tests
public function setup() {
$app = new Zend_Application(APPLICATION_ENV, APPLICATION_PATH . '/configs/application.ini');
$app->bootstrap();
$dbAdapter = $app->getBootstrap()->getResource('db'); /* #var $db Zend_Db_Adapter_Pdo_Mysql */
$this->dbConnection = new Zend_Test_PHPUnit_Db_Connection($dbAdapter, 'unittests');
$dbAdapter->exec("CREATE TABLE IF NOT EXISTS `villains` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) NOT NULL,
`mode` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1");
$dbAdapter->exec('DELETE FROM villains'); // clean out db data
$dbAdapter->exec("
INSERT INTO `villains` VALUES(1, 'Joker', 'employer');
INSERT INTO `villains` VALUES(2, 'Kingpin', '');
INSERT INTO `villains` VALUES(3, 'Penguin', '');
");
}
// ensure the above setup is working as expected
public function assertPreConditions() {
$this->assertEquals(3, $this->getRowCount('villains'));
}
public function provideTest1()
{
return [
// form data is valid? isRequired?
[['id' => '1'], false, false],
[['id' => '2'], true, false],
[['id' => '3'], true, false],
[['id' => ''], true, false],
[[], true, false],
[['id' => '856'], false, false],
[['id' => '856'], false, true],
[['id' => ''], false, true],
[[], false, true],
];
}
public function provideTest2()
{
return [
// form data is valid? isRequired?
[['id' => '1'], true, false],
[['id' => '2'], false, false],
[['id' => '3'], false, false],
[['id' => ''], true, false],
[[], true, false],
[['id' => '856'], false, false],
[['id' => '856'], false, true],
[['id' => ''], false, true],
[[], false, true],
];
}
/**
* #dataProvider provideTest1
*/
public function test1(array $data, $isValid, $isRequired)
{
$form = new Zend_Form();
$id = new Zend_Form_Element_Select("id");
$id->setRegisterInArrayValidator(false);
$id->addValidator(new Zend_Validate_Db_RecordExists(array(
'table' => 'villains',
'field' => 'id',
'exclude' => "mode != 'employer'"
)));
$id->setRequired($isRequired);
$form->addElement($id);
// produces the query
// SELECT `villains`.`id`
// FROM `villains`
// WHERE (`id` = :value) AND (mode != 'employer')
// LIMIT 1
$this->assertSame($isValid, $form->isValid($data));
}
/**
* #dataProvider provideTest2
*/
public function test2(array $data, $isValid, $isRequired)
{
$form = new Zend_Form();
$id = new Zend_Form_Element_Select("id");
$id->setRegisterInArrayValidator(false);
$id->addValidator(new Zend_Validate_Db_RecordExists(array(
'table' => 'villains',
'field' => 'id',
'exclude' => "mode = 'employer'"
)));
$id->setRequired($isRequired);
$form->addElement($id);
// produces the query
// SELECT `villains`.`id`
// FROM `villains`
// WHERE (`id` = :value) AND (mode = 'employer')
// LIMIT 1
$this->assertSame($isValid, $form->isValid($data));
}
}
I think you got confused here. Logic is fine with the validator.
How?
Well exclude here adds negation to your condition.
'exclude' => "mode = 'employer'"
OR
'exclude' => array(
'field' => 'mode',
'value' => 'employer'
)
is interpreted as validator should stand valid for other than mode = employer condition.
EDIT: If you see In the manual you'll see a description: "The above example will check the table to ensure no records other than the one where id = $user_id contains the value $username." this statement should clear things for you I assume.
You pass an array in the exclude option. Eg.
$this->getElement('username')->addValidator(
new Zend_Validate_Db_NoRecordExists(
array(
'table' => 'user',
'field' => 'username',
'exclude' => array(
'field' => 'username',
'value' => $username
)
)
)
);
Check here for more info.
Edit: After reading the question again, I see I didn't actually answer it. The only other advice I would give is using the quoteInto statement as in the documentation.
$email = 'user#example.com';
$clause = $db->quoteInto('email != ?', $email);
$validator = new Zend_Validate_Db_RecordExists(
array(
'table' => 'users',
'field' => 'username',
'exclude' => $clause
)
);
Also here is a link to another question about Zend_Validate_Db_RecordExistswhich has a very good answer.
How to modify zend Db_RecordExists validator where clause?
As per documentation (search for The above example will check the table to ensure no records other than the one where id = $user_id contains the value $username.) the exclude option indeed works as WHERE statement.
I can only say that whoever though that using such keyword for this functionality is a good idea had a wicked sense of humor.