Creating a table in BigQuery using the PHP api with schema - php

I'm trying to create a table in bigquery using the php api.
I can create a table without a schema just fine, but when I provide a schema I get errors. It looks like I'm using the wrong syntax, however I tried just about any formatting I could think of and couldn't find a single example of what I'm trying to achieve.
I use a string literal for the fields parameter for testing. My code looks like this:
$bigQuery = new BigQueryClient([
'keyFilePath' => [keyfilepath],
'projectId' => [projectid],
'location' => [location]
]);
/** #var Dataset $dataSet */
$dataSet = $bigQuery->dataset('my-data-set');
$fieldString = '{"name": "myfield","type": "STRING","mode": "REQUIRED"}' . "\n" . '{"name": "anotherfield", "type": "STRING", "mode": "REQUIRED"}';
$options = [
'fields' => $fieldString
];
$dataSet->createTable('mytable', $options);
Which gives the error:
"Invalid field selection {\"name\":\"myfield\""
Or alternatively, when I format the "$fieldString" like this:
$fieldString = '[{"name": "myfield","type": "STRING","mode": "REQUIRED"}, {"name": "anotherfield", "type": "STRING", "mode": "REQUIRED"}]';
I get the error:
Invalid FieldMask '[{\"name\":\"myfield\",\"type\":\"STRING\",\"mode\":\"REQUIRED\"},{\"name\":\"anotherfield\",\"type\":\"STRING\",\"mode\":\"REQUIRED\"}]'. Map keys should be represented as [\"some_key\"].
I've also tried to create the table first and then update it like so:
$table = $dataSet->createTable('mytable');
$table->update($options);
But I get the same errors. Even when I use the the json representation exactly like shown here the problem persists.
What am I doing wrong here?
UPDATE:
I actually first tried this, before I switched to a string literal for the fields:
$fields = [
['name'=> 'myfield', 'type' => 'INTEGER', 'mode' => 'REQUIRED'],
['name'=> 'anotherfield', 'type' => 'INTEGER', 'mode' => 'REQUIRED']
];
$options = [
'schema' => $fields
];
$dataSet->createTable('mytable', $options);
This yields the error:
"Invalid JSON payload received. Unknown name \"schema\" at 'table': Proto field is not repeating, cannot start list."
Then I edited the code to look like this:
$fields = [
['name'=> 'myfield', 'type' => 'INTEGER', 'mode' => 'REQUIRED'],
['name'=> 'anotherfield', 'type' => 'INTEGER', 'mode' => 'REQUIRED']
];
$options = [
'fields' => $fields
];
$dataSet->createTable('mytable', $options);
Which gives:
Warning: rawurlencode() expects parameter 1 to be string, array given
I didn't mention this in my question before because I thought it wasn't relevant. In hindsight it probably is, but my problem still persists.

There is no such thing as $fieldString that is $fields array
like this:
$fields = [
[
'name' => 'field1',
'type' => 'string',
'mode' => 'required'
],
[
'name' => 'field2',
'type' => 'integer'
],
];
then
$schema = ['fields' => $fields];
$table = $dataset->createTable($tableId, ['schema' => $schema]);
See example here.

Related

Validate array if already exists in mysql php laravel

Currently I have set of array.
and
I can easily insert these data to my database using Laravel without doing any validations
here's the sample array
CODE:
$excel1 = Importer::make('Excel');
$excel1->hasHeader(true);
$excel1->load($savePath.$fileName);
$excel1->setSheet(2);
$collection1 = $excel1->getCollection();
$arr1 = json_decode($collection1,true);
foreach ($arr1 as $row1) {
$insert_data1[] = array(
'projCode' => $projCode,
'emp_id' => $row1['company_id'],
'type' => 'EMP',
'deleted' => 0,
'by_id' => auth()->user()->id,
'updated_by' => auth()->user()->name,
'created_at' => now(),
'updated_at' => now(),
);
}
dd($insert_data1);
OUTPUT:
and I'm using this code to insert these data to my table
DB::table('tbl_emp_proj')->insert($insert_data1);
and this works fine but the problem is,
I'm trying to validate if emp_id exists or not in my users table
Here's my users table
The value of emp_id from array should check if it already exists in my users using company_id field from users. How can I validate it if $insert_data1 is an array and should be check if it exists on database?
UPDATE
currently i have this validator and I tried to add up the $Insert_data1 but gives me undefined var for $insert_data1.
$validator = Validator::make(
[
'file' => $request->file,
'extension' => strtolower($request->file->getClientOriginalExtension()),
],
[
'file' => 'required|max:5000',
'extension' => 'required|in:,csv,xlsx,xls',
],
$insert_data1,
[
'*.emp_id' => "required|exists:users,company_id",
]
);
You can use Laravel Validator to validate any arrays as if its a request input.
use Illuminate\Support\Facades\Validator;
$validator = Validator::make(
$insert_data1,
[
'*.emp_id' => "required|integer|exists:users,company_id",
]
);
EDIT:
You can receive error messages and error items with the validator APIs.
$failed = $validator->fails(); //boolean
$errors = $validator->errors();
$validated = $validator->validated();
$invalid = $validator->invalid();

Search Filter not working in Elastic Search PHP

Stored a new index by the following code.
$data = array(
array("Name"=>"Norman","Email"=>"in.dolor#vulputatemauris.ca","Created"=>"2019-12-29 10:28:03","Modified"=>"2020-11-07 01:45:23"),
array("Name"=>"Drake","Email"=>"posuere#sedorcilobortis.co.uk","Created"=>"2020-11-08 14:37:00","Modified"=>"2019-08-10 06:42:07"),
array("Name"=>"Wynne","Email"=>"ligula.Donec#adipiscingenim.net","Created"=>"2019-05-19 23:30:42","Modified"=>"2019-06-09 08:13:58"),
array("Name"=>"Kirsten","Email"=>"lobortis#Suspendisseeleifend.net","Created"=>"2020-01-09 23:34:19","Modified"=>"2020-04-16 10:23:07"),
array("Name"=>"Ainsley","Email"=>"elit.dictum.eu#Quisquelibero.org","Created"=>"2019-01-22 18:14:39","Modified"=>"2019-09-02 18:44:30"),
array("Name"=>"Walker","Email"=>"ullamcorper#luctussitamet.org","Created"=>"2020-11-05 23:04:46","Modified"=>"2020-01-03 09:29:36"),
array("Name"=>"Evelyn","Email"=>"amet.metus.Aliquam#dui.org","Created"=>"2020-06-28 13:23:09","Modified"=>"2019-04-02 05:41:33"),
array("Name"=>"James","Email"=>"amet.risus#nullaCras.net","Created"=>"2020-04-20 10:15:54","Modified"=>"2020-07-22 12:04:49"),
array("Name"=>"Melvin","Email"=>"nec.eleifend.non#elit.edu","Created"=>"2020-03-07 05:19:53","Modified"=>"2018-12-30 19:33:29"),
);
$hosts = ['http://localhost:9200'];
$client = ClientBuilder::create()->setHosts($hosts)->build();
$params = [
'index' => 'dummy_data',
'id' => 'my_id',
'body' => ['data' => $data]
];
$response = $client->index($params);
Trying to search the first record by the following:
$params = [
'index' => 'dummy_data',
'body' => [
'query' => [
'bool' => [
'must' => [
[ 'match' => [ 'data.Name' => 'Norman' ] ],
],
]
]
]
];
$results = $client->search($params);
The issue is it's returning all the records where the query is matching with only the first one.
Please help with this.
Im guessing you are using the official Elasticsearch-PHP library. In that case it looks like you are indexing all provided data as exactly 1 document with the id being 'my_id'. That is why you get the the entire dataset back when your search matches.
If you want to index multiple documents at the same time, you should look at the bulk_index endpoint.
Here is the official example for reference:
for($i = 0; $i < 100; $i++) {
$params['body'][] = [
'index' => [
'_index' => 'my_index',
]
];
$params['body'][] = [
'my_field' => 'my_value',
'second_field' => 'some more values'
];
}
$responses = $client->bulk($params);

How can I make typeahead show database information in yii2?

I want to make search in my project. I use typeahead but it's not working. This is my code:
<?php
echo '<label class="control-label">Select Repository</label>';
$template = '<div><p class="repo-language">{{no_telepon}}</p>' .
'<p class="repo-name">{{nama}}</p>' .
'<p class="repo-description">{{email}}</p></div>';
echo Typeahead::widget([
'name' => 'twitter_oss',
'options' => ['placeholder' => 'Filter as you type ...'],
'dataset' => [
[
'prefetch' => Penerima::find()->all(),
'datumTokenizer' => "Bloodhound.tokenizers.obj.whitespace('value')",
'display' => 'value',
'templates' => [
'notFound' => '<div class="text-danger" style="padding:0 8px">Unable to find repositories for selected query.</div>',
'suggestion' => new JsExpression("Handlebars.compile('{$template}')")
]
]
]
]);
?>
This question was asked long time a go.
I also faced the same problem, but i could figure-out this.
for future reference i add this post.
in your controller
$result = SampleModel::find()
->select('Attribute_name')
->where('name LIKE "%' . $searchParameter .'%"')
->asArray()
->all();
return Json::encode($result);
here you need to get the database value as "associative array", you can get that from using "asArray()".
then as you see return value as Json encode.
in your "View"
<?php
echo Typeahead::widget([
'name' => 'sampleName',
'options' => ['placeholder' => 'Filtering data ...'],
'scrollable' => true,
'pluginOptions' => ['highlight'=>true],
'dataset' => [
[
'remote' => [
'url' => Yii::$app->urlManager->createUrl(['sample/action']) .
'?searchParameter=%QUERY',
'wildcard' => '%QUERY'
],
'datumTokenizer' => "Bloodhound.tokenizers.obj.whitespace('Atribute_name')",
'display' => 'Atribute_name',
'limit' => 10,
],
],
'pluginEvents' => [
'typeahead:select' => 'function(e, s) {
EnableUserDetailsTypeAhead(s);
}',
]
]);
?>
here few things to be consider.
calling to the controller action. you can do that.
Yii::$app->urlManager->createUrl(['sample/action']) .
'?searchParameter=%QUERY',
'wildcard' => '%QUERY'
],
the below lines inside data set must be provide.
'datumTokenizer' => "Bloodhound.tokenizers.obj.whitespace('Atribute_name')",
'display' => 'Atribute_name',
you will get the expected data.
this sample code i have tested and this is working
From the docs:
prefetch: array, configuration for the prefetch options object. Refer documentation for the options you can set for this parameter. The return data must be Json encoded and converted to an associative array of the format [['value' => 'data1'], ['value' => 'data2'],...], where value is the fixed key set in display
You are passing an array of objects instead of an array of key value pairs. You can use asArray to create a list of objects. You will need to change display to the name of the field containing the data:
'prefetch' => Penerima::find()->select('title')->asArray()->all(),

Merge the inner array to the parent array using hash of cakephp

I have the following array
$array['projects'] = [
'name1' => [
'task' => [
'tags' => ['value1', 'value2'],
'email' => 'email2',
'description' => 'mpla'
],
'email' => 'email1',
'tags' => ['value1', 'value3'],
'title' => 'mpla'
]
];
Is there anyway I could use the Hash class of CakePHP 3 or maybe another class of CakePHP framework to achieve the following result:
$array['projects'] = [
'name1' => [
'email' => 'email2',
'tags' => ['value1', 'value2'],
'title' => 'mpla'
'desciption' => 'mpla'
]
];
If you also know anyother package that it can handle arrays and get my job done it will do.
Not sure that this can be easily achieved using Cake's Hash utility. You can easily extract the array items indexed by task using combine(), but not sure how you would then go about extracting the title values and combining those with the other array elements using Hash:-
Hash::combine($array, 'projects.{s}', 'projects.{s}.task');
Perhaps the simplest solution is to use a foreach loop like this:-
$data = [];
foreach ($array['projects'] as $value) {
$data['projects'] = $value['task'] + ['title' => $value['title']];
}

Using elasticsearch, how to create an index for a document that contains an array, and append to that array in the future

In my example code I am using the php client library, but it should be understood by anyone familiar with elasticsearch.
I'm using elasticsearch to create an index where each document contains an array of nGram indexed authors. Initially, the document will have a single author, but as time progresses, more authors will be appended to the array. Ideally, a search could be executed by an author's name, and if any of the authors in the array get matched, the document will be found.
I have been trying to use the documentation here for appending to the array and here for using the array type - but I have not had success getting this working.
First, I want to create an index for documents, with a title, array of authors, and an array of comments.
$client = new Client();
$params = [
'index' => 'document',
'body' => [
'settings' => [
// Simple settings for now, single shard
'number_of_shards' => 1,
'number_of_replicas' => 0,
'analysis' => [
'filter' => [
'shingle' => [
'type' => 'shingle'
]
],
'analyzer' => [
'my_ngram_analyzer' => [
'tokenizer' => 'my_ngram_tokenizer',
'filter' => 'lowercase',
]
],
// Allow searching for partial names with nGram
'tokenizer' => [
'my_ngram_tokenizer' => [
'type' => 'nGram',
'min_gram' => 1,
'max_gram' => 15,
'token_chars' => ['letter', 'digit']
]
]
]
],
'mappings' => [
'_default_' => [
'properties' => [
'document_id' => [
'type' => 'string',
'index' => 'not_analyzed',
],
// The name, email, or other info related to the person
'title' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'authors' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
'comments' => [
'type' => 'list',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
]
],
]
]
];
// Create index `person` with ngram indexing
$client->indices()->create($params);
Off the get go, I can't even create the index due to this error:
{"error":"MapperParsingException[mapping [_default_]]; nested: MapperParsingException[No handler for type [list] declared on field [authors]]; ","status":400}
HAD this gone successfully though, I would plan to create an index, starting with empty arrays for authors and title, something like this:
$client = new Client();
$params = array();
$params['body'] = array('document_id' => 'id_here', 'title' => 'my_title', 'authors' => [], 'comments' => []);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$ret = $client->index($params);
return $ret;
This seems like it should work if I had the desired index to add this structure of information to, but what concerns me would be appending something to the array using update. For example,
$client = new Client();
$params = array();
//$params['body'] = array('person_id' => $person_id, 'emails' => [$email]);
$params['index'] = 'document';
$params['type'] = 'example_type';
$params['id'] = 'id_here';
$params['script'] = 'NO IDEA WHAT THIS SCRIPT SHOULD BE TO APPEND TO THE ARRAY';
$ret = $client->update($params);
return $ret;
}
I am not sure how I would go about actually appending a thing to the array and making sure it's indexed.
Finally, another thing that confuses me is how I could search based on any author in the array. Ideally I could do something like this:
But I'm not 100% whether it will work. Maybe there is something fundemental about elasticsearch that I am not understanding. I am completely new to so any resources that will get me to a point where these little details don't hang me up would be appreciated.
Also, any direct advice on how to use elasticsearch to solve these problems would be appreciated.
Sorry for the big wall of text, to recap, I am looking for advice on how to
Create an index that supports nGram analysis on all elements of an array
Updating that index to append to the array
Searching for the now-updated index.
Thanks for any help
EDIT: thanks to #astax, I am now able to create the index and append to the value as a string. HOWEVER, there are two problems with this:
the array is stored as a string value, so a script like
$params['script'] = 'ctx._source.authors += [\'hello\']';
actually appends a STRING with [] rather than an array containing a value.
the value inputted does not appear to be ngram analyzed, so a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hello';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // SUCCESS
will find the new value but a search like this:
$client = new Client();
$searchParams['index'] = 'document';
$searchParams['type'] = 'example_type';
$searchParams['body']['query']['match']['_all'] = 'hel';
$queryResponse = $client->search($searchParams);
print_r($queryResponse); // NO RESULTS
does not
There is no type "list" in elasticsearch. But you can use "string" field type and store array of values.
....
'comments' => [
'type' => 'string',
'analyzer' => 'my_ngram_analyzer',
'term_vector' => 'yes',
'copy_to' => 'combined'
],
....
And index a document this way:
....
$params['body'] = array(
'document_id' => 'id_here',
'title' => 'my_title',
'authors' => [],
'comments' => ['comment1', 'comment2']);
....
As for the script for apending an element to array, this answer may help you - Elasticsearch upserting and appending to array
However, do you really need to update the document? It might be easier to just reindex it as this is exactly what Elasticsearch does internally. It reads the "_source" property, does the required modification and reindexes it. BTW, this means that "_source" must be enabled and all properties of the document should be included into it.
You also may consider storing comments and authors (as I understand these are authors of comments, not the document authors) as child document in ES and using "has_child" filter.
I can't really give you specific solution, but strongly recommend installing Marvel plugin for ElasticSearch and use its "sense" tool to check how your overall process works step by step.
So check if your tokenizer is properly configured by running tests as described at http://www.elastic.co/guide/en/elasticsearch/reference/1.4/indices-analyze.html.
Then check if your update script is doing what you expect by retrieving the document by running GET /document/example_type/some_existing_id
The authors and comments should be arrays, but not strings.
Finally perform the search:
GET /document/_search
{
'query' : {
'match': { '_all': 'hel' }
}
}
If you're building the query yourself rather than getting it from the user, you may use query_string with placeholders:
GET /document/_search
{
'query' : {
'query_string': {
'fields': '_all',
'query': 'hel*'
}
}
}

Categories