I am having trouble bulk inserting documents with the PHP SDK. I have to insert each one individually like
$temp = $db->upsert("key here", "value here");
I can achieve about 500 / second. I have attempted to use the cbcdocloader but I cant generate json faster than just using the SDK to insert them directly. I am trying to insert multiple documents with one insert so I dont have to generate all the traffic. Is there a way to do this with an array structure like below?
$data = array("key 1"=> "value 1", "key 2" => "value 2")
I just did not believe it as I stumbled upon it somewhere a time ago, when I was searching for something else... But now I tried it, and it seems to work and I got a great performance increase (when doing a lot of inserts). It seems, you have to do it that way (using SDK 2.0 and upsert()):
$bucket->upsert(array(
'key1' => array('value' => 'value1'),
'key2' => array('value' => array('jsonkey' => 'jsonvalue')));
You have to use 'value' literally in the second array.
It seems, you also can specify options using:
$bucket->upsert(array(...), null, array('expiry' => 3600));
Maybe someone can verify this...
There is no bulk upload api call as such.
There is a semo of inserting in bulk http://docs.couchbase.com/couchbase-devguide-2.1/#performing-a-bulk-set.
Even, this does what you are already doing, in loops.
Related
I am pretty sure this challenge has been solved by someone already but even searching with different words, I could not find a solution for this problem:
I try to give users the possibility to run certain functions of a class based on an argument like
service_class::do_this( "selection-argument" );
but the user shall be able to use "clear words" as well as "aliases" and even "well known" abbreviations or synonyms.
I use switch-case construction to call the "real" function.
Example: To get the contens of a folder, The user can use "getdir", "dir", "Directory", "getfolder", "getcontent", "content", "d-cont" and a number of more other "matching words" to start the function(s) underlaying and getting back the very same result.
Capture-ing lowercase/uppercase is simple. What I search for is an efficient way to capture all possible "variations" - that are, of course different number of variations for different functions called.
At the moment I use multiple "case "": lines after each other, but that makes the code quite long, and further I would like the user to be able to "enahnce" the recognition set for a certain function.
That's why I thought about "stripos" to determine first what "internal word" to use and only then run into the switch-case construction.
Anyone had that issue and can direct me to a "good and efficient" solution?
Seems that Stck-exchange itself had a similar challenge (https://codereview.stackexchange.com/tags/php/synonyms) ... maybe I can simply re-use the underlying code?
Thanks in advance and sorry if I overlooked a solution already posted.
You could use a database or array. Let's do the latter. So to determine whether an user wants to get a directory you would define an array like this:
$getDirVariants = ['getdir',
'dir',
'directory',
'getfolder',
'getcontent',
'content',
'd-cont'];
It is easy to add more of these arrays. To test the query word you would do:
$queryWord = strtolower($queryWord);
if (in_array($queryWord, $getDirVariants)) service_class::getDir(<arguments>);
elseif (in_array($queryWord, $deleteVariants)) service_class::delete(<arguments>);
You can easily add to the arrays or make it a 2D array to contain more commands. That array could also be placed in a database.
Especially when there are many commands, with many variants, a database will be the better solution, because you can find the query word with one database query.
There's a variation I can think of that will also simplify the code when there are many commands. You could use an associative array to find the command:
$commandVariants = ['getdir' => 'getdir',
'dir' => 'getdir',
'directory' => 'getdir',
'getfolder' => 'getdir',
'getcontent' => 'getdir',
'content' => 'getdir',
'd-cont' => 'getdir',
'delete' => 'delete',
'del' => 'delete',
'remove' => 'delete',
'unlink' => 'delete'];
$queryWord = strtolower($queryWord);
if (isset($commandVariants[$queryWord])) {
$command = $commandVariants[$queryWord];
service_class::$command(<arguments>);
}
else echo "I don't recognize that command.";
This uses a variable identifier.
I have a php backend using phpredis (a php client for the redis server) to store key value pairs to a Redis server. The data I need to store is of this form:
"key1" => "v1", "v2", "v3"
"key2" => "m1", "m2", "m3"
"key3" => "n1", "n2", "n3"
...
Based on my research, I can set multiple keys in a redis using the mset command like so:
$redis->mSet(array('key0' => 'value0', 'key1' => 'value1'));
But what I actually need is something like this:
$redis->mSet(array('key0' => array('v1','v2','v3') , 'key1' => array('m1', 'm2', 'm3')));
But this just stores the value for each key as "Array" instead of the actual array specified.
Is this possible to do with a single command like mset or do I need to iterate my data and set each key separately using something like lPush?
phpredis documentation: https://github.com/phpredis/phpredis
So rather than using mSet you can probably use sADD to get your desired functionality.
$redis->sAdd($key, ...$data);
Full documentation on it here.
This would mean iterating and doing it in multiple steps for which I'd reccomend reading into Redis Pipelines and the non-shameless plug link which contains more information.
Which would look something like;
$redis = new Redis();
$pipeline = $redis->multi(Redis::PIPELINE);
foreach ($dataset as $data) {
$pipeline->sAdd($data['key'], ...$data['values']);
}
$pipeline->exec();
I can't think off the top of my head a way to do this in a singular operation, someone else might come along though who knows more than me :)
Edit: Looks like I misunderstood your question a little as it was more focused on doing this in a single operation. Hopefully the above is still useful but to my knowledge you'll have to do this with multiple.
I have an 2D array with a few sub-arrays (about 30, and sub-arrays have 10 elements).
I need to get quite frequently basic data from the array , I have a function that return the contents of it (or partial) all around my scripts. The function looks like:
function get_my_data($index = false){
$sub0 = array(
'something' => 'something',
'something else' => 'else',
...
);
$sub1 = array(
'something' => 'something different',
'something else' => 'else different',
...
);
...
$sub30 = array(
'something' => 'something 30 times different',
'something else' => 'else 30 times different',
...
);
$data = array($sub0,$sub1,$sub2,...,$sub30);
if($index !== false)
return $data[$index];
else
return $data;
?>
And then I call to it using include:
<?php
include 'my_data.php';
$id = $_GET['id'];
$mydata = get_my_data($id);
...
?>
I've done this because when I was starting this project, I didn't imagined I would have more that 10 sub-arrays, and I neither that I would need to have it dynamic. In fact, now I have to add a dynamic column (an index to sub-arrays) and it is not a great idea to use array declaration in this case. I immediately thought to use database, transferring data would not difficult, but if I do that, then I need to change my function get_my_data and insert a query in it, so, for it's called many times, I would have a lot of queries, pretty much every script of my website have one of it. I think performance would be worst (cause mysql is already largely used in my website). The dynamic data would change not too frequently (client do that).
The ideas I have to solve this problem are:
save all data in database and get it through mysql queries,
leave on php side and use files to manage dynamic data,
leave the static part on php side, add a logical connector (such 'id' index in sub-arrays) and id column in mysql database, and get the dynamic data on mysql
I don't want to lose much performance, do yo have any advice or suggestions?
Putting data like this in code is the worst possible plan. Not only do you create a whole bunch of junk and then throw out almost all of it, but if any of this changes it's a nightmare to maintain. Editing source code, checking it into version control, and deploying it is a lot of work to make a simple change to some data.
At the very least store this in a data format like JSON, YAML or XML so you can read it in on-demand and change a data-only file as necessary.
Ideally you put this in a database and query against it when necessary. Databases are designed to store, update, and preserve data like this.
You can also store JSON in the database, MySQL 5.7 even has a native column type for it, which makes this sort of thing even easier.
Not sure about how to title this post.
I just inherited a set of internationalization documents, each containing an array of key value pairs. I recognize the methodology is not ideal, it is however, what I'm stuck with given my role, the time and resources I have available. These lists were created by hand and items were constantly being added haphazardly to keep up with demand. The examples below are simplified, there are ~21 language files, each with 100+ entries.
The array from one file will have elements something like the below:
'download_high' => 'Vysoké',
'download_low' => 'Nízké',
'download_med' => 'Strední',
'download_video' => 'Stáhnout video',
While another file will have something like the following:
'download_video' => 'Descargar Video',
'download_high' => 'Alta',
'download_med' => 'Media',
'download_low' => 'Baja',
For the most part, the elements are ordered consistently, but there are plenty of exceptions and it complicates maintaining the files. I was wondering if there's a way to make the order / formatting of these arrays consistent either via an IDE or notepad++ plugin, etc. I don't want to order them alphabetically necessarily as there are logical groupings that won't translate well to alphabetical sorting since only some of the variables are namespaced. Doing it by hand is almost out of the question, but it would be nice if there were some way to specify an order and have some kind of text manipulation tool batch process the files. I'm not all that familiar with php, so I don't know that it would be worth it for me to do this via code unless it's very simple.
PHP has a function called var_export. You could run the code and then print it with the function.
Personally, I would run some regex in notepad++ to do it.
Edit: In notepad++, you can do a find/replace with regex.
In the "Find what" field, put \s*'(.+?)'\s*=>\s*'(.+?)'\s*(?:(,)|\s*$)\s*
In the "Replace with" field, put '$1' => '$2'$3\n
This will turn something like:
'download_high' => 'Vysoké',
'download_low'=>
'Nízké',
'download_med'
=>
'Strední'
,
'download_video' => 'Stáhnout video',
'filter_by' => 'Filtrovat podle'
,
'footer_contact' => 'Kontakt'
into
'download_high' => 'Vysoké',
'download_low' => 'Nízké',
'download_med' => 'Strední',
'download_video' => 'Stáhnout video',
'filter_by' => 'Filtrovat podle',
'footer_contact' => 'Kontakt'
*note: This is written with the assumption that all keys and values use single quotes and that neither keys nor values have any escaped single quotes inside.
Given the situation I'd do it semi-automatically on an as-needed basis. That is, it sounds to me like you're charged with maintaining these files, and it's in the maintenance that the sorting becomes an issue.
For each language file formatted like:
<?php
$something = array(
'download_video' => 'Descargar Video',
'download_high' => 'Alta',
...
'download_med' => 'Media',
'download_low' => 'Baja',
);
Issue a sort lang.es.php command in bash, or via however you prefer to sort something.
Open the file for editing.
Delete the array declaration.
Copy/paste the sorted lines from #1.
Honestly, it was a super bad idea for your predecessor to effectively hard-code something like this. It would be much better if this were stored via CSV, XML, pg/mySQL, etc where you could at least invoke an editor that understands the data format.
i wouldn't suggest using arrays and php files etc if you are going to have a continuously growing list. take the couple minutes now to set up mysql and pull from a database with your desired parameters. NEAT, FAST, EFFECIENT AND SCALABLE.
My suggestion would be to simply build a script to load all those arrays into a database where they can be maintained more easily. Of course you would need to make sure you database table/fields were set up for UTF-8 character sets and collations.
Am I correct in understanding that you have already gotten your data into PHP arrays? At this point you could just do this to sort the keys. This would make the ordering consistent:
ksort($array);
I'm implementing a very simple conversation system on mongodb.
The idea should be that when I'm opening a convo, it should display send and received messages. It's OK so far and should be pretty easy, by using a simple query like this pseudocode:
(from "my_id" AND to "friend_id") OR (from "friend_id" AND to "my_id")
this should be pretty straightforward and simple, but querying just looks so complicated to me with mongodb (I'm coming from mysql).
I'm trying this, but it's not working at all, and can't find out where the error is.
$cursor =$collection->find
(
array('$or' =>
array('$and' => array("from"=>"$profile", "to"=>"$loggeduser")),
array('$and' => array("to"=>"$profile", "from"=>"$loggeduser"))
)
)->limit(50)->sort(array('date' => -1));
this returns nothing.... Where's the mistake?
Thanks in advance.
Take a look at this page on how to do advanced MongoDB queries: http://www.mongodb.org/display/DOCS/Advanced+Queries
You can use a combination of the $and and $in operators to get what you need. Using the mongo shell, your query would look something like this:
db.yourCollectionName.find({$and: {from: {$in: ["toUser", "loggedOnUser"]}}, {to: {$in: ["toUser", "loggedOnUser"]}}})
I believe this may also give you the equivalent:
db.yourCollectionName.find({$and: {$or: [{from: "toUser"}, {to: "toUser"}]}}, {$or: [{from: "loggedOnUser"}, {to: "loggedOnUser"}]}}})
From there it's a matter of converting the above to the language/DSL that you're using, and sorting by date.
In your code, you don't need the ($and => array()) wrapping each of the objects that you're trying to find. Remove them, so it looks like this:
$cursor = $collection->find(
array('$or' =>
array(
array("from"=>"$profile", "to"=>"$loggeduser"),
array("to"=>"$profile", "from"=>"$loggeduser")
)
)
) ->limit(50)->sort(array('date' => -1));