I apologize if this has been asked before, however I'm wondering if anyone has any insight to the performance of having one nested set table for potentially thousands of users?
I need every registered user to be able to create infinite nested categories. Right now, my structure is a belongsToMany() relation:
users -> user_categories (pivot) -> categories
Users would only have access to their categories, and cannot modify other users categories.
Would there be a massive performance hit for (potentially) tens of thousands of records inside one nested set table? Should each user get their own nested set categories table?
Thanks in advance!
Nested sets allow to fetch all descendants of a node in a single query so for reading categories the performance hit will be similar to the one on a normal (non nested set) table. The drawback of nested sets comes when you are inserting because it requires updating left and right values for all records in the table after the insert.
So the performance hit will depend on how often do you insert and how big your insets are.
You can significantly reduce the overhead if, instead of having a single big tree, you have one root node per user, all stored in the same table, and therefore inserts will involve updating only a small subtree.
If you want to test the performance by yourself you can use the Laravel Baum package and use this seeder to see how inserting 26*10*3 categories perform:
<?php
use App\Category;
use Illuminate\Database\Seeder;
class CategoriesTableSeeder extends Seeder
{
public function run()
{
// Helper function to populate model attributes
$node = function () {
$args = implode(' ', func_get_args());
return ['name' => "Category $args"];
};
// Create first level nodes
foreach(range('A', 'Z') as $letter)
{
$node0 = Category::create($node($letter));
// Create second level nodes
foreach(range(1, 10) as $number)
{
$node1 = $node0->children()->create($node($letter, $number));
// Create third level nodes
foreach(['Δ', 'Σ', 'Ω'] as $greek)
{
$node2 = $node1->children()->create($node($letter, $number, $greek));
}
}
}
}
}
Related
I'm trying to figure out how these nestedsets in Laravel works. I've an many to many relation between organizations and departments. An organization can have many departments. An department can have many departments. For this I'm using Nestedsets.
What I'm trying to do, is retrieving all organizations from a user. On this query I'd like to retrieve all departments attached to these organizations. I'd like the structure, so I've a infinite parent -> child relation on my departments, so I'm able to build a structuretree using treant.js.
I'm pretty sure I've everything build correctly in my database, so my first thought were to use with. However it seems like I'm only getting the first children. Here is an example:
$currentUser->organizations()->with(
'departments.children',
'departments.commodities',
'departments.children.commodities',
)->get()
I've to include children.[model] for every nested department. So if I've two levels, I've to add departments.children.children.commodities, and so on. This seems pretty retarded!
I've been trying pretty many different approches to get a proper solution, but the one below is my best solution for now. I just feel like I'm using the nestedset-library wrong.
public function getUserDepartmentTree() {
foreach ( $this->organizations()->get() as $organization ) {
$dep[] = $organization->departments()->get()->toTree();
}
return $dep;
}
So my question is, how should I get all relational data from my departments tree-structured?
For this you'll want to use descendants instead of children as children will only return the direct/first child models whereas descendants will return everything under a certain node.
Since this will add the relationship as descendants instead of children you'll need to tweak it slightly i.e. change the name of the relationship and then use the toTree() method:
$organizations = $currentUser->organizations()
->with('departments.commodities', 'departments.descendants.commodities')
->get()
->map(function ($organization) {
$organization->departments->map(function ($department) {
return $department->setRelation('children', $department->descendants->toTree())->unsetRelation('descendants');
});
return $organization;
});
We have a COMMON database and then tenant databases for each organization that uses our application. We have base values in the COMMON database for some tables e.g.
COMMON.widgets. Then in the tenant databases, IF a table called modified_widgets exists and has values, they are merged with the COMMON.widgets table.
Right now we are doing this in controllers along the lines of:
public function index(Request $request)
{
$widgets = Widget::where('active', '1')->orderBy('name')->get();
if(Schema::connection('tenant')->hasTable('modified_widgets')) {
$modified = ModifiedWidget::where('active', '1')->get();
$merged = $widgets->merge($modified);
$merged = array_values(array_sort($merged, function ($value) {
return $value['name'];
}));
return $merged;
}
return $countries;
}
As you can see, we have model for each table and this works OK. We get the expected results for GET requests like this from controllers, but we'd like to merge at the Laravel MODEL level if possible. That way id's are linked to the correct tables and such when populating forms with these values. The merge means the same id can exist in BOTH tables. We ALWAYS want to act on the merged data if any exists. So it seems like model level is the place for this, but we'll try any suggestions that help meet the need. Hope that all makes sense.
Can anyone help with this or does anyone have any ideas to try? We've played with overriding model constructors and such, but haven't quite been able to figure this out yet. Any thoughts are appreciated and TIA!
If you put this functionality in Widget model you will get 2x times of queries. You need to think about Widget as an instance, what I am trying to say is that current approach does 2 queries minimum and +1 if tenant has modified_widgets table. Now imagine you do this inside a model, each Widget instance will pull in, in a best case scenario its equivalent from different database, so for bunch of Widgets you will do 1 (->all())+n (n = number of ModifiedWidgets) queries - because each Widget instance will pull its own mirror if it exists, no eager load is possible.
You can improve your code with following:
$widgets = Widget::where('active', '1')->orderBy('name')->get();
if(Schema::connection('tenant')->hasTable('modified_widgets')) {
$modified = ModifiedWidget::where('active', '1')->whereIn('id', $widgets->pluck('id'))->get(); // remove whereIn if thats not the case
return $widgets->merge($modified)->unique()->sortBy('name');
}
return $widgets;
OK, here is what we came up with.
We now use a single model and the table names MUST be the same in both databases (setTable does not seem to work even though in exists in the Database/Eloquent/Model base source code - that may be why it's not documented). Anyway = just use a regular model and make sure the tables are identical (or at least the fields you are using are):
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Model;
class Widget extends Model
{
}
Then we have a generic 'merge controller' where the model and optional sort are passed in the request (we hard coded the 'where' and key here, but they could be made dynamic too). NOTE THIS WILL NOT WORK WITH STATIC METHODS THAT CREATE NEW INSTANCES such as $model::all() so you need to use $model->get() in that case:
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Config;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;
class MergeController extends Controller
{
public function index(Request $request)
{
//TODO: add some validations to ensure model is provided
$model = app("App\\Models\\{$request['model']}");
$sort = $request['sort'] ? $request['sort'] : 'id';
$src_collection = $model->where('active', '1')->orderBy('name')->get();
// we setup the tenants connection elsewhere, but use it here
if(Schema::connection('tenant')->hasTable($model->getTable())) {
$model->setConnection('tenant');
$tenant_collection = $model->get()->where('active', '1');
$src_collection = $src_collection->keyBy('id')->merge($tenant_collection->keyBy('id'))->sortBy('name');
}
return $src_collection;
}
}
If you dd($src_collection); before returning it it, you will see the connection is correct for each row (depending on data in the tables). If you update a row:
$test = $src_collection->find(2); // this is a row from the tenant db in our data
$test->name = 'Test';
$test->save();
$test2 = $src_collection->find(1); // this is a row from the tenant db in our data
$test2->name = 'Test2'; // this is a row from the COMMON db in our data
$test2->save();
dd($src_collection);
You will see the correct data is updated no matter which table the row(s) came from.
This results in each tenant being able to optionally override and/or add to base table data without effecting the base table data itself or other tenants while minimizing data duplication thus easing maintenance (obviously the table data and population is managed elsewhere just like any other table). If the tenant has no overrides then the base table data is returned. The merge and custom collection stuff have minimal documentation, so this took some time to figure out. Hope this helps someone else some day!
What I am trying to do
I want to query a specific set of records using active model like so
$jobModel = Jobs::find()->select('JOB_CODE')->distinct()->where(['DEPT_ID'=>$dept_id])->all();
Then I want to assign a flag attribute to the records in this activerecord based on whether they appear in a relationship table
What I have tried
So in my job model, I have declared a new attribute inAccount. Then I added this function in the job model that sets the inAccount flag to -1 or 0 based on whether a record is found in the relationship table with the specified account_id
public function assignInAccount($account_id){
if(JobCodeAccounts::find()->where(['JOB_CODE'=>$this->JOB_CODE])->andWhere(['ACCOUNT_ID'=>$account_id])->one() == null){
$this->inAccount=0;
}
else{
$this->inAccount = -1;
}
}
What I have been doing is assigning each value individually using foreach like so
foreach($jobModel as $job){
$job->assignInAccount($account_id);
}
However, this is obviously very slow because if I have a large number of records in $jobModel, and each one makes a db query in assignInAccount() this could obviously take some time if the db is slow.
What I am looking for
I am wondering if there is a more efficient way to do this, so that I can assign inAccount to all job records at once. I considered using afterFind() but I don't think this would work as I need to specify a specific parameter. I am wondering if there is a way I can pass in an entire model (or at least array of models/model-attributes and then do all the assignations running only a single query.
I should mention that I do need to end up with the original $jobModel activerecord as well
Thanks to scaisEdge's answer I was able to come up with an alternative solution, first finding the array of jobs that need to be flagged like so:
$inAccountJobs = array_column(Yii::$app->db->createCommand('Select * from job_code_accounts where ACCOUNT_ID = :account_id')
->bindValues([':account_id' => $account_id])->queryAll(), 'JOB_CODE');
and then checking each job record to see if it appears in this array like so
foreach($jobModel as $job){
if(in_array($job->JOB_CODE, $inAccountJobs))
$job->inAccount = -1;
else
$job->inAccount = 0;
}
Does seem to be noticeably faster as it requires only a single query.
I'm wondering if Yii has an efficient method for grouping items by type.
Let's say I have the following model:
Tag
------
id
name
type_id
And let's say there are 5 different types of Tags. I want to be able to display in my index all tags in sections by type_id. Is there a Yii-way of accomplishing this?
Outside a framework I would write a function such that results fetched from the DB were stored like this:
$tags[$typeID][] = $tag;
Then in each section I could do something like:
foreach( $tags[$typeID] as $tag )
{
// Here are all tags for one $typeID
}
But I'm having difficulty figuring out how to do this in Yii without:
A) looping through the entire result set first and rewriting it or,
B) running 5 different queries.
When using ActiveRecord simply specify the "index" in the DBCriteria. So in a query do:
ActiveRecordClass::model()->findAll(array('index'=>'type_id'));
That will return an assoc array that your after. TBF it probably executes exactly the same code, but this is obviously easier to use that performing it everywhere.
Assuming that your active record class is called MyActiveRecordClass, the simplest approach should be sufficient:
$models = MyActiveRecordClass::model()->findAll();
$groupedModels = array();
foreach ($models as $model) {
$groupedModels[$model->typeID][] = $model;
}
If you give more specific information about how you intend to display the grouped results it might be that a better approach can be worked out.
I've recently started using Zend Framework (1.8.4), to provide admin tools for viewing the orders of a shopping cart site.
What I'd like to do is to efficiently create multiple model (Zend_Db_Table_Row_Abstract) objects from a single database result row.
The relationship is simple:
an Order has one Customer (foreign key order_custid=customer.cust_id);
a Customer has many Orders.
Loading the orders is easy enough. Using the method documented here:
Modeling objects with multiple table relationships in Zend Framework
...I could then grab the customer for each.
foreach ($orderList as $o)
{
cust = $o->findParentRow('Customers');
print_r ($cust); // works as expected.
}
But when you're loading a long list of orders - say, 40 or more, a pageful - this is painfully slow.
Next I tried a JOIN:
$custTable = new Customers();
$orderTable = new Orders();
$orderQuery = $orderTable->select()
->setIntegrityCheck(false) // allows joins
->from($orderTable)
->join('customers', 'cust_id=order_custid')
->where("order_status=?", 1); //incoming orders only.
$orders = $orderTable->fetchAll($orderQuery);
This gives me an array of order objects. print_r($orders) shows that each of them contains the column list I expect, in a protected member, with raw field names order_* and cust_*.
But how to create a Customer object from the cust_* fields that I find in each of those Order objects?
foreach ($orders as $o) {
$cols = $o->toArray();
print_r ($cols); // looks good, has cust_* fields...
$cust = new Customer(array( 'table' => 'Customer', 'data' => $cols ) );
// fails - $cust->id, $cust->firstname, etc are empty
$cust->setFromArray($cols);
// complains about unknown 'order_' fields.
}
Is there any good way to create an Order and a Customer object simultaneously from the joined rows? Or must I run the query without the table gateway, get a raw result set, and copy each of the fields one-by-one into newly created objects?
Zend_Db doesn't provide convenience methods to do this.
Hypothetically, it'd be nifty to use a Facade pattern for rows that derive from multiple tables. The facade class would keep track of which columns belong to each respective table. When you set an individual field or a whole bunch of fields with the setFromArray() method, the facade would know how to map fields to the Row objects for each table, and apply UPDATE statements to the table(s) affected.
Alternatively, you could work around the problem of unknown fields by subclassing Zend_Db_Table_Row_Abstract, changing the __set() behavior to silently ignore unknown columns instead of throwing an exception.
You can't have an OO interface to do everything SQL can do. There must be some line in the sand where you decide a reasonable set of common cases have been covered, and anything more complex should be done with SQL.
I use this method to assign database row fields to objects. I use setter methods, but this could probably be also done with only properties on object.
public function setOptions(array $options){
$methods = get_class_methods($this);
foreach ($options as $key => $value) {
$method = 'set' . ucfirst($key);
if (in_array($method, $methods)) {
$this->$method($value);
}
}
return $this;
}