I got an array of symfony3 entities which is 50000 in total. I need to iterate this array and find some entities that match certain criteria.
code is shown below:
$p_r = $em->getRepository('AppBundle:Product_region')->findall();//50000 elements in array
$rowid_2 = $product->getRowId();//some product entity
foreach($p_r as $pr){
$rowid_1 = $pr->getProductid()->getRowId();
if($rowid_1 == $rowid_2){
$regions[] = $pr->getRegionid()->getName();
$filial_name = $pr->getRegionid()->getFilial()->getName();
if(!in_array($filial_name, $filials)){
$filials[] = $filial_name;
}
}
}
The problem is when iteration takes approx 1 second which is very long.it takes about 25% of a CPU 1 core to finish the operation. I tested it on a different array of numbers and observed it took about 0.002 sec to iterate through 50 000 elements. Any ideas how to speed it up and where to search for a delay would be welcome. Thank you.
If You want to find some entities that match certain criteria You could use Doctrine's 2 Criteria to fetch those entities.
I found it really usefull. Here you have link to Doctrine's docs:
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/working-with-associations.html#filtering-collections
Hope this helps. :)
You have two options. Create a Region/Product Symfony Repository and craft a finder function using QueryBuilder/DQL that you can specify the ID you're searching for OR use the ArrayCollection functions and Criteria to filter your results.
use Doctrine\Common\Collections\Criteria;
$criteria = Criteria::create()->where(Criteria::expr()->eq('product_id', $product_id))->setFirstResult(0);
$items = $p_r->matching($criteria);
Iterating (foreach) over that array of objects will trigger lazy loading which will hog CPU/Memory. Check out the code for Doctrine ArrayCollections and Criteria
Related
I queried to get info from a table with a manytomany relationship like this
$userList = UserListing::where('user_id', $user->id)->with("objects")->paginate(10);
Now, i want to limit the amount of results in the "Objects" table, but at the same time i want to know how many objects are in total.
$userList = UserListing::where('user_id', $user->id)->with(["objects"=> function($query) {
$query->take(2);
}])->paginate(10);
But by doing this, i can't get the total of objects since i limited it to 2, then i tried to process the info like this
$userList = UserListing::where('user_id', $user->id)->with("objects")->paginate(10);
foreach ($userList as $key => $value) {
$l = count($value["objects"]);
$value["objects"] = $value["objects"]->take(2);
$value["number_objects"] = $l;
}
But apparently this did not replace the collection value["objects"], since it still returned 3 objects, despite supposedly being reduced with $value["objects"] = $value["objects"]->take(2);. How can i replace the collection with the reduced one?
So, i kept investigating, and noted that userList was a LengthAwarePaginator object, which by property apparently is inmutable in its original fields(Meaning you can add new ones, but not delete/modify the already existent). Knowing this, i searched a little more and found this answer:
https://stackoverflow.com/a/49133519/7228093
Which basically creates a new LenghtAwarePaginator from the original one, allowing you to modify the fields. If someone finds in this situation, this may be a good option(The transform method of collections did not work by the way, only this one).
I'm building timeline for my app and got this issue, that my collection returns me only 11 values.
I have tried to change merged data and it always returns me 11 values that makes me very confused, bc their is no limits in my code.
public function timeline($company_id)
{
// USER COMPANY
$company = auth()->user()->companies()->findOrFail($company_id);
//GETTING DATA TO MERGE COLLECTION
$equities = Share::where('company_id', $company_id)->get();
$equityGrants = EquityGrant::whereIn('share_id', $equities->pluck('id')->toArray())->get();
$warrants = Warrant::where('company_id', $company_id)->orderBy('created_at', 'DESC')->get();
$warrantGtrants = WarrantGrant::whereIn('warrant_id', $warrants->pluck('id')->toArray())->get();
$convertibles = Convertible::where('company_id', $company_id)->get();
// CREATING COLLECTION
$operations = $equities->merge($equityGrants)->merge($warrants)->merge($warrantGtrants)->merge($convertibles);
$operations = $operations->sortByDesc('created_at');
// RETURNS ME ONLY 11 VALUES
return $operations->values()->all();
}
I tried to merge() less instances, like $operations = $equities->merge($equityGrants)->merge($warrants)->merge($warrantGtrants) but always maximum 11 values. I need to return all data for my timeline.
HELP ME Please! :)
Cheers, love :)
SOLUTION:
function concat() instead of merge() fixed the problem.
Using merge() on collections in Laravel will cause overiding all elements with the same ids comming from Eloquent query.
I believe that is why you are getting only 11 elements because this is the count off all elements having distinct ids.
According to the documentation of merge, when a key matches, it will be overwritten by the last item with that key (the collection you are merging in).
A solution to your problem would be the keyBy method (documentation). If you use keyBy('created_at'), it will give you an array where the keys are the created_at timestamps.
Although it sounds like they would be unique, there is a great change that some of the linked resources (like warrant and warrentGrants) are created at the exact same moment. If that is the case, you should try to find another key or something that is unique across all resources.
SOLUTION: function concat() instead of merge() fixed the problem.
I am having an array that I make a collection out of it with CakePHP 3 then I use match to extract a new collection containing only the elements that id=2.
What I am unable to understand is after I use match if I use each to iterate with my original collection i see the element with id=2. \
Shouldn't it be removed from the original collection ?
How can I iterate my new collection cause each, compile, foreach are not working, and when I use debug all I get is
\src\Controller\ComlibsController.php (line 51)
object(Cake\Collection\Collection) {
'count' => (int) 0
}
The collection code id :
//get the current answer and remove it from the query array
$mycollection = new Collection($query[0]['answers']);
$answer = $mycollection->match(['answers.id' => $theid]);
Cookbook is not as simple as they claimed.
Any sort of help would be appreciated.
Performing a match on a collection returns a new collection with the elements that match, but it doesn't alter the original collection in any way.
When you're creating your collection, you're specifying the ['answers'] key, which means that won't be part of the path any more for elements in the collection. You will therefore simply want to do ->match(['id' => $theid]) to find the matches.
I am using laravel framework and I need to get 2 arrays, one with premium themes and one with free themes.. so I would do:
$premium_themes = \App\Theme::where('premium', '=', '1')->get();
$free_themes = \App\Theme::where('premium', '=', '0')->get();
This will work Ok, but will perform two queries on the database. Since I'm an optimization geek, I think it might be better to have a single query... which I would get all themes by using:
$themes = \App\Theme::all();
And then I'd to process this in php to split based on the theme premium property.
So I have 2 questions:
1) A single query is better than 2 queries in this case, or am I over-thinking this?
2) Is there a fast simple way to split the resulting collection into two collections based on the premium property? (I know Laravel has many shortcuts but I'm still new to the framework)
Single query would be better as both of the queries will go over all the rows in the database. Except the 2 queries to split them will go over them for a second time.
You can simply filter them like so;
The simple one line solution $themes = \App\Theme::all()->groupBy('premium');.
Or into separate collections if you need to filter by another element etc just add more to the following;
$themes = \App\Theme::all();
$premium = new Collection;
$free = new Collection;
$themes->each(function ($item) use ($premium, $free){
if($item->premium == '1'){
$premium->push($item);
}
else {
$free->push($item);
}
});
And your items will be filtered into the relevant Collection. Be sure you use the Collection class at the top.
The only reason I can think to keep it as separate queries would be if you need to paginate the data - not something you can do easily if its all mixed together.
A "short cut" would be to use the collection filter() method, I put short cut in quotes because it's not short per-se, more syntatic sugar - but Larvel is nothing if not full of sugar so why not?
Code would look something like this:
$allThemes = \App\Theme::all();
$premiumThemes = $allThemes->filter(function($theme)
{
return $theme->premium;
});
$freeThemes = $allThemes->filter(function($theme)
{
return !$theme->premium;
});
Edit: I'd recommend using Matt Burrow's answer, but I'll leave mine here as the solution is different.
I was hoping for an example on how to bulk insert new "beans" in readbeanphp without looping over each instance.
It shows an example creating and saving a beans here: http://redbeanphp.com/manual/create_a_bean
It makes mention of storeAll($beans) method, but I am unsure exactly how I am suppose to format the data in $beans.
I have tried googling for this and can not find anything related to bulk inserts. Maybe I have searched for the wrong terms.
I am new to this ORM, any help with would appreciated, thanks!
You are definitely right on track. Create a new bean using $bean=R::dispense('bean'); or multiple beans as an array $beans=R::dispense('bean',5);
Then you populate the beans with data:
$bean->title='Hello World!';
//or with an array
$beans[0]->title='Hello World!';
$beans[1]->title='Hello World! Bean 1';
//etc
Then store the bean(s):
R::store($bean);
//or
R::storeAll($beans);
All the beans must be the same type if you have multiples as far as I know, so you can do something like:
$beans=array();
$beans[]=R::dispense('bean');
$beans[]=R::dispense('bean');
$beans[0]->title='Hello World!';
$beans[1]->title='Hello World!1';
R::storeAll($beans);
I could be wrong about that though. The main thing is that this is all a typical ORM, but redbean also supports regular SQL if you need to use it. Hope that helps!
Some real data behind this approach.
FIRST APPROACH.
foreach item found
$bean = R::dispense('bean');
$bean->title = "hello";
R::store("bean");
time taken for 5660 rows = 43s on my mac
SECOND APPROACH.
$beans=array();
$beans[]=R::dispense('bean');
$beans[]=R::dispense('bean');
$beans[0]->title='Hello World!';
$beans[1]->title='Hello World!1';
R::storeAll($beans);
For 5660 rows, 46s. The storeAll is where all the time is. So its taking ages to store these beans.
THIRD APPROACH
$beans=R::dispense('bean',5560);
for loop
$bean[$i]->title = "hello world";
end for
R::storeAll($beans);
For 5660 rows 45s.
Result. None of these approaches are any quicker. : (
RedBean Transactions didn't seem to make this any quicker either
From the creator of RedBean https://stackoverflow.com/a/18811996/445492 Bulk Insert is not supported, use pure sql.
FOURTH APPROACH
for loop
R::exec("insert into bean(title) values (1,'hello world')");
end for
for 5660 rows 7.3s <----- WOW
(please note: I am actually doing some stuff prior so all these results are -4.3 seconds.)
Hence every bean needs to be created first and the method to create a bean is dispense
$bean = R::dispense('customers');
$bean->name = "John";
R::store($bean);
$bean->name = "Walter"
R::store($bean);
the code above creates only one bean even after storing it. Still $bean refers to the same object, so for each record you have to create a new been by using dispense method.
Luckily we have storeAll method that stores all the beans but it requires an array of beans. So we create a bean in each iteration and push it to the array and then at the end of loop we just pass that array to storeAll function.
//create empty array
$beans = array();
//for each customer post create a new bean as a row/record
foreach ($post as $customer) {
$bean = R::dispense('customers');
//assign column values
$bean->firstName = $customer['first_name'];
$bean->lastName = $customer['last_name'];
//push row to array
$beans[] = $bean;
}
//store the whole array of beans at once
R::storeAll($beans);
In the approaches 1, 2 and 3 suggested by John Ballinger, one way to optimize the run time is to put all the insertions performed by storeAll($beans) inside one database transaction. This could be done as follows: replace the line "R::storeAll($beans)" by the following three lines:
R::begin();
R::storeAll($beans);
R::commit();
This approach reduces dramatically the run time when the array $beans is large, AND is not necessary to use SQL "explicitly".