So, basically, i have to loop thought an array of 25000 items, then compare each item with another array's ID and if the ID's from the first array and the second match then create another array of the matched items. That looks something like this.
foreach ($all_games as $game) {
foreach ($user_games_ids as $user_game_id) {
if ($user_game_id == $game["appid"]) {
$game_details['title'][] = $game["title"];
$game_details['price'][] = $game["price"];
$game_details['image'][] = $game["image_url"];
$game_details['appid'][] = $game["appid"];
}
}
}
I tested this loop with only 2500 records from the first array ($all_games) and about 2000 records from the second array ($user_games_ids) and as far as i figured, it takes about 10 seconds for the execution of that chunk of code, only the loops execution. Is that normal? Should that take that long or I'm I approaching the issue from the wrong side? Is there a way to reduce that time? Because when i apply that code to 25000 records that time will significantly increase.
Any help is appreciated,
Thanks.
EDIT: So there is no confusion, I can't use the DB query to improve the performance, although, i have added all 25000 games to the database, i can't do the same for the user games ids. There is no way that i know to get all of the users through that API i'm accessing, and even there is, that would be really a lot of users. I get user games ids on the fly when a user enters it's ID in the form and based on that i use file_get_contents to obtain those ids and then cross reference them with the database that stores all games. Again, that might not be the best way, but only one i could think of at this point.
If you re-index the $game array by appid using array_column(), then you can reduce it to one loop and just check if the data isset...
$game = array_column($game,null,"appid");
foreach ($user_games_ids as $user_game_id) {
if (isset( $game[$user_game_id])) {
$game_details['title'][] = $game[$user_game_id]["title"];
$game_details['price'][] = $game[$user_game_id]["price"];
$game_details['image'][] = $game[$user_game_id]["image_url"];
$game_details['appid'][] = $game[$user_game_id]["appid"];
}
}
Related
I can't figure out how to efficiently get SQL data for a Room/Rates/Dates=Amount...
First I load all the "RateData" for a date range with a PDO select. There are many rows, for many rooms, each with many rates... maybe, or maybe it is all empty except a couple of Amounts. It needs to display $0 for missing dates, so next...
I load the Rooms with PDO and loop through them, and for each room I load the Rates with PDO and loop through them (not a ton of rates per room, and not a ton of rooms, but possibly a very long date-range).
So then I loop through the date range and add $0 to the giant UI grid of Amounts by Rate/Date, nested under each Room. I have to do this anyway, as I also have a ton of logic on what to display in the parent Room row that averages the Rates and such.
So what I need to do is instead of using $0, I need to see if the Room/Rate/Date exists in RateData...
$RateAmount = 0;
$RateDataRow = $RateData.filter('Room=1 && Rate=1 && Date=2022-10-01');
If ($RateDataRow exists) {$RateAmount = $RateDataRow['Amount']}
How to I write the above sudocode in PHP?
The only alternative I can think of would be to do 1000's of SQL calls to populate the grid... which seems bad. Maybe it is not that bad though if PDO caches and doesn't actually query the DB for each grid cell. Please advise. Thanks.
I tried this:
$currcost = 0;
//if $ratedata exists for currdate + currrate + currroom
function ratematch($row)
{if (($row['RoomType_Code']=$currrate)
&& ($row['RatePlan_Code']=$currroom)
&& ($row[ 'Rate_Date']=$currdate->format("Y-m-d")))
{return 1;}
else {return 0;}
}
$match = array_filter($ratedata, 'ratematch');
if (!empty($match)) {$currcost = $match['Rate_Amount'];}
But got an error about redeclaring a function. I have to redeclare it because it is in a loop of currdate under a loop of currrate under a loop of currroom (about 1000 cells).
I made it work, I just have to manually loop through the RateData...
//Get Amount from DB
$currcost = 0;
foreach($ratedata as $datarow)
{if (($datarow['RoomType_Code']==$currroom)
&& ($datarow['RatePlan_Code']==$currrate)
&& ($datarow[ 'Rate_Date']==$currdate->format("Y-m-d")))
{$currcost = $datarow['Rate_Amount'];
break;
}
}
If anyone knows a more efficient way to "query" a previous query without a trip to the SQL server, please post about it. Looping through the fetchAll 1000's times seems bad, but not as bad as doing 1000's of WHERE queries on the SQL Server.
I suppose I could make a 3-dimentional array of $0, then just loop through the RateData once to update that array, and finally loop through the 3-dimentional array to do my other calculations (average rate per room per week sat-sun). Sounds hard though.
Based on what I have read about the internals of php arrays this may not be possible, but I wanted to run this past you and see if there's a chance...
I am pulling two results sets from a SQL query. One is an array of customers (an array of arrays, actually - with each customer having and ID and some other personal data), and the second array is an array of customer orders (also an array of arrays). Basically my foreach loop is matching the customer ID values from the customer array with all of the orders that customer made in the second array and pushing them into a new third data structure.
So let's say I make my SQL query and I pull an array of 500 customers and 3,000 orders. On average then, one customer will have six orders to match. For each iteration through the customer array I need to also iterate through the entire orders array to get all the matches. Then I will push these into a different data structure which is ultimately what I will use.
I wanted to know if unsetting the matched rows from both of the original arrays would speed up the foreach loop since it would in principle have less to iterate over with each cycle. Based on how PHP uses hashes and buckets for its arrays, and the way it makes a copy of arrays in foreach loops, I am wondering if a performance increase would actually occur. I plan to test this with some dummy data but I wanted to know if anyone here has run across a similar situation.
Thanks!!
EDIT - thanks for your answers. Yes, joining the tables is probably the best way, but I am asking this question to gain a better understanding of how PHP handles arrays, etc.
I wrote a test script and I can now see that using unset() doesn't help me:
$customers = Array('1' =>'Jim', '2' => 'Bill', '3' => 'John', '4' => 'Ed', '5' => 'Greg');
$orders = Array();
$final = Array();
for ($x = 0; $x < 1000000; $x++) {
$orders[$x] = rand(1,5);
}
$start = microtime(true);
$counter = 0;
foreach ($customers as $key=>$customer) {
$final[$customer] = Array();
foreach ($orders as $key=>$order) {
$counter++;
if ($order == $key) {
$final[$customer][] = $order;
unset($orders[$key]);
}
}
}
echo $counter; // I usually get the same number of iterations whether unset() is used or not
$finish = microtime(true) - $start; // similar or worse performance if unset() is used
What I see is no performance increase with unset(), but rather a decrease. On top of that, unset() doesn't actually remove the row from the array. When I do a count() of the $orders array it is the same at the end as it is at the beginning.
As per my under standing you are getting results of Customers separately and orders data separately from DB.
Customer
ID Name Add..
1 one ...
2 two ...
3 three ....
Orders
ID CID ord
1 1 ...
2 3 ....
3 3 ....
4 2 ....
While retrieving data itself Join the Tables and get the required results, for example..
SELECT c.Name, c.ID, o.ord FROM customer c
LEFT JOIN orders o ON c.ID = o.CID;
You can add where condition as per requirement;
This is just off the top of my head:
Appending data on to another array using the 'arrayxyz[]' syntax could be faster than 'unset' due to the overhead of logic that needs to run to keep the 'foreach' in a useful state.
Also, unsetting an array item removes the key/value, and if you are using numeric indexes then this leaves a 'hole' which may or may not affect the 'count' function.
You may need to use 'array_values' to reindex the array.
i dont know if i am doing right or wrong, please dont judge me...
what i am trying to do is that if a record belongs to parent then it will have parent id assosiated with it.. let me show you my table schema below.
i have two columns
ItemCategoryID &
ItemParentCategoryID
Let Suppose a record on ItemCategoryID =4 belongs to ItemCategoryID =2 then the column ItemParentCategoryID on ID 4 will have the ID of ItemCategoryID.
I mean a loop with in its own table..
but problem is how to run the select query :P
I mean show all the parents and childs respective to their parents..
This is often a lazy design choise. Ideally you want a table for these relations or/and a set number of depths. If a parent_id's parent can have it's own parent_id, this means a potential infinite depth.
MySQL isn't a big fan of infinite nesting depths. But php don't mind. Either run multiple queryies in a loop such as Nil'z's1, or consider fetching all rows and sorting them out in arrays in php. Last solution is nice if you pretty much always get all rows, thus making MySQL filtering obsolete.
Lastly, consider if you could have a more ideal approach to this in your database structure. Don't be afraid to use more than one table for this.
This can be a strong performance thief in the future. An uncontrollable amount of mysql queries each time the page loads can easily get out of hands.
Try this:
function all_categories(){
$data = array();
$first = $this->db->select('itemParentCategoryId')->group_by('itemParentCategoryId')->get('table')->result_array();
if( isset( $first ) && is_array( $first ) && count( $first ) > 0 ){
foreach( $first as $key => $each ){
$second = $this->db->select('itemCategoryId, categoryName')->where_in('itemParentCategoryId', $each['itemParentCategoryId'])->get('table')->result_array();
$data[$key]['itemParentCategoryId'] = $each['itemParentCategoryId'];
$data[$key]['subs'] = $second;
}
}
print_r( $data );
}
I don't think you want/can to do this in your query since you can nest a long way.
You should make a getChilds function that calls itself when you retrieve a category. This way you can nest more than 2 levels.
function getCategory()
{
// Retrieve the category
// Get childs
$childs = $this->getCategoryByParent($categoryId);
}
function getCategorysByParent($parentId)
{
// Get category
// Get childs again.
}
MySQL does not support recursive queries. It is possible to emulate recursive queries through recursive calls to a stored procedure, but this is hackish and sub-optimal.
There are other ways to organise your data, these structures allow very efficient querying.
This question comes up so often I can't even be bothered to complain about your inability to use Google or SO search, or to offer a wordy explanation.
Here - use this library I made: http://codebyjeff.com/blog/2012/10/nested-data-with-mahana-hierarchy-library so you don't bring down your database
I am wondering if there is a way to do what I am doing more efficiently. Right now, I have a class that retrives statuses from the database. It's pretty simple and shouldn't really effect performance all that much.
public function get ($var1, $var2, $var3)
{
$feed = array(); //Initialize an empty array
//Query the database
$Statement = $this->Database->prepare("SELECT id, name, excerpt, post, timestamp, tags, title FROM posts WHERE col1 = ? AND col2 = ? AND col3 = ? ORDER BY id DESC LIMIT 15");
$Statement->execute(array($var1, $var2, $var3));
while($row = $Statement->fetch(PDO::FETCH_ASSOC))
{
$posts[] = array( "id" => $row["id"], /*etc...*/ );
}
return $posts;
} //end get
And then my page set up something like this which I know is not efficient at all:
<?php for ($count = 1; $count <= $total; $count++): //Display the calendar
echo $count;
$feed = $Feed->get($count, $total, $var3);
foreach ($feed as $post):
echo $post["id"];
endforeach;
endfor; ?>
I hope that makes sense. There's a lot more html thrown in there and everything. Right now there are only 18 rows in my database, and it takes 10 seconds to load the page. Which is really bad. I have to set it up this way because of the design of the site. So the foreach loop has to be within the for loop because the whole thing is set up as a calendar.
My question is whether it would be more efficient to select all of the rows, save them outside of the for loop and then work with that array, or whether it's better to run each query inside the foreach loop the way I'm doing it now (i've read a lot, and know that most people say this is a huge no no). And what kind of issues would I run into if I used the former option and there were say a million rows in the database.
I hope that makes sense. I'll update the question if it doesn't. Right now though about 30 queries are being made to only access 1 or 2 rows. But the only other option I could come up with is selecting all of the rows in the table, and then working with that array, but if there are pretend 1 million rows in the db, I feel like that would affect performance a lot more.
Am I right, and what are some solutions? Thanks
I just want to point out that I did resolve the issue. If anyone is wondering why the foreach loop was querying so sow it was because I accidentally deleted a line where I connected to the Facebook api within the foreach loop every time to gather the poster's information. So if anyone ever stumbles upon this question, just to be sure I want to clarify that making many facebook->api calls is a bad thing. save the info in your database and query that instead.
I have two msyql tables, Badges and Events. I use a join to find all the events and return the badge info for that event (title & description) using the following code:
SELECT COUNT(Badges.badge_ID) AS
badge_count,title,Badges.description
FROM Badges JOIN Events ON
Badges.badge_id=Events.badge_id GROUP
BY title ASC
In addition to the counts, I need to know the value of the event with the most entries. I thought I'd do this in php with the max() function, but I had trouble getting that to work correctly. So, I decided I could get the same result by modifying the above query by using "ORDER BY badgecount DESC LIMIT 1," which returns an array of a single element, whose value is the highest count total of all the events.
While this solution works well for me, I'm curious if it is taking more resources to make 2 calls to the server (b/c I'm now using two queries) instead of working it out in php. If I did do it in php, how could I get the max value of a particular item in an associative array (it would be nice to be able to return the key and the value, if possible)?
EDIT:
OK, it's amazing what a few hours of rest will do for the mind. I opened up my code this morning, and made a simple modification to the code, which worked out for me. I simply created a variable on the count field and, if the new one was greater than the old one, changed it to the new value (see the "if" statement in the following code):
if ( $c > $highestCount ) {
$highestCount = $c; }
This might again lead to a "religious war", but I would go with the two queries version. To me it is cleaner to have data handling in the database as much as possible. In the long run, query caching, etc.. would even out the overhead caused by the extra query.
Anyway, to get the max in PHP, you simply need to iterate over your $results array:
getMax($results) {
if (count($results) == 0) {
return NULL;
}
$max = reset($results);
for($results as $elem) {
if ($max < $elem) { // need to do specific comparison here
$max = $elem;
}
}
return $max;
}