I am currently working on a project that has a very large database of users and their different levels, however we where using RAND() and we are starting to get very high MySQL CPU usage warnings from our web host due to the growing demand of the database.
So we need to optimize this one particular query to help reduce the mysqli calls …
The row LINK where most of the data is stored is close to 50,000 + entries and using RAND() is slowly killing the website.
How would we optimize this to reduce the number of mysqli calls..
$CALLROW = $db->CacheGetAll("SELECT `ID`, `TITLE` FROM `{$tables['link']['name']}` WHERE (`FEATURED` = '1' OR `FEATURED_ADVANCED`= '1' OR `FEATURED_NORMAL`= '1') AND `STATUS` = '2' {$expire_where} ORDER BY RAND() LIMIT 20");
$tpl->assign('CALLROW', $CALLROW);
How can we reduce the memory imprint on the above SQL statement?
Well after trial an error I came up with this
$CALLROW = $db->CacheGetAll("SELECT * FROM `{$tables['link']['name']}` WHERE `FEATURED` = '1' OR `FEATURED_ADVANCED`= '1' OR `FEATURED_NORMAL`= '1' AND `STATUS` = '2' AND ID >= RAND() * (SELECT MAX(ID) FROM `{$tables['link']['name']}`) LIMIT 15");
$tpl->assign('CALLROW', $CALLROW);
However after I refresh it pulls some different ones but never the FEATURED_ADVANCED or FEATURED_NORMAL it seems to only pull the FEATURED IDs first
Any idea of how to truly shuffle these
I have a database that is already in use and I have to improve the performance of the system that's using this database.
There are 2 major queries running about 1000 times in a loop and this queries have inner joins to 3 other tables each. This in turn is making the system very slow.
I tried actually to remove the query from the loop and fetch all the data only once and process it in PHP. But this is putting to much load on the memory (RAM) and the system is hanging if 2 or more clients try to use the system.
There is a lot of data in the tables even after removing the expired data .
I have attached the query below.
Can anyone help me with this issue ?
select * from inventory
where (region_id = 38 or region_id = -1)
and (tour_opp_id = 410 or tour_opp_id = -1)
and room_plan_id = 141 and meal_plan_id = 1 and bed_type_id = 1 and hotel_id = 1059
and FIND_IN_SET(supplier_code, 'QOA,QTE,QM,TEST,TEST1,MQE1,MQE3,PERR,QKT')
and ( ('2014-11-14' between from_date and to_date) )
order by hotel_id desc ,supplier_code desc, region_id desc,tour_opp_id desc,inventory.inventory_id desc
SELECT * ,pinfo.fri as pi_day_fri,pinfoadd.fri as pa_day_fri,pinfochld.fri as pc_day_fri
FROM `profit_markup`
inner join profit_markup_info as pinfo on pinfo.profit_id = profit_markup.profit_markup_id
inner join profit_markup_add_info as pinfoadd on pinfoadd.profit_id = profit_markup.profit_markup_id
inner join profit_markup_child_info as pinfochld on pinfochld.profit_id = profit_markup.profit_markup_id
where profit_markup.hotel_id = 1059 and (`booking_channel` = 1 or `booking_channel` = 2)
and (`rate_region` = -1 or `rate_region` = 128)
and ( ( period_from <= '2014-11-14' and period_to >= '2014-11-14' ) )
ORDER BY profit_markup.hotel_id DESC,supplier_code desc, rate_region desc,operators_list desc, profit_markup_id DESC
Since we have not seen your SHOW CREATE TABLES; and EXPLAIN EXTENDED plan it is hard to give you 1 answer
But generally speaking in regard to your query "BTW I re-wrote below"
SELECT
hotel_id, supplier_code, region_id, tour_opp_id, inventory_id
FROM
inventory
WHERE
region_id IN (38, -1)
AND tour_opp_id IN (410, -1)
AND room_plan_id IN (141, 1)
AND bed_type_id IN (1, 1059)
AND supplier_code IN ('QOA', 'QTE', 'QM', 'TEST', 'TEST1', 'MQE1', 'MQE3', 'PERR', 'QKT')
AND ('2014-11-14' BETWEEN from_date AND to_date )
ORDER BY
hotel_id DESC, supplier_code DESC, region_id DESC, tour_opp_id DESC, inventory_id DESC
Do not use * to get all the columns. You should list the column that you really need. Using * is just a lazy way of writing a query. limiting the columns will limit the data size that is being selected.
How often is the records in the inventory are being updates/inserted/delete? If not too often then you can use consider using SQL_CACHE. However, caching a query will cause you problems if you use it and the inventory table is updated very often. In addition, to use query cache you must check the value of query_cache_type on your server. SHOW GLOBAL VARIABLES LIKE 'query_cache_type';. If this is set to "0" then the cache feature is disabled and SQL_CACHE will be ignored. If it is set to 1 then the server will cache all queries unless you tell it not too using NO_SQL_CACHE. If the option is set to 2 then MySQL will cache the query only where SQL_CACHE clause is used. here is documentation about query_cache_type
If you have an index on those following column in this order it will help you (hotel_id, supplier_code, region_id, tour_opp_id, inventory_id)
ALTER TABLE inventory
ADD INDEX (hotel_id, supplier_code, region_id, tour_opp_id, inventory_id);
If possible increase sort_buffer_size on your server as most likely you issue here is that your are doing too much sorting.
As for the second query "BTW I re-wrote below"
SELECT
*, pinfo.fri as pi_day_fri,
pinfoadd.fri as pa_day_fri,
pinfochld.fri as pc_day_fri
FROM
profit_markup
INNER JOIN
profit_markup_info AS pinfo ON pinfo.profit_id = profit_markup.profit_markup_id
INNER JOIN
profit_markup_add_info AS pinfoadd ON pinfoadd.profit_id = profit_markup.profit_markup_id
INNER JOIN
profit_markup_child_info AS pinfochld ON pinfochld.profit_id = profit_markup.profit_markup_id
WHERE
profit_markup.hotel_id = 1059
AND booking_channel IN (1, 2)
AND rate_region IN (-1, 128)
AND period_from <= '2014-11-14'
AND period_to >= '2014-11-14'
ORDER BY
profit_markup.hotel_id DESC, supplier_code DESC, rate_region DESC,
operators_list DESC, profit_markup_id DESC
Again eliminate the use of * from your query
Make sure that the following columns have the same type/collation and same size. pinfo.profit_id, profit_markup.profit_markup_id, pinfoadd.profit_id, pinfochld.profit_id and each one have to have an index on every table. If the columns have different types then MySQL will have to convert the data every time to join the records. Even if you have index it will be slower. Also, if those column are characters type (ie. VARCHAR()) make sure they are of the CHAR() with a collation of latin1_general_ci as this will be faster for finding ID, but if you are using INT() even better.
Use the 3rd and 4th trick I listed for the previous query
Try using STRAIGHT_JOIN "you must know what your doing here or it will bite you!" Here is a good thread about this When to use STRAIGHT_JOIN with MySQL
I hope this helps.
For the first query, I am not sure if you can do much (assuming you have already indexed the fields you are ordering by) apart from replacing the * with column names (Don't expect this to increase the performance drastically).
For the second query, before you go through the loop and put in selection arguments, you could create a view with all the tables joined and ordered then make a prepared statement to select from the view and bind arguments in the loop.
Also, if your php server and the database server are in two different places, it is better if you did the selection through a stored procedure in the database.
(If nothing works out, then memcache is the way to go... Although I have personally never done this)
Here you have increase query performance not an database performance.
For both queries first check index is available on WHERE and ON(Join) clause columns, if index is missing then you have to add index to improve query performance.
Check explain plane before create index.
If possible show me the explain plane of both query that will help us.
I am making a feed system similar to the way facebook works now what I am trying to implement is the duplicate functionality.
if a user writes the same post= then don't display it duplicatly as PHP normally does by default but bump it up the top so users knows another user as re-wrote it
Here's my query
$select_posts_from_groups_query = $db->query("SELECT * FROM spud_groups_posts LEFT JOIN spud_groups_members
ON spud_groups_posts.group_url = spud_groups_members.gurl WHERE member_name='$mybb_username' GROUP BY post_body ORDER BY time_posted DESC" );
how can I get it to bump it self as the latest update once a user re duplicates it it would let users know that some one else has shared it
Thanks ;)
Three solutions
Assuming you have a update timing field on your data record called time_updated, there are a couple of things you could do:
you could simply max the two:
ORDER BY max(time_posted,time_updated) DESC
This might give issues with NULL values though
substitute NULLs in the time_updated
ORDER BY nvl(time_updated, time_posted) DESC
(Oracle sql syntax nvl() substitutes NULL in the 1st param with the 2nd param - don't know what DB you are using)
always store time_updated (the best solution IMO). When creating the post (initially) set time_updated = time_created AND use
ORDER BY time_updated DESC
i want to get data from my database where one field is max, at this moment i do this in 2 queries. the thing is i dont want to overload the server so i am looking for a way to make it in 1 query. any suggestions?
as you can see i am looking for entry where the timestamp is max.
$query = "SELECT MAX(TIMESTAMP) AS timestamp FROM `data`";
$run_query = mysql_query($query);
$highest = mysql_result($run_query,'0','timestamp');
$query = "SELECT * FROM `data` where `timestamp`='$highest'";
$run_query = mysql_query($query);
thanks in advance.
An alternative, if you can guarantee that there will never be two records with the same timestamp:
SELECT *
FROM data
ORDER BY timestamp DESC
LIMIT 1
If you can have duplicate timestamps, then the other answers with the sub-select are the better solution.
This will simply work as you desired.
SELECT *
FROM data
WHERE timestamp = (SELECT MAX(timestamp) FROM data)
Backticks on this case are optionals. But actually timestamp is a reserved keyword but is permitted to be used even without escaping it with backtick.
SELECT * FROM `data` WHERE `timestamp` = (SELECT MAX(`timestamp`) FROM `data`)
If your goal is to minimize server resources, the difference between one query and two queries is really minor. The engine needs to do pretty much the same work. The difference would be the slight overhead of compiling two queries rather than one.
Regardless of the solution, you will minimize server resources by building an index on data(timestamp).
I'm new to this, sorry if the title is confusing. I am building a simple php/mysql gallery of sorts. It will show the newest 25 entries when a user first goes to it, and also allows off-site linking to individual items in the list. If the URL contains an ID, javascript will scroll to it. But if there are 25+ entries, it's possible that my query will fetch the newest results, but omit an older entry that happens to be in the URL as an ID.
That means I need to do something like this...
SELECT * FROM `submissions` WHERE uid='$sid'
But after that has successfully found the submission with the special ID, also do
SELECT * FROM `submissions` ORDER BY `id` DESC LIMIT 0, 25`
So that I can populate the rest of the gallery.
I could query that database twice, but I am assuming there's some nifty way to avoid that. MySQL is also ordering everything (based on newest, views, and other vars) and using two queries would break that.
You could limit across a UNION like this:
(SELECT * FROM submissions WHERE uid = '$uid')
UNION
(SELECT * FROM submissions WHERE uid <> '$uid' ORDER BY `id` LIMIT 25)
LIMIT 25
Note LIMIT is listed twice as in the case that the first query returns a result, we would have 26 results in the union set. This will also place the "searched for" item first in the returned sort result set (with the other 24 results displayed in sort order). If this is not desirable, you could place an ORDER BY across the union, but your searched for result would be truncated if it happened to be the 26th record.
If you need 25 rows with all of them being sorted, my guess is that you would need to do the two query approach (limiting second query to either 24 or 25 records depending on whether the first query matched), and then simply insert the uid-matched result into the sorted records in the appropriate place before display.
I think the better solution is:
SELECT *
FROM `submissions`
order by (case when usid = $sid then 0 else 1 end),
id desc
limit 25
I don't think the union is guaranteed to return results in the order of the union (there is no guarantee in the standard or in other databases).