I am currently working on a project that has a very large database of users and their different levels, however we where using RAND() and we are starting to get very high MySQL CPU usage warnings from our web host due to the growing demand of the database.
So we need to optimize this one particular query to help reduce the mysqli calls …
The row LINK where most of the data is stored is close to 50,000 + entries and using RAND() is slowly killing the website.
How would we optimize this to reduce the number of mysqli calls..
$CALLROW = $db->CacheGetAll("SELECT `ID`, `TITLE` FROM `{$tables['link']['name']}` WHERE (`FEATURED` = '1' OR `FEATURED_ADVANCED`= '1' OR `FEATURED_NORMAL`= '1') AND `STATUS` = '2' {$expire_where} ORDER BY RAND() LIMIT 20");
$tpl->assign('CALLROW', $CALLROW);
How can we reduce the memory imprint on the above SQL statement?
Well after trial an error I came up with this
$CALLROW = $db->CacheGetAll("SELECT * FROM `{$tables['link']['name']}` WHERE `FEATURED` = '1' OR `FEATURED_ADVANCED`= '1' OR `FEATURED_NORMAL`= '1' AND `STATUS` = '2' AND ID >= RAND() * (SELECT MAX(ID) FROM `{$tables['link']['name']}`) LIMIT 15");
$tpl->assign('CALLROW', $CALLROW);
However after I refresh it pulls some different ones but never the FEATURED_ADVANCED or FEATURED_NORMAL it seems to only pull the FEATURED IDs first
Any idea of how to truly shuffle these
Related
I have a table in my database with approx 400K rows, and I am executing the following statement (query that laravel is executing):
select * from `activities` where `device_id` = ? and `battery_level` is not null order by `created_at` desc limit 1
This takes less than 100ms when I execute it directly using a MySQL client. But laravel takes 1,5 - 2 seconds
My table looks like this:
And how I use the models: (takes 1500ms)
Activity::query()
->where('device_id', 288)
->whereNotNull('battery_level')
->orderByDesc('created_at')
->first();
And with DB: (also takes 1500 ms)
DB::table('activities')
->where('device_id', 288)
->whereNotNull('battery_level')
->orderByDesc('created_at')
->first();
I am looking to reduce this query to the same speed as MySQL between 100 and 200ms max. Removing the ordering desc reduces the time of the query, but I need to use order desc to get the latest.
Any idea what I am doing wrong here?
It may be helpful to provide SHOW CREATE TABLE.
Replacing INDEX(device_id) by
INDEX(device_id, battery_leve, created_at)
may help.
Try this:
$record = DB::raw("SELECT * FROM activities WHERE device_id = 1 AND battery_level IS NOT NULL ORDER BY created_at DESC LIMIT 1")
I recently design a php/mysql website.
I the index page I used this sql query
SELECT * FROM upload WHERE aid = '206' AND pin = '0' AND format like '%video%' ORDER BY created DESC
SELECT * FROM upload WHERE aid = '206' AND pin = '0' AND format like '%image%' ORDER BY created DESC
SELECT * FROM upload WHERE aid = '206' AND pin = '0' AND format like '%image%' ORDER BY created DESC
I do not have a big data yet...maybe 1000 records in DB.
But when I load the page..it takes 8-20 seconds to load.
when I remove that sal queries,the load time will be normal.
please advice me best method to resolve that.
You are using 3 different SQL queries to get the data from same table.
You can get the same result using only one SQL query.
So SQL query should be as below.
SELECT * FROM `upload` WHERE `aid` = '206' AND `pin` = '0' AND (`format` like '%video%' OR `format` like '%image%') ORDER BY `created` DESC
There may be chances that you might have included many CSS,Images or JavaScript libraries.
You can check your page performance issue using below site.
http://tools.pingdom.com/fpt/
Things that might be worth trying to speed things up:
add appropriate indexes to your table (perhaps aid should be an index?)
combine these three queries into a single query (as per Ali Khanusiya's answer)
add an appropriate LIMIT clause to your query (eg. LIMIT 1 or LIMIT 10) to fetch a subset of the results (you can use an OFFSET to paginate the results)
I'm trying to make a quiz using mysql and php
I want to fetch questions randomly from the database
I have a truble with rand function
sometimes it seturns no value and returns also duplications
I tried to find a solution on the net but couldn't make it
this is the part of code that generate the problem
$link=mysqli_connect("localhost","root","","database");
$req="SELECT DISTINCT *
FROM `qst_s`
WHERE `id_qst` = ROUND( RAND()*49 ) + 1 AND `level` = '1' LIMIT 40";
$result=mysqli_query($link,$req);
$question=$result->fetch_assoc();
I have 50 questions in my database level 1 by the way
The simplest way of selecting random rows from the MySQL database is to use "ORDER BY RAND()" clause in the query.
SELECT * FROM `table` ORDER BY RAND() LIMIT 0,1;
The problem with this method is that it is very slow. The reason for it being so slow is that MySQL creates a temporary table with all the result rows and assigns each one of them a random sorting index. The results are then sorted and returned.
Your query would look like this:
$req="SELECT DISTINCT *
FROM `qst_s`
WHERE `level` = '1'
ORDER BY RAND()
LIMIT 40";
I have query that gets 10 random posts , and as you know this is very slow and heavy query, is there any alternatives to submit this query without any slow appearance?
my current rand query :
SELECT * FROM posts ORDER BY RAND() LIMIT 10
From MySQL document:
SELECT * FROM tablename ORDER BY RAND() LIMIT 1
works for small tables, but once the tables grow larger than 300,000 records or so this will be very slow because MySQL will have to process ALL the entries from the table, order them randomly and then return the first row of the ordered result, and this sorting takes long time. Instead you can do it like this (atleast if you have an auto_increment PK):
SELECT MIN(id), MAX(id) FROM tablename;
Fetch the result into $a
//php code
$id=rand($a[0],$a[1]);
SELECT * FROM tablename WHERE id>='$id' LIMIT 1
I'm currently displaying a random row from all the entries and that works fine.
SELECT * FROM $db_table where live = 1 order by rand() limit 1
now, i'd like to limit it to the last 100 entries in the db.
every row in the db has an ID and a timestamp.
it's a small database, so overhead-minimization is not a priority.
thanks!
EDIT:
Still can't get it running.. I get a mysql_fetch_array error:
"Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource
Here's all of my code:
<?php $sql = "SELECT * FROM
(SELECT * FROM $db_table ORDER BY $datetime DESC LIMIT 100)
ORDER BY rand() LIMIT 1";
$query = mysql_query($sql);
while($row = mysql_fetch_array($query)) {
echo "".$row['familyname']."";
} ?>
Thanks again!
This is what I came up with off the top of my head. I've tested it and it works in SQLite, so you shouldn't have much trouble with MySQL. The only change was that SQLite's random function is random() not rand():
SELECT * FROM
(SELECT * FROM $db_table ORDER BY $timestamp DESC LIMIT 100)
ORDER BY rand() LIMIT 1
This page has a pretty detailed writeup on how to optimize an ORDER BY RAND()-type query. It's actually too involved for me to explain adequately on SO (also, I don't fully understand some of the SQL commands used, though the general concept makes sense), but the final optimized query makes use of several optimizations:
First, ORDER BY RAND(), which uses a filesort algorithm on the entire table, is dropped. Instead, a query is constructed to simply generate a single random id.
At this stage, an index scan is being used, which is even less efficient than a filesort in many cases, so this is optimized away with a subquery.
The WHERE clause is replaced with a JOIN to reduce the number of rows fetched by the outer SELECT, and the number of times the subquery is executed, to just 1.
In order to account for holes in the ids (from deletions) and to ensure an equal distribution, a mapping table is created to map row numbers to ids.
Triggers are used to automatically update & maintain the mapping table.
Lastly, stored procedures are created to allow multiple rows to be selected at once. (Here, ORDER BY is reintroduced, but used only on the result rows.)
Here are the performance figures:
Q1. ORDER BY RAND()
Q2. RAND() * MAX(ID)
Q3. RAND() * MAX(ID) + ORDER BY ID
100 1.000 10.000 100.000 1.000.000
Q1 0:00.718s 0:02.092s 0:18.684s 2:59.081s 58:20.000s
Q2 0:00.519s 0:00.607s 0:00.614s 0:00.628s 0:00.637s
Q3 0:00.570s 0:00.607s 0:00.614s 0:00.628s 0:00.637s