I have this MySQL query which I am loading in to my home controller and after running Codeigniter's $this->output->enable_profiler(TRUE); I get an execution time of 5.3044
The Query inside my model:
class Post extends CI_Model {
function stream($uid, $updated, $limit) {
$now = microtime(true);
$sql = "
SELECT
*
FROM
vPAS_Posts_Users_Temp
WHERE
post_user_id = ?
AND post_type !=4
AND post_updated > ?
AND post_updated < ?
UNION
SELECT
u.*
FROM
vPAS_Posts_Users_Temp u
JOIN
PAS_Follow f
ON f.folw_followed_user_id = u.post_dynamic_pid
WHERE u.post_updated > ?
AND post_updated < ?
AND (( f.folw_follower_user_id = ? AND f.folw_deleted = 0 )
OR ( u.post_passed_on_by = f.folw_follower_user_id OR u.post_passed_on_by = ? AND u.post_user_id != ? AND u.post_type =4 ))
ORDER BY
post_posted_date DESC
LIMIT ?
";
$query = $this->db->query($sql, array($uid, $updated, $now, $updated, $now, $uid, $uid, $uid, $limit));
return $query->result();
}
}
Is there anything I can do here to improve the execution time and therefore increase my page load?
Edit
Explain Results
MySQL Workbench Visual Explain
Maybe you won't believe it, but DON'T retrieve SELECT * in your SQL. Just write the fields you want to retrieve and I think it'll speed up a lot.
I've seen increases in speed of more than 20 times when executing a query (from 0.4secs to 0.02 secs) just changing * for required fields.
Other thing: If you have an auto_increment id on INSERT in your tables, DON'T use post_posted_date as ORDER field. Ordering by DATETIME fields is slow, and if you may use an INT id (which hopefully you will have as an index) you will achieve the same result quicker.
UPDATE
As required in the question, technical reasons:
For not using SELECT *: Which is faster/best? SELECT * or SELECT column1, colum2, column3, etc. This is for SQL, but for MySQL (not as complete as question before) mySQL Query - Selecting Fields
For Ordering by Datetime: SQL, SQL Server 2008: Ordering by datetime is too slow, and again, related to MySQL: MySQL performance optimization: order by datetime field
Bonus: Learning how to set the indexes: http://ronaldbradford.com/blog/tag/covering-index/
I would add indexes on post_user_id, post_updated and folw_follower_user_id.
In this case it may also be better to not use union and separate the query into two separate ones and then use PHP to combine to two result sets.
If you switched to using active record you could also look into caching, to get better performance
The rows column shows an estimate for how many rows needs to be examined, which as I understand, means that in your case, it has to scan 72 * 1 * 2627 * 1 * 2 rows, which is quite a lot.
Now, the trick is to bring down this number, and one way is to add indexes. In your case, I would suggest adding an index which contains:
post_user_id, post_type, post_updated, post_updated.
This should bring down the first result set, of 72 rows.
Now for the UNION, try using UNION ALL instead, as it is claimed to be faster.
If that doesn't fix the problem, I would suggest rewriting the query to not use a UNION call at all.
Try the query with left join as you are trying to union on same table
"SELECT
u.*
FROM
vPAS_Posts_Users_Temp u
LEFT JOIN PAS_Follow f ON f.folw_followed_user_id = u.post_dynamic_pid
WHERE (
u.post_user_id = ?
AND u.post_type !=4
AND u.post_updated > ?
AND u.post_updated < ?
)
OR
(
u.post_updated > ?
AND post_updated < ?
AND (( f.folw_follower_user_id = ? AND f.folw_deleted = 0 )
OR ( u.post_passed_on_by = f.folw_follower_user_id OR u.post_passed_on_by = ? AND u.post_user_id != ? AND u.post_type =4 ))
)
ORDER BY
u.post_posted_date DESC
LIMIT ?"
I think you can remove the UNION from the query and make use of left join instead and avoid the unnecessary conditions:
SELECT U.*
FROM vPAS_Posts_Users_Temp AS U
LEFT JOIN PAS_Follow AS F ON F.folw_followed_user_id = U.post_dynamic_pid
WHERE U.post_updated > ?
AND U.post_updated < ?
AND (
(
F.folw_follower_user_id = ? AND F.folw_deleted = 0
)
OR
(
U.post_passed_on_by = F.folw_follower_user_id OR U.post_passed_on_by = ?
)
)
ORDER BY
U.post_posted_date DESC
LIMIT ?
Also identify and set proper indexes in your tables.
Related
I'm trying to change mySQL query for a faster data retrieval from the server. I have a table with more than 500,000 records but it takes forever to retrieve the data.
I want to change the query from
SELECT
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
FROM
loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email
WHERE
loanapply.approve = 'ongoing'
AND loanapply.del = 'false'
AND loanapply.archive = 0
AND loanapply.loanDate = '$date'
GROUP BY
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
ORDER BY
loanapplyId DESC
LIMIT
$currentSize,
$limit
to
SELECT
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
FROM
loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email AS data1
INNER JOIN (
SELECT
loanapply.loanapplyId
FROM
loanapply
LIMIT
$currentSize,
$limit
) AS data2 ON data1.loanapplyId = data2.loanapplyId
WHERE
loanapply.approve = 'ongoing'
AND loanapply.del = 'false'
AND loanapply.archive = 0
AND loanapply.loanDate = '$date'
ORDER BY
loanapplyId DESC
for faster data retrieval, The second query only returns blank values
for fast data retrival first you should check for valid indexes
be sure you have a proper composite index on loanapply eg:
create index myidx on loanapply ( loanDate, archive, del, approve, email )
and
table mkopakenya column ( email)
anyway (not for better perfomance but for proper use of SQL) you should not use group by without aggregation function (this behaviour is allowed for mysql version <5.7 but, by default, produce error for version >= 5.7) if you need distinct result use DISTINCT clause
SELECT DISTINCT loanapply.loanapplyId
, loanapply.loanAmount
, loanapply.email
, loanapply.approve
, loanapply.loanDate
, mkopakenya.name
, mkopakenya.idno
, mkopakenya.phoneNo
, mkopakenya.verification
FROM loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email
WHERE loanapply.approve='ongoing'
AND loanapply.del='false'
AND loanapply.archive=0
AND loanapply.loanDate='$date'
ORDER BY loanapplyId DESC LIMIT $currentSize, $limit
I have an SQL query that fetches posts from a database. Everything works fine, but now I need to order the results by the number of comments each post has. The comments are on a separate table and they have a post_id column that links it to the post. I need to order the posts by the count of the comments table based on a shard ID? I have tried everything but every time I try to add something to my query it stops running completely and leaves my page blank. I need help to know where to put the other JOIN statement. This is my query:
$union = "UNION ALL
(
SELECT DISTINCT wallposts.p_id,wallposts.is_profile_notes,wallposts.times_viewed,wallposts.columnTimesShared,
wallposts.marked,wallposts.secure_id,wallposts.reshared,wallposts.group_id,
wallposts.totaluploads,wallposts.WallUploadID,wallposts.type,
wallposts.value,wallposts.media,wallposts.youtube,wallposts.post_type,
wallposts.privacy,wallposts.tagedpersons,wallposts.with_friends_tagged,wallposts.emotion_head,wallposts.selected_emotion,wallposts.title,
wallposts.url,wallposts.description,wallposts.cur_image,
wallposts.uip,wallposts.likes,wallposts.userid,
wallposts.posted_by,wallposts.post as postdata,wallusers.*,
UNIX_TIMESTAMP() - wallposts.date_created AS TimeSpent,
PosterTable.mem_pic as posterPic, PosterTable.gender as posterGender,PosterTable.oauth_uid as poster_oauth_uid, PosterTable.username as posterUsername,
PosterTable.mem_fname as posterFname,PosterTable.work as posterWork,
PosterTable.mem_lname as posterLname,walllikes_track.id as PostLikeFound,wallposts.date_created
FROM
wallusers,wallusers as PosterTable, wallposts
LEFT JOIN walllikes_track
ON wallposts.p_id = walllikes_track.post_id AND walllikes_track.member_id = ".$user_id."
WHERE
wallusers.active = 1
AND
PosterTable.active = 1
AND
wallposts.group_id IN (".$groups.")
AND
wallposts.group_id != 0
AND
PosterTable.mem_id = wallposts.posted_by
AND
wallposts.marked < ".$this->flagNumber."
AND
wallusers.mem_id = wallposts.posted_by ) ";
The comments table is called wallcomments and it has a column called post_id. I know I need to use JOIN and COUNT but I don't know where to put it within my current code.
Try this query, I didn't run but i updated it.
SELECT wallposts.p_id,wallposts.is_profile_notes,wallposts.times_viewed,wallposts.columnTimesShared,
wallposts.marked,wallposts.secure_id,wallposts.reshared,wallposts.group_id,
wallposts.totaluploads,wallposts.WallUploadID,wallposts.type,
wallposts.value,wallposts.media,wallposts.youtube,wallposts.post_type,
wallposts.privacy,wallposts.tagedpersons,wallposts.with_friends_tagged,wallposts.emotion_head,wallposts.selected_emotion,wallposts.title,
wallposts.url,wallposts.description,wallposts.cur_image,
wallposts.uip,wallposts.likes,wallposts.userid,
wallposts.posted_by,wallposts.post as postdata,wallusers.*,
UNIX_TIMESTAMP() - wallposts.date_created AS TimeSpent,
PosterTable.mem_pic as posterPic, PosterTable.gender as posterGender,PosterTable.oauth_uid as poster_oauth_uid, PosterTable.username as posterUsername,
PosterTable.mem_fname as posterFname,PosterTable.work as posterWork,
PosterTable.mem_lname as posterLname,walllikes_track.id as PostLikeFound,wallposts.date_created
FROM
wallusers,wallusers as PosterTable, wallposts
WHERE
wallusers.active = 1
AND
PosterTable.active = 1
AND
wallposts.group_id IN (".$groups.")
AND
wallposts.group_id != 0
AND
PosterTable.mem_id = wallposts.posted_by
AND
wallposts.marked < ".$this->flagNumber."
AND
wallusers.mem_id = wallposts.posted_by ) " AND wallposts.p_id = walllikes_track.post_id AND walllikes_track.member_id = ".$user_id.";
A more readable query might look like this...
At least then we'd have a chance.
SELECT DISTINCT p.p_id
, p.is_profile_notes
, p.times_viewed
, p.columnTimesShared
, p.marked
, p.secure_id
, p.media...
FROM wallposts p...
I'm trying to get rows from a table according to some basic where clauses, and now I want to include an "AND EXISTS" clause on the end. My code is the following:
$stmt = $mysqli->prepare("SELECT object_id FROM ".
$db_table_prefix."user_events
WHERE LPAD(start_timestamp,15,'0') < LPAD(?,15,'0')
AND event_id = ?
AND EXISTS (
SELECT * FROM ".$db_table_prefix."user_events
WHERE LPAD(start_timestamp,15,'0') > LPAD(?,15,'0')
AND event_id = ? )
");
The problem I'm having is that I don't know how to specify a column value from the main query within the AND EXISTS subquery.
I'm looking for a way to tack on this bit into the subquery:
AND object_id = **object_id from main query**
Any help appreciated
Also, added an alias to the subquery's table to avoid confusion
$stmt = $mysqli->prepare("SELECT object_id FROM ".
$db_table_prefix."user_events
WHERE LPAD(start_timestamp,15,'0') < LPAD(?,15,'0')
AND event_id = ?
AND EXISTS (
SELECT * FROM ".$db_table_prefix."user_events AS u
WHERE LPAD(u.start_timestamp,15,'0') > LPAD(?,15,'0')
AND u.object_id = " .$db_table_prefix.".object_id
AND u.event_id = ? )
");
Rather than EXISTS, you might find a self-join syntax is clearer here, I cannot deduce exactly what you want from your code, but to get object_id for an event that started before a specific time, and was also started again later:
SELECT ue1.object_id
FROM user_events ue1 join user_events ue2
WHERE ue1.event_id = ue2.event_id AND
ue1.object_id = ue2.object_id AND
ue1.event_id = ? AND
LPAD(ue1.start_timestamp, 15, '0') < LPAD(ue2.start_timestamp, 15, '0') AND
LPAD(ue1.start_timestamp, 15, '0') < LPAD(?, 15, '0')
and object_id in (select object_id from ...
I have a query that uses PostgreSQL generate_series function but when it comes to large amounts of data, the query can be slow. An example of code the generates the query is below:
$yesterday = date('Y-m-d',(strtotime ( '-1 day' ) ));
$query = "
WITH interval_step AS (
SELECT gs::date AS interval_dt, random() AS r
FROM generate_series('$yesterday'::timestamp, '2015-01-01', '1 day') AS gs)
SELECT articles.article_id, article_title, article_excerpt, article_author, article_link, article_default_image, article_date_published, article_bias_avg, article_rating_avg
FROM development.articles JOIN interval_step ON articles.article_date_added::date=interval_step.interval_dt ";
if (isset($this -> registry -> get['category'])) {
$query .= "
JOIN development.feed_articles ON articles.article_id = feed_articles.article_id
JOIN development.rss_feeds ON feed_articles.rss_feed_id = rss_feeds.rss_feed_id
JOIN development.news_categories ON rss_feeds.news_category_id = news_categories.news_category_id
WHERE news_category_name = $1";
$params = array($category_name);
$query_name = 'browse_category';
}
$query .= " ORDER BY interval_step.interval_dt DESC, RANDOM() LIMIT 20;";
This series looks for only content that goes one day back and sorts the results in random order. My question is what are was that generate_series can be optimized to improve performance?
You don't need that generate_series at all. And do not concatenate query strings. Avoid it by making the parameter an empty string (or null) if it is not set:
if (!isset($this -> registry -> get['category']))
$category_name = '';
$query = "
select articles.article_id, article_title, article_excerpt, article_author, article_link, article_default_image, article_date_published, article_bias_avg, article_rating_avg
from
development.articles
inner join
development.feed_articles using (article_id)
inner join
development.rss_feeds using (rss_feed_id)
inner join
development.news_categories using (news_category_id)
where
(news_category_name = $1 or $1 = '')
and articles.article_date_added >= current_date - 1
order by
date_trunc('day', articles.article_date_added) desc,
random()
limit 20;
";
$params = array($category_name);
Passing $yesterday to the query is also not necessary as it can be done entirely in SQL.
If $category_name is empty it will return all categories:
(news_category_name = $1 or $1 = '')
Imho, try removing that random() in your order by statement. It probably has a much larger performance impact than you think. As things are it's probably ordering the entire set by interval_dt desc, random(), and then picking the top 20. Not advisable...
Try fetching e.g. 100 rows ordered by interval_dt desc instead, then shuffle them per the same logic, and pick 20 in your app. Or wrap the entire thing in a subquery limit 100, and re-order accordingly along the same lines.
i am using Codeigniter i am facing an issue while setting the values of "Limit" in a query,Limit is only showing "limit NULL"
Here is the snippet of my code.
SELECT block.loc, owner.name , block.dist_name FROM house INNER JOIN block ON house.block_id = block.block_id INNER JOIN owner ON owner.house_id = house.house_id WHERE
block.dist = ? AND house.status = 5 limit ? , ?
$result = $this->db->query($qry, array($this->getDist(), (int) $this->getLimitStart(), (int) $this->getLimitOffset()));
dump for
(int) $this->getLimitStart() is '0' and (int) $this->getLimitOffset() is '10'
As I understand that you make your own getter setter of the object, the getter which u are providing in you query, is returning NULL is only because you are not using the same setter.
For Example:
if You use it ($this->getLimitOffset()) you have to set it also like this yourObject->setLimitOffset(10). I think it will work for you now.
You should swap the start and offset values like this
SELECT block.loc, owner.name , block.dist_name FROM house INNER JOIN block ON house.block_id = block.block_id INNER JOIN owner ON owner.house_id = house.house_id WHERE
block.dist = ? AND house.status = 5 limit ? , ?
$result = $this->db->query($qry, array($this->getDist(), (int) $this->getLimitOffset(),(int) $this->getLimitStart()));
Becsuse codeigniter active record limit's first parameter is limit abd second is offset.
http://codeigniter.com/user_guide/database/active_record.html
Try this: $this->db->limit($nrecords, $offset);