Mysql higher LIMIT offset very slow with join in PHP - php

I'm trying to change mySQL query for a faster data retrieval from the server. I have a table with more than 500,000 records but it takes forever to retrieve the data.
I want to change the query from
SELECT
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
FROM
loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email
WHERE
loanapply.approve = 'ongoing'
AND loanapply.del = 'false'
AND loanapply.archive = 0
AND loanapply.loanDate = '$date'
GROUP BY
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
ORDER BY
loanapplyId DESC
LIMIT
$currentSize,
$limit
to
SELECT
loanapply.loanapplyId,
loanapply.loanAmount,
loanapply.email,
loanapply.approve,
loanapply.loanDate,
mkopakenya.name,
mkopakenya.idno,
mkopakenya.phoneNo,
mkopakenya.verification
FROM
loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email AS data1
INNER JOIN (
SELECT
loanapply.loanapplyId
FROM
loanapply
LIMIT
$currentSize,
$limit
) AS data2 ON data1.loanapplyId = data2.loanapplyId
WHERE
loanapply.approve = 'ongoing'
AND loanapply.del = 'false'
AND loanapply.archive = 0
AND loanapply.loanDate = '$date'
ORDER BY
loanapplyId DESC
for faster data retrieval, The second query only returns blank values

for fast data retrival first you should check for valid indexes
be sure you have a proper composite index on loanapply eg:
create index myidx on loanapply ( loanDate, archive, del, approve, email )
and
table mkopakenya column ( email)
anyway (not for better perfomance but for proper use of SQL) you should not use group by without aggregation function (this behaviour is allowed for mysql version <5.7 but, by default, produce error for version >= 5.7) if you need distinct result use DISTINCT clause
SELECT DISTINCT loanapply.loanapplyId
, loanapply.loanAmount
, loanapply.email
, loanapply.approve
, loanapply.loanDate
, mkopakenya.name
, mkopakenya.idno
, mkopakenya.phoneNo
, mkopakenya.verification
FROM loanapply
LEFT JOIN mkopakenya ON loanapply.email = mkopakenya.email
WHERE loanapply.approve='ongoing'
AND loanapply.del='false'
AND loanapply.archive=0
AND loanapply.loanDate='$date'
ORDER BY loanapplyId DESC LIMIT $currentSize, $limit

Related

How can I order the results of an SQL query by the count of another table based on a shard ID?

I have an SQL query that fetches posts from a database. Everything works fine, but now I need to order the results by the number of comments each post has. The comments are on a separate table and they have a post_id column that links it to the post. I need to order the posts by the count of the comments table based on a shard ID? I have tried everything but every time I try to add something to my query it stops running completely and leaves my page blank. I need help to know where to put the other JOIN statement. This is my query:
$union = "UNION ALL
(
SELECT DISTINCT wallposts.p_id,wallposts.is_profile_notes,wallposts.times_viewed,wallposts.columnTimesShared,
wallposts.marked,wallposts.secure_id,wallposts.reshared,wallposts.group_id,
wallposts.totaluploads,wallposts.WallUploadID,wallposts.type,
wallposts.value,wallposts.media,wallposts.youtube,wallposts.post_type,
wallposts.privacy,wallposts.tagedpersons,wallposts.with_friends_tagged,wallposts.emotion_head,wallposts.selected_emotion,wallposts.title,
wallposts.url,wallposts.description,wallposts.cur_image,
wallposts.uip,wallposts.likes,wallposts.userid,
wallposts.posted_by,wallposts.post as postdata,wallusers.*,
UNIX_TIMESTAMP() - wallposts.date_created AS TimeSpent,
PosterTable.mem_pic as posterPic, PosterTable.gender as posterGender,PosterTable.oauth_uid as poster_oauth_uid, PosterTable.username as posterUsername,
PosterTable.mem_fname as posterFname,PosterTable.work as posterWork,
PosterTable.mem_lname as posterLname,walllikes_track.id as PostLikeFound,wallposts.date_created
FROM
wallusers,wallusers as PosterTable, wallposts
LEFT JOIN walllikes_track
ON wallposts.p_id = walllikes_track.post_id AND walllikes_track.member_id = ".$user_id."
WHERE
wallusers.active = 1
AND
PosterTable.active = 1
AND
wallposts.group_id IN (".$groups.")
AND
wallposts.group_id != 0
AND
PosterTable.mem_id = wallposts.posted_by
AND
wallposts.marked < ".$this->flagNumber."
AND
wallusers.mem_id = wallposts.posted_by ) ";
The comments table is called wallcomments and it has a column called post_id. I know I need to use JOIN and COUNT but I don't know where to put it within my current code.
Try this query, I didn't run but i updated it.
SELECT wallposts.p_id,wallposts.is_profile_notes,wallposts.times_viewed,wallposts.columnTimesShared,
wallposts.marked,wallposts.secure_id,wallposts.reshared,wallposts.group_id,
wallposts.totaluploads,wallposts.WallUploadID,wallposts.type,
wallposts.value,wallposts.media,wallposts.youtube,wallposts.post_type,
wallposts.privacy,wallposts.tagedpersons,wallposts.with_friends_tagged,wallposts.emotion_head,wallposts.selected_emotion,wallposts.title,
wallposts.url,wallposts.description,wallposts.cur_image,
wallposts.uip,wallposts.likes,wallposts.userid,
wallposts.posted_by,wallposts.post as postdata,wallusers.*,
UNIX_TIMESTAMP() - wallposts.date_created AS TimeSpent,
PosterTable.mem_pic as posterPic, PosterTable.gender as posterGender,PosterTable.oauth_uid as poster_oauth_uid, PosterTable.username as posterUsername,
PosterTable.mem_fname as posterFname,PosterTable.work as posterWork,
PosterTable.mem_lname as posterLname,walllikes_track.id as PostLikeFound,wallposts.date_created
FROM
wallusers,wallusers as PosterTable, wallposts
WHERE
wallusers.active = 1
AND
PosterTable.active = 1
AND
wallposts.group_id IN (".$groups.")
AND
wallposts.group_id != 0
AND
PosterTable.mem_id = wallposts.posted_by
AND
wallposts.marked < ".$this->flagNumber."
AND
wallusers.mem_id = wallposts.posted_by ) " AND wallposts.p_id = walllikes_track.post_id AND walllikes_track.member_id = ".$user_id.";
A more readable query might look like this...
At least then we'd have a chance.
SELECT DISTINCT p.p_id
, p.is_profile_notes
, p.times_viewed
, p.columnTimesShared
, p.marked
, p.secure_id
, p.media...
FROM wallposts p...

SQL query with join shows only one result

I have sql query which should shows all records from table swt_modules, but it shows only first row.
$query1 = mysql_query ("SELECT swt_modules.id, swt_modules.name, swt_exam_regs.name AS exam_regs
FROM `swt_modules`
JOIN `swt_exam_regs`
USING ( `id` )
WHERE swt_exam_regs.id = swt_modules.exam_regulation
ORDER BY swt_modules.name DESC , swt_modules.id DESC
LIMIT " . $limit . "");
while ($fetch1 = mysql_fetch_array ($query1))
{
...
}
I have in this table (swt_modules) 3 rows and in each of them value of field "exam_regulation" is 1. In table swt_exam_regs I have only 1 row with 2 columns - id and name. Swt_modules.id stores id number. Which join I should use to be able to see all records?
I would also suggest using mysqli or pdo instead of the now deprecated mysql.
$query1 = mysql_query ("
SELECT
swt_modules.id,
swt_modules.name,
swt_exam_regs.name AS exam_regs
FROM swt_modules
LEFT JOIN swt_exam_regs on swt_exam_regs.id = swt_modules.exam_regulation
ORDER BY
swt_modules.name DESC,
swt_modules.id DESC
LIMIT $limit");
You need to use LEFT JOIN instead of INNER JOIN. Change your query as below. Notice that, I have removed LIMIT since you are trying to fetch all rows.
SELECT swt_modules.id, swt_modules.name, swt_exam_regs.name AS exam_regs
FROM `swt_modules`
LEFT JOIN `swt_exam_regs`
ON swt_exam_regs.id = swt_modules.exam_regulation
ORDER BY swt_modules.name DESC , swt_modules.id DESC

MySQL Query Performance and Codeigniter

I have this MySQL query which I am loading in to my home controller and after running Codeigniter's $this->output->enable_profiler(TRUE); I get an execution time of 5.3044
The Query inside my model:
class Post extends CI_Model {
function stream($uid, $updated, $limit) {
$now = microtime(true);
$sql = "
SELECT
*
FROM
vPAS_Posts_Users_Temp
WHERE
post_user_id = ?
AND post_type !=4
AND post_updated > ?
AND post_updated < ?
UNION
SELECT
u.*
FROM
vPAS_Posts_Users_Temp u
JOIN
PAS_Follow f
ON f.folw_followed_user_id = u.post_dynamic_pid
WHERE u.post_updated > ?
AND post_updated < ?
AND (( f.folw_follower_user_id = ? AND f.folw_deleted = 0 )
OR ( u.post_passed_on_by = f.folw_follower_user_id OR u.post_passed_on_by = ? AND u.post_user_id != ? AND u.post_type =4 ))
ORDER BY
post_posted_date DESC
LIMIT ?
";
$query = $this->db->query($sql, array($uid, $updated, $now, $updated, $now, $uid, $uid, $uid, $limit));
return $query->result();
}
}
Is there anything I can do here to improve the execution time and therefore increase my page load?
Edit
Explain Results
MySQL Workbench Visual Explain
Maybe you won't believe it, but DON'T retrieve SELECT * in your SQL. Just write the fields you want to retrieve and I think it'll speed up a lot.
I've seen increases in speed of more than 20 times when executing a query (from 0.4secs to 0.02 secs) just changing * for required fields.
Other thing: If you have an auto_increment id on INSERT in your tables, DON'T use post_posted_date as ORDER field. Ordering by DATETIME fields is slow, and if you may use an INT id (which hopefully you will have as an index) you will achieve the same result quicker.
UPDATE
As required in the question, technical reasons:
For not using SELECT *: Which is faster/best? SELECT * or SELECT column1, colum2, column3, etc. This is for SQL, but for MySQL (not as complete as question before) mySQL Query - Selecting Fields
For Ordering by Datetime: SQL, SQL Server 2008: Ordering by datetime is too slow, and again, related to MySQL: MySQL performance optimization: order by datetime field
Bonus: Learning how to set the indexes: http://ronaldbradford.com/blog/tag/covering-index/
I would add indexes on post_user_id, post_updated and folw_follower_user_id.
In this case it may also be better to not use union and separate the query into two separate ones and then use PHP to combine to two result sets.
If you switched to using active record you could also look into caching, to get better performance
The rows column shows an estimate for how many rows needs to be examined, which as I understand, means that in your case, it has to scan 72 * 1 * 2627 * 1 * 2 rows, which is quite a lot.
Now, the trick is to bring down this number, and one way is to add indexes. In your case, I would suggest adding an index which contains:
post_user_id, post_type, post_updated, post_updated.
This should bring down the first result set, of 72 rows.
Now for the UNION, try using UNION ALL instead, as it is claimed to be faster.
If that doesn't fix the problem, I would suggest rewriting the query to not use a UNION call at all.
Try the query with left join as you are trying to union on same table
"SELECT
u.*
FROM
vPAS_Posts_Users_Temp u
LEFT JOIN PAS_Follow f ON f.folw_followed_user_id = u.post_dynamic_pid
WHERE (
u.post_user_id = ?
AND u.post_type !=4
AND u.post_updated > ?
AND u.post_updated < ?
)
OR
(
u.post_updated > ?
AND post_updated < ?
AND (( f.folw_follower_user_id = ? AND f.folw_deleted = 0 )
OR ( u.post_passed_on_by = f.folw_follower_user_id OR u.post_passed_on_by = ? AND u.post_user_id != ? AND u.post_type =4 ))
)
ORDER BY
u.post_posted_date DESC
LIMIT ?"
I think you can remove the UNION from the query and make use of left join instead and avoid the unnecessary conditions:
SELECT U.*
FROM vPAS_Posts_Users_Temp AS U
LEFT JOIN PAS_Follow AS F ON F.folw_followed_user_id = U.post_dynamic_pid
WHERE U.post_updated > ?
AND U.post_updated < ?
AND (
(
F.folw_follower_user_id = ? AND F.folw_deleted = 0
)
OR
(
U.post_passed_on_by = F.folw_follower_user_id OR U.post_passed_on_by = ?
)
)
ORDER BY
U.post_posted_date DESC
LIMIT ?
Also identify and set proper indexes in your tables.

How to union 2 SQL-query in Codeigniter? [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
UNION query with codeigniter's active record pattern
I have the following code:
$language_id=$this->get_language_id($language_code);
$english_id=$this->get_language_id('en');
$query="SELECT e.label_value, t.user_id, t.votes, t.approved, t.language_value FROM labels e left outer join labels t on e.label_value=t.label_value WHERE e.language=$english_id and t.language=$language_id and (t.approved=1 or t.user_id=$user_id) and e.label_value in (select distinct label_value from labels WHERE language=$english_id order by label_value limit $start_index, 30) order by e.label_value, t.votes";
$query=$this->db->query($query);
$data=$query->result_array();
But I have got the following error:
This version of MySQL doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery'
So, I need to do the folowoing part "select distinct label_value from labels WHERE language=$english_id order by label_value offset $start limit 30" in another query. Please, help me, how can I do it using CodeIgniter?
UPDATE:
There are is table labels
(label_value, language_value, language) - PK,
user_id,
timestamp,
approved,
votes
and I need to get all queries from this table (for example, it's name is t and e) with labels t.label_value, e.label_value (is exists), e.user_id, e.votes, e.timestamp where t.label_value=e.label_value(same label), t.language=45 (english language), e.language=24 (my language) and (e.user_id=121234 or e.approved=1). But I need all entries, and if (t.label_value!=e.label_value) I need to get this entry with NULL fields.
This is a limitation of MySQL and not PHP or CI. In order to get around it, you need to wrap your sub query in an aliased sub query so it becomes a derived table:
$language_id = $this->get_language_id($language_code);
$english_id = $this->get_language_id('en');
$query = "
SELECT e.label_value, t.user_id, t.votes, t.approved, t.language_value
FROM labels e
LEFT OUTER JOIN labels t on e.label_value=t.label_value
WHERE
e.language = $english_id
AND t.language = $language_id
AND (t.approved = 1 OR t.user_id = $user_id)
AND e.label_value IN (
SELECT label_value
FROM (
SELECT DISTINCT label_value
FROM labels
WHERE language = $english_id
ORDER BY label_value
LIMIT $start_index, 30
) i
)
ORDER BY e.label_value, t.votes
";
$query = $this->db->query($query);
$data = $query->result_array();
I think that will work, let me know if it doesn't and I will take another look at it.
EDIT
I'm having a little difficulty working out exactly what you are trying to do, but I think it might be something more like this:
SELECT t.label_value, t.user_id, t.votes, t.approved, t.language_value
FROM (
SELECT DISTINCT label_value
FROM labels
WHERE language = $english_id
) e
LEFT JOIN labels t ON e.label_value = t.label_value
WHERE
t.language = $language_id
AND (t.approved = 1 OR t.user_id = $user_id)
ORDER BY t.label_value, t.votes
LIMIT $start_index, 30
If this is still not correct, please show some example rows, and the result set you would like to retrieve from those rows.

Query issue in Codeigniter

I want to random show 6 news/reviews on my front page but it shows the same content 6 times random but I will not have duplication of content. Here is the SQL query:
SELECT
anmeldelser.billed_sti ,
anmeldelser.overskrift ,
anmeldelser.indhold ,
anmeldelser.id ,
anmeldelser.godkendt
FROM
anmeldelser
LIMIT 0,6
UNION ALL
SELECT
nyheder.id ,
nyheder.billed_sti ,
nyheder.overskrift ,
nyheder.indhold ,
nyheder.godkendt
FROM nyheder
ORDER BY rand() LIMIT 0,6
showing my example with active record for simplicity,
try randomizing your offset instead of the order, while still limiting to 6
// get the total number of rows
$total_rows = $this->db->count_all_results('my_table');
// offset random point within the total rows
$offset = rand( 0 , $total_rows - 6 );
$q = $this->db->offset( $offset )->limit( 6 )->get( 'my_table' );
print_r( $q->result_array() );
//initialize query builder
$sql1=$sql2=$this->db;
$sql1->select('anmeldelser.billed_sti ,anmeldelser.overskrift ,anmeldelser.indhold ,anmeldelser.id ,anmeldelser.godkendt');
$sql1->from('anmeldelser');
$sql1->order_by('rand()');
$sql1->limit(3);
//get only sql string
$query1=$sql1->get_compiled_select();
$sql2->select('nyheder.id ,nyheder.billed_sti ,nyheder.overskrift ,nyheder.indhold ,nyheder.godkendt');
$sql2->from('nyheder');
$sql2->order_by('rand()');
$sql2->limit(3);
$query2=$sql2->get_compiled_select();
//combine two query
$query = $this->mydb->query("($query1) UNION ($query2)");
$result = $query->result();
I am assuming that you need to join a Two table by this comment of yours.
You didn't mention your foreign key so I am assuming that also.
It is also not clear that the column name of your tables are same or not.
So, I am posting an join query for your table in which I assume your foreign key and column name, so please correct that before using it.
Here is your query to join your table:
$query = $this->db
->select('an.billed_sti,an.overskrift,an.indhold,an.id,an.godkendt, ny.id as ny_id,ny.billed_sti as ny_billed_sti, ny.overskrift as ny_overskrift, ny.indhold as ny_indhold , ny.godkendt as ny_godkendt ')
->from('anmeldelser as an')
->join('nyheder as ny', 'ny.id_fk = an.id', 'left outer') // I am assuming here that the [id_fk] field is the foreign key
->limit(0, 6)
->order_by('puttablename.tablecolumn', 'asc') // Your you table name and column name by which you want to order, you can use [asc/desc] as your need
->get();
And If you want to UNION here is the solution for it.

Categories