Make Query More Efficent (Really slow query taking forever!) - php

What I am trying to do is to get all of the users with the right conditions
so I'm building with a foreach statment a sub_queries to make it work.
the problem is that I got 100,000+ records in the Database.
and this kind of query takes forever to run.
I know I'm not doing it in the best way but I also tried left joins, which was really slow too..
this is the function I'm using:
public function get_affected_users_by_conditions($conditions, $mobile_type)
{
// Basic Query
// Selecting all of the users from `enswitch_mobile users` table
// The total and the `users` with the conditions
$sql = "SELECT COUNT(*) AS `users`,
(SELECT COUNT(*) FROM `enswitch_mobile_users`) AS `total`
FROM
`enswitch_mobile_users` AS `musers`
WHERE
`musers`.`phone_type` = :mobile_type";
$value_counter = 0;
$values = array();
// This is the foreach loop I was talking about
// I am looping all the conditons.
// and when theres a value i'm adding it as a subquery.
foreach($conditions as $cnd) {
switch ($cnd['condition']) {
// REALLY SLOW SUB-QUERY:
case 'talked_atleast':
$value_counter++;
// Here I'm trying to CUT the query by users who talked atleast $value seconds
$sql .= " AND (SELECT SUM(TIME_TO_SEC(TIMEDIFF(`finished_call`,`start_call`))) FROM `enswitch_calls` WHERE `user_id` = `musers`.`id`) >= :value".$value_counter;
$values[$value_counter] = $cnd['value'];
break;
// REALLY SLOW SUB-QUERY:
case 'purchase_atleast':
// Here I am trying to CUT the users by subquery who check if the users has bought at least $value times
$value_counter++;
$sql .= " AND (SELECT COUNT(*) FROM (SELECT user_id FROM enswitch_new_iphone_purchases
UNION
SELECT user_id FROM enswitch_new_android_purchases) AS p WHERE `status` > 0 AND user_id` = `musers`.`id`) >= :value".$value_counter;
$values[$value_counter] = $cnd['value'];
break;
// REALLY SLOW SUB-QUERY:
case 'never_purchase':
// Here I am trying to CUT the users by subquery to get only the users who never made a puchase.
$sql .= ' AND (SELECT COUNT(*) FROM (SELECT user_id FROM enswitch_new_iphone_purchases
UNION
SELECT user_id FROM enswitch_new_android_purchases) AS p WHERE `status` = 0 AND `user_id` = `musers`.`id`) = 0';
break;
}
}
$query = DB::query(Database::SELECT, $sql);
$query->bind(':mobile_type', $mobile_type);
// Looping the values and binding it into the SQL query!
foreach ($values as $k => $v) {
$query->bind(':value'.$k, $values[$k]);
}
// Executing query
$result = $query->execute();
return array('total_users' =>$result[0]['total'], 'affected_users'=>$result[0]['users']);
}
EDIT:
The Slowest Query as Requested: (MySQL)
SELECT COUNT(*) AS `users`,
( SELECT COUNT(*)
FROM `enswitch_mobile_users`
) AS `total`
FROM `enswitch_mobile_users` AS `musers`
WHERE `musers`.`phone_type` = 'iphone'
AND ( SELECT COUNT(*)
FROM ( SELECT `status`,
`user_id`
FROM `enswitch_new_iphone_purchases`
UNION
SELECT `status`,
`user_id`
FROM `enswitch_new_android_purchases`
) AS `p`
WHERE `p`.`status` > 0
AND `p`.`user_id` = `musers`.`id`
) >= 2

The subquery in the second SELECT column will execute for every m_users row that passes the WHERE condition:
SELECT
COUNT(*) AS users,
(SELECT COUNT(*) FROM enswitch_mobile_users) AS total <-- here's the problem
FROM enswitch_mobile_users AS musers
WHERE musers.phone_type = whatever
If I'm reading this correctly, you need a one-row result with the following columns:
users - number of enswitch_mobile_users rows with the specified phone_type
total - count of all enswitch_mobile_users rows
You can get the same result with this query:
SELECT
COUNT(CASE WHEN musers.phone_type = whatever THEN 1 END) AS users,
COUNT(*) AS total
FROM enswitch_mobile_users
The CASE checks for the phone type, and if it matches the one you're interested it it yields a 1, which is counted. If it doesn't match, it yields a NULL, which is not counted.

Related

selecting sql variable returning an empty array

I'm attempting to SET 3 variables in MySQL and get the sum of two of them.
The fist two variables, #cFollow and #cComment, should return an integer value each (the count of how many rows are returned); the third one is the sum of those two integers.
This is my SQL:
SET #cFollow = (SELECT COUNT(*) FROM followers WHERE unix > :unix AND following = :user);
SET #cComment = (SELECT COUNT(*) FROM comments WHERE comment_unix > :unix AND comment_track IN (SELECT upload_id FROM uploads WHERE upload_artist = :user));
SET #total = #cFollow + #cComment;
SELECT #total;
When I tested this on PHPMyAdmin, it returned the correct values and worked perfectly fine. However, when I tested it within PHP, it returned an empty array.
This is my PHP:
$holdPoint = (int)Input::get("hold_point");
$_SQL = "
SET #cFollow = (SELECT COUNT(*) FROM followers WHERE unix > :unix AND following = :user);
SET #cComment = (SELECT COUNT(*) FROM comments WHERE comment_unix > :unix AND comment_track IN (SELECT upload_id FROM uploads WHERE upload_artist = :user));
SET #total = #cFollow + #cComment;
SELECT #total;";
$_PARAMS = [":unix" => $holdPoint, ":user" => $user_id];
$check = DB::getInstance()->queryPro($_SQL, $_PARAMS);
var_dump($check);
This is the result of that var_dump:
array(0){} // not very impressive...
// should be something like int(1) instead
I've been searching around all night learning how to return a variable in PHP from a MySQL query, and this is as far as I've gotten.
All help is appreciated,
Cheers.
This answer is not really meant as a answer but more as a comment.
Also note that your queries
SET #cFollow = (SELECT COUNT(*) FROM followers WHERE unix > :unix AND following = :user);
SET #cComment = (SELECT COUNT(*) FROM comments WHERE comment_unix > :unix AND comment_track IN (SELECT upload_id FROM uploads WHERE upload_artist = :user));
SET #total = #cFollow + #cComment;
SELECT #total;
Can be most likely be rewritten as one query
SELECT
SUM(alias.c) AS total
FROM (
SELECT COUNT(*) AS c FROM followers WHERE unix > :unix AND following = :user
UNION ALL
SELECT COUNT(*) AS c FROM comments WHERE comment_unix > :unix AND comment_track IN (SELECT upload_id FROM uploads WHERE upload_artist = :user)
) AS alias

Select a fixed number of records from a particular user in a sql result

I have 2 tables - users and articles.
users:
user_id (int)
name (varchar)
articles:
article_id (int)
user_id (int)
title (varchar)
description (text)
In my application I need to display 20 RANDOM articles on a page.
My query is like this:
SELECT a.title
, a.description
, u.name
FROM articles a
JOIN users u
USING (user_id)
ORDER
BY RAND()
LIMIT 20
A user can have any number of articles in the database.
Now the problem is sometimes out of 20 results, there are like 9-10 articles from one single user.
I want those 20 records on the page to not contain more than 3 (or say 4) articles from a particular user.
Can I achieve this through SQL query. I am using PHP and MySQL.
Thanks for your help.
You could try this?
SELECT * FROM
(
SELECT B.* FROM
(
SELECT A.*, ROW_NUMBER() OVER (PARTITION BY A.USER_ID ORDER BY A.R) USER_ROW_NUMBER
FROM
(
SELECT a.title, a.description, u.name, RND() r FROM articles a
INNER JOIN users u USING (user_id)
) A
) B
WHERE B.USER_ROW_NUMBER<=4
) C
ORDER BY RAND() LIMIT 20
Mmm, intresting I don't think this is possible through a pure sql query.
My best idea would be to have an array of the articles that you'll eventually display query the database and use the standard SELECT * FROM Articles ORDER BY RAND() LIMIT 20
The go through them, making sure that you have indeed got 20 articles and no one has breached the rules of 3/4 per user.
Have another array of users to exclude, perhaps using their user id as an index and value of a count.
As you go through add them to your final array, if you find any user that hits you rule add them to the array.
Keep running the random query, excluding users and articles until you hit your desired amount.
Let me try some code (it's been a while since I did php)
$finalArray = [];
$userArray = [];
while(count($finalArray) < 20) {
$query = "SELECT * FROM Articles ";
if(count($finalArray) > 0) {
$query = $query . " WHERE articleID NOT IN(".$finalArray.")";
$query = $query . " AND userID NOT IN (".$userArray.filter(>4).")";
}
$query = $query . " ORDER BY Rand()";
$result = mysql_query($query);
foreach($row = mysql_fetch_array($result)) {
if(in_array($finalArray,$row) == false) {
$finalArray[] = $row;
}
if(in_array($userArray,$row[userId]) == false) {
$userArray[$row[userId]] = 1;
}
else {
$userArray[$row[userId]] = $userArray[$row[userId]] + 1;
}
}

ORDER BY COUNT with condition

I have two tables, users and clients_closed_1.
I need to order the result by the count of the rows on the table client_closed_1 where meeting=1.
I did it for time_closed field but that was easy because there was no condition.
It's a part of a search code so I'll show you all of it.
With this code I manage to order it by meeting - but users who has no rows with meeting=1 isn't pull out from the database and I need them to show even if they doesn't have meetings.
if (project_type($_GET['project']) == 1) {
$table = 'clients_closed_1';
} else {
$table = 'clients_closed_2';
}
$s_query = "SELECT *,COUNT(time_closed) as numc FROM `".$table."` FULL JOIN `users` ON users.ID=user_c WHERE 1=1";
if (isset($_POST['search'])) {
if ($_POST['tm'] == 'da') {
$dd = strtotime($_POST['y']."-".$_POST['m']."-".$_POST['d']);
$s_query = $s_query." && DATE(FROM_UNIXTIME(time_closed)) = DATE(FROM_UNIXTIME(".$dd."))";
}
elseif ($_POST['tm'] == 'mon') {
$s_query = $s_query." && YEAR(FROM_UNIXTIME(time_closed))=".$_POST['y']." && MONTH(FROM_UNIXTIME(time_closed))=".$_POST['m'];
}
if (!empty($_POST['search_name'])) {
$s_query = $s_query." && CONCAT(p_name,' ',l_name) LIKE '%".$_POST['search_name']."%'";
}
if (!empty($_POST['level'])) {
$query = "&& (level=3 && project IN (SELECT `project` FROM `project` WHERE type='2')) || level=4";
}
} else {
$s_query = $s_query." && YEAR(FROM_UNIXTIME(time_closed))=YEAR(NOW()) && MONTH(FROM_UNIXTIME(time_closed))=MONTH(NOW())";
}
if (isset($_GET['order'])) {
if ($_GET['order'] == 'closing') {
$s_query = $s_query." GROUP BY users.ID ORDER BY numc DESC";
}
elseif ($_GET['order'] == 'meeting') {
$s_query = $s_query." && meeting='1' GROUP BY users.ID ORDER BY numd DESC";
}
}
$query = $db->query($s_query);
If you need any more code/doedn't understand something comment please and I'll fix it.
Thank you.
EDIT:
example of $s_query:
SELECT *,COUNT(time_closed) as numc, COUNT(meeting) as numd FROM `clients_closed_1`
FULL JOIN `users` ON users.ID=user_c WHERE 1=1 &&
YEAR(FROM_UNIXTIME(time_closed))=YEAR(NOW()) &&
MONTH(FROM_UNIXTIME(time_closed))=MONTH(NOW())
GROUP BY users.ID ORDER BY numc DESC
Im not sure I understand 100% of the criteria youre looking for but here is a rough draft of the query:
SELECT c.id, c.meeting, temp1.time_closed_count, temp2.meeting_count, temp3.status_count
FROM `clients_closed_1` c
FULL JOIN `users` u
ON c.user_c=u.ID
LEFT JOIN (SELECT time_closed, count(time_closed) time_closed_count FROM clients_closed_1 GROUP BY time_closed) temp1
ON c.time_closed = temp1.time_closed
LEFT JOIN (SELECT meeting, count(meeting) meeting_count FROM clients_closed_1 GROUP BY meeting) temp2
ON c.meeting = temp2.meeting
LEFT JOIN (SELECT status, count(status) status_count FROM clients_closed_1 GROUP BY status) temp3
ON c.status = temp3.status
WHERE 1=1
AND YEAR(FROM_UNIXTIME(time_closed))=YEAR(NOW())
AND MONTH(FROM_UNIXTIME(time_closed))=MONTH(NOW())
ORDER BY {$order_criteria} DESC
Whats happeneing here is, we are doing the count of all distinct meeting values in a subquery and joining it to the original query based on the value of "meeting" for each row.
This gives us the total "meetings" grouped by distinct meeting values, without cutting out rows. Such is the same for the other 2 subqueries.
This also cleans things up a bit and allows us to just insert the $order_criteria, where that could be time_closed count, meeting_count, or status_count. Just set a default (id) in case your user does not choose anything :)
Edit: Id also recommend trying to get out of the SELECT * habit. Specify the columns you need and your output will be much nicer. Its also far more efficient when you start dealing with larger tables.
After I wrote a really long query to do this I found the perfect soulution.
SELECT SUM(IF(meeting='1' && user_c=users.ID, 1,0)) as meeting_count FROM clients_closed_1 JOIN users
This query return as meeting_count the number of meeting which their value is '1'.
I didn't know I can do such thing until now, so I shared it here. I guess it can be helpull in the future.

How do I connect two tables to see if a value exist in any of the two

I having problems connecting the below query together so that It works more efficient.Can someone please tell me how I can connect these two queries so that it is only one?
$rs_duplicate = mysql_query("select count(*) as total
from advertisers_account
where user_email='$user_email' ") or die(mysql_error());
list($total) = mysql_fetch_row($rs_duplicate);
}
$rs_duplicate_pub = mysql_query("select count(*) as total
from publishers_account
where user_email='$user_email' ") or die(mysql_error());
list($totalpub) = mysql_fetch_row($rs_duplicate_pub);
if ($totalpub || $total > 0)
{
echo "Not Available ";
} else {
echo "Available";
}
Use a UNION:
SELECT 'advertisers' AS which, count(*) AS total
FROM advertisers_account
WHERE user_email = '$user_email'
UNION
SELECT 'publishers' AS which, count(*) AS total
FROM publishers_account
WHERE user_email = '$user_email'
This query will return two rows, you can use the which column to tell whether it's advertisers or publishers.
This is how you could do it. you need to use joins, but you should make sure to not let any variables in the query be directly from an outside user like from a form submit. That will open you up to SQL Injection. Use Prepared Statements instead.
select count(*) as total from publishers_account INNER JOIN advertisers_account ON advertisers_account.user_email = publishers_account.user_email WHERE user_email='$user_email'
in response to:
Can someone please tell me how I can connect these two queries so that it is only one?
Why not:
Select
(select count(*) as total from advertisers_account where user_email='$user_email') +
(select count(*) as total from publishers_account where user_email='$user_email') as sumofCount
SELECT count(advertisers_account.id)
FROM publishers_account
LEFT JOIN advertisers_account ON publisher_account.email = advertisers_account.email
WHERE publisher_account.email = '$user_email';
If the count is greater than zero, then the email exists in both tables at least once. If it exists only in the left table (publishers), then the counter would be zero. If it doesn't exist at all in the left table, then you'll get no rows at all, even if it does exist in the right table (advertisers)

Getting total in while statement with UNION query

I am trying to calculate how much a user has earned so it reflects on the users home page so they know how much their referrals have earned.
This is the code I have.
$get_ref_stats = $db->query("SELECT * FROM `members` WHERE `referral` = '".$user_info['username']."'");
$total_cash = 0;
while($ref_stats = $get_ref_stats->fetch_assoc()){
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."' UNION SELECT * FROM `completed_repeat` WHERE `user` = '".$ref_stats['username']."'");
$countr_cash = $get_ref_cash->fetch_assoc();
$total_cash += $countr_cash['cash'];
$countr_c_rate = $setting_info['ref_rate'] * 0.01;
$total_cash = $total_cash * $countr_c_rate;
}
It worked fine when I just had
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."'");
but as soon as I added in the UNION it no longer calculated correctly.
For example, there is 1 entry in completed and 1 entry in completed_repeat both of these entries have a cash entry of 0.75. The variable for $countr_c_rate is 0.10 so $total_cash should equal 0.15 but instead it displays as 0.075 with and without the UNION it acts as if it is not counting from the other table as well.
I hope this makes sense as I wasn't sure how to explain the issue, but I am very unsure what I have done wrong here.
In your second query instead of UNION you should use UNION ALL since UNION eliminates duplicates in the resultset. That is why you get 0.075 instead of 0.15.
Now, instead of hitting your database multiple times from client code you better calculate your cash total in one query.
It might be inaccurate without seeing your table structures and sample data but this query might look like this
SELECT SUM(cash) cash_total
FROM
(
SELECT c.cash
FROM completed c JOIN members m
ON c.user = m.username
WHERE m.referral = ?
UNION ALL
SELECT r.cash
FROM completed_repeat r JOIN members m
ON r.user = m.username
WHERE m.referral = ?
) q
Without prepared statements your php code then might look like
$sql = "SELECT SUM(cash) cash_total
FROM
(
SELECT c.cash
FROM completed c JOIN members m
ON c.user = m.username
WHERE m.referral = '$user_info['username']'
UNION ALL
SELECT r.cash
FROM completed_repeat r JOIN members m
ON r.user = m.username
WHERE m.referral = '$user_info['username']'
) q";
$result = $db->query($sql);
if(!$result) {
die($db->error()); // TODO: better error handling
}
if ($row = $result->fetch_assoc()) {
$total_cash = $row['cash_total'] * $setting_info['ref_rate'];
}
On a side note: make use of prepared statements in mysqli instead of building queries with concatenation. It's vulnerable for sql-injections.
With $countr_cash = $get_ref_cash->fetch_assoc(); you only fetch the first row of your result. However, if you use UNION, you get in your case two rows.
Therefore, you need to iterate over all rows in order to get all values.
Ok, So there is only one row in members table. You are iterating only once on the members table. Then you are trying to get rows using UNION clause which will result in two rows and not one. Then you are just getting the cash column of the first row and adding it to the $total_cash variable.
What you need to do is iterate over the results obtained by executing the UNION query and add the $total_cash variable. That would give you the required result.
$get_ref_stats = $db->query("SELECT * FROM `members` WHERE `referral` = '".$user_info['username']."'");
$total_cash = 0;
while($ref_stats = $get_ref_stats->fetch_assoc()){
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."' UNION SELECT * FROM `completed_repeat` WHERE `user` = '".$ref_stats['username']."'");
while($countr_cash = $get_ref_cash->fetch_assoc()){
$total_cash += $countr_cash['cash'];
}
$countr_c_rate = $setting_info['ref_rate'] * 0.01;
$total_cash = $total_cash * $countr_c_rate;
}

Categories