Optimize a count SQL query on a big table - php

I have a table with over 10 thousand registers right now, and they start to run so slow.
I have the following code:
COUNT
$SqlCount = "SELECT tabnews.New_Id
FROM tabnew WHERE New_Id <> '' AND New_Status = 1";
$QueryCount = mysql_query($SqlCount, $Conn) or die(mysql_error($Conn));
$NumCount = mysql_num_rows($QueryCount);
$recordCount = $NumCount;
PAGINATION
if (!$id) $p = 1;
else $p = $id;
$pageSize = 16;
$itemIni = ($pageSize*$p)-$pageSize;
$totalPage = ceil($recordCount/$pageSize);
SHOW
$Sql52 = "SELECT New_Id, New_Nome, New_Data, New_Imagem FROM tabnews WHERE New_Status = 1 ORDER BY New_Id DESC LIMIT $itemIni, $pageSize ";
$Query52 = mysql_query($Sql52, $Conn);
while($Rs52 = mysql_fetch_array($Query52)){
// ECHO RESULTS
}
MY DATABASE:
CREATE TABLE IF NOT EXISTS `tabnews` (
`New_Id` int(11) NOT NULL AUTO_INCREMENT,
`Franquia_Id` text NOT NULL,
`New_Slide` int(2) NOT NULL,
`Categoria_Id` int(2) NOT NULL,
`New_Nome` varchar(255) NOT NULL,
`New_Data` date NOT NULL,
`New_Imagem` varchar(75) NOT NULL,
`New_Status` int(11) NOT NULL,
PRIMARY KEY (`New_Id`),
KEY `idx_1` (`New_Status`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=10490 ;
Any ideas on how I can make this run faster?
I have a dedicated server running CENTOS.

This:
New_Id <> ''
What does this do? It casts every single one of your INT primary key to string to compare it to a string. Why would you compare it to a string? It cannot be '' by definition, omit that New_Id <> '' from your WHERE clause.

20 seconds is very weird for such a little table.
I have a very similar table with almost 4 million rows and your both SQL statements takes less than 0.002 sec.
CREATE TABLE IF NOT EXISTS `tasks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`status` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'open',
`method` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'GET',
`url` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`params` text COLLATE utf8_unicode_ci,
`response` text COLLATE utf8_unicode_ci,
`executed_by` varchar(50) COLLATE utf8_unicode_ci DEFAULT '',
`execute_at` datetime DEFAULT NULL,
`created` datetime NOT NULL,
`modified` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `status` (`status`),
KEY `modified` (`modified`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3839270 ;
-
SELECT COUNT(id) FROM tasks WHERE status='done';
---> Query took 0.0008 sec
-
SELECT id, status, method, url FROM tasks WHERE status='done' ORDER BY id DESC LIMIT 200, 100;
---> Query took 0.0011 sec
Observations:
You should use SELECT COUNT(New_Id)...
New_id <> '' doesn't make sense. New_id can't be empty or NULL
Set the length of New_Status to something that match the values you store there
Try turning off logging: SET GLOBAL general_log = 'OFF';
Update your server packages (specially MySQL)
Is it a dedicated server only for the database?
Is the server running other things? (run 'top' and 'uptime' to check it status)

Related

Reduce MySQL request time with Codeigniter

I use Codeigniter 3.1.11 and have some code which makes one big query to MySQL with an array. The time of a query with a limit of 30000 is about 9 seconds.
How can I reduce the request time? Maybe by using some indexes on my table, or do you know another method? If I need to use indexes, what indexes would I need to use and how can I use these indexes in my query on Codeigniter?
Code from model:
function rows_update() {
$query = $this->db->order_by('rating', 'DESC')->get_where('members', 'game_rating_and_balance_update_last_time <= now() - INTERVAL 1 DAY', '30000', '0');
$arrUpdateBatchData = [];
while ($row = $query->unbuffered_row('array'))
{
// some code here
$arrUpdateData = [
'UserID' => $row['UserID'],
'game_vault_balance' => $new_game_vault_balance,
'game_available_balance' => $new_game_available_balance,
'rating' => $rating_member,
'game_rating_and_balance_update_last_time' => date('Y-m-d H:i:s')
];
$arrUpdateBatchData[] = $arrUpdateData;
if (count($arrUpdateBatchData) > 500)
{
$this->db->update_batch('members', $arrUpdateBatchData, 'UserID');
$arrUpdateBatchData = [];
}
}
//update last items
if (count($arrUpdateBatchData) > 0)
{
$this->db->update_batch('members', $arrUpdateBatchData, 'UserID');
$arrUpdateBatchData = [];
}
return;
}
Raw query to MySQL with update_batch (I simply write only one row from an array):
UPDATE members SET game_vault_balance = CASE WHEN UserID = '9915075' THEN 803.60516004772 ELSE game_vault_balance END, game_available_balance = CASE WHEN UserID = '9915075' THEN 4.1253850908788 ELSE game_available_balance END, rating = CASE WHEN UserID = '9915075' THEN 0.24 ELSE rating END, game_rating_and_balance_update_last_time = CASE WHEN UserID = '9915075' THEN '2020-07-24 22:00:36' ELSE game_rating_and_balance_update_last_time END WHERE UserID IN('9915075')
Table structure:
CREATE TABLE `members` (
`id` int(10) UNSIGNED NOT NULL,
`UserID` varchar(64) NOT NULL,
`telegram_id` varchar(64) DEFAULT NULL,
`first_name` varchar(64) DEFAULT NULL,
`last_name` varchar(64) DEFAULT NULL,
`language` varchar(64) DEFAULT NULL,
`currency` varchar(64) DEFAULT NULL,
`status` varchar(64) DEFAULT NULL,
`rating` varchar(64) DEFAULT NULL,
`game_vault_balance` decimal(32,8) DEFAULT 0.00000000,
`game_available_balance` decimal(32,8) DEFAULT 0.00000000,
`game_rating_and_balance_update_last_time` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`created` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Indexes of this table:
ALTER TABLE `members`
ADD PRIMARY KEY (`id`),
ADD UNIQUE KEY `UserID` (`UserID`) USING BTREE;
AUTO_INCREMENT for the members table:
ALTER TABLE `members`
MODIFY `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT;
COMMIT;

PHP SQL query slow?

Here is my function which i am using to un-follow users.It first DELETE the relationship between users and all the notifications that are related to this relationship.Then it INSERT a new notification for user which we are going to un-follow and then UPDATE his followers count (as one follower has left).I am using multi_query and this query seems to be bit slower on large database and i want to know whether it's a good practice or not or is there is any more complex form of query to get the job done.
PHP Function
// 'By' is the array that hold logged user and 'followed' is the user id which we are going to unfollow
function unFollowUser($followed,$by) {
$following = $this->getUserByID($followed);// Return fetch_assoc of user row
if(!empty($following['idu'])) { // if user exists
// return user followers as number of rows
$followers = $this->db->real_escape_string($this->numberFollowers($following['idu'])) - 1;
$followed_esc = $this->db->real_escape_string($following['idu']);
$by_user_esc = $this->db->real_escape_string($by['idu']);
// delete relationship
$query = "DELETE FROM `relationships` WHERE `relationships`.`user2` = '$followed_esc' AND `relationships`.`user1` = '$by_user_esc' ;" ;
// delete notification (user started following you )
$query.= "DELETE FROM `notifications` WHERE `notifications`.`not_from` = '$by_user_esc' AND `notifications`.`not_to` = '$followed_esc' ;" ;
// Insert a new notification( user has unfollowed you)
$query.= "INSERT INTO `notifications`(`id`, `not_from`, `not_to`, `not_content_id`,`not_content`,`not_type`,`not_read`, `not_time`) VALUES (NULL, '$by_user_esc', '$followed_esc', '0','0','5','0', CURRENT_TIMESTAMP) ;" ;
// update user followers (-1)
$query .= "UPDATE `users` SET `followers` = '$followers' WHERE `users`.`idu` = '$followed_esc' ;" ;
if($this->db->multi_query($query) === TRUE) {
return 1;
} else {
return 0;
}
} else {
return 0;
}
}
Table structures
--
-- Table structure for table `notifications`
--
CREATE TABLE IF NOT EXISTS `notifications` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`not_from` int(11) NOT NULL,
`not_to` int(11) NOT NULL,
`not_content_id` int(11) NOT NULL,
`not_content` int(11) NOT NULL,
`not_type` int(11) NOT NULL,
`not_read` int(11) NOT NULL,
`not_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
--
-- Table structure for table `relationships`
--
CREATE TABLE IF NOT EXISTS `relationships` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user1` int(11) NOT NULL,
`user2` int(11) NOT NULL,
`status` int(11) NOT NULL,
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
--
-- Table structure for table `users`
--
CREATE TABLE IF NOT EXISTS `users` (
`idu` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(32) NOT NULL,
`password` varchar(256) NOT NULL,
`email` varchar(256) NOT NULL,
`first_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`last_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
`verified` int(11) NOT NULL,
`posts` text CHARACTER SET utf32 NOT NULL,
`photos` text CHARACTER SET utf32 NOT NULL,
`followers` text CHARACTER SET utf32 NOT NULL,
UNIQUE KEY `id` (`idu`),
UNIQUE KEY `idu` (`idu`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
In my testing, multi_query has been the fastest way to execute multiple different queries. Why do you feel it's running slow? Compared to what?
Anyway, improvements could come from adding indexes to some of the columns you search frequently:
relationships.users2
relationships.users1
notifications.not_from
notifications.not_to
users.idu
Adding indexes makes searching faster, but it has at least two downsides:
Makes the DB a lot more resource hungry, which could affect your server performance
Makes writing operations take longer
I don't see any problem with your current queries. Really consider whether the slow performance you're seeing comes from the DB queries themselves, or from the rest of your PHP process. Try measuring the script time with the queries, then skipping the queries and taking another measurement (you could hardcode query results). It will give you an idea of whether the slowness is attributable to something else.
Either way, benchmark.
Try creating index on user where deletes are running , this may speed up query

MySQL query not returning any results even though data exists in database

Why is this query not showing more than one data. Even I have 10/12 data but this line only shows 1. Check I have limited it to 3 but it only shows 1.
$getAds = mysql_query("SELECT *
FROM advertises
WHERE status='RUNNING'
AND adult='0'
AND (country LIKE '%$test%' OR country='ALL')
AND (device LIKE '%$pabu%' OR device='ALL')
ORDER BY rand()
LIMIT 0,3");
my database structure
Table structure for table advertises --
CREATE TABLE `advertises` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userid` int(11) NOT NULL, `name` varchar(5000) NOT NULL, `url` varchar(5000) NOT NULL, `type` varchar(500) NOT NULL, `device` varchar(500) NOT NULL, `country` varchar(500) NOT NULL, `time` varchar(500) NOT NULL, `status` varchar(500) NOT NULL, `dset` varchar(500) NOT NULL, `cset` varchar(500) NOT NULL, `acpc` varchar(500) NOT NULL, `ucpc` varchar(500) NOT NULL, `adult` tinyint(10) unsigned NOT NULL DEFAULT '0', `title` varchar(500) NOT NULL, `pcpc` varchar(500) NOT NULL DEFAULT '0', `spent` varchar(500) NOT NULL DEFAULT '0', `adc` varchar(500) NOT NULL DEFAULT '0', PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=8 DEFAULT CHARSET=latin1;
------ -- -- Dumping data for table advertises --
INSERT INTO `advertises` VALUES ('1','1','dqui.com/adultb.php','lt.adqui.com','banner','ALL','ALL','Saturday , September 19 , 2015','RUNNING','no','no','0.015','0.012','1','Android Adult','0.005','49999.925',''); INSERT INTO `advertises` VALUES ('2','1','http://adqui.com/banner.php','http://kid.adqui.com','banner','ALL','ALL','Saturday , September 19 , 2015','RUNNING','no','no','0.01','0.008','0','Android Non','0.002','49999.88','');
INSERT INTO `advertises` VALUES ('3','1','qui.com/adultb.php','lt.adqui.com','banner','ALL','ALL','Saturday , September 19 , 2015','RUNNING','no','no','0.0002','0.0002','1','Adult','0.00','4999.874','0.0002'); INSERT INTO `advertises` VALUES ('4','1','adqu i.com/banner.php','kid.adq ui.com','banner','ALL','ALL','Saturday , September 19 , 2015','RUNNING','no','no','0.0002','0.002','0','non','','4999.923','0.0008');
Status is a keyword in mysql. so try this
$getAds=mysql_query("SELECT * FROM advertises WHERE `status`='RUNNING' AND adult='0' AND (country LIKE '%test%' OR country='ALL') AND (device LIKE '%pabu%' OR device='ALL') ORDER BY rand() LIMIT 0,3");
Have you tried to use mysql_fetch_assoc on the result returned by mysql_query?
Refer to the mysql-query manual in particular Example #2
mysql_query() has already been deprecated, you can use PDO instead.
Try below,
<?php
$dbcon = new PDO("mysql:host=localhost;dbname=test", 'root', '');
$test = 'All';
$pabu = 'All';
$qry = "SELECT *
FROM advertises
WHERE status='RUNNING'
AND adult='0'
AND (country LIKE '%$test%' OR country='ALL')
AND (device LIKE '%$pabu%' OR device='ALL')
ORDER BY rand()
LIMIT 0,3";
$rq = $dbcon->query($qry);
$op = ( $rq ) ? $rq->fetchAll(PDO::FETCH_ASSOC) : '';
echo '<pre>'; print_r($op);
?>

do query in another query, if another query not exists return 0 and compare value between 2 query

I have searched over and over but still can't write a correct query that actually work ! :|
I write this one and it seems ok but it's dosen't worked...
select
s.surf_id,
s.surf_dailyuser,
s.surf_url,
s.surf_cpc,
(if exists(
select surfed_count from `surfed` where s.surf_id = surfed_site and surfed_date = 'today'
) then select surfed_count as surfedcount;
else select 0 as surfedcount; end if
)
from `surfs` s where s.surf_status = 1 and surfedcount < s.surf_dailyuser order by s.surf_rand limit 1
any suggestion will be a big help :)
tables are like this
surfs
CREATE TABLE IF NOT EXISTS `surfs` (
`surf_id` int(11) NOT NULL,
`surf_user` int(11) NOT NULL,
`surf_title` varchar(128) CHARACTER SET latin1 NOT NULL,
`surf_url` varchar(500) CHARACTER SET latin1 NOT NULL,
`surf_dailyuser` int(11) NOT NULL,
`surf_cpc` int(11) NOT NULL,
`surf_status` int(11) NOT NULL,
`surf_date` varchar(32) NOT NULL,
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
surfed
CREATE TABLE IF NOT EXISTS `surfed` (
`surfed_id` int(11) NOT NULL,
`surfed_code` int(11) NOT NULL,
`surfed_user` int(11) NOT NULL,
`surfed_site` int(11) NOT NULL,
`surfed_count` int(11) NOT NULL,
`surfed_date` int(11) NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;
You can use CASE statement as below
select
s.surf_id,
s.surf_dailyuser,
s.surf_url,
s.surf_cpc,
CASE WHEN
(select surfed_count from `surfed` where s.surf_id = surfed_site and surfed_date = 'today') = 1 then (select surfed_count as surfedcount)
ELSE 0 END AS surfedcount
from `surfs` s where s.surf_status = 1 and surfedcount < s.surf_dailyuser order by s.surf_rand limit 1

MySQL delete troubleshooting

I restarted the MySQL service and I attempted to use my PHP programs delete function to delete an existing row but I'm finding although the delete queries were counted the row was not deleted. I tried applying on delete cascade to the foreign key of the child table but that did not seem to have an effect. I'm wondering why the delete would be doing nothing.
CREATE TABLE `customers` (
`idcustomers` int(11) NOT NULL AUTO_INCREMENT,
`firstname` varchar(45) DEFAULT NULL,
`lastname` varchar(45) DEFAULT NULL,
`address1` varchar(45) DEFAULT NULL,
`address2` varchar(45) DEFAULT NULL,
`city` varchar(45) DEFAULT NULL,
`state` varchar(45) DEFAULT NULL,
`zip` varchar(45) DEFAULT NULL,
`phone` varchar(45) DEFAULT NULL,
`email` varchar(45) DEFAULT NULL,
`cell` varchar(45) DEFAULT NULL,
PRIMARY KEY (`idcustomers`),
UNIQUE KEY `idcustomers_UNIQUE` (`idcustomers`)
) ENGINE=InnoDB AUTO_INCREMENT=54 DEFAULT CHARSET=latin1
CREATE TABLE `events` (
`idevents` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(250) DEFAULT NULL,
`start` datetime DEFAULT NULL,
`end` datetime DEFAULT NULL,
`allday` varchar(50) DEFAULT NULL,
`url` varchar(1000) DEFAULT NULL,
`customerid` int(11) NOT NULL,
`memo` longtext,
`dispatchstatus` varchar(45) DEFAULT NULL,
PRIMARY KEY (`idevents`),
KEY `FK_events` (`customerid`),
CONSTRAINT `FK_events` FOREIGN KEY (`customerid`) REFERENCES `customers` (`idcustomers`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1
Com_delete 2
The PHP looks like this:
<?php
session_start();
date_default_timezone_set("America/Los_Angeles");
if($_SESSION['loggedin'] != TRUE)
{
header("Location: index.php");
}
require_once('../php.securelogin/include.securelogin.php');
$mysqli = new mysqli($ad_host, $ad_user, $ad_password, "samedaycrm");
if ($mysqli->connect_errno) {
printf("Connect failed: %s\n", $mysqli->connect_error);
exit();
}
$customerid = $_SESSION['customer_id'];
$tSQL = "delete from events where customerid = \"$customerid\"";
$result = $mysqli->query($tSQL);
$tSQL = "delete from customers where idcustomers = \"$customerid\"";
$result = $mysqli->query($tSQL);
echo $mysqli->error;
?>
Assuming that the customerid and idcustomers columns are both numeric it should be fine. You should not need to quote the variables in those queries btw, then you wouldnt need to escape them. You may try:
$tSQL = "delete from events where customerid = $customerid";
but it should not be any different than what you used already. Of course if you are not sure of the type of the column you can use:
$tSQL = "delete from events where customerid = '".$customerid."'";
or you can get away with:
$tSQL = "delete from events where customerid = '$customerid'";
but I have always hated that for some reason.
if all of that fails troubleshoot by spitting out the $customerid (or even the whole $tSQL) variable and then trying the query manually in phpmyadmin or toad or whatever db client you use, and see what it tells you. If it just says 0 rows affected, then run it like a select instead. Tailor to fit.

Categories