I was just hoping someone could help me speed up 4 queries with a multi query.
GOAL: a single multi query to function as the single queries below.
Simple queries, i am checking one table to see if user is banned, then if not, i am getting row for the id and updating it's view count by 1. If user is banned, i do not want the last to queries to complete.
Thank you in advance for your help.
current performance is around 1200ms. (+1000ms avg for facebook graph api query).
NOTE: af_freefeed.pageid & af_ban.pageid are both indexed in database.
ALSO: I have been studying and referencing from http://www.php.net/manual/en/mysqli.multi-query.php i just can not see how to get this config into multi with the if()
$fconn = new mysqli($fdbhost, $fdbuser, $fdbpass, $fdbname) or die ('Error connecting to mysqli');
// 12,000 rows for af_ban - bigint(255) : indexed
$q = sprintf('SELECT COUNT(pageid) AS numrowst FROM af_ban WHERE pageid = %s', $banpage);
$readtop = $fconn->query($q);
$rowtop = $readtop->fetch_assoc();
// 1.17 million rows for af_freefeed - bigint(255) : indexed
if($rowtop[numrowst] == 0){
$q = sprintf('SELECT COUNT(pageid) AS numrowsf FROM af_freefeed WHERE pageid = %s', $banpage);
$readf = $fconn->query($q);
$rowf = $readf->fetch_assoc();
// increment views
$read = $fconn->query("Update af_freefeed SET views = views + 1 WHERE pageid = ".$banpage."");
}
$q=$fconn->query("SELECT pagename,views,pageid FROM af_freefeed ORDER BY views DESC LIMIT 0, 20");
unset($q);
unset($rowf);
unset($rowtop);
mysqli_close($fconn);
actual request times.
grah api: 1127.04610825ms.
conncect: 1.20711326599ms.
check banned: 0.405788421631ms.
get row: 418.189229965ms.
increment views: 472.24655151ms.
get top20: 94.31447983ms.
Multi_query #1 How to stop the multi query if user is banned?
Possible Contender: 943.8181ms. if added : 933.1279ms. if banned
10ms difference if exit loop for banned. This leads me to believe the loop is completing all the queries before they are actually supposed to be executed, "next_result". Or i have an error in how i looped the functions.
replaced exit; with $thread_id = $fconn->thread_id; $fconn->kill($thread_id);
if banned 953.4719ms. no gain.
$banpage='234232874008';
$query = "SELECT pagename,views,pageid FROM af_freefeed ORDER BY views DESC LIMIT 0, 2;";
$query .= "SELECT pageid AS isbanned FROM af_ban WHERE pageid = \"".$banpage."\";";
$query .= "SELECT pageid AS isadded FROM af_freefeed WHERE pageid = \"".$banpage."\";";
$query .= "Update af_freefeed SET views = views + 1 WHERE pageid = \"".$banpage."\"";
/* execute multi query */
if ($fconn->multi_query($query)) {
if ($result = $fconn->store_result()) {
while ($row = $result->fetch_row()) {
print_r($row).'<br />';
}
$result->free();
}
if ($fconn->more_results()) {
while ($fconn->next_result()){
if($thisresult = $fconn->store_result()){
while (is_array($row = $thisresult->fetch_array())) {
if(isset($row['isbanned'])){
if($row['isbanned']===''.$banpage.''){
$thread_id = $fconn->thread_id;
$fconn->kill($thread_id);
// exit;
}
}
}
}
}
}
}
unset($query);
unset($result);
unset($thisresult);
Multi_query #2 "current for benchmark" How to remove duplicate fields in result set after next_result()?
2.667ms. / 1032.2499ms. but print_r is showing duplicate fields in $thisresults?
**Array
(
[0] => 37
[id] => 37
[1] => 159616034235
[pageid] => 159616034235
[2] =>
[userid] =>
[3] => 30343
[views] => 30343
[4] => Walmart
[pagename] => Walmart
)**
$query = "SELECT pageid AS isbanned FROM af_ban WHERE pageid = \"".$banpage."\";";
$query .= "SELECT pageid AS isadded FROM af_freefeed WHERE pageid = \"".$banpage."\";";
$query .= "SELECT * FROM af_freefeed ORDER BY views DESC LIMIT 0, 20";
//$query .= "Update af_freefeed SET views = views + 1 WHERE pageid = \"".$banpage."\"";
/* execute multi query */
echo '<pre>';
$i=0;
if ($fconn->multi_query($query)) {
if ($result = $fconn->store_result()) {
//$row = $result->fetch_assoc();
while ($row = $result->fetch_assoc()) {
print_r($row).'<br />';
}
$result->free();
}
if ($fconn->more_results()) {
while ($fconn->next_result()){
if($thisresult = $fconn->store_result()){
while ($row2 = $thisresult->fetch_array()) {
if(isset($row2['isadded'])){
if($row2['isadded']===''.$banpage.''){
$addone = $fconn->query("Update af_freefeed SET views = views + 1 WHERE pageid = ".$banpage."");
}
}
print_r($row2);
}
}
}
}
}
/* determine our thread id */
$thread_id = $fconn->thread_id;
/* Kill connection */
$fconn->kill($thread_id);
//
echo '</pre><hr/>';
Try to use index in getting count. Like for example COUNT(pageid). It will speed up your query.
Update
You can also try this link for further explanation
EDIT : So now, the conclusion: (test case below)
You cannot control the execution of subsequent statements of a multi-statement query.
You can therefore not use multi_query() in the way you wanted to.
Execute them all, or execute none.
Regarding
Multi_query #2 "current for benchmark" How to remove duplicate fields in result set after next_result()?
Use fetch_assoc() or fetch_array(MYSQLI_ASSOC) (both practically the same) instead of fetch_array().
About multi_query():
I recently worked on a program using the MySQL C API, which mysqli uses, too.
About multiple-statement query support the documentation states:
Executing a multiple-statement string can produce multiple result sets or row-count indicators. Processing these results involves a different approach than for the single-statement case: After handling the result from the first statement, it is necessary to check whether more results exist and process them in turn if so. To support multiple-result processing, the C API includes the mysql_more_results() and mysql_next_result() functions. These functions are used at the end of a loop that iterates as long as more results are available. Failure to process the result this way may result in a dropped connection to the server.
(emphasize added)
This leads to the conclusion, that aborting a multiple-statement query is not an intended feature.
Moreover, I didn't find any resource explaining when subsequent queries are actually executed.
Calling next_result() doesn't neccessarily mean that the query hasn't been executed already.
EDIT : TEST CASE
To prove what I previously assumed, I created a test case:
<?php
$db = new mysqli('localhost', 'root', '', 'common');
$query = 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
$query .= 'SELECT NOW() as time;';
if($db->multi_query($query)) {
// Current time
echo "'multi_query()' executed at:\n\t\t"
.date('Y-m-d H:i:s')."\n";
// First result
if($result = $db->store_result()) {
$i = 1;
$row = $result->fetch_assoc();
echo "'NOW()' of result $i:\n\t\t".$row['time']."\n";
$result->free();
// Wait 5 seconds
sleep(5);
// Subsequent results
while($db->more_results() && $db->next_result()) {
$result = $db->store_result();
$row = $result->fetch_assoc();
$i++;
echo "'NOW()' of result $i:\n\t\t".$row['time']."\n";
// Wait 5 seconds
sleep(5);
$result->free();
}
}
}
$db->close();
?>
This results in:
'multi_query()' executed at:
2013-05-10 10:18:47
'NOW()' of result 1:
2013-05-10 10:18:47
'NOW()' of result 2:
2013-05-10 10:18:47
'NOW()' of result 3:
2013-05-10 10:18:47
'NOW()' of result 4:
2013-05-10 10:18:47
Given that, it is obvious that all four statements of the query were executed directly after the call to multi_query().
If they were only executed after calling next_result() there would be a 5 second delay caused by sleep(5) calls I added between the loop iterations.
Please run following query in mysql and check your query run time :
CREATE INDEX pagidIndex ON af_ban (pageid(11));
CREATE INDEX pagidFeedIndex ON af_freefeed (pageid(11));
CREATE INDEX viewsIndex ON af_freefeed (views(11));
i am checking one table to see if user is banned, then if not, i am getting row for the id and updating it's view count by 1.
The following query may help you to update the view count. I assume that you already know the page_id.
UPDATE af_freefeed SET views=views+1 WHERE page_id=%s and page_id not in (select page_id from af_ban WHERE page_id=%s);
You could try something along these lines:
$sql = sprintf("SELECT af_freefeed.pageid FROM af_freefeed left join af_ban ".
"on (af_freefeed.pageid = af_ban.pageid) ".
"where af_freefeed.pageid = %s and ".
"af_ban.pageid is null limit 1", $pageid);
to replace your first two queries.
The existence of a record in the results should indicate an unbanned user requesting the resource. Then you can do your update your views.
Hope this helps.
Related
I have a simple php code below
$sql_items = "SELECT id FROM item_masterfile"; /* this query has 7000 rows */
$result_items = mysqli_query($con,$sql_items);
while ($row_items = mysqli_fetch_row($result_items)) {
$sql_qty = "SELECT qty FROM inventory WHERE id = ".$row_items[0];
/* rest of the code here */
}
}
this is working but due to a lot data my server cannot handle it and other queries took long to respond. how can I fix this? My target here is like batch select? to prevent clogged?
What I see in the process is a lot of select waiting to initiate.
How can I fix this?
try with batches using limit and offset like below
$offset = $i * 5000; // $i is run batch number .. 1, 2, 3 etc in the foreach() loop.
if ($i == 0) {
$offset = "";
}
$result_items="SELECT id FROM item_masterfile LIMIT 5000 OFFSET $offset;";
The code in your question shows that there is one_to_one or one_to_many relation between tables, so using pagination and join statement would resolve the problem.
Check below code hope to be helpful
$sql = "
SELECT im.id, i.qty
FROM item_masterfile AS im
JOIN inventory AS i ON
im.id = i.item_masterfile_id
LIMIT $offset, $limit
";
$result_items = mysqli_query($con,$sql);
you can set $offset and $limit dynamically in your code and go on ...
Instead of using the loop and providing id in there, you should collect all the ids in an array and then pass all of them to the query in one go using the IN operator.
Example: SELECT qty FROM inventory WHERE id IN (1,2,3,4,5). By doing this you can avoid loop and you code will not exit with timeout error.
OR
You can achieve the same using a subquery with your main query
Example: SELECT qty FROM inventory WHERE id IN (SELECT id FROM item_masterfile)
Try with follow step:
get result of first query and get id values like (1,2,3,4,...)..
and out side where clause execute second query with WHERE condition IN clause
Like
$sql_items = "SELECT id FROM item_masterfile"; /* this query has 7000 rows */
$result_items = mysqli_query($con,$sql_items);
while ($row_items = mysqli_fetch_row($result_items)) {
$ids[] = $row_items['id'];
}
$sql_qty = "SELECT qty FROM inventory WHERE id IN ".$ids;
/* rest of the code here */
}
I am trying to get single rank for a user for each stat "column" in the table. I am trying to do this as more efficiently because i know you can.
So I have a table called userstats. in that table i have 3 columns user_id, stat_1, stat_2, and stat_3. I want to me able to get the rank for each stat for the associated user_id. with my current code below i would have to duplicate the code 3x and change the column names to get my result. please look at the examples below. Thanks!
this is how i currently get the rank for the users
$rankstat1 = getUserRank($userid);
<code>
function getUserRank($userid){
$sql = "SELECT * FROM ".DB_USERSTATS." ORDER BY stat_1 DESC";
$result = mysql_query($sql);
$rows = '';
$data = array();
if (!empty($result))
$rows = mysql_num_rows($result);
else
$rows = '';
if (!empty($rows)){
while ($rows = mysql_fetch_assoc($result)){
$data[] = $rows;
}
}
$rank = 1;
foreach($data as $item){
if ($item['user_id'] == $userid){
return $rank;
}
++$rank;
}
return 1;
}
</code>
I believe there is a way for me to get what i need with something like this but i cant get it to work.
$rankstat1 = getUserRank($userid, 'stat_1'); $rankstat2 =
getUserRank($userid, 'stat_2'); $rankstat3 = getUserRank($userid,
'stat_3');
You can get all the stat ranks using one query without doing all the PHP looping and checking.
I have used PDO in this example because the value of the $userid variable needs to be used in the query, and the deprecated mysql database extension does not support prepared statements, which should be used to reduce the risk of SQL injection.
The function could be adapted to use the same query with mysqli, or even mysql if you must use it.
function getUserRanks($userid, $pdo) {
$sql = "SELECT
COUNT(DISTINCT s1.user_id) + 1 AS stat_1_rank,
COUNT(DISTINCT s2.user_id) + 1 AS stat_2_rank,
COUNT(DISTINCT s3.user_id) + 1 AS stat_3_rank
FROM user_stats f
LEFT JOIN user_stats s1 ON f.stat_1 < s1.stat_1
LEFT JOIN user_stats s2 ON f.stat_2 < s2.stat_2
LEFT JOIN user_stats s3 ON f.stat_3 < s3.stat_3
WHERE f.user_id = ?"
$stmt = $pdo->prepare($sql);
$stmt->bindValue(1, $userid);
$stmt->execute();
$ranks = $stmt->fetchObject();
return $ranks;
}
This should return an object with properties containing the ranks of the given user for each stat. An example of using this function:
$pdo = new PDO($dsn, $user, $pw);
$ranks = getUserRanks(3, $pdo); // get the stat ranks for user 3
echo $ranks->stat_2_rank; // show user 3's rank for stat 2
$sql = "SELECT user_id, stat_1, stat_2, stat_3 FROM ".DB_USERSTATS." ORDER BY stat_1 DESC";
Also, unless there is a reason you need ALL users results, limit your query with a WHERE clause so you're only getting the results you actually need.
Assuming you limit your sql query to just one user, this will get that user's stats.
foreach($data as $item){
$stat_1 = $item['stat_1'];
$stat_2 = $item['stat_2'];
$stat_3 = $item['stat_3'];
}
If you get more than one user's stats with your sql query, consider passing your $data array back to the calling function and loop through the array to match the users stats to particular user id's.
I have a SELECT statement that pulls a limited number of items based on the value of one of the fields. (ie ORDER BY rate LIMIT 15).
However, I need to do some comparisons that and change the value of rate, and subsequently could alter the results that I want.
I could pull everything (without the LIMIT), alter the rate, re-sort, and then just process the number that I need. However, I don't know if it's possible to alter values in a php result array. I'm using:
$query_raw = "SELECT dl.dragon_list_id, dl.dragon_id, dl.dragon_name, dl.dragon_level, d.type, d.opposite, d.image, dr.dragon_earn_rate
FROM dragon_list dl
LEFT JOIN dragons d ON d.dragon_id = dl.dragon_id
LEFT JOIN dragon_rates dr ON dr.dragon_id = dl.dragon_id
AND dr.dragon_level = dl.dragon_level
WHERE dl.dragon_id IN (
SELECT dragon_id
FROM dragon_elements
WHERE element_id = 3
)
AND dl.dragon_list_id NOT IN (
SELECT dh.dragon_list_id
FROM dragon_to_habitat dh, dragon_list dl
WHERE dl.user_id = 1
AND dh.dragon_list_id = dl.dragon_list_id
AND dl.is_deleted = 0
)
AND dl.user_id = " . $userid . "
AND dl.is_deleted = 0
ORDER BY dr.dragon_earn_rate DESC, dl.dragon_name
LIMIT 15;";
$query = mysqli_query($link, $query_raw);
if (!$query) {
echo "DB Error, could not query the database\n";
echo 'MySQL Error: ' . mysqli_error($link);
exit;
}
$d = mysqli_fetch_array($d_query);
Well, after a lot of research and some trial and error I found my answers....
Yes, I CAN alter the result rows using something like:
$result['field'] = $newvalue;
I also learned I could reset the pointer by using:
mysqli_data_seek($d_query,0);
However, when I reset the counter, I lost the changes I made. So ultimately, I'm still a little stuck, but individually I had the answers.
I need to return data from 2 separate tables at the same time. The info I need from the 2nd table is determined by what is returned from the first table. Here's what I'm working with..
$query = "SELECT * FROM pending WHERE paymentid = '".$_GET['vnum']."'";
$result = mysql_query($query);
$num = #mysql_num_rows($result);
$linkid = $res['paymentid'];
if ($num==0) {
echo "Hello, ".$_SESSION['Fname']."<br />There was an error, I cannot find this payment in the records.";
} else {
$picquery = mysql_query("SELECT * FROM _uploads_log WHERE linkid = '".$linkid."'");
$numb = #mysql_num_rows($picquery);
if ($numb==0) {
echo "there is no picture"; }
else {
echo "<img src=\"".$res['log_filename']."\" width=\"100\">"; }
I don't understand how to return the results as an array, if $res[] returns the results for the first query, then what returns the results for the second one?
or is there a better way to do this entirely?
Thank you
You need to do a join, but in order to still get results from your first query even if there is no picture (I assume that's why you split it up) you want a left join.
select * from pending left join _uploads_log on pending.paymentID=$_GET['vnum'] and _uploads_log.linkid = pending.paymentID
(note: php markings removed for readability - you'll have to add them back in)
This should (untested since I don't have your tables) return the full row for your vnum variable and also include the picture data if there is one.
I have a table containing 4 articles with id 1,2,3 and 4 as well as ordering value 1,2,3,4.
They have separate columns for their title, image etc. I need to get them distinctly with where clause. So i did:
For article 1:
//topstory1
$sql_topstory1 ="SELECT * FROM topstory WHERE story_active='1' && story_order='1'";
$result_topstory1 = mysql_query($sql_topstory1);
$row_topstory1 = mysql_fetch_array($result_topstory1);
$story1_title = $row_topstory1['story_title'];
$story1_abstract = $row_topstory1['story_text'];
And for article 2
//topstory2
$sql_topstory2 ="SELECT * FROM topstory WHERE story_active='1' && story_order='2'";
$result_topstory2 = mysql_query($sql_topstory2);
$row_topstory2 = mysql_fetch_array($result_topstory2);
$story2_title = $row_topstory2['story_title'];
$story2_abstract = $row_topstory2['story_text'];
As I have to reuse them in a page.
PROBLEM IS, the first query works but the second one doesn't. It seems like MySql cannot execute two consecutive queries on the same table in a single php file. But I think there is a simple solution to this...
Please help me soon :( Love you guys :)
There are several possible reasons for the second query to fail, but the fact that it's the second query in the file does not cause it to fail.
I would expect that article 2 does not have the active flag set to 1, causing you to get an empty result set.
Another option is that you may have closed the mysql connection after the first query, then you can't execute another query. (General rule: don't close database connections. PHP takes care of that.)
Why not just get them both with 1 query?
$sql_topstory ="SELECT * FROM topstory WHERE story_active='1' && story_order IN(1, 2) ORDER BY story_order DESC";
$result_topstory = mysql_query($sql_topstory) or trigger_error('Query Failed: ' . mysql_error());
while ($row = mysql_fetch_assoc($result_topstory)) {
$title[] = $row['story_title'];
$abstract[] = $row['story_abstract'];
}
// Then to display
echo 'Story 1 is ' . $title[0] . ' with an abstract of ' . $abstract[1];
There are plenty of ways to do this, this is just a simple demonstration.
$query = <<<SQL
SELECT
story_title
, story_text
FROM
topstory
WHERE
story_active
ORDER BY
story_order ASC
SQL;
$result = mysql_query($query);
$stories = array();
while ($row = mysql_fetch_assoc($result)) {
$stories[] = $row;
}
Now you have an array of stories like so:
array(
0 => array(
'story_title' => ?
, 'story_text' => ?
)
, 1 => array(
'story_title' => ?
, 'story_text' => ?
)
)
Should be pretty easy to iterate through.