how would I count the number of sql queries executed on one page load?
I have a similar script to time taken for page to be generated, but not for how many queries have been executed.
You know what I mean, such as on SMF forums, in the footer, they have:
Page created in 0.136 seconds with 7 queries.
in the footer?
Replacing all of the mysql_query(ies) isn't really an option, there are way too many mysql_queries to replace, although I could spent a day doing it if needs be.
Thanks
SHOW SESSION STATUS LIKE 'Questions'
SMF does its query counting by having its own custom query function:
function db_query($db_string, $file, $line)
{
global $db_cache, $db_count, $db_connection, $db_show_debug, $modSettings;
// One more query....
$db_count = !isset($db_count) ? 1 : $db_count + 1;
...
The simplest way to achieve what you're trying to do would be to do the same; make a wrapper for mysql_query and use that instead of mysql_query.
Here's an example which might be easier to follow than the SMF one.
class QueryLogger
{
public $queries = array();
public function query($sql)
{
$start = microtime(true);
$query = mysql_query($sql);
$queries[] = microtime(true) - $start;
return $query;
}
public function getCount()
{
return sizeof($this->queries);
}
public function getTime()
{
return array_sum($this->queries);
}
}
$queryLogger = new QueryLogger;
$query1 = $queryLogger->query('...');
$query2 = $queryLogger->query('...');
echo 'Ran '.$queryLogger->getCount().' queries in '.$queryLogger->getTime().' seconds.';
I think the following command is a per session count:
SHOW STATUS LIKE 'com_select%'
After Quassnoi comment i put this in the start of script:
$res = mysql_query("SHOW SESSION STATUS LIKE 'Questions'");
$row = mysql_fetch_array($res, MYSQL_ASSOC);
define("START_QUERIES",$row['Value']);
and this:
$res = mysql_query("SHOW SESSION STATUS LIKE 'Questions'");
$row = mysql_fetch_array($res, MYSQL_ASSOC);
define("STOP_QUERIES",$row['Value']);
you can see total num of queries with:
echo "No of queries: ".(STOP_QUERIES-START_QUERIES-1);
I don't know why use -1, maybe it counts mysql_select_db statement (or smth) also, but it works pretty good on my localhost
You can get the number of queries ever executed by calling.
show session status like "Queries";
Call this at the beginning and at the end of page creation, and then you can see how many queries there have been. Don't forget that this command itself is also counted as one.
To count the number of queries, you need to count the number of queries. Sorry to sound redundant, but it really is that simple.
Make a counting function (mysql_query_counted?), then use grep to search through your codebase for mysql_query(, and it shouldn't take more than an hour or two. Possibly even think about using sed or similar to replace the function calls.
A note on SMF and similar that have this built-in. They use DB abstraction layers, so they are already using their own query function, and adding query counting at a later date would have been as simple as adding a line incrementing a counter to to that function. +1 for abstraction and encapsulation I suppose.
Related
I am using php to get special records from Database.
which one is better?
1.
Select * From [table] Limit 50000, 10;
while($row = $stmt->fetch()){
//save in array, total 10 times
}
or
2.
Select * From [table];
$start = 50000;
$length = 10;
while($row = $stmt->fetch()){
if($i < $start+$length && $j >=$start){
//save in array, total 50010 times
}
}
In this case, which one should I use?
Which one using DB with less resources?
which one is better?
Too vague: what is "better"?
Which one using DB with less resources?
You're much better off with the first approach. It's efficient to select as little data as you need and no more. Selecting the whole table will force your script to use a lot more memory because all that data needs to be kept live
The best answer you'll get is: test! You can run your queries multiple times in multiple ways and see for yourself. Just use SELECT SQL_NO_CACHE... instead of the generic SELECT... to force the DB to restart the work from scratch. Measure how long it takes to run the query and process results
function wayOne(){
// execute your 1st query and loop through results
}
function wayTwo(){
// execute 2nd query and loop through results
}
//Measures # of milliseconds it takes to execute another function
function timeThis(callable $callback){
$start_time = microtime();
call_user_func($callback);
$microsecs = microtime()-$start_time; //duration in microseconds
return round($microsecs*1000);//duration in milliseconds
}
$wayOneTime = timeThis('wayOne');
$wayTwoTime = timeThis('wayTwo');
You can then compare the two times. Generally (not always) a process that takes significantly less time uses fewer resources
I have quite an issue I can not seem to solve. I am trying to get a row count from a select statement.
I should start by saying I have tried most all methods resulting from google searches on this issue.
I am using the result set so I would prefer not to make a second query.
The query uses a prepared select statement which seems to be a main issue if I understand it correctly.
I decided to try a simple approach using PHP's native count() function. Which lead me here because I finally reached the end of the rope on this.
On to the details...within a class of mine, I make the query like this.
// Create database connection
$database = DatabaseFactory::getFactory()->getConnection();
// Set and execute database query
$sql = "SELECT * FROM `service_orders` WHERE `agency_id` = :agency_id $filter ORDER BY $sort $order $per_page";
$query = $database->prepare($sql);
$query->execute($query_array);
// If results
if ($query->rowCount() > 0) {
$results = $query->fetchAll();
self::$order_count = "Count: " . count($results);
return $results;
}
// Default, return false
return false;
Findings
If I perform count($results) like I did above, I get the total rows in the database (Let's say 50).
If I print_r($results), it shows the array with the proper number of entries (Let's say 10) that of course differs from the total rows in the database.
How can these two differ? It's as if the count($results) is misreading the result array.
More Findings
Within my actual php page, I call the class to retrieve the data like this.
$results = OrderModel::getServiceOrders();
echo count($results);
Strangely enough, if I then perform count($results) it gives me the correct reading of the result array (which in my example here would be 10).
I am perplexed by this as the count function is being performed on the exact same array. The only difference is one is called on the array within the class, and the other is called on the array returned from the class.
Does anyone have any ideas on how to solve this or why there is the discrepancy when using count() in this instance?
Thank you all in advance!
James
Additional Info
This is another mind numbing scenario. If I return the count along with the actual results, I can access it on the page with the correct value (10 rows). Yet, if I set it into a session variable, and access it that way, the count is the whole data set (50 rows). How is it even possible these two values are not the same?!
$results = $query->fetchAll();
Session::set("order_count", $total[0]); // Yields 50 (incorrect)
return [
"results"=> $results,
"count"=> $total[0], // Yields 10 (correct)
];
I have five different queries running on my about page showing basic data like the number of news stories we have on the site. I am using queries like this:
$sql4 = "SELECT `ride_id` FROM `tpf_rides` WHERE `type` LIKE '%Roller Coaster%'" ;
$result4 = $pdo->query($sql4);
$coasters = $result4->rowCount();
but wonder if there is a more efficient way. I've tried to minimize the load by only pulling id's but because I only need the count can the load be lightened even more?
Also these queries only really need to run once or twice per day, not every time the page is loaded. Can someone point me in the direction of setting this up? I've never had to do this before. Thanks.
Yes there is a more efficient way. Let the database do the counting for you:
SELECT count(*) as cnt
FROM `tpf_rides`
WHERE `type` LIKE '%Roller Coaster%';
If all the counts you are looking for are from the tpf_rides table, then you can do them in one query:
SELECT sum(`type` LIKE '%Roller Coaster%') as RollerCoaster,
sum(`type` LIKE '%Haunted House%') as HauntedHouse,
sum(`type` LIKE '%Ferris Wheel%') as FerrisWheel
FROM `tpf_rides`;
That would be even faster than running three different queries.
If you want to run those queries only every now and then you need to keep the result stored somewhere. This can take a form of a pre-calculated sum you manage yourself or a simple cache.
Below is a very simple and naive cache implementation that should work reliably on linux. Many things can be improved here but maybe this will give you an idea of what you could do.
The below is not compatible with the query suggested by Gordon Linoff which returns multiple counts.
The code has not been tested.
$cache_directory = "/tmp/";
$cache_lifetime = 86400; // time to keep cache in seconds. 24 hours = 86400sec
$sql4 = "SELECT count(*) FROM `tpf_rides` WHERE `type` LIKE '%Roller Coaster%'";
$cache_key = md5($sql4); //generate a semi-unique identifier for the query
$cache_file = $cache_directory . $cache_key; // generate full cache file path
if (!file_exists($cache_file) || time() <= strtotime(filemtime($cache)) + $cache_lifetime)
{
// cache file doesn't exist or has expired
$result4 = $pdo->query($sql4);
$coasters = $result4->fetchColumn();
file_put_contents($cache_file, $coasters); // store the result in a cache file
} else {
// file exists and data is up to date
$coasters = file_get_contents($cache_file);
}
I would strongly suggest you break this down into functions that take care of different aspects of the problem.
When I have a loop like this:
foreach(...) {
$r1 = $zend_db->fetchRow("SELECT ... ");
$zend_table->insert($data_array, $where);
}
... running a few thousand times. Is it possible, that $r1 doesn't contain a record inserted in the previous loop?
At http://dev.mysql.com/doc/refman/5.1/en/query-cache.html they write "The query cache does not return stale data. When tables are modified, any relevant entries in the query cache are flushed." But maybe ZEND does some unexpected caching for SELECT or INSERT?
Do I need to use transactions to solve this?
I had an issue with double records and there is no other explanation where they came from. But I can't reproduce it, cause it happened two months ago, importing csv-data that no longer exists.
As said by drew010 in comment Zend_Db does no caching unless you implement it yourself . For next time to be sure about the problem try something like this
try {
foreach(...) {
$r1 = $zend_db->fetchRow("SELECT ... ");
$zend_table->insert($data_array, $where);
}
} catch(Zend_Db_Exception $e)
{
$logger->log($e->getMessage(),Zend_Log::CRIT); //$logger is instance of Zend_Log
}
Given 5,000 IDs of records fetch in the database, which query , in your opinion is faster?
Loop through 5000 IDs using php and perform a SELECT query for each one,
foreach($ids as $id){
// do the query
$r = mysql_query("SELECT * FROM TABLE WHERE ID = {$id}");
}
Or collect all ids in an array, and use SELECT * FROM TABLE WHERE ID IN (1 up to 5000)
//assuming $ids = array(1,2 ---- up to 5000);
$r = mysql_query("SELECT * FROM TABLE WHERE ID IN (".join(",",$ids).")");
Without a shadow of a doubt, loading them all in one go will be faster. Running 5,000 queries is going to be a lot slower as each query will carry a certain amount of overhead.
Also, to speed it up even more, DON'T use the * operator! Select the fields you are going to use, if you only need the ID column, specify this! If you want all the columns, specify them all, because you may later add fields in and you do not need to retrieve this new field.
option 2 is definitely going to be faster. 5000 separate db queries are going to have huge network connection overhead.
The fastest way is not to request 5000 rows at all.
You barely need 100 to display them on one page. 5000 is way overkill
Sure measure it, but I'd certainly recommend letting the database doing the job.
All depends, I hope you're not creating a connection for each call though.
Loop is faster if you use a Query Statement using bind variables. Declare the Statement off the loop; then inside the loop bind the variable per each id.
Do not underestimate the time going into SQL parsing; especially on these long winded things.
Option 2 is faster. With option 1 you do a full roundtrim to the server for each iteration.
I'd point out that in this case you might consider using paging to display the data.
Hint: Measure, Measure, Measure. With a code worth 10 minutes of your time you will have the answer right away.
Which is faster for many queries?
Try measure it for example like this:
<?php
$start = getmicrotime();
for ($i=0;$i<100000;$i++)
{
foreach($ids as $id){
// do the query
$r = mysql_query("SELECT * FROM TABLE WHERE ID = {$id}");
}
}
$end = getmicrotime();
echo 'Time (1): '.($end- $start).' sec';
$start = getmicrotime();
for ($i=0;$i<100000;$i++)
{
//assuming $ids = array(1,2 ---- up to 5000);
$r = mysql_query("SELECT * FROM TABLE WHERE ID IN (".join(",",$ids).")");
}
$end = getmicrotime();
echo 'Time (2): '.($end- $start).' sec';
?>