On my website, I want to use a lot of different data from my database. Currently, I'm using four queries to gather different data. But is there a way to make it more efficient and put them into one big query? And how would I do that?
Edit: So the answer was to simply put all queries together into one and use as much data manipulation as possible in the database queries, and not in php.
$qry = "SELECT COUNT(*) cnt,
AVG(level) avg_lvl,
SUM(IF(onlinestatus=1, 1, 0)) online_cnt,
(SELECT Max(time) FROM refreshes) refresh_time
FROM players";
foreach ($db->query($qry) as $row){
$amount_total = $row['cnt'];
$average_level = floor($row['avg_lvl']);
$online_amount = $row['online_cnt'];
$milliseconds = $row['refresh_time'] + 1800000;
$update_time = DateTime::createFromFormat('U', intval($milliseconds / 1000));
}
You could combine all queries into one, like this:
$qry = "SELECT COUNT(*) cnt,
AVG(level) avg_lvl,
SUM(IF(onlinestatus=1, 1, 0)) online_cnt,
(SELECT Max(time) FROM refreshes) refresh_time
FROM rookstayers";
foreach ($db->query($qry) as $row){
$amount_total = $row['cnt']
$level = $row['avg_lvl'];
$online_amount = $row['online_cnt'];
$milliseconds = $row['refresh_time'] + 1800000;
$update_time = DateTime::createFromFormat('U', intval($milliseconds / 1000));
}
The last query you have seems to assume there is only one record in the result, as the loop would overwrite the previous result in each iteration. And as there is no order by in that query, it would be a bit of a gamble what the outcome would be. So I have taken the most recent time from the table in case there are multiple records there.
Note that that the above loop only executes once, as there is a guarantee to get exactly one result from the query.
The first and third queries can be combined into one:
select count(*) as num, sum(onlinestatus = 1) as numOnline
from rookstayers;
The second should be an aggregation:
select level, count(*) as cnt
from rookstayers
group by level;
The fourth is also an aggregation; I'm not sure exactly what the data looks like, but it seems to be something like:
select sum(time + 1800000)
from refreshes;
In general, you should do as much data manipulation in the database as you can. That is what databases are designed for.
EDIT:
The first, second, and third can be combined into:
select count(*) as num, sum(onlinestatus = 1) as numOnline,
avg(level) as avgLevel
from rookstayers;
Related
I have a question about using a MySQL Query to convert my data into a JSON Object. The Query I have is converting to a JSON Object, but it is not working the way I would like.
I have multiple tables in my database that I would like to graph on a chart using the date as the X axis and the values as the Y axis. I am currently joining the tables by date. However, some tables may have multiple submissions per day while others may not have any. Currently, the Query I have is only showing results for dates that data was submitted to all 4 tables.
I would also like to graph the information on a scale of 0-10. Three of the 4 tables only have values from 0-10 so I am taking the average of each value per day. The nutrition table, which holds nf_sugars and nf_total_carbohydrates has larger numbers that I will be using normalization to convert them into a 0-10 scale. For now, I am just attempting to get the SUM per day and will complete the rest of the calculation after this part is working. However, the query I am currently running is giving me results that are much higher than the SUM of the actual numbers in my database.
Any help would be greatly appreciated! Here is the PHP I am currently using to create the JSON Object. As a side note, I did successfully connect to my database, I just did not include that here.
$myquery = "SELECT track_ticseverity.date,
AVG(track_ticseverity.ticnum) as average_ticnum,
track_fatigue.date,
AVG(track_fatigue.fatiguenum) as average_fatiguenum,
track_stress.date,
AVG(track_stress.stressnum) as average_stressnum,
track_nutrition.date,
((SUM(track_nutrition.nf_sugars) ) ) as sum_nf_sugars,
((SUM(track_nutrition.nf_total_carbohydrate) ) ) as sum_nf_total_carbohydrate
FROM track_ticseverity
INNER JOIN track_fatigue
ON track_ticseverity.date=track_fatigue.date
INNER JOIN track_stress
ON track_fatigue.date=track_stress.date
INNER JOIN track_nutrition
ON track_stress.date=track_nutrition.date
WHERE track_ticseverity.user_id=1
AND track_fatigue.user_id=1
AND track_stress.user_id=1
AND track_nutrition.user_id=1
GROUP BY track_ticseverity.date";
$query = mysqli_query($conn, $myquery);
if ( ! $query ) {
echo mysqli_error(s);
die;
}
$data = array();
for ($x = 0; $x < mysqli_num_rows($query); $x++) {
$data[] = mysqli_fetch_assoc($query);
}
echo json_encode($data);
mysqli_close($conn);
EDIT - The Query is successfully returning a JSON object. My issue is that the query I wrote does not output the data in the correct way. I need the query to select information from multiple tables, some with multiple submission per day and others with only one or no submissions.
EDIT2 - I am thinking another way to handle this is to combine multiple SELECT statements into a single JSON Object, but I am not sure how to do this.
The sum is larger than expected because of the joins. Imagine that a certain date
occurs in one track_nutrition record and two track_fatigue records, then the join
will make that the data from the first table is once combined with the first track_fatigue
record, and then again with the second record. Thus the same nf_sugars
value will be counted twice in the sum. This behaviour will also affect the averages.
You should therefore first perform the aggregations, and only then perform the joins.
Secondly, to ensure you catch all data, even if for a certain date not all tables have
values, you should use full outer joins. This will guarantee that each record in each table
will find its way in the result. Now, MySQL does not support such full outer joins, so
I use an extra sub-select to select all different dates from the 4 tables and then
"left join" them with the other aggregated data:
SELECT dates.date,
IFNULL(average_ticnum_n, 0) as average_ticnum
IFNULL(average_fatiguenum_n, 0) as average_fatiguenum
IFNULL(average_stressnum_n, 0) as average_stressnum
IFNULL(sum_nf_sugars_n, 0) as sum_nf_sugars
IFNULL(sum_nf_total_carbohydrate_n, 0) as sum_nf_total_carbohydrate
FROM (
SELECT DISTINCT user_id,
date
FROM (
SELECT user_id,
date
FROM track_ticseverity
UNION
SELECT user_id,
date
FROM track_fatigue
UNION
SELECT user_id,
date
FROM track_stress
UNION
SELECT user_id,
date
FROM track_nutrition
) as combined
) as dates
LEFT JOIN (
SELECT user_id,
date,
AVG(ticnum) as average_ticnum_n
FROM track_ticseverity
GROUP BY user_id,
date) as grp_ticseverity
ON dates.date = grp_ticseverity.date
AND dates.user_id = grp_ticseverity.user_id
LEFT JOIN (
SELECT user_id,
date,
AVG(fatiguenum) as average_fatiguenum_n
FROM track_fatigue
GROUP BY user_id,
date) as grp_fatigue
ON dates.date = grp_fatigue.date
AND dates.user_id = grp_fatigue.user_id
LEFT JOIN (
SELECT user_id,
date,
AVG(stressnum) as average_stressnum_n
FROM track_stress
GROUP BY user_id,
date) as grp_stress
ON dates.date = grp_stress.date
AND dates.user_id = grp_stress.user_id
LEFT JOIN (
SELECT user_id,
date,
SUM(nf_sugars) as sum_nf_sugars_n,
SUM(nf_total_carbohydrate) as sum_nf_total_carbohydrate_n
FROM track_nutrition
GROUP BY user_id,
date) as grp_nutrition
ON dates.date = grp_nutrition.date
AND dates.user_id = grp_nutrition.user_id
WHERE dates.user_id = 1
ORDER BY dates.date;
Note that you will get 0 values in some of the columns when there is no data for that
particular date. If you prefer to get NULL instead, remove the Nvl() from those columns
in the query above.
Then, to normalize all data on a 0 - 10 scale, you could look at the maximum
found for each type of value and use that for a conversion, or if you know beforehand
what the ranges are per type, then it is probably better to use that information, and
maybe code that in the SQL as well.
However, it always looks a bit odd to have values combined in a graph that actually use
different scales. One might easily jump to wrong conclusions with such graphs.
I will prefer to use this way (assuming all other things are working fine e.g query is working fine)
$query = mysqli_query($conn, $myquery);
if ( ! $query ) {
echo mysqli_error($conn); //You need to put $conn here to display error.
die;
}
$data = array();
while($row = mysqli_fetch_assoc($query)) {
$data[] = $row;
}
echo json_encode($data);
mysqli_close($conn);
if you are using PDO (PHP Data Objects) for database operations; then following code can be used to create the JSON from PDO.
$array = $pdo->query("SELECT * FROM employee")->fetchAll(PDO::FETCH_ASSOC);
echo json_encode($array);
For Multiple Select we need to try it like that way :
$array = $pdo->query("SELECT 1; SELECT 2;");
$array->nextRowset();
var_dump( $array->fetchAll(PDO::FETCH_ASSOC) );
I need get specific values and count all values from a MySQL table, i need get the best performance, my question is: What is better and faster?
- Use two separate queries:
$TBCount = $Resps = $MySQL->query('SELECT COUNT(*) FROM T1');
$Resps = $MySQL->query('SELECT id, name FROM T1 LIMIT 1');
while ($Resp = $Resps->fetch_assoc()) {
...
}
- Use One query with two SELECT:
$Resps = $MySQL->query('SELECT id, name, (SELECT COUNT(*) FROM T1) AS count FROM T1 LIMIT 1');
while ($Resp = $Resps->fetch_assoc()) {
$TBCount = $Resp['count'];
...
}
- Or someone have some best idea?
In the case of "One query, two SELECT", in the "while" loop, how can i get the count value outside of the loop? (to avoid unnecessary rewrite of the variable).
I would suggest first option with minor modification.
$TBCount = $Resps = $MySQL->query('SELECT COUNT(id) FROM T1');
$Resps = $MySQL->query('SELECT id, name FROM T1 LIMIT 1');
while ($Resp = $Resps->fetch_assoc()) {
...
}
note that I have mentioned just id in count query, this can be a good performance impact while dealing with large data.
You can do the same in second option, but as per my thought process second option will give count in all row returned with the main query. And that value will be same for all rows which is really not required.
That is the reason I would go with first option for separate queries.
$Resps = $MySQL->query('SELECT id, name, (SELECT COUNT(*) as tot FROM T1) AS count FROM T1 LIMIT 1');
$Resp = $Resps->fetch_assoc()
$TBCount = $Resp['tot'];
mysqli_data_seek($Resps, 0);
for finding the count value you do not need to repeatedly loop again and again for the rest of the values use the while loop.
while ($Resp = $Resps->fetch_assoc()) {
//rest of your code
...
}
Take a look at SQL_CALC_FOUND_ROWS. It does everything you want. It will give you the total results, even though you're limiting the data returned.
> SELECT SQL_CALC_FOUND_ROWS id, name FROM T1 LIMIT 1
> SELECT FOUND_ROWS();
To get the total number of records, I usually use this query:
$total= mysql_num_rows(mysql_query("SELECT id FROM t_statistic WHERE pageid = $pid"));
but I got one the other query like below:
$data = mysql_fetch_object(mysql_query("SELECT COUNT(id) AS num_rows FROM t_statistic WHERE pageid = $pid"));
$total = $data->num_rows;
Between the two queries above. Which is more quickly and effectively (when the total number of records in the millions)?
I prefer the second query. It gives you already the record count, while the first query gives you the list of IDs (not the count), although it has been filtered but there are some cases when ID exist more than once in the table.
The Second query is quick and efficient:
SELECT COUNT(id) AS num_rows FROM t_statistic WHERE pageid = $pid
If you know about query optimisation. The query will only keeps only count in memory while calculating the answer. And directly gives number of rows.
Where as first query:
SELECT id FROM t_statistic WHERE pageid = $pid
Keeps all the selected rows in memory. then number of rows are calculated in further operation.
So second query is best in both ways.
Definitely the second one.
Some engines, like MySQL can do a count just by looking at an index rather than the table's data.
I've used something like the following on databases with millions of records.
SELECT count(*) as `number` FROM `table1`;
Way faster than: mysql_num_rows($res);
BTW: The * in Count(*) basically means it won't look at the data, it will just count the records, as opposed to Count(colname).
1) SELECT COUNT(*) FROM t_statistic WHERE pageid = $pid" --> count(*) counts all rows
2)SELECT COUNT(id) FROM t_statistic WHERE pageid = $pid" --> COUNT(column) counts non-NULLs only
3) SELECT COUNT(1) FROM t_statistic WHERE pageid = $pid" -->COUNT(1) is the same as COUNT(*) because 1 is a non-null expressions
Your use of COUNT(*) or COUNT(column) should be based on the desired output only.
So. Finally we have result is count(column) is more faster compare to count(*) .
Part of my page I have lots of small little queries, probably about 6 altogether, grabbing data from different tables. As an example:
$sql_result = mysql_query("SELECT * FROM votes WHERE voted_on='$p_id' AND vote=1", $db);
$votes_up = mysql_num_rows($sql_result);
$sql_result = mysql_query("SELECT * FROM votes WHERE voted_on='$p_id' AND vote=0", $db);
$votes_down = mysql_num_rows($sql_result);
$sql_result = mysql_query("SELECT * FROM kids WHERE (mother_id='$p_id' OR father_id='$p_id')", $db);
$kids = mysql_num_rows($sql_result);
Would it be better if these were all grabbed in one query to save trips to the database? One query is better than 6 isn't it?
Would it be some kind of JOIN or UNION?
Its not about number of queries but amount of useful datas you transfer. If you are running database on localhost, is better to let sql engine to solve queries instead computing results in additional programs. The same if you are thinking about who should be more bussy. Apache or mysql :)
Of course you can use some conditions:
SELECT catName,
SUM(IF(titles.langID=1, 1, 0)) AS english,
SUM(IF(titles.langID=2, 1, 0)) AS deutsch,
SUM(IF(titles.langID=3, 1, 0)) AS svensk,
SUM(IF(titles.langID=4, 1, 0)) AS norsk,
COUNT(*)
FROM titles, categories, languages
WHERE titles.catID = categories.catID
AND titles.langID = languages.
example used from MYSQL Bible :)
If you really want to lower the number of queries, you can put the first two together like this:
$sql_result = mysql_query("SELECT * FROM votes WHERE voted_on='$p_id'", $db);
while ($row = mysql_fetch_array($sql_result))
{
extract($row);
if ($vote=='0') ++$votes_up; else ++$votes_down;
}
The idea of joining tables is that these tables are expected to have something in between (a relation, for example).
Same is for the UNION SELECTS, which are prefered to be avoided.
If you want your solution to be clean and scalable in future, I suggest you to use mysqli, instead of mysql module of PHP.
Refer to: mysqli::multi_query. There is OOP variant, where you create mysqli object and call the function as method.
Then, your query should look like:
// I use ; as the default separator of queries, but it might be different in your case.
// The above could be set with sql statement: DELIMITER ;
$query = "
SELECT * FROM votes WHERE voted_on='$p_id' AND vote=1;
SELECT * FROM votes WHERE voted_on='$p_id' AND vote=0;
SELECT * FROM kids WHERE (mother_id='$p_id' OR father_id='$p_id');
";
$results = mysqli_multi_query($db, $query); // Returns an array of results
Fewer queries are (generally, not always) better, but it's also about keeping your code clear enough that others can understand the query. For example, in the code you provided, keep the first two together, and leave the last one separate.
$sql_result = mysql_query("SELECT vote, COUNT(*) AS vote_count
FROM votes
WHERE voted_on='$p_id'
GROUP BY vote", $db);
The above will return to you two rows, each containing the vote value (0 or 1) and the vote count for the value.
Hey guys, I created a list for fixtures.
$result = mysql_query("SELECT date FROM ".TBL_FIXTURES." WHERE compname = '$comp_name' GROUP BY date");
$i = 1;
$d = "Start";
while ($row = mysql_fetch_assoc($result))
{
$odate = $row['date'];
$date=date("F j Y", $row['date']);
echo "<p>Fixture $i - $d to $date</p>";
}
As you can see from the query, the date is displayed from the fixtures table.
The way my system works is that when a fixture is "played", it is removed from this table. Therefore when the entire round of fixtures are complete, there wont be any dates for that round in this table. They will be in another table.
Is there anyway I can run an other query for dates at the same time, and display only dates from the fixtures table if there isnt a date in the results table?
"SELECT * FROM ".TBL_CONF_RESULTS."
WHERE compid = '$_GET[id]' && type2 = '2' ORDER BY date"
That would be the second query!
EDIT FROM HERE ONWARDS...
Is there anyway I can select the date from two tables and then only use one if there are matches. Then use the rows of dates (GROUPED BY) to populate my query? Is that possible?
It sounds like you want to UNION the two result sets, akin to the following:
SELECT f.date FROM tbl_fixtures f
WHERE f.compname = '$comp_name'
UNION SELECT r.date FROM tbl_conf_results r
WHERE r.compid = '$_GET[id]' AND r.type2 = '2'
GROUP BY date
This should select f.date and add rows from r.date that aren't already in the result set (at least this is the behaviour with T-SQL). Apparently it may not scale well, but there are many blogs on that (search: UNION T-SQL).
From the notes on this page:
//performs the query
$result = mysql_query(...);
$num_rows = mysql_num_rows($result);
//if query result is empty, returns NULL, otherwise,
//returns an array containing the selected fields and their values
if($num_rows == NULL)
{
// Do the other query
}
else
{
// Do your stuff as now
}
WHERE compid = '$_GET[id]' presents an oportunity for SQL Injection.
Are TBL_FIXTURES and TBL_CONF_RESULTS supposed to read $TBL_FIXTURES and $TBL_CONF_RESULTS?
ChrisF has the solution!
One other thing you might think about is whether it is necessary to do a delete and move to another table. A common way to solve this type of challenge is to include a status field for each record, then rather than just querying for "all" you query for all where status = "x". For example, 1 might be "staging", 2 might be "in use", 3 might be "used" or "archived" In your example, rather than deleting the field and "moving" the record to another table (which would also have to happen in the foreach loop, one would assume) you could simply update the status field to the next status.
So, you'd eliminate the need for an additional table, remove one additional database hit per record, and theoretically improve the performance of your application.
Seems like what you want is a UNION query.
$q1 = "SELECT DISTINCT date FROM ".TBL_FIXTURES." WHERE compname = '$comp_name'";
$q2 = "SELECT DISTINCT date FROM ".TBL_CONF_RESULTS.
"WHERE compid = '$_GET[id]' && type2 = '2'";
$q = "($q1) UNION DISTINCT ($q2) ORDER BY date";