mysql - multiple select with UNION ALL and count() in the same query - php

Question

I have a DB-table formed like this:
ID | pairname | timeframe | value1 | value2 | value3
I have about 290 pairnames and 4 timeframes and for each timeframe I will have 250 rows.
Table looks like this:
1 | appusd | 1h | 12319 | 129312 | 11111 | 11111
2 | appusd | 1h | 12444 | 123912 | 11221 | 111231
3 | appusd | 4h | 12312 | 123123 | 11111 | 12321
4 | gttusd | 1h | 12342 | 123123 | 11111 | 12321
5 | gttusd | 4h | 12342 | 123123 | 11111 | 12321
I run a PHP script, that will first fill the DB with values and then check every hour if values are on current.
In PHP I loop first to get all pairnames I need from an other table. Inside of this loop, I loop through the 4 different timeframes to create this SELECT:
(
SELECT value1, timeframe
FROM db1
WHERE pairname = 'blabla'
AND timeframe = '1h'
ORDER BY value1 DESC
LIMIT 2
)
UNION ALL (
SELECT value1, timeframe
FROM db1
WHERE pairname = 'blabla'
AND timeframe = '4h'
ORDER BY value1 DESC
LIMIT 2
)
UNION ALL (
SELECT value1, timeframe
FROM db1
WHERE pairname = 'blabla'
AND timeframe = '8h'
ORDER BY value1 DESC
LIMIT 2
)
UNION ALL (
SELECT value1, timeframe
FROM db1
WHERE pairname = 'blabla'
AND timeframe = '2h'
ORDER BY value1 DESC
LIMIT 2
)
it works nicely. But because I have 290 queries out of the loop, it takes some time. (about 9 secs with all the other script part, that's ok.)
Now I do not only want the last 2 entries of each pairname/timeframe but also to count how many entries I have, I need to count each row of each pairname/timeframe.
(
SELECT COUNT( * ) AS rawnr
FROM db1
WHERE pairname = 'blabla'
AND timeframe = '1h'
)
UNION ALL (
...
)
Now I have another 290 queries. and script-time doubles. It's getting slow. Is it possible to count() and select at the same time, if I have this UNION ALL inside of my SELECT?
I've tried it, and it does not work, maybe because of the "limit 2" inside of the same select? I also tried to UNION ALL the count() into a new select, and this did not work either, because UNION wants the same amount of columns.
My solution works with about 17 seconds script-time. After this, I will have to check each result if it's up to date, pull new data from another server and then also delete/add new entries. This will definitely exceed my script-time, and I will have to find a solution for refreshing the script after certain amount of pairname/timeframe editing. (if you know how to reload a php-script at a certain point, that would be also fine!)
EDIT to see the other tables and the script:
The pairname is combined out of 2 names. Name one could be "app" and second "usd".
1st table with names formed like this:
nameID | name | active | value1 | value2 | value3
2nd table contains which names can be paired.
pairID | nameID1 | nameID2 | active | somevalue
I check first which "names" are active and then which pairs are "active". After this I combine the names to get: "appusd". This pairname I use in the third table, there I save all the data to this pairnames.
The reason I use "pairname" as a own raw is, that the data I fill in comes from an API that wants exactly this "pairname", so I can handle data better later on.
The thing is, that I do not want all names, and not all pairname-combinations.
All this has a script-time of 0,7 sec. So it's fine.
$tradeagainst = array("ABC","USD");
$timeframearr = array("1h","2h","4h","8h");
$tradeagainstsql = "";
$namepairarr = array();
$insert = "";
$ticker = array();
$tabelle = "names";
$tabelle2 = "namepairs";
$sqlsc = "SELECT NameID, shortname, nametyp
FROM ".$tabelle."
ORDER BY shortname ASC";
$resultsc = mysqli_query($conn, $sqlsc);
$nameshortnamearr = array();
$nameIDarr = array();
while($rowc = mysqli_fetch_assoc($resultsc)){
$nameshortnamearr []= $rowc["shortname"];
$nameIDarr []= $rowc["NameID"];
if(in_array($rowc["shortname"],$tradeagainst)){
if($tradeagainstsql == ""){
$tradeagainstsql .= " AND (".$tabelle2.".tradenameID = '".$rowc["NameID"]."'";
}else{
$tradeagainstsql .= " OR ".$tabelle2.".tradenameID = '".$rowc["NameID"]."'";
}
}
}
if($tradeagainstsql != ""){
$tradeagainstsql .= ") ";
}
$sqls = "SELECT ".$tabelle.".*, GROUP_CONCAT(".$tabelle2.".tradenameID) as trID
FROM ".$tabelle."
LEFT JOIN ".$tabelle2." ON ".$tabelle.".NameID = ".$tabelle2.".NameID
WHERE ".$tabelle.".active != '1'
AND ".$tabelle2.".active != '1'
".$tradeagainstsql."
GROUP BY ".$tabelle.".NameID
ORDER BY ".$tabelle.".shortname";
$results = mysqli_query($conn, $sqls);
if (mysqli_num_rows($results) > 0) {
while($row = mysqli_fetch_assoc($results)) {
$nameID = $row["NameID"];
$shortname = $row["shortname"];
$active = $row["active"];
$nametyp = $row["nametyp"];
$namepairsarr = explode(",",$row["trID"]);
foreach($namepairsarr as $trnameID){
$ct++;
$namepair = $shortname.$nameshortnamearr[array_search($trnameID, $nameIDarr)];
$namepairarr []= $namepair;
}
}
}
this first part I do, to get the $namepairarr. It is also dynamically, to be able to set up, the $tradeagainst, $timeframearr;
The second part is this one:
$tabelle3 = "namepairdata";
$updatearray = array();
foreach($namepairarr as $namepair){
$sqlnp = "";
$sqlnpcount = "";
foreach($timeframearr as $timeframe){
if($sqlnp == ""){
$sqlnp .= "(SELECT value1, timeframe
FROM ".$tabelle3."
WHERE pair = '".$namepair."' AND timeframe = '".$timeframe."' ORDER BY value1 DESC LIMIT 2)";
$sqlnpcount .= "(SELECT count(*) AS rawcount
FROM ".$tabelle3."
WHERE pair = '".$namepair."' AND timeframe = '".$timeframe."')";
}else{
$sqlnp .= " UNION ALL (SELECT value1, timeframe
FROM ".$tabelle3."
WHERE pair = '".$namepair."' AND timeframe = '".$timeframe."' ORDER BY value1 DESC LIMIT 2)";
$sqlnpcount .= " UNION ALL (SELECT count(*) AS rawcount
FROM ".$tabelle3."
WHERE pair = '".$namepair."' AND timeframe = '".$timeframe."')";
}
}
$resultnp = mysqli_query($conn, $sqlnp);
$resultnpc = mysqli_query($conn, $sqlnpcount);
$rowsvalue1 = array();
$rowstimeframe = array();
$rowscount = array();
while($rowkl = mysqli_fetch_assoc($resultnp)){
$rowsvalue1 []= $rowkl['value1'];
$rowstimeframe []= $rowkl['timeframe'];
}
while($rowklc = mysqli_fetch_assoc($resultnpc)){
$rowscount []= $rowklc['rawcount'];
}
$rc = 0;
foreach($timeframearr as $timeframe){
$value11 = "";
$value12 = "";
$timeframecount = $rowscount[$rc]; $rc++;
for($ints = 0; $ints < count($rowstimeframe); $ints++){
if($timeframe == $rowstimeframe[$ints]){
if($value11 == ""){
$value11 = $rowsvalue1[$ints];
}else{
$value12 = $rowsvalue1[$ints];
}
}
}
$updatearray []= array($namepair,$timeframe,$value11,$value12,$timeframecount);
}
Finally I get an array $updatearray, which I will have to do something with.