I'm trying to build a MySQL SELECT that will only pull in rows in 10 minute increments (or sufficiently close to 10 min increments). My data is recorded about once a minute per row, but the timing of the INSERT isn't always exact. Each row has a datetime column.
My first idea was to pull in a whole bunch of rows then weed them out with a PHP function. That approach is messy and I'd much rather do this with a smarter MySQL query. Is it possible?
$result = mysqli_query($con,"SELECT * FROM pairs ORDER BY pair_id DESC LIMIT 2000;");
while($row = mysqli_fetch_array($result)) {
echo $row['SomeTimestamp'] . " " . $row['SomeValue'];
echo "<br>";
}
This will fetch all the records with timestamp within last 10min.
SELECT * FROM pairs WHERE SomeTimestamp > DATE_SUB(CURTIME(), INTERVAL 10 MINUTE)
You could use WHERE and date function
SELECT ... FROM ... WHERE /* select the lines that in the range of 10 mins */
Related
I have run into a delayed processing time for a PHP program,
I have a MySQL record with over 1000 tables;
Each table is created once a new device is added, e.g assets_data_imeixx - to assets_data_imeixx1000th table
Each table contains about 45,000 rows of records inserted every 10 seconds,
Below is my PHP code to query the database and fetch all these records based on datetime.
Issue: The program executes without error but it takes about 1.3minutes to 4mins for very large records.
PHP Code:
$ms = mysqli connection string in config.php //$ms is OKAY
$user_id = '5';
$q = "SELECT * FROM `user_assets` WHERE `user`='".$user_id ."' ORDER BY `imei` ASC";
$r = mysqli_query($ms,$q);
$result = array(); //$result array to contain all data
while($row =mysqli_fetch_array($r)){
//fetch 7 days record
for ($i=1; $i < 7; $i++) {
$date = "-" . $i . " days";
$days_ago = date('Y-m-d', strtotime($date, strtotime('today')));
$sql1 = "SELECT * FROM assets_data_" . $row["imei"] . " WHERE dt_time LIKE '" . $days_ago . "%' LIMIT 1"; // its correct
//$result1 = $conn->query($sql1);
$result1 = mysqli_query($ms,$sql1);
$row2 = mysqli_fetch_array($result1);
echo $row['imei']." ".$row2['dt_server']."<br/>";
}
}
Above code fetches over 1000 devices from user_assets table, These IMEI each has its own table that contains over 45,000 records in each table of location data.
The for loop iterates over each IMEI table and records.
Above code runs without error but take so much time to complete, I want to find a solution to optimize and have code execute in a very short time max 5 seconds.
I need help and suggestions on optimizing and running this large scale of data and iteration.
(from Comment)
CREATE TABLE gs_object_data_863844052008346 (
dt_server datetime NOT NULL,
dt_tracker datetime NOT NULL,
lat double DEFAULT NULL,
lng double DEFAULT NULL,
altitude double DEFAULT NULL,
angle double DEFAULT NULL,
speed double...
(From Comment)
gs_object_data_072101424612
gs_object_data_072101425049
gs_object_data_072101425486
gs_object_data_072101445153
gs_object_data_111111111111111
gs_object_data_1234567894
gs_object_data_222222222222222
gs_object_data_2716325849
gs_object_data_2716345818
gs_object_data_30090515907
gs_object_data_3009072323
gs_object_data_3009073758
gs_object_data_352093088838221
gs_object_data_352093088839310
gs_object_data_352093088840045
gs_object_data_352121088128697
gs_object_data_352121088132681
gs_object_data_352621109438959
gs_object_data_352621109440203
gs_object_data_352625694095355
gs_object_data_352672102822186
gs_object_data_352672103490900
gs_object_data_352672103490975
gs_object_data_352672103490991
gs_object_data_352887074794052
gs_object_data_352887074794102
gs_object_data_352887074794193
gs_object_data_352887074794417
gs_object_data_352887074794425
gs_object_data_352887074794433
gs_object_data_352887074794441
gs_object_data_352887074794458
gs_object_data_352887074794474
gs_object_data_352887074813696
gs_object_data_352887074813712
gs_object_data_352887074813720
gs_object_data_352887074813753
gs_object_data_352887074813761
gs_object_data_352887074813803
900+ tables each having different location data.
Requirement: Loop through each table, fetch data for selected date range say:
"SELECT dt_server FROM gs_object_data_" . $row["imei"] . " WHERE dt_server BETWEEN '2022-02-05 00:00:00' AND '2022-02-12 00:00:00'";
Expected Result: Return result set containing data from each table containing information for the selected date range. That means having 1000 tables will have to be looped through each table and also fetch data in each table.
I agree with KIKO -- 1 table not 1000. But, if I understand the rest, there are really 2 or 3 main tables.
Looking at your PHP -- It is often inefficient to look up one list, then go into a loop to find more. The better way (perhaps 10 times as fast) is to have a single SELECT with a JOIN to do both selects at once.
Consider some variation of this MySQL syntax; it may avoid most of the PHP code relating to $days_ago:
CURDATE() - INTERVAL 3 DAY
After also merging the Selects, this gives you the rows for the last 7 days:
WHERE date >= CURDATE() - INTERVAL 7 DAY
(I did not understand the need for LIMIT 1; please explain.)
Yes, you can use DATETIME values as strings, but try not to. Usually DateTime functions are more efficient.
Consider "composite" indexes:
INDEX(imei, dt)
which will be very efficient for
WHERE imei = $imei
AND dt >= CURDATE() - INTERVAL 7 DAY
I would ponder ways to have less redundancy in the output; but that should mostly be done after fetching the raw data from the table(s).
Turn on the SlowLog with a low value of long_query_time; it will help you locate the worst query; then we can focus on it.
An IMEI is up to 17 characters, always digits? If you are not already using this, I suggest BIGINT since it will occupy only 8 bytes.
For further discussion, please provide SHOW CREATE TABLE for each of the main tables.
Since all those 1000 tables are the same it would make sense to put all that data into 1 table. Then partition that table on date, use proper indexes, and optimize the query.
See: Normalization of Database
Since you limit results to one user, and one row per device, it should be possible to execute a query in well below one second.
I am trying to figure out how to determine the total number of rows within a specific number range using PHP.
I have a simple MySQL table with a single column. The column contains thousands of rows, each row containing a number between 0 and 100.
I figured out how to find the number of rows for a specific number, using array_count_values, but I can't figure out how to find the number of rows for a range.
For example, how many numbers are there between 60 and 80?
Here is the code that I put together to find a single value. What code should I add to find range values?
$query = "SELECT numbers FROM table";
$result = mysqli_query($conn, $query) or die("Error in $query");
$types = array();
while(($row = mysqli_fetch_assoc($result))) {
$types[] = $row['numbers'];
}
$counts = array_count_values($types);
echo $counts['12'];
If you need to count within multiple ranges you can use UNION so you don't have to send 5 queries.
$query = "SELECT COUNT(numbers) FROM `table` WHERE numbers between 00 and 20
UNION ALL
SELECT COUNT(numbers) FROM `table` WHERE numbers between 20 and 40
UNION ALL
SELECT COUNT(numbers) FROM `table` WHERE numbers between 40 and 60
UNION ALL
SELECT COUNT(numbers) FROM `table` WHERE numbers between 60 and 80
UNION ALL
SELECT COUNT(numbers) FROM `table` WHERE numbers between 80 and 100";
You can do this several ways.
Simple way (one full table scan)
SELECT SUM(IF(x BETWEEN 20 AND 30, 1, 0)) AS b_20_30,
SUM(...) AS b_31_40,
...
FROM tableName...
will return only one row with all your results in the time of a table scan.
Fancy way (not really recommended)
If you can come up with a rule to map your intervals to a single number, for example:
0...9 => interval i = 0
10...19 => interval i = 1 => the rule is "i = FLOOR(X/10)"
20...29 => interval i = 2
...and you don't need to scan too many rows, you might do something not very maintainable like this:
SELECT SUM(FLOOR(POW(100, FLOOR(x / 10)))) AS result FROM tableName;
Here, a value of 25 (between 20 and 29) will become 2, and the total sum will be increased by 1002. So long as you never have more than 100 rows in each group, the final result will be a univocal sum of powers and, if you have - say - 17 rows between 0 and 9, 31 rows between 10 and 19, and 74 between 20 and 29, you'll get a "magical parlor trick" answer of
743117
from whence you can recover the number of rows as 74,31,17 in that order.
Using 1000 instead of 100 would yield 74031017 (and the possibility of coping with up to 999 numbers in each group).
Note that the use of functions inside the SELECT pretty much guarantees you'll need a full, slow table scan.
Using indexes for speed
But we can get rid of the table scan, and simplify generation, by judiciously using indexed WHEREs - this is identical, performance-wise, to a UNION, but the result is simpler since it is only one row:
SELECT (SELECT COUNT(*) FROM tableName WHERE x BETWEEN ...) AS b_20_30,
(...)
; -- no table name on the outer query
This will need several subqueries (one per interval), but those subqueries will all use an index on x where available, which can make the overall query very fast. You just need
CREATE INDEX x_ndx ON tableName(x);
The same index will greatly improve the performance of the "simple" query above, which will no longer need a table scan but only a much faster index scan.
Build the query using PHP
Supposing we have the intervals specified as convenient arrays, we can use PHP to generate the query in the first place. No need of manually entering all those SELECTs.
$intervals = [ [ 20, 30 ], [ 17, 25 ], ... ];
function queryFromIntervals(array $intervals) {
$index = 0;
return 'SELECT ' .
implode(',', array_map(
function($interval) use ($tableName, &$index) {
return "(SELECT COUNT(1) FROM {$tableName} WHERE x BETWEEN {$interval[0]} AND {$interval[1]}) AS b_" . ($index++);
},
$intervals
))
// . ", (SELECT COUNT(*) FROM {$tableName}) AS total"
. ';';
}
This will again yield a single row, with fields named b_0, b_1, and so on. They will contain the number of rows where the value of x is between the bounds of the corresponding interval in $intervals (the intervals may overlap).
So by executing the query and retrieving a tuple called $result, you might get
[ 'b_0' => 133, 'b_1' => 29 ]
meaning that there are 133 rows with x between 20 and 30, and 29 with x between 17 and 25. You can add to the query a last total field (commented in the code above):
, ... ( SELECT COUNT(*) FROM tableName ) AS total;
to also get the total number of rows.
The same function, changing the inner return value, can be adapted to generate the "simple" query which uses IFs instead of SELECTs.
Why not let the database do the heavy lifting for you?
$query = "SELECT COUNT(*) FROM table WHERE numbers BETWEEN 60 AND 80";
$result = mysqli_query($conn, $query) or die("Error in $query");
$row = mysqli_fetch_array($result, MYSQLI_NUM);
echo $row[0];
If I understood well, you try to put a condition in your request sql.
Look that https://www.w3schools.com/sql/sql_where.asp
$query = "SELECT numbers FROM table WHERE numbers >= 60 AND numbers <= 80";
hello all i am a php developer and right now i am having a simple issue like i want to get the last 10 data out of my database table and then show random 5 out of them .
like see the code below.
$random=rand(0,18);
mysqli_query(
$connection,
"SELECT something1,something2
FROM `table`
where (id !='0')
ORDER BY time DESC
LIMIT 4 OFFSET $random");
this query selects last 4 updated fields randomely out of 18 but i want a something better solution to this
First, define "the last 10"; for now let's assume the last ten when ordered by time. So, that is in fact the first ten when I order them the order way (I guess you got that):
SELECT * FROM `table` WHERE (id != '0') ORDER BY time DESC LIMIT 10;
then we subselect that to randomize and limit:
SELECT * FROM (
SELECT * FROM `table` WHERE (id != '0') ORDER BY time DESC LIMIT 10;
) A ORDER BY RAND() LIMIT 5
Remember, your database is extremely powerful and fast, use it for any data-processing.
Not sure there is a great solution with Select. Your sample take 4 line in a row with an OFFSET which is not a truly random selection. You should limit your select to 10 and then get 5 row of the 10 by another php code.
$rows = mysqli_query($connection,
"SELECT something1,something2 FROM `table` where (id !='0')
ORDER BY time DESC LIMIT 10");
//then, get 5 random key from your array of row from the mysql select
$rand_keys = array_rand($rows, 5);
In case you prefer randomizing in PHP, you can use PHP's shuffle and array_slice.
<?php
$last_10_rows = mysqli_query($connection, "simple query to get last 10 rows");
shuffle($last_10_rows); // randomize the array
$random_5_rows = array_slice($last_10_rows, 0, 5); // take first 5
?>
For
I have a live chat and i need to ban if the users chatted more than 5 times in a row
this is the sql:
function countMessages() {
global $tsUser;
$querys = db_exec(array(__FILE__, __LINE__), 'query', 'SELECT * FROM c_chat_messages where msg_user = "'.$tsUser->uid.'" ORDER BY msg_id ASC');
$counts = db_exec('num_rows', $querys);
if($counts > 5) { $this->banUser(); }
}
$tsUser->uid is the id of the user
i need to check if the user chatted for more than 5 times in a row so the php executes the $this->banUser(); function
you have to take timestamp of each message sent by the user in your database table.
And than you can fire a query checking if the user has sent more than 5 messages in last 15 seconds, by calculating the time different between the messages.
to get last five messages sent by that user,
SELECT * FROM c_chat_messages where msg_user = "'.$tsUser->uid.'" ORDER BY msg_id DESC limit 0,5
than get the difference between last and first record returned by above query.
to get time difference in seconds you can use
SELECT TIME_TO_SEC(TIMEDIFF('2010-08-20 12:01:00', '2010-08-20 12:00:00')) diff;
you can call a ajax file every one or two seconds to check this.
Let me know if further clarification needed.
Okay guys, this probably has an easy answer but has been stumping me for a few hours now.
I am using PHP/HTML to generate a table from a MySQL Table. In the MySQL table (TimeRecords) I have a StartTime and EndTime column. In my SELECT statement I am subtracting the EndTime from the StartTime and aliasing that as TotalHours. Here is my query thus far:
$query = "SELECT *,((EndTime - StartTime)/3600) AS TotalPeriodHours
FROM TimeRecords
WHERE Date
BETWEEN '{$CurrentYear}-{$CurrentMonth}-1'
AND '{$CurrentYear}-{$CurrentMonth}-31'
ORDER BY Date
";
I then loop that through an HTML table. So far so good. What I would like to do is to add up all of the TotalHours and put that into a separate DIV. Any ideas on 1) how to write the select statement and 2) where to call that code from the PHP/HTML?
Thanks in advance!
Try this
$query= "
SELECT ((EndTime - StartTime)/3600) AS Hours, otherFields, ...
FROM TimeRecords
WHERE
Date BETWEEN '{$CurrentYear} - {$CurrentMonth} - 1'
AND '{$CurrentYear}-{$CurrentMonth} - 31' ";
$records =mysql_query($query);
$sum= 0;
while($row=mysql_fetch_array($records))
{
echo"$row['otherFields']";
echo"$row['Hours']";
$sum+=$row['Hours'];
}
echo" Total Hours : $sum ";
Just use a single query with a Sum(). You could also manually calculate it if you're already displaying all rows. (If paginating or using LIMIT, you'll need a separate query like below.)
$query = "
SELECT Sum(((EndTime - StartTime)/3600)) AS SumTotalPeriodHours
FROM TimeRecords
WHERE
Date BETWEEN '{$CurrentYear} - {$CurrentMonth} - 1'
AND '{$CurrentYear}-{$CurrentMonth} - 31'
";
You can do this in the same query if you have a unique id using GROUP BY WITH ROLLUP
$query = "
SELECT unique_id,SUM((EndTime - StartTime)/3600) AS TotalPeriodHours
FROM TimeRecords
WHERE Date BETWEEN '{$CurrentYear}-{$CurrentMonth}-1'
AND '{$CurrentYear}-{$CurrentMonth}-31'
GROUP BY unique_id WITH ROLLUP
ORDER BY Date
";
In this instance the last result from your query with contain NULL and the overall total. If you don't have a unique ID you will need to do it in PHP as per Naveen's answer.
A few comments on your code:
Using SELECT * is not considered good practice. SELECT the columns you need.
Not all months have a day 31 so this may produce unexpected results. If you're using PHP5.3+, you can use
$date = new DateTime();
$endDate = $date->format( 'Y-m-t' );
The "t" flag here gets the last day of that month. See PHP docs for more on DateTime.