How to improve execution time for this SQL query? - php

I have a table with the following example:
ID
name
task
startDate
endDate
1
John
Day
2022-09-20
2022-09-21
2
Joe
Midday
2022-09-20
2022-09-21
3
John
Day
2022-09-22
2022-09-23
4
Sara
Night
2022-09-20
2022-09-21
5
Joe
Night
2022-09-24
2022-09-25
I would like to count the rows that have a name in a given name list AND task in a given task list AND within a date range.
I am using php and the code and sql query I am currently using is:
$taskList = list of all the tasks I want to search for
$nameList = list of all the names I want to search for
SELECT COUNT(ID) FROM table WHERE
`task` IN ('".implode("', '", $taskList)."')
AND `name` IN ('".implode("', '", $nameList)."')
AND (startDate >= '".$startDate."')
AND (startDate <= '".$endDate."')
It takes ~40 milliseconds per execution but I need to do this multiple times. Is there a way I can shorten the execution time?

Looks like your query is fast enough, but on some case it also slow.
Please make sure:
Add index group to your table columns that where query executed:
task, name, startDate
Consider don't use asterisk to select count query with the column name that not nullable values, try to use select count(primary_key_column)
Just Test The difference.
But, before start please doing escape the values before doing direct query to sql to prevent sql injection and or invalid data value.
/**
* #var PDO $pdo
*/
$taskList = array_map([$pdo, 'quote'], $taskList);
$nameList = array_map([$pdo, 'quote'], $nameList);
$startDate = $pdo->quote($startDate);
$endDate = $pdo->quote($endDate);
$quotedTaskList = implode(',', $taskList);
$quotedNameList = implode(',', $nameList);
$query = "SELECT COUNT(primary_key_or_non_nullable_column) FROM table WHERE
`task` IN ({$quotedTaskList}')
AND `name` IN ({$quotedNameList})
AND (`startDate` >= {$startDate})
AND (`startDate` <= {$endDate})
";

Related

Processing millions of data records with PHP MySQL issue

I have run into a delayed processing time for a PHP program,
I have a MySQL record with over 1000 tables;
Each table is created once a new device is added, e.g assets_data_imeixx - to assets_data_imeixx1000th table
Each table contains about 45,000 rows of records inserted every 10 seconds,
Below is my PHP code to query the database and fetch all these records based on datetime.
Issue: The program executes without error but it takes about 1.3minutes to 4mins for very large records.
PHP Code:
$ms = mysqli connection string in config.php //$ms is OKAY
$user_id = '5';
$q = "SELECT * FROM `user_assets` WHERE `user`='".$user_id ."' ORDER BY `imei` ASC";
$r = mysqli_query($ms,$q);
$result = array(); //$result array to contain all data
while($row =mysqli_fetch_array($r)){
//fetch 7 days record
for ($i=1; $i < 7; $i++) {
$date = "-" . $i . " days";
$days_ago = date('Y-m-d', strtotime($date, strtotime('today')));
$sql1 = "SELECT * FROM assets_data_" . $row["imei"] . " WHERE dt_time LIKE '" . $days_ago . "%' LIMIT 1"; // its correct
//$result1 = $conn->query($sql1);
$result1 = mysqli_query($ms,$sql1);
$row2 = mysqli_fetch_array($result1);
echo $row['imei']." ".$row2['dt_server']."<br/>";
}
}
Above code fetches over 1000 devices from user_assets table, These IMEI each has its own table that contains over 45,000 records in each table of location data.
The for loop iterates over each IMEI table and records.
Above code runs without error but take so much time to complete, I want to find a solution to optimize and have code execute in a very short time max 5 seconds.
I need help and suggestions on optimizing and running this large scale of data and iteration.
(from Comment)
CREATE TABLE gs_object_data_863844052008346 (
dt_server datetime NOT NULL,
dt_tracker datetime NOT NULL,
lat double DEFAULT NULL,
lng double DEFAULT NULL,
altitude double DEFAULT NULL,
angle double DEFAULT NULL,
speed double...
(From Comment)
gs_object_data_072101424612
gs_object_data_072101425049
gs_object_data_072101425486
gs_object_data_072101445153
gs_object_data_111111111111111
gs_object_data_1234567894
gs_object_data_222222222222222
gs_object_data_2716325849
gs_object_data_2716345818
gs_object_data_30090515907
gs_object_data_3009072323
gs_object_data_3009073758
gs_object_data_352093088838221
gs_object_data_352093088839310
gs_object_data_352093088840045
gs_object_data_352121088128697
gs_object_data_352121088132681
gs_object_data_352621109438959
gs_object_data_352621109440203
gs_object_data_352625694095355
gs_object_data_352672102822186
gs_object_data_352672103490900
gs_object_data_352672103490975
gs_object_data_352672103490991
gs_object_data_352887074794052
gs_object_data_352887074794102
gs_object_data_352887074794193
gs_object_data_352887074794417
gs_object_data_352887074794425
gs_object_data_352887074794433
gs_object_data_352887074794441
gs_object_data_352887074794458
gs_object_data_352887074794474
gs_object_data_352887074813696
gs_object_data_352887074813712
gs_object_data_352887074813720
gs_object_data_352887074813753
gs_object_data_352887074813761
gs_object_data_352887074813803
900+ tables each having different location data.
Requirement: Loop through each table, fetch data for selected date range say:
"SELECT dt_server FROM gs_object_data_" . $row["imei"] . " WHERE dt_server BETWEEN '2022-02-05 00:00:00' AND '2022-02-12 00:00:00'";
Expected Result: Return result set containing data from each table containing information for the selected date range. That means having 1000 tables will have to be looped through each table and also fetch data in each table.
I agree with KIKO -- 1 table not 1000. But, if I understand the rest, there are really 2 or 3 main tables.
Looking at your PHP -- It is often inefficient to look up one list, then go into a loop to find more. The better way (perhaps 10 times as fast) is to have a single SELECT with a JOIN to do both selects at once.
Consider some variation of this MySQL syntax; it may avoid most of the PHP code relating to $days_ago:
CURDATE() - INTERVAL 3 DAY
After also merging the Selects, this gives you the rows for the last 7 days:
WHERE date >= CURDATE() - INTERVAL 7 DAY
(I did not understand the need for LIMIT 1; please explain.)
Yes, you can use DATETIME values as strings, but try not to. Usually DateTime functions are more efficient.
Consider "composite" indexes:
INDEX(imei, dt)
which will be very efficient for
WHERE imei = $imei
AND dt >= CURDATE() - INTERVAL 7 DAY
I would ponder ways to have less redundancy in the output; but that should mostly be done after fetching the raw data from the table(s).
Turn on the SlowLog with a low value of long_query_time; it will help you locate the worst query; then we can focus on it.
An IMEI is up to 17 characters, always digits? If you are not already using this, I suggest BIGINT since it will occupy only 8 bytes.
For further discussion, please provide SHOW CREATE TABLE for each of the main tables.
Since all those 1000 tables are the same it would make sense to put all that data into 1 table. Then partition that table on date, use proper indexes, and optimize the query.
See: Normalization of Database
Since you limit results to one user, and one row per device, it should be possible to execute a query in well below one second.

MySql timestamp query and comparison in monthly and weekly

i have a table where i enter entries for applicants. it has a column date_added where the type is timestamp and the default is current_timestamp. i want to make a weekly and monthly report but i can't decide what would be the best thing to do. i need an sql statement that return the number of entry like this..
switch($searchby){
case "monthly": $qry = "select * from tblapplicant where date_added > $month"; break;
case "weekly": $qry = "select * from tblapplicant where date_added > $week"; break;
}
$res = mysql_query($select);
$rec_count = mysql_num_rows($res);
echo "There are <font color='red' size='3'>".$rec_count."</font> matching records found.";
i know this is incorrect but this is all i can think at the moment.
another thing i want is, i want it to be the exact year and month report..
edit:
output should be:
There are 13 applicants for the whole month of June 2013. - or if weekly -
There are 3 applicants for the third quarter of July 2013.
I suggest the following approach. First, use your php code to determine the day that your search period begins, and the day after it ends. For example, if it's a monthtly report for September 2013, your start date would be 2013-09-01, and the end date would be 2013-10-01. Then you simply have one query.
where date_added >= $startdate
and date_added < $enddate
The major benefits of this approach are:
The time component of your timestamp field gets handled.
Your query will run reasonably quickly, especially if date_added is indexed.
It's easier to sort out programmnig logic with application code than with sql
You can write your query as a stored procedure to make it even faster.
This can be your sql statement
select * from tblapplicant where month(date_added) > $month
select * from tblapplicant where DAYOFWEEK(date_added) = X
You should replace X whith your weeek day 1 = Sunday, 2 = Monday, …, 7 = Saturday

Two appointments a week php

I have the following situation:
A patient is allowed to make 2 online appointments a week from today.
My code is:
$user_id = $_SESSION['id'];
$datum1 = date("d-n-Y");
$datum2 = date("d-n-Y", strtotime(date("Y-m-d", strtotime($datum)) . " +7 days"));
$query = "SELECT * FROM afspraken WHERE user_id='$user_id' AND datum BETWEEN '$datum1' AND '$datum2'";
The query with values:
SELECT * FROM afspraken WHERE user_id='2' AND datum BETWEEN '24-4-2012' AND '01-5-2012'
Still PhpMyAdmin gives 0 results whilst there are more then 10 records.
id user_id datum begintijd opmerking
55 2 24-4-2012 9:30 Deze afspraak is online gemaakt.
56 2 24-4-2012 10:00 Deze afspraak is online gemaakt.
Does someone see whats wrong in my queries?
Thanks in advance.
You can exclude date representation errors by leaving it to MySQL.
$query = "SELECT * FROM afspraken WHERE user_id='$user_id' AND datum BETWEEN CURRENT_DATE AND DATE_ADD(CURRENT_DATE, INTERVAL 1 WEEK)";
In general you probably need format 'YYYY-MM-DD'.
Another thing: your code is vulnerable for SQL injection. Use a prepared statement. They are nicer to read too, and escaping single quotes and backslashes in string fields etcera.
From the MYSQL Doc:
For best results when using BETWEEN with date or time values, use CAST() to explicitly convert the values to the desired data type.
http://dev.mysql.com/doc/refman/5.5/en/comparison-operators.html#operator_between
I would try casting the dates as DATE MySQL datatype in your WHERE clause.
So change you query to:
SELECT * FROM afspraken WHERE user_id='2' AND datum BETWEEN CAST('24-4-2012' AS DATE) AND CAST('01-5-2012' AS DATE);
For first check if in your database the field 'datum' is a date field.
Replace your query like this:
SELECT * FROM afspraken WHERE user_id='2' AND datum BETWEEN '24/4/2012' AND '01/5/2012'

MySQL & PHP: summing up data from a table

Okay guys, this probably has an easy answer but has been stumping me for a few hours now.
I am using PHP/HTML to generate a table from a MySQL Table. In the MySQL table (TimeRecords) I have a StartTime and EndTime column. In my SELECT statement I am subtracting the EndTime from the StartTime and aliasing that as TotalHours. Here is my query thus far:
$query = "SELECT *,((EndTime - StartTime)/3600) AS TotalPeriodHours
FROM TimeRecords
WHERE Date
BETWEEN '{$CurrentYear}-{$CurrentMonth}-1'
AND '{$CurrentYear}-{$CurrentMonth}-31'
ORDER BY Date
";
I then loop that through an HTML table. So far so good. What I would like to do is to add up all of the TotalHours and put that into a separate DIV. Any ideas on 1) how to write the select statement and 2) where to call that code from the PHP/HTML?
Thanks in advance!
Try this
$query= "
SELECT ((EndTime - StartTime)/3600) AS Hours, otherFields, ...
FROM TimeRecords
WHERE
Date BETWEEN '{$CurrentYear} - {$CurrentMonth} - 1'
AND '{$CurrentYear}-{$CurrentMonth} - 31' ";
$records =mysql_query($query);
$sum= 0;
while($row=mysql_fetch_array($records))
{
echo"$row['otherFields']";
echo"$row['Hours']";
$sum+=$row['Hours'];
}
echo" Total Hours : $sum ";
Just use a single query with a Sum(). You could also manually calculate it if you're already displaying all rows. (If paginating or using LIMIT, you'll need a separate query like below.)
$query = "
SELECT Sum(((EndTime - StartTime)/3600)) AS SumTotalPeriodHours
FROM TimeRecords
WHERE
Date BETWEEN '{$CurrentYear} - {$CurrentMonth} - 1'
AND '{$CurrentYear}-{$CurrentMonth} - 31'
";
You can do this in the same query if you have a unique id using GROUP BY WITH ROLLUP
$query = "
SELECT unique_id,SUM((EndTime - StartTime)/3600) AS TotalPeriodHours
FROM TimeRecords
WHERE Date BETWEEN '{$CurrentYear}-{$CurrentMonth}-1'
AND '{$CurrentYear}-{$CurrentMonth}-31'
GROUP BY unique_id WITH ROLLUP
ORDER BY Date
";
In this instance the last result from your query with contain NULL and the overall total. If you don't have a unique ID you will need to do it in PHP as per Naveen's answer.
A few comments on your code:
Using SELECT * is not considered good practice. SELECT the columns you need.
Not all months have a day 31 so this may produce unexpected results. If you're using PHP5.3+, you can use
$date = new DateTime();
$endDate = $date->format( 'Y-m-t' );
The "t" flag here gets the last day of that month. See PHP docs for more on DateTime.

SQL Items within the Last Day

In my code, I am trying to find items in an activities table that are within the last day. This query is not returning any results, are there any problems with it? Is there a better query?
$curday = time() - (24*3600);
$query = "SELECT * FROM activities WHERE userid = '$userid' AND 'timestamp' > '$curday'";
There are two choices here, you can get and format the date through PHP or use SQL language to do it. I prefer to do it within the SQL, it also allows me to use the same query in a MySQL client.
This question is essentially the same thing: MySQL SELECT last few days?
This would be the new query:
$query = "SELECT * FROM activities WHERE userid = '$userid' AND 'timestamp' > DATE_ADD(CURDATE(), INTERVAL -1 DAY)";
you can try with unix function 'mktime' to get value of yesterday ..
as
$curday = mktime(0,0,0,date("m"),date("d")-1,date("Y"));
for reference
if your database will mysql only then you can extract yesterday in sql itself..
SELECT * FROM activities
WHERE userid = '$userid'
AND timestamp > DATE_SUB(CONCAT(CURDATE(), ' 00:00:00'), INTERVAL 1 DAY)
one more thing if timestamp is your column name don't put this column inside single quote ..
What you can use is DATE_SUB. This can be used as follows
SELECT * FROM activities
WHERE userid = '$userid'
AND timestamp > date_sub(current_date, interval 1 day)
This way you don't need to work with current date in PHP
in Informix it would be (TODAY - 1) if the column is type DATE

Categories