I have table in MySQL which has columns
id(primary key) ,deviceid(varchar) ,date(varchar),time(varchar),value(varchar) parameter(varchar)
I made mistake by separating time and date fields in my DB columns and then with my business logic where with lot of repetitive code
A new record is inserted every minute in database every day. My major query is to fetch data between two dates for a given number of deviceids such that data only one value is selected per interval ( this interval is decide by user input for exampe if interval is 1 hour i need 1 value between interval 00-01 hrs,1 value in terval hrs 01-02 hrs and so on 1 value from 23-24(00) hours )
My query is taking almost a minute to get data for and display for 2-3 days of data . And shorter the interval gets more time it takes .
I am new to databases and only knows basic CRUD operation . I have designed tabled through php myadmin default options.
I read about the concept of indexing to improve search performance and i am confused on which columns should I apply indexing.
Also my value columun is currently varchar but values of value column are all floating numbers(temperature/humidity of a boiler room). Can changing its type from varchar to int increase my speed ?
here is more explanation
I am using my php to display report
my time is in string format and so is date
for example date is stored as '08.31.2020'
and time is stored '23.05' (hours,min)
let say I want data from '08.01.2020' to '08.20.2020' and interval of 1 hour =60 minutes
Here is my business logic and querys
$time1 ='00.00'
$time2 ='01.00' // here I have increased time by adding 60 minutes to $time1 .
while(fromDate <= toDate)
{
for(each device in array)
{
$query = "Select parameter,value,time,date from currentdata where deviceid='device[$i]' and date='fromDate' and (time > '$time1' and time <= '$time2' ) order by time limit 1 "
}
$time1=$time2; //change interval
$time2 =$time2 +60 minutes //
If(time1>=time2)
{
// increment From date by 1
}
}
No of devices are about 30 -35 .
Parameter is just a name which will be either Temperate of humidity
Any general advice ,suggestion ,criticism will be helpful.
Use appropriate datatypes in the table. Numeric quantities should not be put in VARCHAR; use INT, DECIMAL, or FLOAT, as needed. (Or variants on them.)
If your incoming data is not "clean", the read it into a VARCHAR and 'cleanse' it as you copy it into the SMALLINT UNSIGNED, etc, in the real table.
Also DATE must have the year first.
Use things like + INTERVAL 1 HOUR for simple date or time arithmetic.
Usually it is better not to split date and time for storage, but to split as needed on output.
On your other Question, I admonished you to do the SELECT with a JOIN, not a loop.
After addressing all those things, provide SHOW CREATE TABLE, the new query, etc. Then I will weigh in on the index.
Related
I would like to store a duration in database, in my case it's a training duration (sport). We set when we do the training (datetime) and how long the session is (duration).
At the moment, I've choose the Time type, then I've: 01:00:00 for an hour of training, and retrieve a Datetime object in PHP (but I think it's not ideal to sum them for exemple) and in my form I've an undesired behavior: the html time type automatically set the current hour when the time is empty.
I hesitate in:
keep time, it seem logical/simple to have something like: 00:00:00
store it as secondes or minutes as an integer, maybe simpler to do sum of training amount for exemple (need to create a converter to display it as 00:00 or 00:00:00, and find a nice way to fill it in form)
a way I've not think ?
Thanks a lot for your help.
The correct data type to store a duration is interval.
This makes it easy to do date arithmetic: you can simply add the duration to the start time to get the end time:
SELECT INTERVAL '01:30:00';
interval
----------
01:30:00
(1 row)
SELECT TIME '09:00:00' + INTERVAL '01:30:00';
?column?
----------
10:30:00
(1 row)
SELECT TIMESTAMP WITH TIME ZONE '2021-03-28 02:00:00+01' + INTERVAL '01:30:00';
?column?
------------------------
2021-03-28 04:30:00+02
(1 row)
I'm not sure what your question is. Postgres has no problem using sum() on the time data type:
select sum(t)
from (values ('03:12:00'::time), ('05:27:00'), ('18:59')) v(t)
You can also use interval.
You might have issues if individual durations exceed 24 hours. If that is a potential issue, just use hours, minutes, or seconds.
If you have a problem with "empty" values, then you will need to set them explicitly. It sounds like a bug on the HTML side.
I am using CodeIgniter and I am calculating the total time from the dates.
Explanation:
What I am doing is, Every login I am inserting the last_Ativity date and time in the database using below code.
$data_login= array('emp_id' =>$result->id ,'last_activity' =>date("Y-m-d H:i:s", STRTOTIME(date('h:i:sa'))));
$this->db->insert('tbl_current_login',$data_login);
last_activity time continuously updating if the user still in the system . (I am using ajax to update the datetime. I haven't shared that code).
Now I have to calculate the total time of the specific user for a single day(current date).
For example- emp_id 26 logged in twice so I have to calculate the time
First-time login date and time:-2018-09-17 07:27:55
Second-time login date and time:- 2018-09-17 07:35:22
It will increase depending upon how many time the user logged in.
I am confused about the time. Am I on the right path to calculate the total hour login in the system?
Should I use an MYSQL query or PHP to calculate? I need some idea.
Would you help me out in this?
This is what I would do
last_activity time continuously updating if the user still in the system . (I am using ajax to update the datetime. I haven't shared that code).
Before you update the row.
check if a row for activity exists
if it does, get the timestamps for the date and subtract the current time (the one you are changing last_activity to, from the one stored in the DB) take that number and add it to an integer column named something like elapsed time (you would have to add this to the DB)
if not then enter a row with 0 elapsed time ( depending how you put the first row in, maybe on login) this may never be an issue.
For the timestamps, you would do a select to get the current row. Take the datetime field and use either
$time = strtotime($row['last_activity']);
OR
$time = (new DateTime($row['last_activity']))->getTimestamp();
Then you simply do the same thing to the date you are going to replace that with and then subtract to get the difference in seconds.
$elapsed = time() - $time;
And then add that to the current rows value, and save it. This way you can keep track of a running total in seconds of the time they spend during that session.
Then when you need to count the total time its a simple matter of doing
SELECT SUM(elapsed_time) FROM {table} WHERE DATE(last_Ativity) = :date
If you were dealing with just two date time fields in the DB it would be easier to just get the difference of those, but sense you already have code to constantly update the last active field this would require less work in the long run IMO.
Option2
The other option is to add another Datetime field to put a start time or login time in. Then when you query you can convert them to their timestamps and subtract to get the difference.
This makes the SQL harder (when doing the SUM ), I can't really think off the top of my head how I would calculate the elapsed time on multiple rows and then sum them up. But it does simplify the PHP quite a bit. So which ever way works best for what you need. Think about if you need the utility to know when they logged in, or if you just want an easier way to calculate the time they spend.
Something like that.
Assuming that the only log happens based on user actions, and so, after 15 minutes (for example) the user is assumed logged out
And assuming you'd want daily total, the solution should be something like this:
SELECT
first.emp_id,
SUM(TIMESTAMPDIFF(MINUTE,first.last_acivity, DATE_ADD(IFNULL(last.last_acivity, first.last_acivity), INTERVAL 15 MINUTE))) as logged_minutes
FROM
(
SELECT
la1.*
FROM
last_acivity la1
LEFT JOIN last_acivity la2 ON
la1.emp_id = la2.emp_id AND la1.last_acivity < la2.last_acivity
AND la2.activity =< #date0
WHERE
la1.last_acivity >= #date0
AND la2.login_id IS NULL
) first
LEFT JOIN
(
SELECT
la1.*
FROM
last_acivity la1
LEFT JOIN last_acivity la2 ON
la1.emp_id = la2.emp_id AND la1.last_acivity > la2.last_acivity
AND la2.activity =< #date0
WHERE
la1.last_acivity >= #date0
AND la2.login_id IS NULL
) last
ON
first.emp_id = last.emp_id
GROUP BY
emp_id
In this query need to set the date seperately:
SET #date0 = DATE(NOW()) ;
To get the first record of the day, or the last, we need to LEFT join the table to itself, on the same emp_id BUT witn with an inequality, which will get for each emp record its ancestors or predecessors
When we add the NULL condition we bring the we get the edge case: first or last
What's left then is just calculating the minutes between the 2 tables
Since I assumed no log out record occurs, I treated the case when the first and last logins are the same, or no last login
I am trying to insert actual hours not the time itself to MySQL database through form fields. So for example
$time1 = '00:00';
$time2 = '27:20';
$time3 = '00:45';
So I can retrieve the different rows and can calculate on the fly whenever require. Either through search query or even in other area of the system.
When I have tried to do addition of above three times, it is not giving the result the way I am looking for
$total = strtotime($time1) + strtotime($time2) + strtotime($time3);
echo date('H:i:s', $total);
The result
14:16:44
While it should be something like
28:05:00
I have used TIME DATATYPE in MySQL table. I may use as a TEXT but I am also concern about the error happen in user input. Where I do not have to force the user to insert the any particular format but they can either insert as below way
27.20
27:20
or
1.5
1:30
My main concern is to calculate the time, the user input can be on second priority but it would be great if can implement.
So is there anyway, idea or hint to achieve this?
date() expects the timestamp in UNIX format, i.e. seconds since January 1 1970 00:00:00 UTC (which is also the value provided by strtotime)
You're passing it the result of adding a series of amounts of time since 1 January 1970 instead of just adding up hours, so (as far as date is concerned) you're generating a random date and time, and printing only the time (try printing the date of $total and see what you get).
Since your time is stored in the database, one possibility is to let MySQL handle the time calculations itself, e.g.:
SELECT ADDTIME('00:00',ADDTIME('27:20','00:45'))
will result in "28:05:00". You can have your database fields as TIME and operate on them directly through SQL, and do the user input conversions into acceptable TIME values in PHP.
If you're only interested in the hours and minutes, why don't you just store the value as an in integer? Just multiply the hours by 60.
You can handle the conversion in PHP.
Alternatively, you can also easily use two (very small) int fields for this.
I have a query that counts the "Xp" difference per day from my database, this all works as it should however it groups from midnight-midnight, what I would like to do is group 3am to 3am.
However another issue I think I may have is that my query may not always have the rows being the exact second at 3am due to the fact that it has to run a huge query and retrieve data from another website per user profile, so it should get all data after 3am, but before maybe 4am or something, so it has enough time to get all of the rows.
my current mysql is:
SELECT FROM_UNIXTIME(date, '%Y%m%d') AS YYYYMMDD, MAX(xp)-MIN(xp) AS xp_gain
FROM skills
WHERE userID = '$checkID'
AND skill = '$skill'
AND date >= '$date'
GROUP BY YYYYMMDD
ORDER BY date ASC
The best way to handle this is to add (if you can) another column that is just a DATE (not a DATETIME) and have this field rollover from one day to the next at 3am, (you can to this by subtracting 3 hours from the current time when doing the INSERT).
This gives you a couple of benefits, especially with a large number of rows:
It is much faster to query or group by a DATE than a range of
DATETIME
It will always query the rows at the exact second of 3am,
regardless of how long the query takes.
my question is more "theoretical" than practical - in other words, Im not really looking for a particular code for how to do something, but more like an advice about how to do it. Ive been thinking about it for some time but cannot come up with some feasible solution.
So basically, I have a MySQL database that saves weather information from my weather station.
Column one contains date and time of measurement (Datetime format field), then there is a whole range of various columns like temp, humidity etc. The one I am interested in now is the one with the temperature. The data is sorted by date and time ascending, meaning the most recent value is always inserted to the end.
Now, what I want to do is using a PHP script, connect to the db and find temperature changes within a certain interval and then find the maximum. In other words, for example lets say I choose interval 3h. Then I would like to find the time, from all the values, where there was the most significant temperature change in those 3 h (or 5h, 1 day etc.).
The problem is that I dont really know how to do this. If I just get the values from the db, Im getting the values one by one, but I cant think of a way of getting a value that is lets say 3h from the current in the past. Then it would be easy, just subtracting them and get the date from the datetime field at that time, but how to get the values that are for example those 3 h apart (also, the problem is that it cannot just simply be a particular number of rows to the past as the intervals of data save are not regular and range between 5-10mins, so 3 h in the past could be various number of rows).
Any ideas how this could be done?
Thx alot
Not terribly hard actually. So I would assume it's a two column table with time and temp fields, where time is a DATETIME field
SELECT MAX(temp) FROM records
WHERE time >= "2013-10-14 12:00:00" and time <= "2013-10-14 15:00:00"
SELECT t1.*, ABS(t1.temperature - t2.temperature) as change
FROM tablename t1
JOIN tablename t2
ON t2.timecolumn <= (t1.timecolumn - INTERVAL 3 HOUR)
LEFT JOIN tablename t3
ON t3.timecolumn <= (t1.timecolumn - INTERVAL 3 HOUR)
AND t2.timecolumn > t3.timecolumn
WHERE
t3.some_non_nullable_column IS NULL
ORDER BY ABS(t1.temperature - t2.temperature) DESC
LIMIT 1;
1 table joined 2 times on itself, t2 is the quaranteed direct predecessor of t1 t2 is the closest record with offset 3h before or more. This could with the proper indexes, and a limited amount of data (where limited is in the eye of the beholder) be quite performant. However, if you need a lot of those queries in a big dataset, this is a prime candidate for denormalization, were you create a table which also stores the calculated offsets compared to the previous entry.