I use MySQL DATETIME column to store date & time. Dates are in UTC. I want to select item from one day. What i'm doing now:
SELECT * FROM data WHERE DATE(CONVERT_TZ(datetime, 'UTC', 'Australia/Sydney')) = '2012-06-01'
note that the timezone depends on user
Problem is that it is quite slow with table growing.
Is there any solution how to make it faster?
Currently your query has to compute the conversion for every row of the database. You probably could make things better by converting the other way round, so that the conversion only occurs once (or actually twice, as you'll have to form a range). Then a proper index on datetime should make things pretty fast.
SELECT * FROM data
WHERE datetime BETWEEN CONVERT_TZ('2012-06-01 00:00:00', 'Australia/Sydney', 'UTC')
AND CONVERT_TZ('2012-06-01 23:59:59', 'Australia/Sydney', 'UTC')
Or if you worry about a 23:60:00 leap second not getting matched by any query, you can do
SELECT * FROM data
WHERE datetime >= CONVERT_TZ('2012-06-01', 'Australia/Sydney', 'UTC')
AND datetime < CONVERT_TZ('2012-06-01' + INTERVAL 1 DAY, 'Australia/Sydney', 'UTC')
In the latter form, you wouldn't have to add the hours PHP-side but instead could simply pass the date as a string parameter.
Depending on your real goal, using TIMESTAMP instead of DATETIME may be a good solution.
TIMESTAMP stores the datetime as UTC, converting as it stores and as it is fetched, from/to the local timezone. This way, what I read from your table is automatically different than what you stored (assuming we are in different timezones).
Yes, use #MvG's approach of flipping the query. Yes, use his second form.
And index the column. In composite indexes, put the timestamp last, even though it is more selective.
DO NOT do SELECT *
Indexing - make sure apropriate colunms/id
fields are indexed.
Do time-conversion php-side.
OR make sure you do 1 & 2 and it may be wrapped into a Stored Proc, passing timezone as param.
Currently MySQL query will be as below:
SELECT * FROM data
WHERE datetime >= CONVERT_TZ('2012-06-01', '+00:00', '+10:00')
AND datetime < CONVERT_TZ('2012-06-01' + INTERVAL 1 DAY, '+10:00', '+00:00')
Related
I am stuck for couple of Days on SQL specific scenario. The scenario is as follows,
I have a table, lets call it traffic which has 2 columns -> date and `vehicle (well many more but those are the two I need to match).
The date column is stored as Unix Timestamp. Now this would have been easy to just compare the current date (obtain from php from time() function) however the trick here is that some of these dates have time attached to them also.
For example if you run strtotime(13-02-2017 13:00) and strtotime(13-02-2017) you will get 2 different results. Basically I only care to match the date and not the time.
So I need some way to select the vehicle and date from the database that are equalled to the current Unix Timestamp but with the trick explained above, so I just need to much the date ONLY if possible.
You can use FROM_UNIXTIME() to convert a timestamp to a datetime, and then use the DATE() function to get the date part of that.
WHERE DATE(FROM_UNIXTIME(date)) = CURDATE()
However, this can't use an index, so another way that can make use of an index is to check if it's in a range of timestamps for the current date:
WHERE date BETWEEN UNIX_TIMESTAMP(CURDATE()) AND UNIX_TIMESTAMP(CURDATE()) + 86399
(there are 86400 seconds in a day).
SELECT * FROM traffic WHERE DATE(date) = DATE(CURRENT_TIMESTAMP);
I currently store my datetimes in UTC, and for all purposes, it's worked fine so far. But now I want to query for records created on a particular day, like today, in my own timezone, which is Eastern.
Due to the difference in hours, records I'm creating at the moment will have tomorrow's date. For instance, the date portion would currently be 2016-06-20 in my timezone, but UTC is 2016-06-21.
How could I accurately get records for a particular day in my own timezone when the records are stored as UTC? I was thinking that I could use the convert_tz function to convert the stored value to Eastern, and use that as part of my where clause. That way the dates match up. Is that a reliable solution?
Convert the datetime boundaries you are searching for into UTC.
The CONVERT_TZ function is convenient. Something like this:
WHERE my_utc_date_time_col >= CONVERT('2016-02-20','EST5EDT','+00:00')
AND my_utc_date_time_col < CONVERT('2016-02-20','EST5EDT','+00:00') + INTERVAL 1 DAY
(This assumes that you've populated the MySQL time zone tables, so named time zones are supported.)
That winds up effectively equivalent to:
WHERE my_utc_date_time_col >= '2016-02-20' + INTERVAL 4 HOUR
AND my_utc_date_time_col < '2016-02-20' + INTERVAL 4 HOUR + INTERVAL 1 DAY
or
WHERE my_utc_date_time_col >= '2016-02-20 04:00:00'
AND my_utc_date_time_col < '2016-02-21 04:00:00'
If you want to reference a function such as NOW() to get the current date, then the timezone of the MySQL connection is going to be important, because the datetime value returned will be in the time_zone of the MySQL connection.
SHOW VARIABLES LIKE 'time_zone' ;
Note that we prefer to do the conversion on the literal side rather than on the column, because if we do it on the column, that forces MySQL to perform the conversion for every row in the table, and compare the literal. When we do the conversion on the literal side, that allows MySQL to make effective use an index range scan operation (with an appropriate index available.)
The CONVERT_TZ function is in fact what you need. But I disagree with:
I could use the convert_tz function to convert the stored value to
Eastern, and use that as part of my where clause.
Because using functions on the WHERE clause might reduce performance.
If you're using stored procedures, I suggest that you convert first the input dates, then use them in your SELECT statement. It would look something like this:
SET #fromDate = CONVERT_TZ(#fromDate, ...) --Convert to UTC
SET #toDate = CONVERT_TZ(#toDate, ...) --Convert to UTC
SELECT *
FROM TableA A
WHERE A.FromDate BETWEEN #fromDate AND #toDate
http://ee1.php.net/manual/en/class.datetime.php
$foo = new DateTime("today", new DateTimeZone("America/Vancouver"));
var_dump($foo->format('c'), $foo->format('U'));
$foo->setTimeZone(new DateTimeZone("UTC"));
var_dump($foo->format('c'), $foo->format('U'), $foo->format('Y-m-d H:i:s'));
Output:
string(25) "2016-06-20T00:00:00-07:00"
string(10) "1466406000"
string(25) "2016-06-20T07:00:00+00:00"
string(10) "1466406000"
string(19) "2016-06-20 07:00:00"
https://dev.mysql.com/doc/refman/8.0/en/time-zone-support.html
SELECT DATE(CONVERT_TZ(NOW(), 'UTC', 'America/Vancouver')) as VancouverToday;
I'm not saying this is optimized, fast, etc. I'm just saying if you've stored UTC values and/or your server is UTC and you're looking for a comparison for Today, this shows you "today" adjusted for your timezone.
What is the best way to store date and time into separate fields in MySQL database.
I will perform following actions in the query :
sort the tables with date, ASC or DSC
sort the tables with respect to time,
select tables by certain conditions such as:
example query :
SELECT * FROM table WHERE starttime > '$starttime AND endtime < $endtime
lets say
$date = "11/12/2016";
$starttime = "09:00 AM";
$endtime = "05:00 PM";
what is the best way to store these in correct format?
In our use cases we have to consider time zones, so a complete date is a combination of date,time and timezone.
Since date and time are related I prefer to translate everything to miliseconds since 1.1.1970 in UTC (epoch timestamp) and store these values. The advantage is an absolute time without the need to remember the timezone. Sorting, filtering, finding intersection of appointments is very easy.
However the moment you translate the long value back to a human readable value you need the time zone for which the date was created. So you must keep this data somewhere. In general the timezone is the for one user so all date related to a user have the same timezone.
I have a DATETIME string and I need only the DATE in my script to perform some searches in my database. Currently, I have two scenarios in my mind, but don't know which of them is faster.
The first scenario:
In my MYSQL database, I have two columns: datetime (which is a DATETIME type) and date (which is a DATE type).
Then, in my PHP script, each time I save a record, I will insert my known string to the datetime field, and then convert it to fit the date field (I was thinking of something like: $date = date("Y-m-d", strtotime($datetime))).
This way, all the necessary pieces are stored in my database and I can retrieve them on the fly (both the datetime and the date fields).
The second scenario:
The MYSQL database should consist only of the datetime column.
My PHP script will insert the known string to the datetime field without any other modifications.
And when I retrieve my data, I would do something like: SELECT datetime, DATE(datetime) FROM ...
Conclusion
Which of these scenarios is faster and therefore should be used? Should date formats be made on save or on retrieve? Is MYSQL faster than PHP on formatting dates? Is it better to store everything in the database and retrieve as it is, or store only the minimum and format on retrieve? Which of these scenarios is the best practice?
Thank you!
It depends of your usecases:
If you are only going to need the date for reading, then go with a single datetime column, conversion from datetime to date is cheap enough.
If you are going to select rows at a given date (like WHERE date = '2011-08-01'), then go for a date column, as this will allow mysql to use the indexes on the date column if you have added one.
If you are going to select rows in a date range, then go for a datetime column. You could do things like WHERE datetime >= '2011-08-01' AND datetime < '2011-08-16'.
The second one is the best and fast as you are getting the value based on the requirement. Rather getting some value and working on it later.
imho
datetime, or even unsigned integer (unix timestamp) is better for range filtering
datetime allow date-time function, it could be useful for aggregate function
avoid formatted data from mysql (that's mean raw)
anything related to presentation is PHP duty
Definitely depends on your situation - if you will be reading (a lot) more than writing, you can store both. But I'd go for storing one field (datetime) and convert that, either in PHP or while retrieving it from MySQL (convert datetime to char in the format you like)
I'm using the America/New York timezone. In the Fall we "fall back" an hour -- effectively "gaining" one hour at 2am. At the transition point the following happens:
it's 01:59:00 -04:00
then 1 minute later it becomes:
01:00:00 -05:00
So if you simply say "1:30am" it's ambiguous as to whether or not you're referring to the first time 1:30 rolls around or the second. I'm trying to save scheduling data to a MySQL database and can't determine how to save the times properly.
Here's the problem:
"2009-11-01 00:30:00" is stored internally as 2009-11-01 00:30:00 -04:00
"2009-11-01 01:30:00" is stored internally as 2009-11-01 01:30:00 -05:00
This is fine and fairly expected. But how do I save anything to 01:30:00 -04:00? The documentation does not show any support for specifying the offset and, accordingly, when I've tried specifying the offset it's been duly ignored.
The only solutions I've thought of involve setting the server to a timezone that doesn't use daylight savings time and doing the necessary transformations in my scripts (I'm using PHP for this). But that doesn't seem like it should be necessary.
Many thanks for any suggestions.
I've got it figured out for my purposes. I'll summarize what I learned (sorry, these notes are verbose; they're as much for my future referral as anything else).
Contrary to what I said in one of my previous comments, DATETIME and TIMESTAMP fields do behave differently. TIMESTAMP fields (as the docs indicate) take whatever you send them in "YYYY-MM-DD hh:mm:ss" format and convert it from your current timezone to UTC time. The reverse happens transparently whenever you retrieve the data. DATETIME fields do not make this conversion. They take whatever you send them and just store it directly.
Neither the DATETIME nor the TIMESTAMP field types can accurately store data in a timezone that observes DST. If you store "2009-11-01 01:30:00" the fields have no way to distinguish which version of 1:30am you wanted -- the -04:00 or -05:00 version.
Ok, so we must store our data in a non DST timezone (such as UTC). TIMESTAMP fields are unable to handle this data accurately for reasons I'll explain: if your system is set to a DST timezone then what you put into TIMESTAMP may not be what you get back out. Even if you send it data that you've already converted to UTC, it will still assume the data's in your local timezone and do yet another conversion to UTC. This TIMESTAMP-enforced local-to-UTC-back-to-local roundtrip is lossy when your local timezone observes DST (since "2009-11-01 01:30:00" maps to 2 different possible times).
With DATETIME you can store your data in any timezone you want and be confident that you'll get back whatever you send it (you don't get forced into the lossy roundtrip conversions that TIMESTAMP fields foist on you). So the solution is to use a DATETIME field and before saving to the field convert from your system time zone into whatever non-DST zone you want to save it in (I think UTC is probably the best option). This allows you to build the conversion logic into your scripting language so that you can explicitly save the UTC equivalent of "2009-11-01 01:30:00 -04:00" or ""2009-11-01 01:30:00 -05:00".
Another important thing to note is that MySQL's date/time math functions don't work properly around DST boundaries if you store your dates in a DST TZ. So all the more reason to save in UTC.
In a nutshell I now do this:
When retrieving the data from the database:
Explicitly interpret the data from the database as UTC outside of MySQL in order to get an accurate Unix timestamp. I use PHP's strtotime() function or its DateTime class for this. It can not be reliably done inside of MySQL using MySQL's CONVERT_TZ() or UNIX_TIMESTAMP() functions because CONVERT_TZ will only output a 'YYYY-MM-DD hh:mm:ss' value which suffers from ambiguity problems, and UNIX_TIMESTAMP() assumes its input is in the system timezone, not the timezone the data was ACTUALLY stored in (UTC).
When storing the data to the database:
Convert your date to the precise UTC time that you desire outside of MySQL. For example: with PHP's DateTime class you can specify "2009-11-01 1:30:00 EST" distinctly from "2009-11-01 1:30:00 EDT", then convert it to UTC and save the correct UTC time to your DATETIME field.
Phew. Thanks so much for everyone's input and help. Hopefully this saves someone else some headaches down the road.
BTW, I am seeing this on MySQL 5.0.22 and 5.0.27
MySQL's date types are, frankly, broken and cannot store all times correctly unless your system is set to a constant offset timezone, like UTC or GMT-5. (I'm using MySQL 5.0.45)
This is because you can't store any time during the hour before Daylight Saving Time ends. No matter how you input dates, every date function will treat these times as if they are during the hour after the switch.
My system's timezone is America/New_York. Let's try storing 1257051600 (Sun, 01 Nov 2009 06:00:00 +0100).
Here's using the proprietary INTERVAL syntax:
SELECT UNIX_TIMESTAMP('2009-11-01 00:00:00' + INTERVAL 3599 SECOND); # 1257051599
SELECT UNIX_TIMESTAMP('2009-11-01 00:00:00' + INTERVAL 3600 SECOND); # 1257055200
SELECT UNIX_TIMESTAMP('2009-11-01 01:00:00' - INTERVAL 1 SECOND); # 1257051599
SELECT UNIX_TIMESTAMP('2009-11-01 01:00:00' - INTERVAL 0 SECOND); # 1257055200
Even FROM_UNIXTIME() won't return the accurate time.
SELECT UNIX_TIMESTAMP(FROM_UNIXTIME(1257051599)); # 1257051599
SELECT UNIX_TIMESTAMP(FROM_UNIXTIME(1257051600)); # 1257055200
Oddly enough, DATETIME will still store and return (in string form only!) times within the "lost" hour when DST starts (e.g. 2009-03-08 02:59:59). But using these dates in any MySQL function is risky:
SELECT UNIX_TIMESTAMP('2009-03-08 01:59:59'); # 1236495599
SELECT UNIX_TIMESTAMP('2009-03-08 02:00:00'); # 1236495600
# ...
SELECT UNIX_TIMESTAMP('2009-03-08 02:59:59'); # 1236495600
SELECT UNIX_TIMESTAMP('2009-03-08 03:00:00'); # 1236495600
The takeaway: If you need to store and retrieve every time in the year, you have a few undesirable options:
Set system timezone to GMT + some constant offset. E.g. UTC
Store dates as INTs (as Aaron discovered, TIMESTAMP isn't even reliable)
Pretend the DATETIME type has some constant offset timezone. E.g. If you're in America/New_York, convert your date to GMT-5 outside of MySQL, then store as a DATETIME (this turns out to be essential: see Aaron's answer). Then you must take great care using MySQL's date/time functions, because some assume your values are of the system timezone, others (esp. time arithmetic functions) are "timezone agnostic" (they may behave as if the times are UTC).
Aaron and I suspect that auto-generating TIMESTAMP columns are also broken. Both 2009-11-01 01:30 -0400 and 2009-11-01 01:30 -0500 will be stored as the ambiguous 2009-11-01 01:30.
I think micahwittman's link has the best practical solution to these MySQL limitations: Set the session timezone to UTC when you connect:
SET SESSION time_zone = '+0:00'
Then you just send it Unix timestamps and everything should be fine.
But how do I save anything to 01:30:00
-04:00?
You can convert to UTC like:
SELECT CONVERT_TZ('2009-11-29 01:30:00','-04:00','+00:00');
Even better, save the dates as a TIMESTAMP field. That's always stored in UTC, and UTC doesn't know about summer/winter time.
You can convert from UTC to localtime using CONVERT_TZ:
SELECT CONVERT_TZ(UTC_TIMESTAMP(),'+00:00','SYSTEM');
Where '+00:00' is UTC, the from timezone , and 'SYSTEM' is the local timezone of the OS where MySQL runs.
Mysql inherently solves this problem using time_zone_name table from mysql db.
Use CONVERT_TZ while CRUD to update the datetime without worrying about daylight savings time.
SELECT
CONVERT_TZ('2019-04-01 00:00:00','Europe/London','UTC') AS time1,
CONVERT_TZ('2019-03-01 00:00:00','Europe/London','UTC') AS time2;
This thread made me freak since we use TIMESTAMP columns with On UPDATE CURRENT_TIMESTAMP (ie: recordTimestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP) to track changed records and ETL to a datawarehouse.
In case someone wonder, in this case, TIMESTAMP behave correctly and you can differentiate between the two similar dates by converting the TIMESTAMP to unix timestamp:
select TestFact.*, UNIX_TIMESTAMP(recordTimestamp) from TestFact;
id recordTimestamp UNIX_TIMESTAMP(recordTimestamp)
1 2012-11-04 01:00:10.0 1352005210
2 2012-11-04 01:00:10.0 1352008810
I was working on logging counts of visits of pages and displaying the counts in graph (using Flot jQuery plugin). I filled the table with test data and everything looked fine, but I noticed that at the end of the graph the points were one day off according to labels on x-axis. After examination I noticed that the view count for day 2015-10-25 was retrieved twice from the database and passed to Flot, so every day after this date was moved by one day to right.
After looking for a bug in my code for a while I realized that this date is when the DST takes place. Then I came to this SO page...
...but the suggested solutions was an overkill for what I needed or they had other disadvantages. I am not very worried about not being able to distinguish between ambiguous timestamps. I just need to count and display records per days.
First, I retrieve the date range:
SELECT
DATE(MIN(created_timestamp)) AS min_date,
DATE(MAX(created_timestamp)) AS max_date
FROM page_display_log
WHERE item_id = :item_id
Then, in a for loop, starting with min_date, ending with max_date, by step of one day (60*60*24), I'm retrieving the counts:
for( $day = $min_date_timestamp; $day <= $max_date_timestamp; $day += 60 * 60 * 24 ) {
$query = "
SELECT COUNT(*) AS count_per_day
FROM page_display_log
WHERE
item_id = :item_id AND
(
created_timestamp BETWEEN
'" . date( "Y-m-d 00:00:00", $day ) . "' AND
'" . date( "Y-m-d 23:59:59", $day ) . "'
)
";
//execute query and do stuff with the result
}
My final and quick solution to my problem was this:
$min_date_timestamp += 60 * 60 * 2; // To avoid DST problems
for( $day = $min_date_timestamp; $day <= $max_da.....
So I am not staring the loop in the beginning of the day, but two hours later. The day is still the same, and I am still retrieving correct counts, since I explicitly ask the database for records between 00:00:00 and 23:59:59 of the day, regardless of the actual time of the timestamp. And when the time jumps by one hour, I am still in the correct day.
Note: I know this is 5 year old thread, and I know this is not an answer to OPs question, but it might help people like me who encountered this page looking for solution to the problem I described.