I'm trying to implement a timeseries db to store simple counters using redis (and php, but the language shouldn't be relevant i think). So I've implemented my redis keys as follows (simplified):
someprefix:YYYY-MM-DD:somecounter
Now when i want to get a range of data for a specific interval i just get all keys for the specific range and that's all working fine. (YYYY-MM-DD is the date as UTC)
Now i want to implement the ability to get data according to some timezone X.
My question is: is there any way this key schema can be used for that with any degree of accuracy?
I'm guessing not, since there's no time information at all so i'll also have to add at least the hours and minutes to the key so timezone conversion works correctly. I also probably should save the information in smaller time intervals otherwise when converting timezones there are cases where I would end up getting all data for a different day when the timezone difference shouldn't be more than 13h therefore giving me wrong results, am I right?
Would it be more appropriate to just use unix timestamps instead of the formatted date on the redis keys? For example, if I later on decide to store data with smaller precision, say per hour or per each 10 minutes, what would be a more flexible key format?
Hope I was able to explain my issue correctly, but please feel free to ask for any clarifications.
Thanks
Its always good to go with epoch (UNIX timestamp) when you have to deal with timezone.
I would suggest bucking timestamps to frame the Keys. For example an event happened at timestamp 1409800502515(Thu, 04 Sep 2014 03:15:02 GMT), you could bucket it at Hour level or Day level like this
Hour bucket = 1409800502515 - (1409800502515 % (60 * 60)) = 1409800500000
Day bucket = 1409800502515 - (1409800502515 % (24 * 60 * 60)) = 1409800464000
and frame keys like
someprefix:1409800500000:somecounter OR
someprefix:1409800464000:somecounter
For example for calculating the page views per hour, find the appropriate hourly bucket and increment the counter
mypage.html:1409800464000:page_views INCR 10
Firstly, I'm not sure how you're doing "get all keys for the specific range and that's all working fine" but if you're using KEYS someprefix:* note that this is not a recommended practice for production. Consider using the SCAN command that's available from v2.8 instead.
Secondly, you could consider using an ordered set for counting. So, following your convention, you'll have a key called someprefix:somecounter that you'll be ZADDing to members with the epoch as their score. Use the epoch and the counter's reading as a unique member name (e.g. '1409800500000:1` where 1409800500000 is the epoch and 1 is the counter's value).
Note that you can measure time resolutions from years to microseconds - it all depends on how much div you apply to the original epoch before setting the score.
Related
Introduction to my website
My website is for visitors in Korea(AKA Republic of Korea).
And the server for My website is in the United States of America.
And PHPMyAdmin displays EDT when it runs a query SELECT ## system_time_zone.
Structure of my website
When I first uploaded my website to this server in October this year, I checked the DB time.
And it seemed that there was a time difference of 13 hours with Korea. So I added 3600 * 13 seconds to DB time(without setting timezone) as follows.
const Offset = 3600 * 13;
$SelectNow = $PDO->prepare('SELECT DATE_ADD(NOW(), INTERVAL '.Offset.' SECOND)');
$SelectNow->execute() or exit;
$DbNow = $SelectNow->fetchColumn();
My website takes $DbNow as above and uses it in various situations.
For example, in the posting situation, enter $DbNow in the datetime field of the INSERT INTO query as follows:
$WriteNote = $PDO->prepare('INSERT INTO table_note(my_datetime, my_contents) VALUES("'.$DbNow.'", :my_contents)');
$WriteNote->bindValue(':my_contents', $my_contents, PDO::PARAM_STR);
$WriteNote->execute();
The problem situation
One day in November of this year, when I wrote a post and checked the date field(my_datetime) of the post, I got an additional time difference of one hour with Korea.
Apparently, at the end of October, I corrected the time difference of 3600 * 13. And then I confirmed that it matches the Korean time. However, in November, There is a time difference of one hour!
Guess the cause
It seems that US summer time is being applied to the DB server of my website. Did I guess right?
My question
1) How can I solve this time difference fundamentally?
Is it correct to convert DB time to KST?
Or is it the correct way to convert to UTC and then added 3600 * x to UTC?
2) Even though the problem is resolved, some of the existing data in my DB has a time difference of one hour with Korean time.
What query do I use if I want to select the data with a time difference?
And how much more or subtract it from the data to get rid of the 1 hour time difference?
Use UTC to store time in Database.
change your queries to insert with UTC datetimes.
Use external libraries to convert UTC to respective timezones.
(below are the my personal recommendation.)
There may be best of it.
PHP : Carbon
Javascript : Moment, moment timezone.
No, it takes timezone of Database server resides in.
little manual verification, or create a job to change all dates in UTC.
Edit:
http://carbon.nesbot.com/docs/
I mean you can create a script and run with cron job.
I'm currently working on a project where I'm recording times into my database, and I want to store the difference between the two times as well. I implemented them using the SQL object: TIMESTAMP and recorded the timestamp using
TIMESTAMP(CURDATE(),CURTIME())
These store fine. Now I want to compute the difference between the two times, but it looks like PHP's TIMESTAMPDIFF() function takes in datetime objects instead of timestamp objects. There are a few ways I could move on from here, but I was wondering if there's a preferred way that SQL developers record and compute time differences. I need both the date and the time, so that I can get the difference accurate to the second.
This seems to be a 2 part question and is definitely duplicates of other questions on SO. A quick search produced a SO question that over 1000 people voted for DATETIME as the storage type (Should I use field 'datetime' or 'timestamp'?)
The main difference between DATETIME and TIMESTAMP in mysql is that TIMESTAMP will store that date at UTC and datetime will store your specific value. So if you don't need to translate between time zones, or have multiple time zones etc. Stick with DATETIME and the you don't have to convert or cast or anything.
As far as getting the difference between 2 datetimes or 2 timestamps I am certain their are SO questions on here I suggest searching a little more.
I am trying to insert actual hours not the time itself to MySQL database through form fields. So for example
$time1 = '00:00';
$time2 = '27:20';
$time3 = '00:45';
So I can retrieve the different rows and can calculate on the fly whenever require. Either through search query or even in other area of the system.
When I have tried to do addition of above three times, it is not giving the result the way I am looking for
$total = strtotime($time1) + strtotime($time2) + strtotime($time3);
echo date('H:i:s', $total);
The result
14:16:44
While it should be something like
28:05:00
I have used TIME DATATYPE in MySQL table. I may use as a TEXT but I am also concern about the error happen in user input. Where I do not have to force the user to insert the any particular format but they can either insert as below way
27.20
27:20
or
1.5
1:30
My main concern is to calculate the time, the user input can be on second priority but it would be great if can implement.
So is there anyway, idea or hint to achieve this?
date() expects the timestamp in UNIX format, i.e. seconds since January 1 1970 00:00:00 UTC (which is also the value provided by strtotime)
You're passing it the result of adding a series of amounts of time since 1 January 1970 instead of just adding up hours, so (as far as date is concerned) you're generating a random date and time, and printing only the time (try printing the date of $total and see what you get).
Since your time is stored in the database, one possibility is to let MySQL handle the time calculations itself, e.g.:
SELECT ADDTIME('00:00',ADDTIME('27:20','00:45'))
will result in "28:05:00". You can have your database fields as TIME and operate on them directly through SQL, and do the user input conversions into acceptable TIME values in PHP.
If you're only interested in the hours and minutes, why don't you just store the value as an in integer? Just multiply the hours by 60.
You can handle the conversion in PHP.
Alternatively, you can also easily use two (very small) int fields for this.
All,
I'm trying to decide how to deal with time in a project which relies on (server) time intervals (in short, some content is available after user completed a specific action at least n hours before). Right now, it seems like the easiest option would be to extract the Unix time stamp with time() and store it as is in MySQL.
Any reason why this is not a good idea? Any gotcha I need to be aware of? Performance impact?
Timestamps are fine. Don't divide them, it's unneeded calculation. If you plan to query (per object) about a timeout more often than update it then you would be better off storing the expiration time instead of the current (so calculating delta only once). Beware about DATETIME columns: they don't regard timezone setting, while your PHP does... so if you happen to have different timezone settings on different requests, then you're out of luck. Timestamps are absolute, and they also account for manace like daylight-savings times, where 3:01 is 2 minutes after 1:59...
Seems fine to me. Though you should probably store it as a DATETIME and use DateTime objects, rather than UNIX timestamps and time().
$time = new DateTime;
echo $time->format("Y-m-d H:i:s"); //Outputs current time, example: 2012-10-13 22:58:34
Actually, this is the best idea. The function time() give you the number of seconds from January 1th, 1970 00:00:00. There's no performance impact because it's only an integer. In MySQL, create a field like that INT, 10, Unsigned.
Time will give you performance on the SELECT and the WHERE. See http://gpshumano.blogs.dri.pt/2009/07/06/mysql-datetime-vs-timestamp-vs-int-performance-and-benchmarking-with-myisam/
The only problem you have is : time is limited to year 2038... but by the time 2038 come, the internal computer clock bytes will be larger ... hope so.
The other thing you may want to worrie about the DATETIME is : PHP time() run under UTC, while DATETIME depend on the timezone...
Stats when you do INSERT with 10000000 rows.
Stats when you SELECT / WHERE with indexes :
This question already has answers here:
Should I use the datetime or timestamp data type in MySQL?
(40 answers)
Closed 9 years ago.
Probably many coders want to ask this question. it is What's the adventages of each one of those MySQL time formats. and which one you will prefer to use it in your apps.
For me i use Unix timestamp because maybe i find it easy to convert & order records with it, and also because i never tried the DATETIME thing. but anyways i'm ready to change my mind if anyone tells me i'm wrong.
Thanks
Timestamp (both PHP ones and MySQL's ones) are stored using 32 bits (i.e. 4 bytes) integers ; which means they are limited to a date range that goes from 1970 to 2038.
DATETIME don't have that limitation -- but are stored using more bytes (8 bytes, if I'm not mistaken)
After, between storing timestamps as seen by PHP, or timestamps as seen by MySQL :
using PHP timestamps means manipulations are easier from PHP -- see Date/Time Functions
using MySQL's timestamps means manipulations are easier from MySQL -- see 11.6. Date and Time Functions
And, for more informations between MySQL's TIMESTAMP and DATETIME datatypes, see 10.3.1. The DATETIME, DATE, and TIMESTAMP Types
As others have said, timestamps can represent a smaller range of datetimes (from 1970 to 2038). However, timestamps measure the number of seconds since the Unix Epoch (1970-01-01 00:00:00 UTC), thereby making them independent of time zone, whereas DATETIME stores a date and time without a time zone. In other words, timestamps unambiguously reference a particular point in time, whereas the exact point in time a DATETIME refers to requires a time zone (which is not stored in a DATETIME field). To see why this can matter, consider what happens if we change our time zone.
Let's say we want to store the datetime 2010-03-27 12:00 UTC. If we store this and retrieve it using a timestamp or DATETIME, then there usually appears to be no difference. However, if the server now changes so that the local time zone is UTC+01, then we get two different results if we pull out the datetime.
If we'd set the field to a DATETIME, it would report the datetime as 2010-03-27 12:00, despite the change in time zone. If we'd set the field to a timestamp, the date would be reported as 2010-03-27 11:00. This isn't a problem with either datatype -- it's just a result of the fact that they store slightly different information.
That really depends. I'll give you 2 examples where one overcome the other:
Timestamp is better than DATETIME when you want to store users session in the database and the session creation time (in Timestamp format) is used for fast row retrieval (with index).
E.g. table may look like this:
[session_create_time AS Timestamp][IP_address AS 32bit Int][etc...]
Having an index on the first two columns can really speed up your queries. If you had a DATETIME value type for the session_create_time field, then it could be taken much more time. Take into account that session queries are executed each time a user request a page, so efficiency is crucial.
DATETIME is better than Timestamp when you want to store a user's date of birth or some historic events that require flexible time range.
Unless digitizing records prior to January 1, 1970, I like the UNIX epoch. Its just a matter of preference, whole unsigned numbers are simpler to deal with when using multiple languages.
Just keep in mind, the epoch starts at January 1, 1970. A lot of companies had been in business for decades, if not longer, prior to that.