Edit: TL;DR
# This query ---------------------------------------
SELECT STR_TO_DATE('2020-10-20T14:43:49+00:00', '%Y-%m-%dT%H:%i:%s') AS date;
# Results in----------------------------------------
+---------------------+
| date |
+---------------------+
| 2020-10-20 14:43:49 |
+---------------------+
# But throws----------------------------------------
+---------+------+-----------------------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------------------+
| Warning | 1292 | Truncated incorrect datetime value: '2020-10-20T14:43:49+00:00' |
+---------+------+-----------------------------------------------------------------+
Why?
Detailed description
I have read through a lot of questions regarding similar issues, but could not find a definitive answer to this problem.
I'm running Doctrine migrations in a Symfony 5.2.x project on a MariaDB 10.2 database. I am trying to extract a date string from a JSON data column into its own column on the same table, but running into error messages when the original date string has a certain format.
ALTER TABLE form
ADD updated_at DATETIME DEFAULT NULL;
UPDATE form AS f
SET updated_at = STR_TO_DATE(
TRIM(BOTH '"' FROM (
SELECT JSON_EXTRACT(f.data, '$.updatedAt')
)),
'%Y-%m-%dT%H:%i:%s+00:00'
);
This works for any date string with a timezone offset of 0, like 2020-12-04T11:14:07+00:00. For obvious reasons, it fails for a non-zero offset like 2020-12-04T11:14:07+01:00, because
Literal characters in format must match literally in str.
-- https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_str-to-date
and results in an error
Warning | 1411 | Incorrect datetime value: '2020-12-04T11:14:07+01:00' for function str_to_date
However, if I understand the documentation correctly, I shouldn't even have to include the timezone offset in the format string:
Extra characters at the end of str are ignored.
But when I change the format string from '%Y-%m-%dT%H:%i:%s+00:00' to '%Y-%m-%dT%H:%i:%s', the update operation fails for all items, even though the dates are parsed correctly (or, at least, look correct):
MariaDB [db]> select STR_TO_DATE('2020-10-20T14:43:49+00:00', '%Y-%m-%dT%H:%i:%s') as date;
+---------------------+
| date |
+---------------------+
| 2020-10-20 14:43:49 |
+---------------------+
1 row in set, 1 warning (0.00 sec)
MariaDB [db]> show warnings;
+---------+------+-----------------------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------------------+
| Warning | 1292 | Truncated incorrect datetime value: '2020-10-20T14:43:49+00:00' |
+---------+------+-----------------------------------------------------------------+
1 row in set (0.00 sec)
The Question
Apart from the historic bug in the application that would in some cases result in a non-UTC updated_at date, what am I doing wrong? As I understand it, anything in the string after the bit matching %s should be ignored by STR_TO_DATE() and irrelevant to the query. Why are my migrations failing when the DB clearly manages to parse the strings to something that looks like the datetime type it understands? How can I make sure it parses every item's date irrespective of its TZ offset (I wouldn't even mind if the result was updated_at times for some items with an hour's difference to the actual datetime)?
Edit
Because I don't fully understand what it does or what the implications are, I've tried changing sql_mode before executing my queries, but got the same results:
SET ##SQL_MODE = REPLACE(##SQL_MODE, 'NO_ZERO_IN_DATE', '');
SET ##SQL_MODE = REPLACE(##SQL_MODE, 'NO_ZERO_DATE', '');
Edit II
I ended up rewriting the migration and manually looping over each entry in PHP, re-setting the timezone and writing the corrected (UTC) value back to the DB.
This is obviously way more verbose and much slower than the SQL one-liner. The lack of answers (or even comments) here suggests I might have stumbled upon either a Maria/MySQL bug or faulty documentation.
Related
I want to convert this format of datetime "31-12-2018 19:30 hs." from Argentina to an UTC timestamp, I am using the following code:
$clean_date = substr($date, 0, -4);
$dt = new DateTime($clean_date, new DateTimeZone('America/Argentina/Buenos_Aires'));
$dt->setTimeZone(new DateTimeZone('UTC'));
$timestamp = $dt->getTimestamp();
But it doesn't work, in the database the record is "0000-00-00 00:00:00", but if I echo the $dt, till there is working perfectly and showing the datetime in UTC.
Could someone please help me?
Thanks.
This has nothing to do with PHP. You're simply using the incorrect date literal format in MySQL. A per the docs:
MySQL recognizes DATETIME and TIMESTAMP values in these formats:
As a string in either 'YYYY-MM-DD HH:MM:SS' or 'YY-MM-DD HH:MM:SS' format. A “relaxed” syntax is permitted here, too: Any
punctuation character may be used as the delimiter between date parts
or time parts.
As a string with no delimiters in either 'YYYYMMDDHHMMSS' or 'YYMMDDHHMMSS' format, provided that the string makes sense as a
date.
As a number in either YYYYMMDDHHMMSS or YYMMDDHHMMSS format, provided that the number makes sense as a date.
1546335960 could be the last case but numbers don't make sense as date because year 1546 did not have 33 months.
To make it worse, many MySQL Servers are configured by default to let these kind of errors slip through:
mysql> CREATE TABLE test (
-> foo TIMESTAMP
-> );
Query OK, 0 rows affected (0.74 sec)
mysql> SET ##SESSION.sql_mode = '';
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO test (foo) VALUES (1546335960);
Query OK, 1 row affected, 1 warning (0.39 sec)
mysql> SHOW WARNINGS;
+---------+------+------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------+
| Warning | 1265 | Data truncated for column 'foo' at row 1 |
+---------+------+------------------------------------------+
1 row in set (0.00 sec)
mysql> SELECT * FROM test;
+---------------------+
| foo |
+---------------------+
| 0000-00-00 00:00:00 |
+---------------------+
1 row in set (0.00 sec)
As you can see, you got a mere warning (that you need to read explicitly) and data corruption.
If you configure your app to use a strict mode you'll get a proper error message just in time:
mysql> SET ##SESSION.sql_mode = 'TRADITIONAL,NO_AUTO_VALUE_ON_ZERO';
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO test (foo) VALUES (1546335960);
ERROR 1292 (22007): Incorrect datetime value: '1546335960' for column 'foo' at row 1
mysql>
Please note that timestamp is just a generic English word:
A digital record of the time of occurrence of a particular event.
It isn't necessarily synonym for Unix time.
I need to store a deadline, that consists of a date and a time (e.g 2016-05-02 19:02). I am currently using a field that has DATETIME as datatype but the problem is that its automatically saving it as 2016-05-02 19:02:00.
as a solution i was thinking to save the date in a String field.
So i am wondering if i should do that ? any performance advantages/disadvantages ?
Don't fight the database. Use the builtin types unless you really need something that they can't offer(I'd say it's unlikely, though). (And by this I mean that you should use TIME, DATE or similar for times and dates. Then you can do calculations without having to convert values, etc)
If you don't need the seconds then just keep them 00 all the time.
Whether you use DATETIME, DATE & TIME or perhaps TIMESTAMP is up to you, how you use the data. Choose the alternative that makes most sense in your current situation.
As mentioned in the other answer, you should always use built-in data types whenever possible.
In your case, stick with DATETIME and then convert it to whatever format you need in the query using the DATE_FORMAT function, like so:
mysql> SELECT * FROM `mytable`;
+----+---------------------+
| id | mydatetime |
+----+---------------------+
| 1 | 2016-06-06 14:12:00 |
+----+---------------------+
1 row in set (0.00 sec)
mysql> SELECT DATE_FORMAT(`mydatetime`,'%b %d %Y %h:%i %p') AS `mydatetime` FROM `mytable`;
+----------------------+
| mydatetime |
+----------------------+
| Jun 06 2016 02:12 PM |
+----------------------+
1 row in set (0.00 sec)
Reference:
http://www.w3schools.com/sql/func_date_format.asp
After writing a PHP function to ensure data parsed from a csv is inputed as its correct format (to match the column data type set when creating the table), I've learned that MySQL by default will output all values as strings anyway.
My question is therefore - is there any need to ensure an integer (for an id column that has been set to store integers only) IS an integer and not a string containing a number ( "1" for example) before inserting into MySQL database?
If not, then what is the thinking behind explicitly stating what values a column should store when creating tables in MySQL?
The values are being converted between string and integer. When inserting a row into MySQL both PHP and MySQL can convert a string of "1" into an integer 1. Try passing a string "notanumber" into an Integer field, it's not going to work because you can't convert that string value into a number. The reason MySQL returns strings in selecting is so everything is in one type of format, there may be another reason for it - but it's easier to know everything in your results is a string and not have to check if it's an integer, or a float, or whatever else. With PHP and implicit conversion this isn't a huge deal, but for a language like C# that is very strongly typed this can save a lot of time. You know it's a string, and convert to what you want if need be instead of checking for tons of different possibilities.
You definitely should be checking data before inserting, or at least handling the MySQL errors if you don't. You can check using isset($var) for null or empty values, is_numeric($var) for integers, is_float($var) for floats. I would recommend validating everything before putting it into the database.
A lot of data types will be automaticly cast towards the correct type in MySQL. I.e. inserting a number to a varchar field will become a string.
The thinking about stating the column types have several reason, mostly for speed and space optimization. Off course you can create all fields as varchars, but storing the number 300000000 in a varchar field would need (at least) 9 bytes while for an integer field a basic 32bit (4 bytes) would be enough. Comparing integer numbers (in the where clause) is easy, but numbers in strings is different. I.e. ordering string cat,cars,car will be: car, cars, cat. But how would you order strings 1000,1200 and 10000? As strings it would be 1000, 10000, 1200. As numbers 1000,1200 and 10000.
For ints/strings, data types aren't too critical while inserting. You can probably find some edge cases where an exotic floating point value-as-string doesn't insert properly. MySQL for the most part will do the right thing when forced to do type juggling while inserting. A string inserted into a numeric-type field will get converted to a number, as best as MySQL can.
The major problem is when it comes time to actually USE the data you've inserted. That's when number v.s. string distinctions become critical:
mysql> create table test (strings varchar(2), integers int);
Query OK, 0 rows affected (0.00 sec)
mysql> insert into test values ('12', 12), ('2', 2), ('112', 112);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> select * from test;
+---------+----------+
| strings | integers |
+---------+----------+
| 12 | 12 |
| 2 | 2 |
| 112 | 112 |
+---------+----------+
3 rows in set (0.00 sec)
Simple select, sorting via the integer field:
mysql> select * from test order by integers;
+---------+----------+
| strings | integers |
+---------+----------+
| 2 | 2 |
| 12 | 12 |
| 112 | 112 |
+---------+----------+
Everything ok with integer sorting. We get nicely sorted ascending list.
But when it comes time for the strings:
mysql> select * from test order by strings asc;
+---------+----------+
| strings | integers |
+---------+----------+
| 112 | 112 |
| 12 | 12 |
| 2 | 2 |
+---------+----------+
Ooops... totally wrong. MySQL (properly) sorted as strings, and by string rules, 112 is smaller than 2.
I'm having a hard time figuring out on how to make a sequential number on my ID field together with the date today and then resets the next day with the date of the next day.
For example:
+------------+------+
| id | name |
+------------+------+
| 0322150001 | John |
| 0322150002 | Mark |
| 0322150003 | Josh |
| 0323150001 | Paul |
| 0323150002 | Bon |
+------------+------+
If you want that value to be implicitly set when inserting a row (i.e. without mentioning id in your query), the only way I know of is to set a default value for that field.
Quoting the MySQL Manual:
The DEFAULT value clause in a data type specification indicates a default value for a column. With one exception, the default value must be a constant; it cannot be a function or an expression. [...] The exception is that you can specify CURRENT_TIMESTAMP as the default for TIMESTAMP and DATETIME columns.
So your options are limited to either using CURRENT_TIMESTAMP and setting the field type to TIMESTAMP or DATETIME, or you set the id field in your SQL query.
I'm starting to design a database and before too many records get inputted, I want to think into the future and collect as much data as possible. I think it would be good for me to know when the record was added. Based on your experience, is it better to do this through mySQL via datetime or through php via the date function. I'll be using php to input all the values, so it would simply be another field.
So far, I like the php approach because I can customize it to take up minimal space: yymmddhhmm & time zone.
Based on your experience, what is the best way to store this data or are the two ways indifferent?
Also, what time zone would you suggest using? The time zone where I am located or GMT? Is it best to use GMT if say I were to move later on or if individuals from multiple timezones administered the database.
Store it as DATETIME/TIMESTAMP in MySQL, it is stored as an integer anyway, just goes in and comes out as a timestamp. Store the data in UTC.
You can manipulate the timestamp in PHP by constructing it with DateTime() and then going from there.
This also allows you to put a NOT NULL DEFAULT CURRENT_TIMESTAMP on the column, which saves you actively having to build it in php.
For simplicity, I suggest using MySQL's timestamp field. Because the database then understands what it is, it's stored much more efficiently than your text version (as a number, rather than a string of characters), and you can do more with it.
For example:
mysql> CREATE TABLE foo (something TEXT NOT NULL, created TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP);
Query OK, 0 rows affected (0.11 sec)
mysql> INSERT INTO foo (something) VALUES ("one");
Query OK, 1 row affected (0.07 sec)
mysql> INSERT INTO foo (something) VALUES ("two");
Query OK, 1 row affected (0.13 sec)
mysql> SELECT * FROM foo;
+-----------+---------------------+
| something | created |
+-----------+---------------------+
| one | 2013-09-18 22:57:01 |
| two | 2013-09-18 22:57:03 |
+-----------+---------------------+
2 rows in set (0.00 sec)
mysql> SELECT something, NOW() - created as seconds_since_insert FROM foo;
+-----------+----------------------+
| something | seconds_since_insert |
+-----------+----------------------+
| one | 136 |
| two | 134 |
+-----------+----------------------+
2 rows in set (0.00 sec)