I have a database named as DB in which there is a table named as log. log table have following structure.
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| id | bigint(64) | NO | PRI | NULL | auto_increment |
| BCReq | varchar(400) | YES | | NULL | |
| BCRes | varchar(400) | YES | | NULL | |
| ServerReq | varchar(400) | YES | | NULL | |
| ServerRes | varchar(400) | YES | | NULL | |
+----------------+--------------+------+-----+---------+----------------+
and the value of BCReq field of one of the record is look like this:
uId-->xxxxxx/regiodId-->yyy/videoTitle-->bcdes
/location-->Asia Pacific/BCreqtime-->24-07-2014
10:30:16/Exception-->null
Now i want to fetch the data from log table on the basis of date .is it possible? if yes then how? Please comment if any more data required.Thanks in advance.
However, despite my dislike for the way the data has been modeled.
As developers, we don't sell code...we offer solutions :P
Requirement:
Fetch the data from log table on the basis of date
SQL LIKE statement it.
DBAs will probably want to kill me for suggesting this, because you're perform a LIKE search on a what is probably a non-indexed column. For example
SELECT * FROM log WHERE BCReq LIKE '%24-07-2014%'
But it will give you all records of log on that day, assuming you want to fetch it by day.
I'm sure you can extend it to search by hour, minute, second or even month.
Also, a DBA behind me just blanched at what I posted :P
So here's me trying to redeem myself in his eyes.
Perform your query using a non-transactional connection, so you don't lock the table for other people.
But all in all, the data model needs to be reviewed and addressed to cater for this.
Additional information to address your search between 2 dates issue:
If you wish to get data within 2 dates, due to your data model, in that the field BCReq is difficult to search, a viable solution is to have a developer write up an application or script to grab that data.
A sample solution (pseudo code) could be:
Grab 2 dates from user. Let's call them start_date && end_date
Format those to dates to DD-MM-YYYY format.
Determine the days_between those start_date and end_date
For every day between start_date and end_date
Run SQL query above to retrieve data
Store data retrieved from each iteration to output
Do something with your output
Related
I have a table of around 6000 records with a date column amongst other columns which represent the deadline for a query. I need to compare the date in the column to todays date which I understand is done something like:
SELECT DATEDIFF(DATE_TO_COMPARE, CURDATE());
However, I then have another comlumn I want to set to that date difference. So for each date, I need to compare, insert the difference in the column difference_in_days, iterate to the next date and repeat.
I am also invoking this function whenever a certain page on my site is loaded using AJAX and PHP/PDO
My SQL knowledge isn't that extensive, how can I achieve this.
Table is kinda of like
field 1, field2, field 3, date_to_compare, field 4, field 5, difference_in_days
| | | | 2016-04-20 | | | |
| | | | 2016-04-25 | | | |
| | | | 2016-04-22 | | | |
| | | | 2016-04-27 | | | |
| | | | 2016-04-29 | | | |
Sonds like you want to do an update?
UPDATE table_name
SET difference_in_days = DATEDIFF(date_to_compare, CURDATE());
This will update every record in the table to the diff of the current date.
However, this will require you running the update every day, if you want that column to maintain relevance.
Alternative Approach:
If you're not querying this a lot, you may be better off using a view, which will update real-time every time you query it.
CREATE VIEW diff_view_name AS
SELECT *, DATEDIFF(date_to_compare, CURDATE()) AS difference_in_days
FROM table_name;
Then you could query it using:
SELECT * FROM diff_view_name;
How much faster (in %) sql will be if I will avoid to used built-in mysql date and time functions ?
What do I mean ? For example: SELECT id FROM table WHERE WEEKOFYEAR(inserted)=WEEKOFYEAR(CURDATE())
MySQL has a lot of buil-in function to work with date and time, and they are suitable as well. But what about peromance ?
Above sql can be rewritten without built-in functions, like: SELECT id FROM table WHERE inserted BETWEEN 'date for 1 day of particular week 00:00:00' AND 'last day of particular week 23:59:59', server side code become worse :( but on db side we could use indexes
I see two problems for usage built-in functions:
1. indexes
I did small test
mysql> explain extended select id from table where inserted between '2013-07-01 00:00:00' and '2013-07-01 23:59:59';
+----+-------------+-------+-------+---------------+------+---------+------+------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+-------+---------------+------+---------+------+------+----------+--------------------------+
| 1 | SIMPLE | table | range | ins | ins | 4 | NULL | 7 | 100.00 | Using where; Using index |
+----+-------------+-------+-------+---------------+------+---------+------+------+----------+--------------------------+
mysql> explain extended select id from table where date(inserted)=curdate();
+----+-------------+-------+-------+---------------+------+---------+------+--------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+-------+---------------+------+---------+------+--------+----------+--------------------------+
| 1 | SIMPLE | table | index | NULL | ins | 4 | NULL | 284108 | 100.00 | Using where; Using index |
+----+-------------+-------+-------+---------------+------+---------+------+--------+----------+--------------------------+
First one took 0.00 sec second one was running after first one and took 0.15. Everything was made with small anout of data.
and second problem, is
time to call that functions
If in table I have 1 billion records it means that WEEKOFYEAR, DATE whatever... would be called so many times, so many records do we have, right ?
So the question will it bring real profit if I will stop to work with mysql built-in date and time functions ?
Using a function of a column in a WHERE clause or in a JOIN condition will prevent the use of indexes on the column(s), if such indexes exist. This is because the raw value of the column is indexed, as opposed to the computed value.
Notice the above does not apply for a query like this:
SELECT id FROM atable WHERE inserted = CURDATE(); -- the raw value of "inserted" is used in the comparison
And yes, on top of that, the function will be executed for each and every row scanned.
The second query is running the date function on every row in the table, while the first query can just use the index to find the rows it needs. Thats where the biggest slowdown would be. Look at the rows column in the explain output
Thanks for reading.
This is not a coding question as much as it is a logic one. But if my current logic is wrong, some coding help would be appreciated.
I have made a table on my database which is a log of everything that happens on my site.
When a user registers, it's saved. When he logs, again. And so on. Each action is represented by a number.
The data looks like this
----------------------------
| id | action | timestamp |
----------------------------
| 1 | 1 | 1299132900 |
| 2 | 2 | 1346876672 |
| 3 | 14 | 1351983948 |
| 4 | 1 | 1359063373 |
----------------------------
ID and action are of type INT(11) and timestamp is TIMESTAMP
I'm using a query to retrieve all records from the last 30 days.
SELECT id, action, timestamp FROM log WHERE timestamp >= DATE_SUB( CURDATE(),INTERVAL 30 DAY)
It works, and gives me all the correct values.
I need to arrange this data to make a graphic in flot.
As I see it, there are 2 steps:
Group the results by action number.
Then, inside each group, separate values by date, so the X axis of the graphic is date and Y axis is count.
With those arrays I could make different javascript data arrys to pass to flot.
Am I on the right track?
Should there be several mysql queries, or a GROUP BY clause?
I'm kind of lost here and would appreciate any help.
I am still new to PHP and I was wondering which alternative would be better or maybe someone could suggest a better way.
I have a set of users and I have to track all of their interactions with posts. If a users taps on a button, it will add the post to a list and if they tap it again, it will remove the post, so would it be better to:
Have a column of a JSON array of postIDs stored in the table for each user (probably thousands).
-or-
Have a separate table with every save (combination of postID and userID) (probably millions) and return all results where the userID's match?
For the purposes of this question, there are two tables: Table A is users and Table B is posts. How should I store all of the user's saved posts?
EDIT: Sorry, but I didn't mention that posts will have multiple user interactions and users will have multiple post interactions (Many to Many relationship). I think that would affect Bob's answer.
This is an interesting question!
The solution really depends on your expected use case. If each user has a list of posts they've tagged, and that is all the information you need, it will be expedient to list these as a field in the user's table (or in their blob if you're using a nosql backend - a viable option if this is your use case!). There will be no impact on transmission time since the list will be the same size either way, but in this solution you will probably save on lookup time, since you're only using one table and dbs will optimize to keep this information close together.
On the other hand, if you have to be able to query a given post for all the users that have tagged it, then option two will be much better. In the former method, you'd have to query all users and see if each one had the post. In this option, you simply have to find all the relations and work from there. Presumably you'd have a user table, a post table and a user_post table with foreign keys to the first two tables. There are other ways to do this, but it necessitates maintaining multiple lists and cross checking each time, which is an expensive set of operations and error-prone.
Note that the latter option shouldn't choke on 'millions' of connections, since the db should be optimized for this sort of quick read. (pro tip: index the proper columns!) Do be careful about any data massage, though. One unnecessary for-loop will kill your performance.
For the purposes of this question, there are two tables: Table A is users and Table B is posts. How should I store all of the user's saved posts?
If each user has a unique ID of some sort (primary key), then ad a field to each post that refers to the unique ID of the user.
mysql> describe users;
+----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| email | varchar(200) | YES | | NULL | |
| username | varchar(20) | YES | | NULL | |
+----------+------------------+------+-----+---------+----------------+
mysql> describe posts;
+---------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| user | int(11) unsigned | NO | | NULL | |
| text | text | YES | | NULL | |
+---------+------------------+------+-----+---------+----------------+
Then to get posts for a user, for example:
SELECT text
FROM posts
WHERE user=5;
Or to get all the posts from a particular organization:
SELECT posts.text,users.username
FROM posts,users
WHERE post.user=users.id
AND users.email LIKE '%#example.com';
I think it would make sense to keep a third table that would be all the post status data.
If your user interface shows, say, 50 posts per page, then the UI only needs to keep track of 50 posts at a time. They'll all have unique IDs in your database, so that shouldn't be a problem.
We are implementing a system that analyses books. The system is written in PHP, and for each book loops through the words and analyses each of them, setting certain flags (that translate to database fields) from various regular expressions and other tests.
This results in a matches table, similar to the example below:
+------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| regex | varchar(250) | YES | | NULL | |
| description | varchar(250) | NO | | NULL | |
| phonic_description | varchar(255) | NO | | NULL | |
| is_high_frequency | tinyint(1) | NO | | NULL | |
| is_readable | tinyint(1) | NO | | NULL | |
| book_id | bigint(20) | YES | | NULL | |
| matched_regex | varchar(255) | YES | | NULL | |
| [...] | | | | | |
+------------------------+--------------+------+-----+---------+----------------+
Most of the omitted fields are tinyint, either 0 or 1. There are currently 25 fields in the matches table.
There are ~2,000,000 rows in the matches table, the output of analyzing ~500 books.
Currently, there is a "reports" area of the site which queries the matches table like this:
SELECT COUNT(*)
FROM matches
WHERE is_readable = 1
AND other_flag = 0
AND another_flag = 1
However, at present it takes over a minute to fetch the main index report as each query takes about 0.7 seconds. I am caching this at a query level, but it still takes too long for the initial page load.
As I am not very experienced in how to manage datasets such as this, can anyone advise me of a better way to store or query this data? Are there any optimisations I can use with MySQL to improve the performance of these COUNTs, or am I better off using another database or data structure?
We are currently using MySQL with MyISAM tables and a VPS for this, so switching to a new database system altogether isn't out of the question.
You need to use indexes, create them on the columns you do a WHERE on most frequently.
ALTER TABLE `matches` ADD INDEX ( `is_readable` )
etc..
You can also create indexes based on multiple columns, if your doing the same type of query over and over its useful. phpMyAdmin has the index option on the structure page of the table at the bottom.
Add multi index to this table as you are selecting by more than one field. Below index should help a lot. Those type of indexes are very good for boolean / int columns. For indexes with varchar values read more here: http://dev.mysql.com/doc/refman/5.0/en/create-index.html
ALTER TABLE `matches` ADD INDEX ( `is_readable`, `other_flag`, `another_flag` )
One more thing is to check your queries by using EXPLAIN {YOUR WHOLE SQL STATEMENT} to check which index is used by DB. So in this example you should run query:
EXPLAIN ALTER TABLE `matches` ADD INDEX ( `is_readable`, `other_flag`, `another_flag` )
More info on EXPLAIN: http://dev.mysql.com/doc/refman/5.0/en/explain.html