I have a MySQL database that holds log data from a vehicle's OBD-II reader. The application that interfaces with the reader collects a large amount (usually well over 2,000) of data point sets. Each of these data point sets gets its own row in the table, resulting in something like this:
+---------------+----------------------------------+---------------+----+----+-----+
| session | id | time | kd | kf | ... |
+---------------+----------------------------------+---------------+----+----+-----+
| 1420236074526 | 5cff4a3cc80b22cecb7de85266b25355 | 1420236074534 | 14 | 8 | ... |
| 1420236074526 | 5cff4a3cc80b22cecb7de85266b25355 | 1420236075476 | 17 | 8 | ... |
| 1420236074526 | 5cff4a3cc80b22cecb7de85266b25355 | 1420236075476 | 19 | 8 | ... |
| 1420236074526 | 5cff4a3cc80b22cecb7de85266b25355 | 1420236075476 | 23 | 8 | ... |
| 1420236074526 | 5cff4a3cc80b22cecb7de85266b25355 | 1420236077477 | 25 | 8 | ... |
+---------------+----------------------------------+---------------+---------+-----+
k5 and kf are the vehicle data types (vehicle speed and ambient air temperature, respectively).
There are two indexes in this table (which is called raw_logs):
session maps to columns session and id
id maps to column id
What I'd like to do is grab all of the rows that have the same timestamp (1420236074526, for example), and lump them together into a "session". The goal here is to create a select list where I can view data by session. Ideally, I'd have something like this as the output:
<select>
<option>January 1, 2015 - 7:43AM</option>
<option>January 1, 2015 - 5:15PM</option>
<option>January 2, 2015, - 7:38AM</option>
...
</select>
This is what I have so far (I'm using Medoo to try and simplify the queries):
$session = $database->query("SELECT * FROM raw_logs USE INDEX (session)");
$sessionArray = array();
foreach($session as $session){
$s_time = $session["session"];
$sessionArray[$s_time] = array(
"vehicle_speed" => $session["kd"],
"ambient_air_temp" => $session["kf"]
);
} print_r($sessionArray);
This works... sort of. I get the session time as the array and kd and kf under that with the correct key/value pairs, but it doesn't seem to want to iterate through the whole thing. There are around 25,000 rows in the table at the moment, but it's only returning a few, and there doesn't seem to be any logical order to the listing... it'll return two results from January 8th, one from the 9th, 4 from the 10th, etc.
What would be the best way to select all sessions with the same time stamp, group them, and create a selectable object that will only display the data for the given session?
If you are not ordering the query, there likely won't be any logical order to the listing. Also, you are overwriting the session data where duplicate values exist in the session column. You likely want to do something like $sessionArray[$s_time][] = array(... to append each row under the session id. Also, if you are having trouble, it is best to limit the results from your query down to like 20-100 and keep massaging until you get the result you want.
Related
I'll use an example for this.
Say I'm taking RSS feeds, going through their items, and storing some information about each item in a DB. I'm using SimplePie for processing the feeds, and Josh Campbell's MySQLi wrapper for the DB actions.
For each entry, I will save the item title, the time the item was published (retrieved from the feed), the time the item was added to my DB (defined at the time of inserting to DB), and an auto-incremented ID assigned to each.
function process_rss($rss_url) {
require_once './feed-parser/vendor/autoload.php';
$feed = new SimplePie();
// some feed processing logic here
$items = $feed->get_items();
$data = array();
foreach ($items as $item) {
$data[] = array(
'name' => $item->get_title(),
'published' => $item->get_date(),
'created' => time()
);
}
$insert = ... // db insert query with the $data array
if ($insert) {
return 'Imported';
} else {
return 'Try again...';
}
}
Now assume I ran this function 3 times with 3 different feeds, and a total of 7 items were processed and inserted to the DB. They are inserted at the order in which they were processed, so my table would appear as follows:
ID | Name | Published | Created
----------------------------------------
1 | Track 1 | 1595147815 | 1594986945
2 | Track 2 | 1594996785 | 1594986945
3 | Track 3 | 1595146567 | 1594986945
4 | Track 4 | 1594996598 | 1594986945
5 | Track 5 | 1594996096 | 1594986945
6 | Track 6 | 1594986545 | 1594986945
7 | Track 7 | 1594986609 | 1594986945
What I want to do now, is to sort that table by the published column permanently (rather than sorting while fetching the entries from the DB).
Taking the above example, the desired output would look like this:
ID | Name | Published | Created
----------------------------------------
6 | Track 6 | 1594986545 | 1594986945
7 | Track 7 | 1594986609 | 1594986945
5 | Track 5 | 1594996096 | 1594986945
4 | Track 4 | 1594996598 | 1594986945
2 | Track 2 | 1594996785 | 1594986945
3 | Track 3 | 1595146567 | 1594986945
1 | Track 1 | 1595147815 | 1594986945
Looking into the answers provided here, it is indicated that using ALTER and ORDER BY will not persist the changes upon new inserts. How would I go about achieving this?
How can MySQL records be reordered by a certain column permanently?
You Can't Do That™. Seriously. And you will be sorry if you try.
Seriously, rows in SQL databases have no inherent order. If you do not specify ORDER BY on your query the server is free to return the set of rows you requested in any order it wants. If you get them in a particular order that somehow matches your expectation, it is a lucky coincidence.
As tables grow, SQL databases sometimes start using different ways of accessing them. And, without any ORDER BY specification, those different ways of accessing them may generate different orderings. So, if you need a particular order but you don't specify it, sometime in the life of your application the order will change. When you don't expect it to. And your users will suddenly be baffled by strange results.
Use ORDER BY.
Right now I have a PHP script that is fetching the first three results from a MYSQL database using:
SELECT * FROM table Order by DATE DESC LIMIT 3;
After that command I wanted PHP to fetch the next three results, initially I was going to use:
SELECT * FROM table Order by DATE DESC LIMIT 3,3;
However there will be a delay between the two commands which means that it is very possible that a new row will be inserted into the table during the delay. My first thought was to store the DATE value of the last result and then include a WHERE DATE > $stored_date but if entry 3 and 4 have the same date it will skip entry 4 and return results from 5 onward. This could be avoided using the primary key field which is an integer which increments automatically.
I am not sure which the best approach is, but I feel like there should be a more elegant and robust solution to this problem, however I am struggling to think of it.
Example table:
-------------------------------------------
| PrimaryKey | Data | Date |
-------------------------------------------
| 0 | abc | 2014-06-17 11:43:00 |
| 1 | def | 2014-06-17 12:43:00 |
| 2 | ghi | 2014-06-17 13:43:00 |
| 3 | jkl | 2014-06-17 13:56:00 |
| 4 | mno | 2014-06-17 14:23:00 |
| 5 | pqr | 2014-06-17 14:43:00 |
| 6 | stu | 2014-06-17 15:43:00 |
-------------------------------------------
Where Data is the column that I want.
Best will be using primary key and select like
SELECT * FROM table WHERE pk < $stored_pk Order by DATE DESC LIMIT 3;
And if you have automatically generated PK you should use ORDER BY pk it will be faster
Two options I can think of depending on what your script does:
You could either use transactions: performing these queries inside a transaction will give you a consistent view of the data.
Alternatively you could just use:
SELECT * FROM table Order by DATE DESC;
And only fetch the results as you need them.
Say if I wanted to add the functionality of logging user actions within a web application. My table schema would look similar to the following:
tbl_history:
+----+---------+--+-----------+
| id | user_id | | action_id |
+----+---------+--+-----------+
| 1 | 1 | | 1 |
| 1 | 1 | | 2 |
| 1 | 2 | | 2 |
+----+---------+--+-----------+
A user can generate many actions so I will need to paginate this history. In order to do this I will need to figure out the total amount of rows for the user then calculate how many pages of data there should be.
Which would method be the most efficient if I were to have hundreds of users generating thousands of rows of data each day?
A)
Using the MYSQL's COUNT() function to query the amount of rows of data in the tbl_history table for a particular user.
B)
Having another table which would keep a count of history for the user within the tbl_history table.
+---------+--+---------------+
| user_id | | history_count |
+---------+--+---------------+
| 1 | | 2 |
| 2 | | 1 |
+---------+--+---------------+
This will allow me to instantly get the total count of rows with a simple query in less than 1ms.
The tradeoff is that I will need to perform more queries updating the count for each user and also again on page load.
Which method is more efficient to use? Or is there any other better method? Any technical explanation would be great.
Thanks in advance.
I'm displaying a record set using Datatables pulling records from two tables.
Table A
sno | item_id | start_date | end_date | created_on |
===========================================================
10523563 | 2 | 2013-10-24 | 2013-10-27 | 2013-01-22 |
10535677 | 25 | 2013-11-18 | 2013-11-29 | 2013-01-22 |
10587723 | 11 | 2013-05-04 | 2013-05-24 | 2013-01-22 |
10598734 | 5 | 2013-06-14 | 2013-06-22 | 2013-01-22 |
Table B
id | item_name |
=====================================
2 | Timesheet testing |
25 | Vigour |
11 | Fabwash |
5 | Cruise |
Now since the number of records returned is going to turn into a big number in near future, I want the processing to be done serverside. I've successfully managed to achieve that but it came at a cost. I'm running into a problem while dealing with filters.
From the figure above, (1) is the column whose value will be in int (item_id), but using some small modifications inside the while loop of the mysql resource, I'm displaying the corresponding string using Table B.
Now if I use the filter (2), it is working fine since those values come from Table A
The Problem
When I try to filter from the field (3), if I enter a string value such as fab it says no record found. But if I enter an int such as 11 I get a single row which contains Fabwash as the item name.
So while filtering I'm required to use the direct value used in Table A and not its corresponding string value stored in Table B. I hope the point that I'm putting across is understandable because it is hard to explain it in words.
I'm clueless on how to solve the issue.
I have a comma delimited list that im storing in a varchar field in a mysql table.
Is it possible to add and remove values from the list directly using sql queries? Or do I have to take the data out of the table, manipulate in PHP and replace it back into mysql?
There is no way to do it in InnoDB and MyIsam engines in mysql. Might be in other engines (check CSV engine).
You can do it in a stored procedure, but, not recommended.
What you should do to solve such an issue is to refactor your code and normalize your DB =>
original table
T1: id | data | some_other_data
1 | gg,jj,ss,ee,tt,hh | abanibi
To become:
T1: id | some_other_data
1 | abanibi
T2: id | t1_id | data_piece
1 | 1 | gg
2 | 1 | jj
3 | 1 | ss
4 | 1 | ee
5 | 1 | tt
6 | 1 | hh
and if data_piece is a constant value in the system which is reused a lot, you need to add there a lookup table too.
I know it looks more work, but then it will save you issues like you have now, which take much more time to solve.