Importing and reporting on larger datasets in a MySQL cluster - php

Question

we have a MySQL (mariaDB/Galera) cluster containing several billion unique data points in one table.
We need to migrate that table to a new one sorting out doublicate entries which takes a very long time and we are constrained in that regard. The next step would be to genereate reports for a given time window and UUID of the correspoinding NAS (a router in the real world/a location) as well as unique IDs (MACs) of users that are recurrent or switch NASes
The MySQL (mariaDB/Galera) DB right now ist about 25GB in size which should not be an issue. But the queries for reports on UIDs/MACs of users in combination with UUIDs NASes/locations takes a very long time.
The table structure is layed out as depicted here. One is the actual table and two would be a possible optimization. But I really don't know if that would do anything.
Is our DB approach the right one or should we use a different one (DB, table structure, stack, whatever ..) (open for suggestions)
The query for the migration (which is very slow) is the following:
INSERT INTO `metric_macs` m
(`uuid`,`shortname`,`mac`,`start`,`stop`,`duration`)
VALUES
SELECT uuid, shortname, mac, a, b, duration
FROM import i
ON DUPLICATE KEY update m.id = m.id
Query for unique users:
SELECT DISTINCT mac FROM `metric_macs` WHERE uuid in ('xxxx','yyyyy') and ( start BETWEEN '2020-01-01' and '2020-02-01' or stop BETWEEN '2020-01-01' and '2020-02-01') ;
Count of all datasets
Query for recurrent users:
SELECT id FROM `metric_macs`
WHERE uuid in ('xxxx','yyyyy')
and ( start BETWEEN '2020-01-01' and '2020-02-01'
or stop BETWEEN '2020-01-01' and '2020-02-01')
GROUP BY `mac`, `uuid`
HAVING COUNT(*) > 1
Count of all datasets
Query for unique location switching users:
SELECT uuid,mac FROM `metric_macs`
WHERE uuid in ('xxxx','yyyyy')
and ( start BETWEEN '2020-01-01' and '2020-02-01'
or stop BETWEEN '2020-01-01' and '2020-02-01')
GROUP BY `mac`, `uuid`
After that php is used to count all users with more than two distinct UUIDS.
The list is updated every 15 minutes with a list of UIDs (MACs) that are connected to a NAS, that list is checked for activity of a given UID(MAC) in the last 20 minutes. If there was we update the stop count of the last entries an add 15 minutes and start the calculation gain.
Sorry for the mess. We are fairly new to this kind of report generation. What are the possible ways to optimize the database or queries for near instant reporting?
Thanks!
Edit:
CREATE TABLE `metric_macs` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`uuid` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`shortname` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`mac` varchar(80) COLLATE utf8mb4_unicode_ci NOT NULL,
`start` datetime NOT NULL,
`stop` datetime NOT NULL,
`duration` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `metric_macs_uuid_index` (`uuid`),
KEY `metric_macs_mac_index` (`mac`),
KEY `metric_macs_start_stop_index` (`start`,`stop`),
KEY `metric_macs_uuid_start_stop_index` (`uuid`,`start`,`stop`),
KEY `metric_macs_uuid_stop_index` (`uuid`,`stop`)
) ENGINE=InnoDB AUTO_INCREMENT=357850432 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Answer 1

It is good to move away from 36-byte ids. However, don't stop at 8 bytes; you probably don't need more than 4 bytes (INT UNSIGNED, max or 4 billion) or 3 (MEDIUMINT UNSIGNED, max of 16M).
While you are at it, remove the dashes and unhex the uuids so they can fit in BINARY(16) (16 bytes).
I think you have 3 problems to tackle in the conversion:
Efficiently changing the current schema to a better one. Assuming this has old, unchanging, rows, you can do this in the background.
Quickly finishing the final step. (We will actually do this last.)
Changing the ingestion to the new format.
Step 0: Grab the latest timestamp so you know where to do steps 2 and 3 after spending time doing step 1.
Step 1: To build users and stations, it might be simply
INSERT INTO users (user_id)
SELECT UUID2BIN(userID)
FROM ( SELECT DISTINCT userID ) FROM log;
(and similarly for stations)
See this for converting uuids: http://mysql.rjweb.org/doc.php/uuid
That may take some time, but it does the de-dup efficiently.
Let me discuss step 3 before filling in step 2.
Step 3: If the ingestion rate is "high", see this for details on a ping-ponging staging table and for bulk normalization, etc:
http://mysql.rjweb.org/doc.php/staging_table
However, your ingestion rate might be not that fast. Do not use IODKU with a trick of using LAST_INSERT_ID to get the id from users and stations. It will "burn" ids and threaten to overflow your INT/MEDIUMINT id. Instead, see the link above.
Inserting into time_table, if no more than 100 per second (HDD) or 1000 per second (SSD), can be a simple INSERT while you get the necessary ids
INSERT INTO time_table (user_id, station_id, start_time, stop_time)
VALUES (
( SELECT id FROM users WHERE userID = uuid2bin('...') ),
( SELECT id FROM stations WHERE userID = uuid2bin('...') ),
'...', '...'
);
Back to step 2. You have saved a bunch of rows in the old table. And you saved the starting date for those. So do the bulk normalization and mass insert from log as if it were a "staging table" as discussed in my link.
That should allow you to convert with zero downtime and only a small amount of time when the new table is "incomplete".
I have not covered why the "reports take a long time". I need to see the SELECTs. Meanwhile, here are two thoughts:
If you build the new INT-like ids, sort them by date so that they are at least somewhat ordered chronologically ordered, therefore better clustered for some types of queries.
In general, building and maintaining a "summary table" allows reports to be run much faster. See http://mysql.rjweb.org/doc.php/summarytables
"Query for recurrent users:" has multiple query performance problems. Unless my approach is not adequate, I don't want do get into the details.