I have a big mysql table ('d_operations') with more than 2 million records (and more to come). I have written a PHP webpage that shows a chart with the number of operations in a day for each half an hour (0:00 0:30 1:00 1:30 ... 23:59).
It works great but takes too much time to get the results so I am wondering if my table and queries could be optimized.
For each half an hour in a day I do a select query asking MySQL for the number of operations done in that period of time.
This takes more than a minute to finish!
This is the table schema:
mysql> describe d_operations;
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| idx | int(11) unsigned | NO | PRI | NULL | auto_increment |
| system_id | int(11) | YES | | NULL | |
| dev_id | varchar(17) | YES | | NULL | |
| name | varchar(17) | YES | | NULL | |
| nond | smallint(6) | YES | | NULL | |
| is_new | smallint(6) | YES | | NULL | |
| tstamp | int(10) unsigned | YES | | NULL | |
+-----------+------------------+------+-----+---------+----------------+
I have a auto_increment primary key, that doesn't seem to help in the queries. The rest of the fields can be repeated (a device can do several operations in that period of time and it can be rows with the same tstamp).
tstamp is UNIX timestamp
This is how I do the queries in PHP:
for($i=$GLOBALS['init_hour'];$i<=($GLOBALS['end_hour']-1800);$i+=1800){
$n=$i+1800;
$sql="SELECT count(*) as num from d_operations where (tstamp >= $i and tstamp < $n);";
$r=mysqli_query($GLOBALS['con'],$sql);
$row = mysqli_fetch_row($r);
$values = ($i == $GLOBALS['init_hour']) ? $row[0] : $values.",".$row[0];
$GLOBALS['a_average'][$i]=$row[0];
}
In the worst case, I loop through every half an hour in that day, that is 48 queries.
This is the MySQL explain command:
mysql> explain select count(*) as num from d_operations where (tstamp >= 1464739200 and tstamp < 1464825599);
+----+-------------+--------------+------+---------------+------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+------+---------------+------+---------+------+---------+-------------+
| 1 | SIMPLE | d_operations | ALL | NULL | NULL | NULL | NULL | 2215384 | Using where |
+----+-------------+--------------+------+---------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)
Is there a more efficient way for doing this? (table definition, MySQL query optimization...)
Thanks
As Jon Stirling and Mark Baker suggested, the solution was as simple as creating an index for the tstamp column:
ALTER TABLE d_operations ADD INDEX ts_index(tstamp);
Thanks!
Related
I have a current problem, I tried to search in the documentations and the answers already given in the same site, but none of the answers helped me.
In fact I have a database, and two tables.
-> gpscoordonnee
MariaDB [leguideduflaneur]> DESCRIBE gpscoordonnee;
+--------------+----------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+----------+------+-----+---------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| Nom_Commerce | int(11) | NO | MUL | NULL | |
| date | datetime | NO | | current_timestamp() | |
+--------------+----------+------+-----+---------------------+----------------+
3 rows in set (0.017 sec)
-> marchantpart
MariaDB [leguideduflaneur]> DESCRIBE marchantpart;
+---------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| Nom | varchar(200) | NO | | NULL | |
| Adresse | varchar(300) | NO | | NULL | |
| Tel | int(11) | NO | | NULL | |
| Email | varchar(100) | NO | | NULL | |
+---------+--------------+------+-----+---------+----------------+
5 rows in set (0.016 sec)
In the gpscoordonnee table, the Nom_Commerce field is a foreign Keys of Name of the marchantpart table.
I want by displaying the Nom_Commerce that it displays the contents of Nom
AND NOT 1
I have already tried all these methods but nothing is displayed, not even an error :
SELECT n.Nom
from gpscoordonnee us
LEFT JOIN marchantpart ON us.NID = n.Nom
OR
SELECT gpscoordonnee, marchantpart.Nom AS Nom_Commerce
FROM gpscoordonnee
JOIN marchantpart ON marchantpart.Nom=gpscoordonnee.Nom_Commerce
I don't want this :
Result that displays integers instead of names
But i want this :
Result with names
I find my error.
I use SELECT Nom FROM gpscoordonnee t1 INNER JOIN marchantpart t2 ON t1.Nom_Commerce=t2.id and my result is this
RESULT
Thanks for all the people who help me.
Hi I have try to follow couple of examples how to solve my problem but with no success.
So here is the situation
I have a table with the following ( the table will increase with each month)
mysql> DESCRIBE poraba;
+------------+------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------+------------+------+-----+-------------------+-----------------------------+
| mesec | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| voda_mrzla | varchar(5) | NO | | NULL | |
| voda_topla | varchar(5) | NO | | NULL | |
+------------+------------+------+-----+-------------------+-----------------------------+
With the following vaules
mysql> SELECT * FROM poraba;
+---------------------+------------+------------+-----------+
| mesec | voda_mrzla | voda_topla | id_poraba |
+---------------------+------------+------------+-----------+
| 2014-03-03 16:19:08 | 5985 | 3417 | 1 |
| 2014-04-03 20:57:51 | 5978 | 3412 | 2 |
I would like to perform the following. Always only on last entry. So I get the difference between current and previous month.
Eg.:
voda_mrzla (from 2014-04-03) - voda_mrzla (from 2014-03-03) = difference_cold
voda_topla (from 2014-04-03) - voda_topla (from 2014-03-03) = difference_hot
in to
mysql> DESCRIBE usage_per_month;
+-----------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+-------------+------+-----+---------+-------+
| difference_cold | varchar(10) | NO | | NULL | |
| difference_hot | varchar(10) | NO | | NULL | |
It looks like poor database design. You should at least have an id column with an incremental value as a key. That way the (nonetheless cumbersome) query would be a little easier. Given your current design, what you need to do is this (very, very slow and inefficient):
insert into usage_per_month
select a.voda_mrzla - b.voda_mrzla, a.voda_topla - voda_topla
from ( select * from poraba where mesec = ( select max(mesec) from poraba) ) as a,
( select * from poraba where mesec = ( select max(mesec) from poraba where mesec <> (select max(mesec) from poraba) ) ) as b;
Yes, it's this ugly.
Changing all "varchar" in to "int" adding ID to all tables and executing the following solved my problem
INSERT INTO usage_per_month (difference_cold, difference_hot)
SELECT p1.voda_mrzla - p2.voda_mrzla AS poraba_mrzla1, p1.voda_topla - p2.voda_topla AS poraba_topla1 FROM poraba p1
LEFT JOIN poraba p2 ON p2.`id_poraba` = (
SELECT MAX(`id_poraba`) FROM poraba p3 WHERE p3.`id_poraba` < p1.`id_poraba`
) ORDER BY p1.id_poraba DESC LIMIT 1
Tnx for help!
I am making a simple database to use as a mock example for an E-Commerce website. One things that I am required to do it delete records from the database using PHP. I can delete a user name or a order or etc.. But when I try to do them all at once I am screwing something up. What I want to happen is delete all the information about a user. Example : User info, Shipping Info, Billing Info, Credit Card Info, and anything else to do with the specific user. My tables are as followed.
mysql> SHOW columns FROM shirt_billing_addresses;
+----------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------------+--------------+------+-----+---------+----------------+
| shirt_billing_addresses_id | mediumint(9) | NO | PRI | NULL | auto_increment |
| house | mediumint(9) | NO | | NULL | |
| street | varchar(100) | NO | | NULL | |
| city | varchar(100) | NO | | NULL | |
| state | char(2) | NO | | NULL | |
| zip | char(5) | NO | | NULL | |
+----------------------------+--------------+------+-----+---------+----------------+
6 rows in set (0.01 sec)
mysql> SHOW columns FROM shirt_credit_cards;
+-----------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+--------------+------+-----+---------+----------------+
| shirt_credit_cards_id | mediumint(9) | NO | PRI | NULL | auto_increment |
| shirt_users_id | mediumint(9) | NO | MUL | NULL | |
| type | varchar(30) | NO | | NULL | |
| no | char(16) | NO | | NULL | |
| security_code | char(3) | NO | | NULL | |
+-----------------------+--------------+------+-----+---------+----------------+
5 rows in set (0.04 sec)
mysql> SHOW columns FROM shirt_orders;
+-----------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------------+--------------+------+-----+---------+----------------+
| shirt_orders_id | int(11) | NO | PRI | NULL | auto_increment
| order_total | double(6,2) | NO | | NULL | |
| payment_date | datetime | NO | | NULL | |
| shirt_credit_cards_id | mediumint(9) | NO | MUL | NULL | |
| shirt_shipping_addresses_id | mediumint(9) | NO | MUL | NULL
| shirt_billing_addresses_id | mediumint(9) | NO | MUL | NULL |
| shirt_shipping_methods_id | tinyint(4) | NO | MUL | NULL |
+-----------------------------+--------------+------+-----+---------+----------------+
7 rows in set (0.02 sec)
mysql> SHOW columns FROM shirt_shipping_addresses;
+-----------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------------+--------------+------+-----+---------+----------------+
| shirt_shipping_addresses_id | mediumint(9) | NO | PRI | NULL | auto_increment
| house | mediumint(9) | NO | | NULL | |
| street | varchar(100) | NO | | NULL | |
| city | varchar(100) | NO | | NULL | |
| state | char(2) | NO | | NULL | |
| zip | char(5) | NO | | NULL | |
+-----------------------------+--------------+------+-----+---------+----------------+
6 rows in set (0.03 sec)
mysql> SHOW columns FROM shirt_users;
+----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+----------------+
| shirt_users_id | mediumint(9) | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(30) | NO | | NULL | |
| email | varchar(30) | NO | | NULL | |
| user_id | varchar(30) | NO | | NULL | |
| password | char(40) | NO | | NULL | |
+----------------+--------------+------+-----+---------+----------------+
6 rows in set (0.03 sec)
mysql> SHOW columns FROM shirt_users_types;
+----------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+--------------+------+-----+---------+----------------+
| shirt_users_types_id | int(11) | NO | PRI | NULL | auto_increment |
| shirt_users_id | mediumint(9) | NO | MUL | NULL | |
| shirt_types_id | smallint(6) | NO | MUL | NULL | |
| shirt_orders_id | int(11) | NO | MUL | NULL | |
| type_quantity | smallint(6) | NO | | NULL | |
| type_total | double(6,2) | NO | | NULL | |
+----------------------+--------------+------+-----+---------+----------------+
6 rows in set (0.03 sec)
The PHP code I have is as follows:
#mysqli_query($link, "SET AUTOCOMMIT=0");
$select_sui = "SELECT
shirt_users.shirt_users_id,
shirt_users_types.shirt_users_types_id,
shirt_orders.shirt_orders_id,
shirt_shipping_addresses.shirt_shipping_addresses_id,
shirt_billing_addresses.shirt_billing_addresses_id,
shirt_credit_cards.shirt_credit_cards_id
from
shirt_users,
shirt_users_types,
shirt_orders,
shirt_shipping_addresses,
shirt_billing_addresses,
shirt_credit_cards
where
shirt_users.shirt_users_id = shirt_users_types.shirt_users_id and
shirt_users_types.shirt_orders_id = shirt_orders.shirt_orders_id and
shirt_orders.shirt_shipping_addresses_id = shirt_shipping_addresses.shirt_shipping_addresses_id and
shirt_orders.shirt_billing_addresses_id = shirt_billing_addresses.shirt_billing_addresses_id and
shirt_orders.shirt_credit_cards_id = shirt_credit_cards.shirt_credit_cards_id and
shirt_users.shirt_users_id = $shirt_users_id";
The only problem is that this query will return an empty set unless all the tables have the correct information in it. This is a problem because if a user has not ordered any items yet but I want to delete them from the database, It will not allow me. Any help on this matter would be greatly appreciated.
You have to use LEFT JOIN to link your tables (MySQL doc).
SELECT [...]
FROM shirt_users
LEFT JOIN shirt_users_types ON shirt_users.shirt_users_id = shirt_users_types.shirt_users_id
LEFT JOIN shirt_orders ON shirt_users_types.shirt_orders_id = shirt_orders.shirt_orders_id
LEFT JOIN shirt_shipping_addresses ON shirt_orders.shirt_shipping_addresses_id = shirt_shipping_addresses.shirt_shipping_addresses_id
LEFT JOIN shirt_billing_addresses ON shirt_orders.shirt_billing_addresses_id = shirt_billing_addresses.shirt_billing_addresses_id
LEFT JOIN shirt_credit_cards ON shirt_orders.shirt_credit_cards_id = shirt_credit_cards.shirt_credit_cards_id
WHERE
shirt_users.shirt_users_id = $shirt_users_id
With this you'll be able to load shirt_users even if there is no linked record in other tables.
EDIT for Delete records
If you want to delete in all your tables in only one request, you must use as syntax which is fairly the same as your first try (by replacing the SELECT statement by DELETE). ANd you will get the same problem (it will only delete records which have a linked record in each table and not null).
Method 1 :
NB : Your DB must support foreign keys (for example, MyISAM doesn't support it but InnoDB does).
You can use this method, if you always need in your app to delete all linked records.
The most beautiful way to achieve this is to add a ON DELETE CASCADE constraint on your relations :
-- Drop the old constraint ("fk_test" must be replace by your constraint name)
ALTER TABLE shirt_users_types DROP FOREIGN KEY `fk_test`;
-- Create the new with ON DELETE
ALTER TABLE shirt_users_types
ADD CONSTRAINT `fk_test`
FOREIGN KEY (`shirt_users_id` )
REFERENCES `shirt_users` (`shirt_users_id` )
ON DELETE CASCADE;
When you will delete a shirt_users entry, all linked records in shirt_users_types.
In a few words, with ON DELETE CASCADE, each time you will delete a parent element (the One side of the relation), all the children (the Many side of the relation or the table wich contains the column <related_record>_id) will be automatically deleted too.
Method 2 :
If your relation can be nullable (for example if shirt_users_types.shirt_orders_id can be null) or if your DB schema not permits to delete all needed records with ON DELETE CASCADE.
You can delete your records with several DELETE using relations to shirt_users in order to retrieve records in each linked table. (For correct syntax of DELETE with relations see MySQL docs).
In your code your unique identifier is $shirt_users_id, so you have to start by deleting in the tables which are not directly linked to shirt_users and by reaching back your relations to finish by the table shirt_users.
If you don't delete records in the correct order, you will not be able to delete all you need.
For example, with schema described in your question, if you delete records in shirt_orders before shirt_shipping_addresses, you will no longer can retrieve shipping addresses with $shirt_users_id because shirt_shipping_addresses is linked to shirt_users due to its relation with shirt_orders.
So the correct order for your current schema is (there is no need to have an order for table on the same line the list above) :
shirt_shipping_addresses / shirt_billing_addresses
shirt_orders / shirt_credit_cards / shirt_users_types
shirt_users
You can have a contraint fails error due to foreign keys. In this case you can make your delete in a TRANSACTION (it's a best practice) or disable foreign key checks for your queries.
SET foreign_key_checks = 0;
I have created a voting system in php and mysql. When a user votes on an id, a record is inserted in "votes" referencing the FK media_id. When I then display the entries I use this query to get the number of votes for each entry:
$sql = "SELECT COUNT(*) FROM insta_votes WHERE media_id ='".$mediaid."'";
if ($res = $db->query($sql)) {
return $res->fetchColumn();
}
return 0;
This works fine, but I want to be able to sort the results by the number of votes they have. Preferably using just one query. How can I achieve this?
The tables are structured like this:
votes table
+-----------+--------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| media_id | varchar(255) | NO | | NULL | |
| ip | varchar(20) | NO | | NULL | |
| c_time | timestamp | NO | | CURRENT_TIMESTAMP | |
| sessionid | varchar(30) | NO | | NULL | |
+-----------+--------------+------+-----+-------------------+----------------+
entries table
+---------------+--------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+-------------------+-------+
| page_id | int(11) | NO | MUL | NULL | |
| media_id | varchar(255) | NO | PRI | NULL | |
| url | varchar(255) | NO | | NULL | |
| c_time | datetime | NO | | NULL | |
| likes | int(11) | YES | | NULL | |
| deleted | tinyint(1) | NO | | 0 | |
| inserted_time | timestamp | YES | | CURRENT_TIMESTAMP | |
| numReports | int(11) | NO | | 0 | |
+---------------+--------------+------+-----+-------------------+-------+
Thank you!
If I understand the tables correctly (and I may not), each entries row may reference multiple votes rows. In that case, the query you need will go something like this:
SELECT
entries.page_id,
COUNT(*) AS VoteCount
FROM entries
INNER JOIN votes ON entries.media_id = votes.media_id
GROUP BY entries.page_id
ORDER BY VoteCount
If you add additional entries columns to the SELECT list, be sure to add them to the GROUP BY list as well.
Addendum: #JuanPabloCalifano pointed out, correctly, that this query won't include entries with zero votes. Here's how to include them:
SELECT
entries.page_id,
COALESCE(COUNT(votes.id), 0) AS VoteCount
FROM entries
LEFT JOIN votes ON entries.media_id = votes.media_id
GROUP BY entries.page_id
ORDER BY VoteCount
SELECT COUNT(*) as CNT, `media_id` FROM `insta_votes` GROUP BY `media_id` order by 1;
SELECT COUNT(*), media_id FROM insta_votes
GROUP BY media_id
ORDER BY COUNT(*);"
The first is the place table where the general information is kept and the second is the wait table where users sign up (like a waiting list)
+---------+--------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+--------------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(30) | YES | | NULL | |
| userid | int(3) | YES | | NULL | |
| address | varchar(300) | YES | | NULL | |
| desc | varchar(550) | YES | | NULL | |
| phone | int(15) | YES | | NULL | |
| image | varchar(50) | YES | | NULL | |
| website | varchar(100) | YES | | NULL | |
| cat | varchar(25) | YES | | NULL | |
| date | timestamp | NO | | CURRENT_TIMESTAMP | |
+---------+--------------+------+-----+-------------------+----------------+
+----------+-----------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------+-----------+------+-----+-------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| userid | int(11) | YES | | NULL | |
| place_id | int(11) | YES | | NULL | |
| date | timestamp | NO | | CURRENT_TIMESTAMP | |
+----------+-----------+------+-----+-------------------+----------------+
For now I m doing a SELECT * FROM place; and displaying the data on the home page. Something like tihs:
<? foreach($places as $place): ?>
<? echo $place->name; ?>; <? echo $place->userid; ?> etc ...
Click this to insert your userid and $place->id into wait table
<? endforeach ?>
This is where I got lost. I would like to do something like:
<? if($current_user_id == $userid_from_wait_that_matches_place_id): ?>
<p>You already registered for this!</p>
<? else: ?>
Click this to insert your userid and $place->id into wait table
<? endif; ?>
Not sure if it's better to check for the user's id in the model that adds data to the wait table or to check in the model that grabs data for the home page. From what I've read, the second option would be better. Or should I use two separate queries ?
I think your database design is wrong: you should create seperate users table with user-specific data (name, image,...) plus an user_id. And an another table with "general" information (as you said): name, desc, map, etc. And in this table doesn't use user-specific information only user_id.
And if your database isn't too large you can use a select tag with valid user_ids so you don't need validation.
EDIT if you want to know what are the user_ids which isn't in wait table, use similar query:
SELECT user.userid
FROM user
LEFT JOIN wait ON user.userid=wait.userid
WHERE ISNULL(wait.place_id)
These userid can put into a select-list.
Please read up on joins in select queries. Looks like you need to use a left outer join between your master table and your temporary table: http://www.tizag.com/mysqlTutorial/mysqlleftjoin.php.
You could use a query like this one:
select *
from wait_table
left join general_info_table on wait_table.user_id = general_info_table.user_id
where wait_table.user_id = 1;
This way, IF the user_id is in the wait_table it would return you the info on the client... if it doesn't exists in the table, well, should return null.
I would filter out which table fields i really need from the query, though.