Adapting formats between an sql database and a ploting service - php

im triyng to figure out an smart way to solve this problem. In my MySql server i have this simplified table:
As you can see, this is an statistic table, im interested on ploting how many visits (profile_visit under the type colunm) recived independently of the ip by each day from the begining of the data, that means that i have to query something like WHERE type=profile_visit AND user_url=xxx. But, this gives me a bunch of rows representing each visit made.
The question is, how can i use this raw data retrived from the query to obtain an array with the total visits by day (i dont care about time)?
Im using PHP, is a good idea to make the adaptation using a php script or it can be done using just MySQL querys?
If i reach the array with the total visits by day i can just simple adapt the format i need by:
$result=mysql_query("SELECT * FROM ".table_stats." WHERE user_url='xxx' AND type='profile_visit'");
echo "data.addRows([";
while($row = mysql_fetch_array($result, MYSQL_ASSOC)) {
$salida = $salida . "['".$row['date']."', $row['total']],";
}
$salida = rtrim($salida, ",");
echo $salida . "]);";
Thanks for any help and orientation about this.

You can easily do this directly from SQL that will run faster than retrieving the information from the DB and then processing it with php. The query should look like:
SELECT datetime, COUNT(id_stat) as numVisits WHERE type="type_profile" AND user_url = "xxx" GROUP BY DATE(DATE_SUB(datetime, INTERVAL 1 DAY))
This will return the number of visits (numVisits) grouped by day, and the lowest datetime recorded that day.
I do not know if you want to display the information just showing the day. If so, you will need to use php to modify the string provided by the DB.
Using your example the result of the query is:
datetime | numVisits
2011-11-10 12:05:44 | 9
2011-11-12 20:06:06 | 3
...

Is this what your after?
SELECT `tbl`.`ip`,COUNT(*) AS `visits` FROM `tbl` WHERE `tbl`.`type` = 'profile_visit' GROUP BY `tbl`.`ip` ORDER BY `visits`
This will return two columns, one with the IP and the other with the respective number of visits (all of type 'profile_vist').
UPDATE
Sorry, I read your question to quickly and missed the time parameter, you'll need something along the line:
SELECT
`tbl`.`ip`,
DATE(`tbl`.`date`) AS `date`,
COUNT(*) AS `visits`
FROM `tbl`
WHERE `tbl`.`type` = 'profile_visit'
AND DATE(`tbl`.`date`) = 'date'
GROUP BY `tbl`.`ip`
ORDER BY `visits`
This will give you a summary on the specific date. If you don't need the IP, remove it from the SELECT-list and GROUP BY(DATE(tbl.date)) instead. To optimize, consider using DATE instead of DATETIME to avoid casting between the two (or adding an additional column).

Related

How to filter out certain rows in MySQL dynamically to query against them?

I have a PHP - MySQL set up . I have a table devicevalue structure of it is like this
devId | vals | date | time
xysz | 23 | 2020.02.17 | 22.06
abcs | 44 | 2020.02.31 | 22.07
The vals columns hold temperature values .
any user loggin in on my webapp have access to only certain devices.
Here are steps
On my website "a user" selects from and to dates for which he wants to see data & submit it
Then these dates are passed a page "getrecords.php " ,where there are lot select queries ( and many are in loop ) to fetch filtered data in required format
The problem is that this table holds almost 2-3 Million records . and in every where clause I have to add to and from conditions. this causes to search in entire table .
My question is there any way that I can get temporary table at step 1 which will have only certain rows based on given two dates and then all my queries on other page will be against that temporary table ?
Edit: If your date column is a text string, you must convert it to a column of type DATE or TIMESTAMP, or you will never get good performance from this table. A vast amount of optimization code is in the MySQL server to make handling of time/date data types efficient. If you store dates or times as strings, you defeat all that optimization code.
Then, put an index on your date column like this.
CREATE INDEX date_from_to ON devicevalue (`date`, devId, vals, `time` );
It's called a covering index because the entire query can be satisfied using it only.
Then, in your queries use
WHERE date >= <<<fromdate>>>
AND date < <<<todate>> + INTERVAL 1 DAY
Doing this indexing correctly gets rid of the need to create temp tables.
If your query has something like `WHERE devId = <<>> in it, you need this index instead (or in addition).
CREATE INDEX date_id_from_to ON devicevalue (devId, `date`, vals, `time` );
If you get a chance to change this table's layout, combine the date and time columns into a single column with TIMESTAMP data type. The WHERE clauses I showed you above will still work correctly if you do that. And everything will be just as fast.
SQL is made to solve your kind of problem simply and fast. With a good data choices and proper indexing, a few million records is a modestly-sized table.
Short answer: No. Don't design temp tables that need to live between sessions.
Longer answer:
Build into your app that the date range will be passed from one page to the next, then use those as initial values in the <form> <input type=text...>
Then make sure you have a good composite index for the likely queries. But, to do that, you must get a feel for what might be requested. You will probably need a small number of multi-column indexes.
You can probably build a SELECT from the form entries. I rarely need to use more than one query, but it is mostly "constructed" on the fly based on the form.
It is rarely a good idea to have separate columns for date and time. It makes it very difficult, for example, to say noon one day to noon the next day. Combine into a DATETIME or TIMESTAMP.
O.Jones has said a lot of things that I would normally add here.

Needing only 1 row from PHP MySQL database

I'm sure I've done this in the past, but it's a few years ago and I don't remember how it's done and the online tutorials aren't helping.
I have a MySQL database. It has 1 table in it called 'data'. In the 'data' table, there are about 15,000 rows, and 31 columns. I need to extract the data from only 1 of these rows, based on a lookup referencing the string in column 1. When the mysql query finds the correct row, I need every single item read into variables that I can show on my page.
I believe this line is the problem:
$sql = "SELECT Mark,Manufacturer,Model FROM data";
Could someone please let me know what it needs to be changed to, to get the desired result? TIA! :)
you can set options of select query
$sql = "SELECT Mark,Manufacturer,Model FROM data WHERE Model (or manufacturer,mark) = 'some text'";
As my colleges have Explained "where' is your friend!
So you can always query as follows :
Select * from Data
Where column_1 = 'Your Desired String'
Alternatively you could use
Select Discinct Limit 1 Mark,Manufacturer,Model FROM data
Order By Mark Asc

Select query takes too long

These 2 querys take too long to produce a result (sometimes 1 min or even sometime end up on some error) and put really heavy load on the server:
("SELECT SUM(`rate`) AS `today_earned` FROM `".PREFIX."traffic_stats` WHERE `userid` = ?i AND from_unixtime(created) > CURRENT_DATE ORDER BY created DESC", $user->data->userid)
("SELECT COUNT(`userid`) AS `total_clicks` FROM `".PREFIX."traffic_stats` WHERE `userid` = ?i", $user->data->userid)
The table has about 4 million rows.
This is the table structure:
I have one index on traffic_id:
If you select anything from traffic_stats table it will take forever, however inserting to this table is normal.
Is it possible to reduce the time spent on executing this query? I use PDO and I am new to all this.
ORDER BY will take a lot of time and since you only need aggregate data (adding numbers or counting numbers is commutative), the ORDER BY will do a lot of useless sorting, costing you time and server power.
You will need to make sure that your indexing is right, you will probably need an index for user_id and for (user_id, created).
Is user_id numeric? If not, then you might consider converting it into numeric type, int for example.
These are improving your query and structure. But let's improve the concept as well. Are insertions and modifications very frequent? Do you absolutely need real-time data, or you can do with quasi-realtime data as well?
If insertions/modifications are not very frequent, or you can do with older data, or the problem is causing huge trouble, then you could do this by running periodically a cron job which would calculate these values and cache them. The application would read them from the cache.
I'm not sure why you accepted an answer, when you really didn't get to the heart of your problem.
I also want to clarify that this is a mysql question, and the fact that you are using PDO or PHP for that matter is not important.
People advised you to utilize EXPLAIN. I would go one further and tell you that you need to use EXPLAIN EXTENDED possibly with the format=json option to get a full picture of what is going on. Looking at your screen shot of the explain, what should jump out at you is that the query looked at over 1m rows to get an answer. This is why your queries are taking so long!
At the end of the day, if you have properly indexed your tables, your goal should be in a large table like this, to have number of rows examined be fairly close to the final result set.
So let's look at the 2nd query, which is quite simple:
("SELECT COUNT(`userid`) AS `total_clicks` FROM `".PREFIX."traffic_stats` WHERE `userid` = ?i", $user->data->userid)
In this case the only thing that is really important is that you have an index on traffic_stats.userid.
I would recommend, that, if you are uncertain at this point, drop all indexes other than the original primary key (traffic_id) index, and start with only an index on the userid column. Run your query. What is the result, and how long does it take? Look at the EXPLAIN EXTENDED. Given the simplicity of the query, you should see that only the index is being used and the rows should match the result.
Now to your first query:
("SELECT SUM(`rate`) AS `today_earned` FROM `".PREFIX."traffic_stats` WHERE `userid` = ?i AND from_unixtime(created) > CURRENT_DATE ORDER BY created DESC", $user->data->userid)
Looking at the WHERE clause there are these criteria:
userid =
from_unixtime(created) > CURRENT_DATE
You already have an index on userid. Despite the advice given previously, it is not necessarily correct to have an index on userid, created, and in your case it is of no value whatsoever.
The reason for this is that you are utilizing a mysql function from_unixtime(created) to transform the raw value of the created column.
Whenever you do this, an index can't be used. You would not have any concerns in doing a comparison with the CURRENT_DATE if you were using the native TIMESTAMP type but in this case, to handle the mismatch, you simply need to convert CURRENT_DATE rather than the created column.
You can do this by passing CURRENT_DATE as a parameter to UNIX_TIMESTAMP.
mysql> select UNIX_TIMESTAMP(), UNIX_TIMESTAMP(CURRENT_DATE);
+------------------+------------------------------+
| UNIX_TIMESTAMP() | UNIX_TIMESTAMP(CURRENT_DATE) |
+------------------+------------------------------+
| 1490059767 | 1490054400 |
+------------------+------------------------------+
1 row in set (0.00 sec)
As you can see from this quick example, UNIX_TIMESTAMP by itself is going to be the current time, but CURRENT_DATE is essentially the start of day, which is apparently what you are looking for.
I'm willing to bet that the number of rows for the current date are going to be fewer in number than the total rows for a user over the history of the system, so this is why you would not want an index on user, created as previously advised in the accepted answer. You might benefit from an index on created, userid.
My advice would be to start with an individual index on each of the columns separately.
("SELECT SUM(`rate`) AS `today_earned` FROM `".PREFIX."traffic_stats` WHERE `userid` = ?i AND created > UNIX_TIMESTAMP(CURRENT_DATE)", $user->data->userid)
And with your re-written query, again assuming that the result set is relatively small, you should see a clean EXPLAIN with rows matching your final result set.
As for whether or not you should apply an ORDER BY, this shouldn't be something you eliminate for performance reasons, but rather because it isn't relevant to your desired result. If you need or want the results ordered by user, then leave it. Unless you are producing a large result set, it shouldn't be a major problem.
In the case of that particular query, since you are doing a SUM(), there is no value of ORDERING the data, because you are only going to get one row back, so in that case I agree with Lajos, but there are many times when you might be utilizing a GROUP BY, and in that case, you might want the final results ordered.

PHP MySQL query - select all users but only display the earliest logon times for each user

I want to be able to bring back the earliest logon time per user, so only 1 record (the earliest record) displays for each user
I've tried various ways of GROUP BY but can't seem to get it quite right (if that is actually the correct way of doing). username is the unique value which can be used to GROUP BY
Here's the code I'm currently working with..
SELECT username, name, logon, added FROM data WHERE (date(added) LIKE '$date') AND (logon = (SELECT MIN(logon) FROM data))
I've also tried (below) but only get one result back, only displaying one user
WHERE (date(added) LIKE '$date') AND logon = (SELECT MIN(logon) FROM data)
The first image is what I'm currently getting, the second image is how I want my results to display, please see below
Let me know if you require anymore information, I've tried to put as much as possible
Thanks,
Tom
You are close. Your query needs a correlation clause:
SELECT d.*
FROM data d
WHERE date(d.added) = '$date' AND
d.logon = (SELECT MIN(d2.logon) FROM data d2 WHERE d2.name = d.name);
Note: The logic for added is confusing. First, you should not use like with dates. And, for that matter, you should not be inserting parameter values into the string, you should be using query parameters. And, actually, I don't see the need for that column; your question doesn't mention added.
use this query , problem will be solved
SELECT username, min(logontime) FROM data where date(logontime) = '2016-08-10' group by username
do this:
<?php
$users = array();
$sql = "select * from users";
$result= mysqli_query($conn,$sql);//$conn is your connection object
while($user = mysqli_fetch_assoc($result)){
if(!in_array($user['name'],$users)){
$users[$user['name']]=$user['logOnTime'];//replace key names with your table row names
}
}
//$users is an array that has your user names as key and their last logon time as value
?>

Queries execute correctly, but performance isn't good

I made a database which collects information on a daily basis. as like calender the database stores members' daily amount. If month doesn't match any existing month(M-Y) the database will create a new html month table. I have solved this problem, as follows:
mysql_query("something goes here");
while(condition)
{
mysql_query("something goes here")
while(condition)
{
mysql_query("something goes here");
while()
{
....................
}
........................
}
}
This algorithm worked well when I discovered it. However, after a few days, it was placing a heavy load on my server. I then tried the same algorithm in PHP (but I can't this). How can I make this run faster?
The code is as follows:
$q2=mysql_query("SELECT a.member_id,a.dates,MONTH(dates) AS months,
YEAR(dates)AS years,sum(amount) as sums
FROM account AS a
left join member as m
on(a.member_id=m.member_id)
GROUP BY (SELECT EXTRACT(YEAR_MONTH FROM dates))
ORDER by dates DESC
");
$k=0;
while($data2=mysql_fetch_assoc($q2))
{
$months=$data2['months'];
$years=$data2['years'];
$daten = new DateTime($data2['dates']);
print "<tr><th align='left'><b>".$daten->format('F-Y')."</b></th>";
$q3=mysql_query("select * from member");
while($data3=mysql_fetch_assoc($q3))
{
$id3=$data3['member_id'];
$q4=mysql_query("
SELECT SUM(amount) AS total FROM account
WHERE member_id=$id3
AND month(dates)=$months
AND year(dates)=$years
");
while($data4=mysql_fetch_assoc($q4))
{
$total=$data4['total'];
print "<td class='total'>".number_format($total)."</td>";
}
}
print "<td class='total'><b>".$data2['sums']."</b></td></tr>";
$k=$k+$data2['sums'];
}
Among other things:
You're running the query SELECT * FROM member for every row in the first query. This query is independent of the loop, so running it again every time is wasteful.
For each result from the SELECT * FROM member query, you're running another query (SELECT SUM(amount) AS total FROM account ...). There are several issues with this query:
First of all, this query could be combined into the previous query using a GROUP BY, to avoid having to run one query for every member. Something like:
SELECT member_id, SUM(amount) AS total FROM account WHERE ... GROUP BY member_id
Second of all, you're using MONTH(dates) = $months AND YEAR(dates) = $years. This is inefficient, as it forces the server to examine every row; converting it to a range on dates (e.g, dates BETWEEN '$year-$months-01' AND '$year-$months-31') would speed things up if there were an appropriate index on dates.
In general: Avoid queries in loops. The number of queries involved in generating a page should, to the degree possible, always be a small, nearly constant number. It should not grow with your data.
Have you setup the appropriate indexes in your MySQL database? This can cause a huge performance difference. http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html

Categories