I usually prepare reports and charts from excel manually using pivot table adding several columns manually from the raw data and then using pivot table on the fields and populating it.
And I would like to see if this can be automated by:
a) Loading the data into a mysql database
b) Using several queries to add additional columns and then prepare the data ready to be used by
c) Chart APIs/JQuery.
Since I know csv to mysql is easier, I now have the raw data file in CSV format.
The raw data basically contains different fields mainly time, date time and strings.
Using a PHP script, I was able to load these data using the LOAD DATA LOCAL INFILE command.
Based on dates, I need to prepare a column y which says months and this month column has to be updated with the month name('jan', etc.) depending on the date field(yyyy-mm-dd hh:mm:ss) on certain x column in the same table.
or maybe just use this and reference in the graphs(Not sure how complex that would be):-
mysql> select count(*) as Count, monthname(date) from alerts;
+-------+---------------------------------+
| Count | monthname(date) |
+-------+---------------------------------+
| 24124 | March |
+-------+---------------------------------+
1 row in set (0.19 sec)
Similarly, I need a column a that says "Duration < 5 minutes" and a column b that says "Duration > 5 min < 10 min" , where I would put a numeric value '1', if it falls within the range.
I looked into the self-join examples but I could not make it work in my case inspite of several efforts.
I need some help to get me going because my belief is that a table with all relevant columns is better off than using queries at runtime.
Also, is it better to format the data first and load it to mysql OR load the data and format it?
Please let me know.
Thanks
Update1
Okay, I got this working with a self join as below
UPDATE t1 p1 INNER JOIN ( select monthname(dt_received) AS EXTMONTHNAME from t1)p2 SET p1.MONTH=p2.EXTMONTHNAME;
but why does it update all the month with the same month name even though dt_received has other months ?
Can someone help?
Update2
Again, still struggling, I was made aware of the 1093 error/constraint. The workarounds are simply not helping
Unlike Excel where manual formatting was required, I found querying the database much easier using queryies
This resolved the issue
UPDATE tablename p1 INNER JOIN ( select monthname(dt_received) AS EXTMONTHNAME from tablename )p2 SET p1.MONTH=p2.EXTMONTHNAME where monthname(p1.dt_received)=p2.EXTMONTHNAME;
But would someone know, why it takes close to 14 minutes to change 36879 rows?
How do I optimize it.
Related
I have a PHP - MySQL set up . I have a table devicevalue structure of it is like this
devId | vals | date | time
xysz | 23 | 2020.02.17 | 22.06
abcs | 44 | 2020.02.31 | 22.07
The vals columns hold temperature values .
any user loggin in on my webapp have access to only certain devices.
Here are steps
On my website "a user" selects from and to dates for which he wants to see data & submit it
Then these dates are passed a page "getrecords.php " ,where there are lot select queries ( and many are in loop ) to fetch filtered data in required format
The problem is that this table holds almost 2-3 Million records . and in every where clause I have to add to and from conditions. this causes to search in entire table .
My question is there any way that I can get temporary table at step 1 which will have only certain rows based on given two dates and then all my queries on other page will be against that temporary table ?
Edit: If your date column is a text string, you must convert it to a column of type DATE or TIMESTAMP, or you will never get good performance from this table. A vast amount of optimization code is in the MySQL server to make handling of time/date data types efficient. If you store dates or times as strings, you defeat all that optimization code.
Then, put an index on your date column like this.
CREATE INDEX date_from_to ON devicevalue (`date`, devId, vals, `time` );
It's called a covering index because the entire query can be satisfied using it only.
Then, in your queries use
WHERE date >= <<<fromdate>>>
AND date < <<<todate>> + INTERVAL 1 DAY
Doing this indexing correctly gets rid of the need to create temp tables.
If your query has something like `WHERE devId = <<>> in it, you need this index instead (or in addition).
CREATE INDEX date_id_from_to ON devicevalue (devId, `date`, vals, `time` );
If you get a chance to change this table's layout, combine the date and time columns into a single column with TIMESTAMP data type. The WHERE clauses I showed you above will still work correctly if you do that. And everything will be just as fast.
SQL is made to solve your kind of problem simply and fast. With a good data choices and proper indexing, a few million records is a modestly-sized table.
Short answer: No. Don't design temp tables that need to live between sessions.
Longer answer:
Build into your app that the date range will be passed from one page to the next, then use those as initial values in the <form> <input type=text...>
Then make sure you have a good composite index for the likely queries. But, to do that, you must get a feel for what might be requested. You will probably need a small number of multi-column indexes.
You can probably build a SELECT from the form entries. I rarely need to use more than one query, but it is mostly "constructed" on the fly based on the form.
It is rarely a good idea to have separate columns for date and time. It makes it very difficult, for example, to say noon one day to noon the next day. Combine into a DATETIME or TIMESTAMP.
O.Jones has said a lot of things that I would normally add here.
I have a question as to a better way of doing this as I have a very large database with a lot symbols. "Hence a, aa... etc"
I would like to know if I can actually query every table also desc order would be nice. In one line. Otherwise I will have to type thousands of unions and it will be a pain later as the database will be changed often. As a table is erased and another joins it place.
Every table has the Date column and would like to search based on a date.
Thank you in advance.
I.E.
SELECT * from a where Date = '2017-07-31' union
SELECT * from aa where Date = '2017-07-31' union
SELECT * from aaap where Date = '2017-07-31' union
SELECT * from aabvf where Date = '2017-07-31' union
I mean, you COULD....
SELECT * FROM a,aa,aaap,aabvf WHERE date='2017-07-21'
Ahmed helped me out. As to why my data structure is like that. Well. If you have better suggested I'm opening to it. So.
Why.
Basically I have data in the form of symbols
I.E. A, AA that are stock tickers
They have dates that are unique keys to open, high, low, various other stock measurements.
So why I would want to grab just a single date. It's basically the top date or "today" to display and chart. So I can do various other things with the data.
If you have another method of storing I'm open.
I written a java program (not normally a web developer) that mines the data and that form and stores how I suggested. Which I could change, if you have a better way. I would love to hear. Also. If you have opinion on how to store data faster with MySQL I would love to hear. Currently I have few hundred threads that basically store data. Each thread handles a symbol. It creates a table if it doesn't exist with the ticker name and puts its data in columns separated date (unique key) open, high, etc... also various other operations the incoming data and stores that. Thank you for the answer and thank you if you have a better method !
Ps sorry I didn't mean chart. I display the top date as a table with corresponding data attached!
I want to know what dates are not present in my db, i have this query
SELECT Hour.date
, User.l_name
, User.f_name
FROM kadry.hours AS Hour
LEFT
JOIN kadry.subordinates Subordinate
ON Subordinate.supervisor_id LIKE "02705"
LEFT
JOIN kadry.users User
ON User.assigned = Subordinate.department_id
WHERE Hour.subordinate_id = User.id
AND Hour.date IN("2015-02-09","2015-02-10","2015-02-11"
,"2015-02-12","2015-02-13","2015-02-16"
,"2015-02-17","2015-02-18","2015-02-19"
,"2015-02-20","2015-02-23")
ORDER
BY User.id ASC
, Hour.date ASC
and in php i can process it and i can check if any date is missing, but i think there is a way to do it in MySQL, but i don't know how. any help would be great.
MySQL is a database, and as such, it can only show you what does it have. There are some advanced select commands, but usually you should want any logic on processing data in PHP.
Remember that selecting only dates and comparing them with something in PHP usually will be much faster than complicated MySQL-sided calculations if they are even possible.
I would recommend you to:
Prepare a list of all possible dates in PHP (either generated or
pre-definied). Store them in array.
Recover all dates from database.
Remove all dates in earlier prepared array that you found in MySQL. Method depends on data from MySQL, but the fastest should be to run
foreach over MySQL records.
I have a column I want to update with results of the difference between 2 mysql columns and count how many rows were affected. In my case it can only be 1. This is the mysql query I am using which is not consistent at all
$connection->query("UPDATE items SET Quantity_Available = Quantity - Quantity_Committed WHERE Item_ID = '$itemid'");
if($count=$connection->affected_rows!=1){echo $count;die('makassi');}
If I replace the Quantity_Committed with a numeric value, I get what I want i.e the code continues. However if I leave it as it is, I get the proper $count figure(1) but it also fails by echoing 'makassi' which it shouldn't.
Is this an improper way of subtracting 2 mysql columns or is this a bug in the php mysqli api??
This is really baffling to me!! Help please
This is a bad practice what you are trying to do. If a column in the database is derived from another column already in the column. Then such is create redundancy is the database. All a database should be normalized as much as possible. Please read here about data normalization.
Whatever you are trying to do can be achieved in a much better way. Like
Filtering the records
SELECT * FROM items WHERE Quantity - Quantity_Column > 5
Or, retrieving the quantity available.
SELECT (Quantify - Quantity_Column) as `Quality_Available` from items
I have an online highscores made with php + mysql but it currently shows the All Time highscores, I want to add Daily/Weekly/Monthly to that and I was wondering what would be the best way todo that?
My current thought is to add 3 new tables and then have the data inserted into each of them, and then having a cron which would run at the appropriate times to delete the data from each of the tables.
Is there any better way I could do this?
Another thing, I want to have it so the page would be highscores.php?t=all t=daily, etc. How would I make it so that the page changed the query depending on that value?
Thanks.
Use one table and add a column with the date of the highscore. Then have three different queries for each timespan, e.g.
SELECT ... FROM highscores WHERE date>"05-12-2011";
If you want to have a generic version without the need to have a fixed date, use this one:
SELECT ...
FROM highscores
WHERE date >= curdate() - INTERVAL DAYOFWEEK(curdate())+6 DAY;