I want to know what dates are not present in my db, i have this query
SELECT Hour.date
, User.l_name
, User.f_name
FROM kadry.hours AS Hour
LEFT
JOIN kadry.subordinates Subordinate
ON Subordinate.supervisor_id LIKE "02705"
LEFT
JOIN kadry.users User
ON User.assigned = Subordinate.department_id
WHERE Hour.subordinate_id = User.id
AND Hour.date IN("2015-02-09","2015-02-10","2015-02-11"
,"2015-02-12","2015-02-13","2015-02-16"
,"2015-02-17","2015-02-18","2015-02-19"
,"2015-02-20","2015-02-23")
ORDER
BY User.id ASC
, Hour.date ASC
and in php i can process it and i can check if any date is missing, but i think there is a way to do it in MySQL, but i don't know how. any help would be great.
MySQL is a database, and as such, it can only show you what does it have. There are some advanced select commands, but usually you should want any logic on processing data in PHP.
Remember that selecting only dates and comparing them with something in PHP usually will be much faster than complicated MySQL-sided calculations if they are even possible.
I would recommend you to:
Prepare a list of all possible dates in PHP (either generated or
pre-definied). Store them in array.
Recover all dates from database.
Remove all dates in earlier prepared array that you found in MySQL. Method depends on data from MySQL, but the fastest should be to run
foreach over MySQL records.
Related
I have a question as to a better way of doing this as I have a very large database with a lot symbols. "Hence a, aa... etc"
I would like to know if I can actually query every table also desc order would be nice. In one line. Otherwise I will have to type thousands of unions and it will be a pain later as the database will be changed often. As a table is erased and another joins it place.
Every table has the Date column and would like to search based on a date.
Thank you in advance.
I.E.
SELECT * from a where Date = '2017-07-31' union
SELECT * from aa where Date = '2017-07-31' union
SELECT * from aaap where Date = '2017-07-31' union
SELECT * from aabvf where Date = '2017-07-31' union
I mean, you COULD....
SELECT * FROM a,aa,aaap,aabvf WHERE date='2017-07-21'
Ahmed helped me out. As to why my data structure is like that. Well. If you have better suggested I'm opening to it. So.
Why.
Basically I have data in the form of symbols
I.E. A, AA that are stock tickers
They have dates that are unique keys to open, high, low, various other stock measurements.
So why I would want to grab just a single date. It's basically the top date or "today" to display and chart. So I can do various other things with the data.
If you have another method of storing I'm open.
I written a java program (not normally a web developer) that mines the data and that form and stores how I suggested. Which I could change, if you have a better way. I would love to hear. Also. If you have opinion on how to store data faster with MySQL I would love to hear. Currently I have few hundred threads that basically store data. Each thread handles a symbol. It creates a table if it doesn't exist with the ticker name and puts its data in columns separated date (unique key) open, high, etc... also various other operations the incoming data and stores that. Thank you for the answer and thank you if you have a better method !
Ps sorry I didn't mean chart. I display the top date as a table with corresponding data attached!
I am working on an timesheet application, and writing a PHP code to fetch all the timesheets till date. This is the query that I have written to fetch the timesheets -
SELECT a.accnt_name, u.username, DATE_FORMAT(t.in_time, '%H:%i') inTime, DATE_FORMAT(t.out_time, '%H:%i') outTime, DATE_FORMAT(t.work_time, '%H:%i') workTime, w.wrktyp_name, t.remarks, DATE_FORMAT(t.tmsht_date, '%d-%b-%Y') tmshtDate, wl.loctn_name, s.serv_name, t.status_code, t.conv_kms convkms, t.conv_amount convamount FROM timesheets t, accounts a, services s, worktypes w, work_location wl, users WHERE a.accnt_code=t.accnt_code and w.wrktyp_code=t.wrktyp_code and wl.loctn_code=t.loctn_code and s.serv_code=t.serv_code and t.usr_code = u. ORDER BY tmsht_date desc
The where clause contains the clauses to get the actual values of respective codes from respective tables.
The issue is that this query is taking a lot of time to execute and the application crashes at the end of few minutes.
I ran this query in the phpmyadmin, there it works without any issues.
Need help in understanding what might be the cause behind the slowness in the execution.
Use EXPLAIN to see the execution plan for the query. Make sure MySQL has suitable indexes available, and is using those indexes.
The query text seems to be missing the name of a column here...
t.usr_code = u. ORDER
^^^
We can "guess" that's supposed to be u.usr_code, but that's just a guess.
How many rows are supposed to be returned? How large is the resultset?
Is your client attempting to "store" all of the rows in memory, and crashing because it runs out of memory?
If so, I recommend you avoid doing that, and fetch the rows as you need them.
Or, consider adding some additional predicates in the WHERE clause to return just the rows you need, rather than all the rows in the table.
It's 2015. Time to ditch the old-school comma syntax for join operation, and use JOIN keyword instead, and move join predicates from the WHERE clause to the ON clause. And format it. The database doesn't care, but it will make it easier on the poor soul that needs to decipher your SQL statement.
SELECT a.accnt_name
, u.username
, DATE_FORMAT(t.in_time ,'%H:%i') AS inTime
, DATE_FORMAT(t.out_time ,'%H:%i') AS outTime
, DATE_FORMAT(t.work_time,'%H:%i') AS workTime
, w.wrktyp_name
, t.remarks
, DATE_FORMAT(t.tmsht_date, '%d-%b-%Y') AS tmshtDate
, wl.loctn_name
, s.serv_name
, t.status_code
, t.conv_kms AS convkms
, t.conv_amount AS convamount
FROM timesheets t
JOIN accounts a
ON a.accnt_code = t.accnt_code
JOIN services s
ON s.serv_code = t.serv_code
JOIN worktypes w
ON w.wrktyp_code = t.wrktyp_code
JOIN work_location wl
ON wl.loctn_code = t.loctn_code
JOIN users
ON u.usr_code = t.usr_code
ORDER BY t.tmsht_date DESC
Ordering on the formatted date column is very odd. Much more likely you want results returned in "date" order, not in the string order with month and day before the year. (Do you really want to sort on the day value first, before the year?)
FOLLOWUP
If this same exact query complete quickly, with the entire resultset (of approx 720 rows) from a different client (same database, same user), then the issue is likely something other than this SQL statement.
We would not expect the execution of the SQL statement to cause PHP to "crash".
If you are storing the entire resultset (for example, using mysqli store_result), you need to have sufficient memory for that. But the thirteen expressions in the select list all look relatively short (formatted dates, names and codes), and we wouldn't expect "remarks" would be over a couple of KB.
For debugging this, as others have suggested, try adding a LIMIT clause on the query, e.g. LIMIT 1 and observe the behavior.
Alternatively, use a dummy query for testing; use a query that is guaranteed to return specific values and a specific number of rows.
SELECT 'morpheus' AS accnt_name
, 'trinity' AS username
, '01:23' AS inTime
, '04:56' AS outTime
, '00:45' AS workTime
, 'neo' AS wrktyp_name
, 'yada yada yada' AS remarks
, '27-May-2015' AS tmshtDate
, 'zion' AS loctn_name
, 'nebuchadnezzar' AS serv_name
, '' AS status_code
, '123' AS convkms
, '5678' AS convamount
I suspect that the query is not the root cause of the behavior you are observing. I suspect The problem is somewhere else in the code.
How to debug small programs http://ericlippert.com/2014/03/05/how-to-debug-small-programs/
phpadmin automatically adds LIMIT to the query, that's why you got fast results.
Check how many rows are in table
Run your query with limit
First of all: modify you query so that it looks like the one given by Spencer
Do you get an error message when your application 'crashes' or does it just stop?
You could try:
ini_set('max_execution_time', 0);
in your php code. This sets the maximum execution time to unlimited. So if there are no errors, your script should execute to the end. So you can see if your query gets the desired results.
Also just as a test end your query with
LIMIT 10
This should greatly speed up your query as it will only take the first ten results.
You can later change this value to one better suited for your needs. Unless you absolutely need the complete result set, I suggest you always use LIMIT in your queries.
I am running a select * from table order by date desc query using php on a mysql db server, where the table has a lot of records, which slows down the response time.
So, is there any way to speed up the response. If indexing is the answer, what all columns should I make indexes.
An index speeds up searching when you have a WHERE clause or do a JOIN with fields you have indexed. In your case you don't do that: You select all entries in the table. So using an index won't help you.
Are you sure you need all of the data in that table? When you later filter, search or aggregate this data in PHP, you should look into ways to do that in SQL so that the database sends less data to PHP.
you need to use caching system.
the best i know Memcache It's really great to speed up your application and it's not using database at all.
Simple answer: you can't speed anything up using software.
Reason: you're selecting entire contents of a table and you said it's a large table.
What you could do is cache the data, but not using Memcache because it's got a limit on how much data it can cache (1 MB per key), so if your data exceeds that - good luck using Memcache to cache a huge result set without coming up with an efficient scheme of maintaining keys and values.
Indexing won't help because you haven't got a WHERE clause, what could happen is that you can speed up the order by clause slightly. Use EXPLAIN EXTENDED before your query to see how much time is being spent in transmitting the data over the network and how much time is being spent in retrieving and sorting the data from the query.
If your application requires a lot of data in order for it to work, then you have these options:
Get a better server that can push the data faster
Redesign your application because if it requires so much data in order to run, it might not be designed with efficiency in mind
Optimizing Query is a big topic and beyond the scope this question
here are some highlight that will boost you select statement
Use proper Index
Limit the number records
use the column name that you require (instead writing select * from table use select col1, col2 from table)
to limit query for large offset is little tricky in mysql
this select statement for large offset will be slow because it have to process large set of data
SELECT * FROM table order by whatever LIMIT m, n;
to optimize this query here is simple solution
select A.* from table A
inner join (select id from table order by whatever limit m, n) B
on A.id = B.id
order by A.whatever
I have to get all entries in database that have a publish_date between two dates. All dates are stored as integers because dates are in UNIX TIMESTAMP format...
Following query works perfect but it takes "too long". It returns all entries made between 10 and 20 dazs ago.
SELECT * FROM tbl_post WHERE published < (UNIX_TIMESTAMP(NOW())-864000)
AND published> (UNIX_TIMESTAMP(NOW())-1728000)
Is there any way to optimize this query? If I am not mistaken it is calling the NOW() and UNIX_TIMESTAMP on evey entry. I thought that saving the result of these 2 repeating functions into mysql #var make the comparison much faster but it didn't. 2nd code I run was:
SET #TenDaysAgo = UNIX_TIMESTAMP(NOW())-864000;
SET #TwentyDaysAgo = UNIX_TIMESTAMP(NOW())-1728000;
SELECT * FROM tbl_post WHERE fecha_publicado < #TenDaysAgo
AND fecha_publicado > #TwentyDaysAgo;
Another confusing thing was that PHP can't run the bove query throught mysql_query(); ?!
Please, if you have any comments on this problem it will be more than welcome :)
Luka
Be sure to have an index on published.And make sure it is being used.
EXPLAIN SELECT * FROM tbl_post WHERE published < (UNIX_TIMESTAMP(NOW())-864000) AND published> (UNIX_TIMESTAMP(NOW())-1728000)
should be a good start to see what's going on on the query. To add an index:
ALTER TABLE tbl_post ADD INDEX (published)
PHP's mysql_query function (assuming that's what you're using) can only accept one query per string, so it can't execute the three queries that you have in your second query.
I'd suggest moving that stuff into a stored procedure and calling that from PHP instead.
As for the optimization, setting those variables is about as optimized as you're going to get for your query. You need to make the comparison for every row, and setting a variable provides the quickest access time to the lower and upper bounds.
One improvement in the indexing of the table, rather than the query itself would be to cluster the index around fecha_publicado to allow MySQL to intelligently handle the query for that range of values. You could do this easily by setting fecha_publicado as PRIMARY KEY of the table.
The obvious things to check are, is there an index on the published date, and is it being used?
The way to optimize would be to partition the table tbl_post on the published key according to date ranges (weekly seems appropriate to your query). This is a feature that is available for MySQL, PostgreSQL, Oracle, Greenplum, and so on.
This will allow the query optimizer to restrict the query to a much narrower dataset.
I agree with BraedenP that a stored procedure would be appropriate here. If you can't use one or really don't want to, you can always either generate the dates on the PHP side, but they might not match exactly with the database unless you have them synced.
You can also do it more quickly as 3 separate queries likely. Query for the begin data, query for the end date, then use those values as input into your target query.
I'm running a sql query to get basic details from a number of tables. Sorted by the last update date field. Its terribly tricky and I'm thinking if there is an alternate to using the UNION clause instead...I'm working in PHP MYSQL.
Actually I have a few tables containing news, articles, photos, events etc and need to collect all of them in one query to show a simple - whats newly added on the website kind of thing.
Maybe do it in PHP rather than MySQL - if you want the latest n items, then fetch the latest n of each of your news items, articles, photos and events, and sort in PHP (you'll need the last n of each obviously, and you'll then trim the dataset in PHP). This is probably easier than combining those with UNION given they're likely to have lots of data items which are different.
I'm not aware of an alternative to UNION that does what you want, and hopefully those fetches won't be too expensive. It would definitely be wise to profile this though.
If you use Join in your query you can select datas from differents tables who are related with foreign keys.
You can look of this from another angle: do you need absolutely updated information? (the moment someone enters new information it should appear)
If not, you can have a table holding the results of the query in the format you need (serving as cache), and update this table every 5 minutes or so. Then your query problem becomes trivial, as you can have the updates run as several updates in the background.