I have two tables,
ir1_police is included messages that reported to admin.
ir1_police_flag, admin can flag reporters from 1 to 2. (1 means medium 2 means low and nothing means high priority).
If someone try to report something that is not real. Admin will flag it as 1 or 2 .
So I would make a list of report that shows first high priority, second medium and at last low.
I use the mysql statement but there is a problem. if there was nothing ir1_police_report nothing will be shown. or if exist only shows they are on ir_police_flags.
I have no idea to select them if no record exists on ir1_police_flags
SELECT * FROM ir1_police
JOIN ir1_police_flags on ir1_police_flags.uid = `ir1_police.uid
WHERE
ir1_police.status=0 AND ir1_police.parent_id=0
ORDER BY ir1_police.time DESC
Replace JOINwith LEFT JOIN. The former only selects rows from the tables where a match is found, whereas the latter selects all rows from the first table, even when there is no match in the other table.
Then you can add a second field to ORDER BY:
SELECT * FROM ir1_police
LEFT JOIN ir1_police_flags ON ir1_police_flags.uid = ir1_police.uid
WHERE ir1_police.status=0 AND ir1_police.parent_id=0
ORDER BY
ir1_police_flags.flag ASC,
ir1_police.time DESC
Notice the LEFT JOIN produces results where all ir1_police_flags's fields are NULL where there is no match in this table. This is perfect in your case, because NULL is considered smaller than any value, as far as ORDER BY is concerned.
Your application might justify this structure, but you should ask yourself whether this flag shouldn't be just a column in the table ir1_police altogether.
Related
I have an orders grid holding 1 million records. The page has pagination, sort and search options. So If the sort order is set by customer name with a search key and the page number is 1, it is working fine.
SELECT * FROM orders WHERE customer_name like '%Henry%' ORDER BY
customer_name desc limit 10 offset 0
It becomes a problem when the User clicks on the last page.
SELECT * FROM orders WHERE customer_name like '%Henry%' ORDER BY
customer_name desc limit 10 offset 100000
The above query takes forever to load. Index is set to the order id, customer name, date of order column.
I can use this solution https://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/ if I don't have a non-primary key sort option, but in my case sorting is user selected. It will change from Order id, customer name, date of order etc.
Any help would be appreciated. Thanks.
Problem 1:
LIKE "%..." -- The leading wildcard requires a full scan of the data, or at least until it finds the 100000+10 rows. Even
... WHERE ... LIKE '%qzx%' ... LIMIT 10
is problematic, since there probably not 10 such names. So, a full scan of your million names.
... WHERE name LIKE 'James%' ...
will at least start in the middle of the table-- if there is an index starting with name. But still, the LIMIT and OFFSET might conspire to require reading the rest of the table.
Problem 2: (before you edited your Question!)
If you leave out the WHERE, do you really expect the user to page through a million names looking for something?
This is a UI problem.
If you have a million rows, and the output is ordered by Customer_name, that makes it easy to see the Aarons and the Zywickis, but not anyone else. How would you get to me (James)? Either you have 100K links and I am somewhere near the middle, or the poor user would have to press [Next] 'forever'.
My point is that the database is not the place to introduce efficiency.
In some other situations, it is meaningful to go to the [Next] (or [Prev]) page. In these situations, "remember where you left off", then use that to efficiently reach into the table. OFFSET is not efficient. More on Pagination
I use a special concept for this. First I have a table called pager. It contains an primary pager_id, and some values to identify a user (user_id,session_id), so that the pager data can't be stolen.
Then I have a second table called pager_filter. I consist of 3 ids:
pager_id int unsigned not NULL # id of table pager
order_id int unsigned not NULL # store the order here
reference_id int unsigned not NULL # reference into the data table
primary key(pager_id,order_id);
As first operation I select all records matching the filter rules from and insert them into pager_filter
DELETE FROM pager_filter WHERE pager_id = $PAGER_ID;
INSERT INTO pager_filter (pager_id,order_id,reference_id)
SELECT $PAGER_ID pager_id, ROW_NUMBER() order_id, data_id reference_id
FROM data_table
WHERE $CONDITIONS
ORDER BY $ORDERING
After filling the filter table you can use an inner join for pagination:
SELECT d.*
FROM pager_filter f
INNER JOIN data_table d ON d.data_id = f.reference id
WHERE f.pager_id = $PAGER_ID && f.order_id between 100000 and 100099
ORDER BY f.order_id
or
SELECT d.*
FROM pager_filter f
INNER JOIN data_table d ON d.data_id = f.reference id
WHERE f.pager_id = $PAGER_ID
ORDER BY f.order_id
LIMIT 100 OFFSET 100000
Hint: All code above is not tested pseudo code
I have a MySQL database with 2 tables:
Table A:
Number
Location
Table B:
Calling Code
Area Code
Location
Initially, I have about 60,000 entries in table A, which has the Location column empty at the beginning. In table B I have about 250,000+ entries with a lot of area codes, calling codes (1, 011) and their respective location in the world. What I want is a FAST way of populating the table A's location column with the location of the number.
So for example if the first entry in Table A is (17324765600, null) I want to read trough table B and get the location for that number. Right now I am getting the location of a number with this query:
SELECT b.location
FROM
tableB b
LEFT JOIN tableA a
ON a.number LIKE CONCAT(b.calling_code, b.code, '%')
ORDER BY CHAR_LENGTH(b.code) DESC
LIMIT 1;
That gives me the proper location (even though I have my doubts that it can fail..). The problem is that performance wise this method is a no go. If I loop over all the 50k number
Update 1
Allow me to put some sample data with the expected output:
Sample Table A:
number location
17324765600 NULL
01134933638950 NULL
0114008203800 NULL
…60k Records + at the moment..
Sample Table B:
calling_code code location
1 7324765 US-NJ
011 34933 Spain
011 400820 China
…250,000+ records at the moment
Expected output after the processing:
Table A:
number location
17324765600 US-NJ
01134933638950 Spain
0114008203800 China
The best I’ve come up with is the following update statement:
UPDATE tableA a JOIN tableB b ON a.location LIKE CONCAT(b.calling_code, b.code, '%') SET a.location = b.location
Of course here I am not sure if it will always return the longest prefix of the code, for example if in the above tables there was another code starting with 73247XX let’s say that code is for Iowa (just as an example).. I am not sure if the query will always return the longest code so here I would also need help.
Let me know if the samples help.
.SQL for the database structure:
Download
Update 2:
I am thinking on doing this the following way:
Before inserting the data in table A I am thinking of exporting Table B into a CSV and sort it by area code, that way I can have 2 pointers one for the array of entries for table A and one for the csv, both sorted by area code that way I can make a kind of parallel search and populate the entry's location on PHP and not having to do this in MySQL.
Let me know if this approach seems like a better option if so I will test it out and publish the answer.
If you want all locations, then you need to remove LIMIT
SELECT b.location
FROM
tableB b
LEFT JOIN tableA a
ON a.number LIKE CONCAT(b.calling_code, b.code, '%')
ORDER BY CHAR_LENGTH(b.code);
If you want the same location name should not come twice then you need to use GROUP BY
SELECT b.location
FROM
tableB b
LEFT JOIN tableA a
ON a.number LIKE CONCAT(b.calling_code, b.code, '%')
GROUP BY b.location ORDER BY CHAR_LENGTH(b.code) ;
You have one join only with 250000 records, its not so stressful. You should take proper indexing for search columns and fine tune your mysql server. A good indexing & server variables well to set will solve your problem easily. Optimize your query well.Generally it creates problems when we have much of joins & many string comparison.
I think you need the query like this-
UPDATE a SET a.location = (
SELECT location from b
WHERE a.number LIKE CONCAT(b.calling_code, b.area_code, '%')
ORDER BY LENGTH(CONCAT(b.calling_code, b.area_code, '%')) desc
limit 1
);
I decided to take the below approach since I did not received any clear response:
Prior to the process I prepared 2 new tables, a table for country codes and a table for state codes (since I also need to know the state in case the number is within the US). Both tables will have: country, state, calling_code, code …
As for these 2 tables I broke down all the numbers with the prefixes and grouped them by area code so instead of having full 6 numbers to identify a country/state I grouped them by the first 3 numbers and if the code is within the USA or not, hence the 2 tables.
With this modifications I was able to break the 250,000 + rows table to only about 300 rows (each table).
After this I will follow these steps:
I get the list of phone numbers
I first execute a query very similar as the one I posted to update all the numbers that belong to the country_code table
I then update the rows that are still without location assigned with the table of state_code
I had to put some kind of cron in order to get this done every x amount of time to avoid having a huge amount of phones.
This may not be the best approach but for the 50k numbers that are in place at the moment I was able to (manually executing query by query with some more polishing) get it down to about 10 seconds, executing this every x amount of time (which will allow performing this process to less than 10k numbers) will make this smoothly.
I will mark this as the answer but if someone else magically comes up with a better answer I will make sure to update this.
Divide and conquer!
I am developing a car rental site. I have two tables test_tbl_cars and test_reservations.
I am using the search query (cribbed from Jon Kloske in "How do I approach this PHP/MYSQL query?"):
$sql = mysql_query("SELECT
test_tbl_cars.*,
SUM(rental_start_date <= '$ddate' AND rental_end_date >= '$adate') AS ExistingReservations
FROM test_tbl_cars
LEFT JOIN test_reservations USING (car_id)
GROUP BY car_id
HAVING ExistingReservations = 0");
This gives me excellent search results but the test_tbl_cars table contains many cars which in any given search returns several of the same car model as being available.
How can I filter the query return such that I get one of each model available?
Use Distict clause
$sql = mysql_query("SELECT
DISTINCT test_tbl_cars.model, test_tbl_cars.*,
SUM(rental_start_date <= '$ddate' AND rental_end_date >= '$adate') AS ExistingReservations
FROM test_tbl_cars
LEFT JOIN test_reservations USING (car_id)
GROUP BY car_id
HAVING ExistingReservations = 0");
Awww, should have tagged me, I only saw this now over a year later! You've probably already figured out how to work this by now, but I'll take a crack at it anyway for completeness sake and because most of the answers here I don't think are doing what you want.
So the problem you are having is that in the other question each room had a unique ID and it was unique rooms people were interested in booking. Here, you're extending the concept of a bookable item to a pool of items of a particular class (in this case, model of car).
There's may be a way to do this without subqueries but by far the easiest way to do it is to simply take the original idea from my other answer and extend it by wrapping it up in another query that does the grouping into models (and as you'll see shortly, we get a bunch of other useful stuff for free out of doing this).
So, firstly lets start by getting the list of cars with counts of conflicting reservations (as per the update to my other answer):
(I'll use your query for these examples as a starting point, but note you really should use prepared statements or at the very least escaping functions supplied by your DB driver for the two parameters you're passing)
SELECT car_id, model_id, SUM(IF(rental_id IS NULL, 0, rental_start_date <= '$ddate' AND rental_end_date >= '$adate')) AS ConflictingReservations
FROM test_tbl_cars
LEFT JOIN test_reservations USING (car_id)
GROUP BY car_id
This will return one row per car_id giving you the model number, and the number of reservations that conflict with the date range you've specified (0 or more).
Now at this stage if we were asking about individual cars (rather than just models of cars available) we could restrict and order the results with "HAVING ConflictingReservations = 0 ORDER BY model_id" or something.
But, if we want to get a list of the availability of ~models~, we need to perform a further grouping of these results to get the final answer:
SELECT model_id, COUNT(*) AS TotalCars, SUM(ConflictingReservations = 0) AS FreeCars, CAST(IFNULL(GROUP_CONCAT(IF(ConflictingReservations = 0, car_id, NULL) ORDER BY car_id ASC), '') AS CHAR) AS FreeCarsList
FROM (
SELECT car_id, model_id, SUM(IF(rental_id IS NULL, 0, rental_start_date <= '$ddate' AND rental_end_date >= '$adate')) AS ConflictingReservations
FROM test_tbl_cars
LEFT JOIN test_reservations USING (car_id)
GROUP BY car_id
) AS CarReservations
GROUP BY model_id
You'll notice all we're doing is grouping the original query by model_id, and then using aggregate functions to get us the model_id, a count of total cars we have of this model, a count of free cars of this model we have which we achieve by counting all the times a car has zero ConflictingReservations, and finally a cute little bit of SQL that returns a comma separated list of the car_ids of the free cars (in case that was also needed!)
A quick word on performance: all the left joins, group bys, and subqueries could make this query very slow indeed. The good news is the outer group by should only have to process as many rows as you have cars for, so it shouldn't be slow until you end up with a very large number of cars. The inner query however joins two tables (which can be done quite quickly with indexes) and then groups by the entire set, performing functions on each row. This could get quite slow, particularly as the number of reservations and cars increases. To alleviate this you could use where clauses on the inner query and combine that with appropriate indexes to reduce the number of items you are inspecting. There's also other tricks you can use to move the comparison of the start and end dates into the join condition, but that's a topic for another day :)
And finally, as always, if there's incorrect edge cases, mistakes, wrong syntax, whatever - let me know and I'll edit to correct!
I have to run this Mysql query on my website to fetch huge amount of data: (3 tables , each with 100,000 + records)
SELECT on_resume.*, on_users.subscribed, on_users.user_avatar, on_resume_page.*
FROM on_resume
LEFT JOIN on_users ON (on_resume.resume_userid = on_users.user_id )
LEFT JOIN on_resume_page ON ( on_resume.resume_userid = on_resume_page.resume_userid)
WHERE on_resume.active= '1'
GROUP BY on_resume.rid
ORDER BY on_resume.rid DESC
LIMIT 0,18
The time I run this at Phpmyadmin sql section, the whole mysqld service will be down and needs to be restarted.
Now I was testing this query and I found out if I don't use Group by and Order by conditions the query will be fine.
SELECT on_resume.*, on_users.subscribed, on_users.user_avatar, on_resume_page.*
FROM on_resume
LEFT JOIN on_users ON (on_resume.resume_userid = on_users.user_id )
LEFT JOIN on_resume_page ON ( on_resume.resume_userid = on_resume_page.resume_userid)
WHERE on_resume.active= '1'
LIMIT 0,18
Showing rows 0 - 17 ( 18 total, Query took 0.4248 sec)
Why is it like this and how can I fix it?...
NOTE : I have tested the SQL query with group by or Order by alone in either case , even with one of them still the query fails and hangs the server.
EDIT : This problem is solved by making column on_resume_page.resume_userid indexed.
This is what i was told, took a while to figure it out:
Look at #jer in Chicago comment
Remember, when there is a GROUP BY clause, there are certain rules that apply for grouping columns. One of those rules is "The Single-Value Rule" -- every column named in the SELECT list must also be a grouping column unless it is an argument for one of the set functions. MySQL extends standard SQL by allowing you to use columns or calculations in a SELECT list that don't appear in a GROUP BY clause. However, we are warned not to use this feature unless the columns you omit from the GROUP BY clause are not unique in the group because you will get unpredictable results.
I'm new to SQL and I want to implement the following query:
I've got two tables, LicenseTbl and UnlockTbl:
LicenseTbl contains information about a purchased software license:
LicenseID, ProgramID, Owner, Location, OrderNo, BlockTime
UnlockTbl contains information about a specific software registration:
UnlockID, LicenseID (foreign key into LicenseTbl), Timestamp, SerialNo, Key, UninstallTime
where BlockTime and UninstallTime contain a timestamp if the license was blocked or the software uninstalled and NULL otherwise.
I want to devise a query that gives me ALL LicenseIDs for which the following conditions hold:
belongs to a given customer,
is not blocked,
is either not listed in the UnlockTbl or there are < X different SerialNo's in lines which are not marked as uninstalled.
I have written this, but I'm not sure if it is absolutely correct (it's one of my first SQL queries ever):
SELECT LicenseID FROM LicenseTbl
JOIN UnlockTbl
ON (LicenseTbl.LicenseID = UnlockTbl.LicenseID)
WHERE LicenseTbl.OrderNo = '$givenOrderNo'
AND LicenseTbl.Owner = '$givenOwner'
AND LicenseTbl.Location = '$givenLocation'
AND LicenseTbl.BlockTime IS NULL
AND UnlockTbl.UninstallTime IS NULL
GROUP BY LicenseTbl.LicenseID, UnlockTbl.Key
HAVING COUNT(*) < $X
(which is supposed to mean, list all licenses which have only been used less than X times simultaneously. I would prefer those that have been used the least first but don't know how to sort like that.)
This is a good start, but I would change the query to the following...
SELECT
LicenseID
FROM
LicenseTbl
LEFT JOIN
UnlockTbl
ON UnlockTbl.LicenseID = LicenseTbl.LicenseID
AND UnlockTbl.UninstallTime IS NULL
WHERE
LicenseTbl.OrderNo = '$givenOrderNo'
AND LicenseTbl.Owner = '$givenOwner'
AND LicenseTbl.Location = '$givenLocation'
AND LicenseTbl.BlockTime IS NULL
GROUP BY
LicenseTbl.LicenseID
HAVING
COUNT(DISTINCT UnlockTbl.SerialNo) < $X
ORDER BY
COUNT(DISTINCT UnlockTbl.SerialNo)
1). LEFT JOIN
A LEFT JOIN ensures that all rows in LicenseTbl are returned, even if there are no matches in the UnlockTbl table. (If there are no matches, the UnlockTbl table's values are all represented as NULL.)
2). UnlockTbl.UninstallTime IS NULL in the JOIN and not the WHERE
The WHERE clause is applied after the JOIN. This means that any records in UnlockTbl where UninstallTime have a real value (NOT NULL) get joined and then get filtered out. This in turn means that if all the relevant records in UnlockTbl have a non-NULL value in UninstallTime, all the rows for that License will get filtered.
3). GROUP BY on just the license, not the Key.
Simply, I don't know why you had it there, and it doesn't appear in the English description of what you want.
As you want a list of LicenseIDs, grouping by only that field ensures that you get one record per LicenseID.
4). HAVING clause modified to look at COUNT(DISTINCT SerialNo)
COUNT(*) counts all records. Even if there was no match (All the UnlockTbl values appearing as NULL), this would return 1.
COUNT(SerialNo) counts only records where SerialNo is NOT NULL. If there was no match (All the UnlockTbl values appearing as NULL), this would return 0.
COUNT(DISTINCT SerialNo) also counts only records where SerialNo is NOT NULL, but treats duplicates of the sme value as just 1 entry.
5). ORDER BY COUNT(DISTINCT SerialNo)
Takes the same value as is being filtered in the HAVING clause, and orders by it.