I am currently building a site for a car dealership. I would like to allow the user to refine the search results similar to amazon or ebay. The ability to narrow down the results with a click would be great. The problem is the way I am doing this now there are many different queries that need to be done each with a COUNT total.
So the main ways to narrow down the results are:
Vehicle Type
Year
Make
Price Range
New/Used
Currently I am doing 5 queries every time this page is loaded to get the numbers of results while passing in the set values.
Query 1:
SELECT vehicle_type, COUNT(*) AS total FROM inventory
[[ Already Selected Search Parameters]]
GROUP BY vehicle_type
ORDER BY vehicle_type ASC
Query 2:
SELECT make, COUNT(*) AS total FROM inventory
[[ Already Selected Search Parameters]]
GROUP BY make
ORDER BY make ASC
Query 3,4,5...
Is there any way to do this in one query? Is it faster?
Your queries seem reasonable.
You can do it in a single query using UNION ALL:
SELECT 'vehicle_type' AS query_type, vehicle_type, COUNT(*) AS total
FROM inventory
...
GROUP BY vehicle_type
UNION ALL
SELECT 'make', make, COUNT(*) AS total FROM inventory ... GROUP BY make
UNION ALL
SELECT ... etc ...
The performance benefit of this will not be huge.
If you find that you are firing off these queries a lot and the results don't change often, you might want to consider caching the results. Consider using something like memcache.
There are a couple ways to rank data along the lines of data warehousing but what you are trying to accomplish in search terms is called facets. A real search engine (which would be used with the sites you mentioned) performs this.
SEE: Faceted searching and categories in MySQL and Solr
Many sites use Lucene (Java-based) search engine with SOLR to accomplish what you are referring to. There is a newer solution called ElasticSearch that has a RESTful API and offers facets but you'd need to install Java, ES, and then could make calls to search engine that returns native JSON.
SEE: http://www.elasticsearch.org/guide/reference/api/search/facets/
Doing it in MySQL without requiring so many joins might need additional tables and perhaps triggers and gets complex. If the car dealership isn't expecting Cars.com traffic (millions/day) then you may be trying to optimize something before it actually needs it. Your recursive query might be fast enough and you haven't reported that there is an actual issue or bottleneck.
Use JOIN syntax:
http://dev.mysql.com/doc/refman/5.6/en/join.html
Or, I think you could write MySQL function for that. Where you will pass your search Parameters.
http://dev.mysql.com/doc/refman/5.1/en/create-function.html
To find where is faster you should do your own speed tests. That helped me to find out that some of my queries faster without joining them.
Related
I am working on big eCommerce shopping website. I have around 40 databases. i want to create search page which show 18 result after searching by title in all databases.
(SELECT id_no,offers,image,title,mrp,store from db1.table1 WHERE MATCH(title) AGAINST('$searchkey') AND title like '%$searchkey%')
UNION ALL (SELECT id_no,offers,image,title,mrp,store from db3.table3 WHERE MATCH(title) AGAINST('$searchkey') AND title like '%$searchkey%')
UNION ALL (SELECT id_no,offers,image,title,mrp,store from db2.table2 WHERE MATCH(title) AGAINST('$searchkey') AND title like '%$searchkey%')
LIMIT 18
currently i am using the above query its working fine for 4 or more character keyword search like laptop nokia etc but takes 10-15 sec for processes but for query with keyword less than 3 characters it takes 30-40sec or i end up with 500 internal server error. Is there any optimized way for searching in multiple databases. I generated two index primary and full text index with title
Currently my search page is in php i am ready to code in python or any
other language if i gets good speed
You can use the sphixmachine:http://sphinxsearch.com/. This is powerfull search for database. IMHO Sphinx this best decision
for search in your site.
FULLTEXT is not configured (by default) for searching for words less than three characters in length. You can configure that to handle shorter words by setting a ...min_token_size parameter. Read this. https://dev.mysql.com/doc/refman/5.7/en/fulltext-fine-tuning.html You can only do this if you control the MySQL server. It won't be possible on shared hosting. Try this.
FULLTEXT is designed to produce more false-positive matches than false-negative matches. It's generally most useful for populating dropdown picklists like the ones under the location field of a browser. That is, it requires some human interaction to choose the correct record. To expect FULLTEXT to be able to do absolutely correct searches is probably a bad idea.
You simply cannot use AND column LIKE '%whatever%' if you want any reasonable performance at all. You must get rid of that. You might be able to rewrite your python program to do something different when the search term is one or two letters, and thereby avoid many, but not all, LIKE '%a%' and LIKE '%ab%' operations. If you go this route, create ordinary indexes on your title columns. Whatever you do, don't combine the FULLTEXT and LIKE searches in a single query.
If this were my project I'd consider using a special table with columns like this to hold all the short words from the title column in every row of each table.
id_pk INT autoincrement
id_no INT
word VARCHAR(3)
Then you can use a query like this to look up short words
SELECT a.id_no,offers,image,title,mrp,store
FROM db1.table1 a
JOIN db1.table1_shortwords s ON a.id_no = s.id_no
WHERE s.word = '$searchkey'
To do this, you will have to preprocess the title columns of your other tables to populate the shortwords tables, and put an index on the word column. This will be fast, but it will require a special-purpose program to do the preprocessing.
Having to search multiple tables with your UNION ALL operation is a performance problem. You will be able to improve performance dramatically by redesigning your schema so you need search only one table.
Having to search databases on different server machines is a performance problem. You may be able to rig up your python program to search them in parallel: that is, to somehow use separate tasks to search each one, then aggregate the results. Each of those separate search tasks requires its own connection to the data base, so this is not a cheap or simple solution.
If this system faces the public web, you will have to redesign it sooner or later, because it will never perform well enough as it is now. (Sorry to be the bearer of bad news.) Many system designers like to avoid redesigning systems after they become enormous. So, if I were you I would get the redesign done.
If your focus is on searching, then bend the schema to facilitate searching rather than the other way around.
Collect all the strings to search for in a single table. Whereas a UNION of 40 tables does work, it will be ~40 times as slow as having the strings collected together.
Use FULLTEXT when the words are long enough, use some other technique when they are not. (This addresses your 3-char problem; see also the Answer discussing innodb_ft_min_token_size. You are using InnoDB, correct?)
Use + and boolean mode to say that a word is mandatory: MATCH(col) AGAINST("+term" IN BOOLEAN MODE)
Do not add on a LIKE clause unless there is a good reason.
As i am a junior PHP Developer growing day by day stuck in a performance problem described here:
I am making a search engine in PHP ,my database has one table with 41 column and million's of rows obviously it is a very large dataset. In index.php i have a form for searching data.When user enters search keyword and hit submit the action is on search.php with results.The query is like this.
SELECT * FROM TABLE WHERE product_description LIKE '%mobile%' ORDER BY id ASC LIMIT 10
This is the first query.After result shows i have to run 4 other query like this:
SELECT DISTINCT(weight_u) as weight from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(country_unit) as country_unit from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(country) as country from TABLE WHERE product_description LIKE '%mobile%'
SELECT DISTINCT(hs_code) as hscode from TABLE WHERE product_description LIKE '%mobile%'
These queries are for FILTERS ,the problem is this when i submit search button ,all queries are running simultaneously at the cost of Performance issue,its very slow.
Is there any other method to fetch weight,country,country_unit,hs_code speeder or how can achieve it.
The same functionality is implemented here,Where the filter bar comes after table is filled with data,How i can achieve it .Please help
Full Functionality implemented here.
I have tried to explain my full problem ,if there is any mistake please let me know i will improve the question,i am also new to stackoverflow.
Firstly - are you sure this code is working as you expect it? The first query retrieves 10 records matching your search term. Those records might have duplicate weight_u, country_unit, country or hs_code values, so when you then execute the next 4 queries for your filter, it's entirely possible that you will get values back which are not in the first query, so the filter might not make sense.
if that's true, I would create the filter values in your client code (PHP)- finding the unique values in 10 records is going to be quick and easy, and reduces the number of database round trips.
Finally, the biggest improvement you can make is to use MySQL's fulltext searching features. The reason your app is slow is because your search terms cannot use an index - you're wild-carding the start as well as the end. It's like searching the phonebook for people whose name contains "ishra" - you have to look at every record to check for a match. Fulltext search indexes are designed for this - they also help with fuzzy matching.
I'll give you some tips that will show useful in many situations when querying a large dataset, or mostly any dataset.
If you can list the fields you want instead of querying for '*' is a better practice. The weight of this increases as you have more columns and more rows.
Always try to use the PK's to look for the data. The more specific the filter, the less it will cost.
An index in this kind of situation would come pretty handy, as it will make the search more agile.
LIKE queries are generally pretty slow and resource heavy, and more in your situation. So again, the more specific you are, the better it will get.
Also add, that if you just want to retrieve data from this tables again and again, maybe a VIEW would fit nicely.
Those are just some tips that came to my mind to ease your problem.
Hope it helps.
I'm creating a search function for my website where the user can put in anything he likes in a textfield. It get's matched against anything (name, title, job, car brand, ... you name it)
I initially wrote the query with an INNER JOIN on every table that needed to be searched.
SELECT column1, column2, ... FROM person INNER JOIN person_car ON ... INNER JOIN car ...
This ended up in a query with 6 or 8 INNER JOINs, and a whole lot WHERE ... LIKE '%searchvalue%'
Now this query seems to cause a time'out in MySql, and I even got a warning from my hosting provider that the queries just taking up too many resources.
Now obviously I'm doing this very wrong, but I was wondering how the correct approach to these kind of search functions is.
Use multiple queries or UNION multiple queries so they go into a single resultset.
Additionally, using FULLTEXT indexes will most likely help to speed up your queries since (LIKE '%string%') - especially with the leading '%' - is extremely slow (all rows need to be checked without using indexes)
I recommend implementing Lucene or Sphynx search engines, they are much faster and scalable than sql queries.
I'm running a sql query to get basic details from a number of tables. Sorted by the last update date field. Its terribly tricky and I'm thinking if there is an alternate to using the UNION clause instead...I'm working in PHP MYSQL.
Actually I have a few tables containing news, articles, photos, events etc and need to collect all of them in one query to show a simple - whats newly added on the website kind of thing.
Maybe do it in PHP rather than MySQL - if you want the latest n items, then fetch the latest n of each of your news items, articles, photos and events, and sort in PHP (you'll need the last n of each obviously, and you'll then trim the dataset in PHP). This is probably easier than combining those with UNION given they're likely to have lots of data items which are different.
I'm not aware of an alternative to UNION that does what you want, and hopefully those fetches won't be too expensive. It would definitely be wise to profile this though.
If you use Join in your query you can select datas from differents tables who are related with foreign keys.
You can look of this from another angle: do you need absolutely updated information? (the moment someone enters new information it should appear)
If not, you can have a table holding the results of the query in the format you need (serving as cache), and update this table every 5 minutes or so. Then your query problem becomes trivial, as you can have the updates run as several updates in the background.
Does anyone have any idea on how can you create a product filtering query (or queries) that will emulate the results on this page?
http://www.emag.ro/notebook_laptop
Explanation
If you press HP as a brand, the page will show you all the HP products, and the rest of the available filters are gathered from this query result. Fine and dandy until now, I got this licked w/o any problems.
Press 4GB Ram, and ofcourse you will see all HP products that have this property/feature. Again fine and dandy, got no problems until here.
BUT if you look closely you will see that the Brand features now show also, let's say Acer, having a few products with the 4GB feature, and maybe more after Acer, and the checkbox isn't yet pressed.
The only ideea that comes to mind is to make that much more queries to the database to get these other possibilities results.
After you start checking the 3rd possible option (let's say Display size) the things start to complicate even more.
I guess my question is:
Does anyone has any idea on how to make this w/o taxing the server with tons of queries ?
Thank you very much for reading this far, I hope I made myself clear in all this little story.
Take a look at sql
UNION
syntax.
"UNION is used to combine the result from multiple SELECT statements into a single result set."
It's not really "tons" of queries, it's one query per attribute type (brand, RAM, HDD). Let's say you have selected HP, 4GB RAM and 250GB disk. Now for each attribute type select products according to the filter, except for the current type, and group by results by the current type. In a simplistic model, the queries could look like this:
SELECT brand, count(*) FROM products WHERE ram='4BG' AND disk='250GB' GROUP BY brand
SELECT ram, count(*) FROM products WHERE brand='HP' AND disk='250GB' GROUP BY ram
SELECT disk, count(*) FROM products WHERE brand='HP' AND ram='4BG' GROUP BY disk
SELECT cpu, count(*) FROM products WHERE brand='HP' AND ram='4BG' AND disk='250BG' GROUP BY cpu
...
You should have indexes on the columns, that every query doesn't do a sequential scan over the table. Of course there are some "popular" combinations and you will likely have to display the same numbers on multiple pages when the user is sorting/navigating the list, so you might want to cache the numbers and invalidate the cache on update/insert/delete.
It could be that there is some sophisticated means of determining some computed distance of a result from your criteria, but maybe it is as simple as using an OR in the query rather than an AND.