I am creating a search engine for my php based website. I need to search a mysql table.
Thing is, the search engine must be pretty 'smart', so that users can easily find their items (it's a classifieds website).
I have currently set up a FULLTEXT search with this piece of code:
MATCH (headline) AGAINST ($querystring)
But this isn't enough...
For instance, lets say the field headline contains something like Bmw 330ci.
If I search for 330, I wont get any results. The ending ('ci') is just one of many endings in car models which must be taken into account when searching the table.
Or what if the headline field is bmw330? Also no results, because it only matches full words.
Or also, what if the headline is bmw 330, and I search for bmw 520, still with FULLTEXT I will get the bmw 330 as a result, even though I searched for bmw 520... Not good!
How should I solve this problem?
When it comes to fulltext search, people who want free solutions often tend to use either Sphinx or Solr.
I've not used any of those two, but I've read several times that they were great, and easy to use from/with PHP and MySQL.
Don't reinvent the wheel: inverted-index search engine are already there, free of charge, open source, easy and powerful. They have all what you need for such kind of search requirements.
Depending on your context, you can choose between a search library like Apache Lucene or a search platform like Apache Solr or Elastic Search.
All of them have a great documentation and they are widely used. That extremely minimizes the learning curve, even if you never worked with fulltext search world.
Related
I have mysql table containing about lots of names of songs.
I want to create smart search, so that i can find specific song from name.
it should return then smart search result like google does.
(not as perfect as google)
I need spell mistake avoidance and ranking according to relevance.
is there any easy way to do this?
Good evening,
I am facing a small problem whilst trying to build a little search algorithm.
I have a database table containing video game names and software names. Now I would like to add new offers by fetching and parsing xml files on other servers. The issue is:
How can I compare the strings for the product name so it works even if the offer name doesn't match the product name stored in my database up to a 100%?
As an example I am currently using this PHP + SQL code to compare the strings:
$query_GID = "select ID,game from gkn_catalog where game like '%$batch_name%' or meta like '%$batch_name%' ";
I am currently using the like operator in conjunction with two wild-cards to compare the offer name (batch_name) with the name in the database (game).
I would like to know how I can improve on this as this method isn't very failsafe or whatever you want to call it, what happens is:
If the database says the game title is:
Deus Ex Human Revolution Missing Link
and the batch_name says:
Deus Ex Human Revolution Missing Link DLC
the result will be empty/wrong/false ... well it won't find the game in my database at all.
Same goes for something like this:
Database = Lego Star Wars The Complete Saga batch_name = Lego
Star Wars : The Complete Saga
Result: False
Is there a better way to do the SQL query? Or how can I try to get that query working so it can deal with strings that come with special characters (like -minus- & [brackets]) and or characters which aren't included in the names within the database (like DLC, CE...)?
You're looking for fuzzy search algorithms and fuzzy search results. This is a whole field of study. However, there are also some straightforward tutorials to get you started if you take a quick google around.
You might be tempted to try something like PHP's wonderful levenshtein method, which calculates the "closeness" of two strings. However, this would require matching it against every record. If there will be thousands of records, that's out of the question.
MySQL has some matching tools which may help. I see that as I'm writing this, somebody has already mentioned FULLTEXT and MATCH() in the comments. Those are a great way to go.
There are a few other good solutions to look into as well. Storing an index of keywords (with all the articles and helpers like of/the/an/am/is/are/was/of/from removed) and then searching on each word in the search is a simple solution. However, it doesn't produce great results in that the returned values are not weighted well, and it doesn't localize at all.
There are lots of cheap and wonderful third party search tools (Lucene comes to mind) as well that will do most of this work for you. You just call an API and they manage the caching, keywords, indexing, fuzzying, et al for searches.
Here are some SO questions that are related to fuzzy searches, which will help you find more terminology and ideas:
Lightweight fuzzy search library
Fuzzy queries to database
Fuzzy matching on string
fuzzy searching an array in php
MySQL queries, as you found out can use the percent character as a joker (%) in conjunction with the LIKE operator.
You have multiple solutions depending on what you want exactly.
you can make a fulltext search
you can search using language algorithm like soundex
you can search by keywords
Remember that you can make a search in multiple passes (search for exact match, then percent on every side, explode in words then insert % between every word, search by keyword, etc.) depending if exact match has priority over close search, etc.
I am trying to optimize my search engine. Right now, I am running a strcmp between the search words the user entered and keywords stored in the database. I am trying to come up with a way so that the more matches the users search words has with the keywords the sooner it will show up in the search results.
For example, if the user search for "red apple painting" and I have two entries for that item with the following keywords 1. "old apple painting green" 2. "apple painting red new york" I would like the second entry to come up first in the search result because all of the users search words were found in the keywords stored in the db.
Any help on how I can achieve this?
Take a look at full text search.
You may also want to consider an external text search engine such as Lucene or Sphinx.
you need to create an index of words. The index would contain word id, doc id, number of hits, position of hits. Then the searcher will be able to give results like you want. There are free indexing tools available in market. But if you want to develop your own then follow the original paper bt google founders-
http://infolab.stanford.edu/~backrub/google.html
Find applicable keywords with accurate seek site visitors capability.
Create and optimize pages for engines like google and customers alike.
Make sure your internet site is offered to each bots and human beings.
Build applicable links from different notable web sites.
As the title says, I need a search engine... for mysql searching.
My website is PHP based.
I was going with sphinx but my hosting company doesn't support full-text indexes!
So a search engine to be used without full-text!
It should be pretty powerful, and must include atleast these functions below:
When searching for 'bmw 520' only matches where these two words come in exactly this order is returned. not matches for only 'bmw' or only '520'.
When searching for 'bmw 330ci' results as the above will be returned, but, WITH AND WITHOUT the ci extension. There are a nr of extensions in cars as you all know (i, ci, si, fi etc).
I want the 'minus sign' to 'exclude' all returns containing the word after the sign, ex: 'bmw -330' will return all 'bmw' results without the '330' ones. (a NOT instead of minus sign is also ok)
all special character accents like 'é' are converted to their simple values, in this case 'e'.
list of words to ignore completely in the search
Thanks guys!
The Zend_Lucene search competent works fairly well. I am not sure how it would cope with your second requirement, however if you customized the tokenized you should be able to do it by treating a change from letters to numbers as a new word.
The one I am really not sure about is the top requirement. Given how it is indexed, order becomes irreverent in the search, so you may not be able to do it without heavy editing of Lucene, writing a filter (using lucene to pull the matches, then checking the order), or writing your own solution. All of these will slow the search down, and add load to your server.
There is also solr, but I have never used it and don't know anything about it. Sphinx was another one, but I see you have already ruled that out.
Xapian is very good (very comprehensive) if you have the time for the initial setup.
It functions as you would expect a search engine to work, tell the indexer what bits of information to index under what namespace/table/object (Page, Profile, Products etc), then issue a query for your users based on keywords, it also supports google style tags e.g. "profile:Mark icecream" would search my profile for the word icecream, i seem to remember it supporting ranges too for data you specify as numeric.
Can be used in local mode which can offer spelling modifications (Did you mean?), or remote mode that many sites can index to and query from.
What really saved me one time was the ability to attach transient non searchable data to an indexed item, e.g. attaching the DB id to all data indexed for that record, very good for then going and getting the whole record from the DB when your matches come back from xapian.
I have used a couple of Search Engines on my site during it's time, but in the next rebuild I'm planning to move to Google Site Search.
There are several reasons for this:
Users are very familiar with the Google style of search result listings which improves usability and hence click-through rates
The Google engine is very good at guessing when to use the page description and when to use a fragment of the page (it also very good at getting relevant fragments compared to some other engines)
It's used by thousands of very popular websites
Google is the most popular search engine around so you know their technology is both reliable and accurate
Google Site Search begins at $100 per annum for 1000 pages or less (and a limit on queries)
or you can use the free Google Custom Search Engine (but this has much less customizability)
I have very simple company index with Zend Lucene using this to create the index:
// store company primary key to identify it in the search results
$doc->addField(Zend_Search_Lucene_Field::Keyword('pk', $this->getId()));
// index company fields
$doc->addField(Zend_Search_Lucene_Field::Unstored('zipcode', $this->getZipcode(), 'utf-8'));
$doc->addField(Zend_Search_Lucene_Field::Unstored('name', $this->getName(), 'utf-8'));
I can search on the company name but not the zipcode. Is there a problem with Zend Lucene Search indexing integers? If s/o could shed some light who was experience, please help me out. I can only imagine using Lucene to search by zipcode is pretty common.
I believe the default text Analyzer for Zend Lucene does not search numbers by default. Zend comes packaged with several different text analyzers. Use the TextNum analyzer to search both numbers and characters. There are also a handful of other analyzers in the zend/search/lucene/analysis/analyzer/common folder that you may find useful.
You can change your default analyzer with the following code:
Zend_Search_Lucene_Analysis_Analyzer::setDefault(
new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum());
I believe your problem is with the Analyzer.
I suggest you use Zend_Search_Lucene_Field::Keyword,
instead of Zend_Search_Lucene_Field::Unstored for the zip code field.
This way, the Lucene analyzer will not modify the zip code while indexing.
The Java Lucene has explain() which can be used to debug searches.
You may have to print some interim values to simulate explain(), and see whether this is indeed the problem.
If you are searching for 123, you will get all hits with 123 as well as 34123 for instance. So you have to make sure, that you're index and your query string is unambiguous.
I suggest to index the zipcode as a string such as "000123". After that you can search on the index with "000123" and you will get the correct resultset and nothing like 34123. you only have to translate the zipcode into the "correct" querystring.