I have an array with a list of keywords,
$arr = array('london','england','football',...);
The array could have Nth keywords.
I need to use this on a sql query but i dont want to do too many repetitions like this.
select * from t1 where title LIKE('%$arr[0]%') OR title LIKE('%$arr[1]%') [...]
just need a better solution as each keyword filters out the records.
thnx
One solution would be to create a lookup table that is populated with the rowid and keywordid whenever a row is created or changed.
Then you can skip all the LIKE and get much more performance.
A trigger on insert and update should work but it will cost a little more performance on insert and update.
You would just use
SELECT * FROM t1 WHERE id in (SELECT id FROM lookup WHERE keywordid IN ())
And if the keywords are many you could use a sub query more to get the keyword id's or if fewer use a cached lookup structure directly in code, dictionary or likewise.
http://www.webdevelopersnotes.com/tutorials/sql/tutorial_mysql_in_and_between.php3
I'm unsure whether you wanted to optimize the actual query itself or the means to insert keywords into the query.
If it's the latter, then what comes to mind quickly is:
$query = "SELECT * FROM t1 WHERE title";
foreach($arr as $keyword)
{
$stack[] = " LIKE '%$keyword%' ";
}
$query .= implode(' OR title ', $stack);
If you want to avoid full table scans based on keywords, that requires a different topic in itself with more explanation at your end.
Depending on the kind (and amount) of searches you'll be doing, you might need to consider using something else than pure-MySQL for that.
MySQL by itself is not quite well-suited when it comes to fulltext search :
Using like '%...%' will result in bad performances (scanning all the lines of your table)
Using a FULLTEXT index means using MyISAM as storage engine.
If you need to do a lot of searches, it might become more interesting to invest in a solution dedicated to indexing/searching through text, like Solr, or Sphinx.
Related
I am trying to search a table for multiple keywords. However, I am not looking for one keyword or even 10. It is around one thousand keywords. These keywords are also in a table and can be controlled. I would rather not hard-code these keywords into my SQL command...
The target table I am searching contains a lot of text and a cell could contain an entire sentences or paragraph... so doing something like a 'full text' search in mySQL seems like a good start.
Very similar to this question, but again, when I speak of multiple keywords, I mean hundreds to thousands. mysql FULLTEXT search multiple words
Can I dump my keyword table into an array and run a FULLTEXT search? Can this even be approached with mySQL or are there limits im not considering? Im open to other technology suggestions too. Sorry that I dont have code or errors to post. I am first trying to understand conceptually how to approach this. -tia
Recently i had to do a similar decision. I decided to go with lucene. I store the indexable fields in lucene, and return an id for mysql row.
Other choice is sphinx , a full tutorial can be found here.
See a related post here. And here.
SELECT * FROM articles WHERE body LIKE '%$keyword%';
you just need to use a for loop in your mysql query to read all keyworks.
its clear that you separated your keywords with comma or -
so you need to explode keyword by comma and put them in a variable .
for example :
$keywords = "key1,key2,key3,..."; // values come from keywords column from db.
now you just need to explode $keywords.
$keys=explode(',',$keywords);
and finally in your query you need to use a for loop:
$query = "SELECT * From targettable";
$i = 0;
foreach ($keys as $key)
{
$query .= "WHERE keywords LIKE '%".$key."%' )";
$i++;
}
i named your keywords column = "keywords".
also you can easily add your other condition into $query.
Watch out with mysql fulltext searches, if your result set is > 50% of the entire data set, mysql sees this is a failed search, and you wont get anything.
This sounds likely to occur if you have a hit list of 1000s of possible words.
I'd say you would be better off investigating the extraction of keywords from text (lets imagine they are articles) as they are stored.
Then store these keywords in their own table.
For best results though, you'd likely want to investigate Natural Language Processing and extract meaning from articles rather than just words.
This is the query I'm using to perform a forum search.
$sql = "SELECT t.title, t.user_id, t.replies, t.views, t.last_poster_name, t.last_post_time, a.username
FROM fm_topics t
LEFT JOIN fm_replies r
ON t.t_id=r.topic_id
LEFT JOIN account a
ON t.user_id=a.id
WHERE
`title` LIKE '%" . sanitize($search_string) . "%' OR
`content` LIKE '%" . sanitize($search_string) . "%'
GROUP BY t_id LIMIT 0, 10";
The above works fine. However, it takes like 5+ seconds to load. Now, if I take out the part where it searches for the title too, it loads way faster (obviously). My question is, how can I improve this query? how would you rewrite it for better performance?
You can use MySql functions like MATCH and AGAINST. By default, the MATCH() function performs a natural language search for a string against a text collection. A collection is a set of one or more columns included in a FULLTEXT index. The search string is given as the argument to AGAINST(). For each row in the table, MATCH() returns a relevance value; that is, a similarity measure between the search string and the text in that row in the columns named in the MATCH() list.
Natural Language Full-Text Searches
The query needs no optimization instead the table might if indexing is not done on that table.
What is an index in SQL Server?
And for your scenario, i.e. text searching using wildcard, Full text indexing is recommended.
When should you use full-text indexing?
Why not make a full-text search and index it? That will make searches much faster. By using a LIKE query, especially one that is prefix wild-carded it means every row has to be sequentially scanned.
https://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html
I would make two-column full-text index on title,content:
CREATE FULLTEXT INDEX index_content ON (title,content)
Than search on it like:
SELECT ... WHERE MATCH(title, content) AGAINST ("your query")
Optionally using IN BOOLEAN MODE - read the docs for reasons on why you would and would not use this modifier.
If you really cannot implement a full-text search+index and need to keep the LIKE then you can consider just adding a regular index on content, but then to take advantage of that index you will have to remove the leading wild-card (you can keep the trailing one); MySQL cannot use an index on a leading-wildcard query.
I have a PHP interface with a keyword search, working off a DB(MySQL) which has a Keywords field.
The way in which the keywords field is set up is as follows, it is a varchar with all the words formatted as shown below...
the, there, theyre, their, thermal etc...
if i want to just return the exact word 'the' from the search how would this be achieved?
I have tried using 'the%' and '%the' in the PHP and it fails to work by not returning all of the rows where the keyword appears in.
is there a better (more accurate) way to go about this?
Thanks
If you want to select the rows that have exactly the keyword the:
SELECT * FROM table WHERE keyword='the'
If you want to select the rows that have the keyword the anywhere in them:
SELECT * FROM table WHERE keyword LIKE '%the%'
If you want to select the rows that start with the keyword the:
SELECT * FROM table WHERE keyword LIKE 'the%'
If you want to select the rows that end with the keyword the:
SELECT * FROM table WHERE keyword LIKE '%the'
Try this
SELECT * FROM tablename
WHERE fieldname REGEXP '[[:<:]]test[[:>:]]'
[[:<:]] and [[:>:]] are markers for word boundaries.
MySQL Regular Expressions
if you also search for the commas, you can be sure you are getting the whole word.
where keywordField like '%, the, %'
or keywordField like '%, the'
or keywordField like 'the, %'
maybe I didn't understand the question properly... but If you want all the words where 'the' appears, a LIKE '%word%' should work.
If the DB of words is HUGE MySQL may fail to retrieve some of the words, that can be solved in 2 ways...
1- get a DB that support bigger sizes (not many ppl would chose this one tho). For example SQL Server has a 'CONTAINS' function that works better than LIKE '%word%'.
2- use a external search tool that uses inverted index search. I used Sphinx for a project and it works quite good. This is better if you rarely UPDATE the rows of the data you want to search from, which should be the case.
Sphinx for example would generate a file from your MySQL table and use this file to solve the search (it's very fast), this file should be re-indexed everytime you do a insert or update on the table, making it a much better solution if you rarely update or insert new rows.
It looks like you have a one to many relationship going on within a column. It might be better to create a separate table for keywords with a row for each keyword and a foreign key to whatever it is you're searching on.
Doing like '%???%' is generally a bad idea because the DB can't make use of an index so it will scan the whole table. Whether this matters will depend on the size of data you're working with but its worth considering up front. The single best way to help DB performance is in the initial table design. This can be tricky to change later.
Taking a PHP array of terms with variable length (i.e. it could be 50 terms, it could be 400), what's the most efficient way of searching my database for each of these terms?
The search I'm trying to do is quite straightforward. For each term, I'd like to do:
SELECT id, post_title FROM wp_posts WHERE post_title LIKE %term%
Obviously I can run a foreach in PHP and run multiple MySQL queries, but I'd imagine this to be hugely inefficient.
The code I've most recently tried involves multiple OR statements, but with ~100ish terms it appears to run very slowly.
I have no idea if something like this would work?
SELECT id, post_title FROM wp_posts WHERE post_title LIKE %term1%, %term2%, %term3%, %term4%, [...]
Can I use a more efficient SQL statement, or should I be looking at this in a different way?
Stock MySQL could handle this kind of search using
MATCH (post_title) AGAINST ('term1 term2 term3 term4')
To do this search you will need to add Full Text index into the table using
ALTER TABLE wp_posts ADD FULLTEXT INDEX ft_key1(post_title);
This would be way faster than LIKE %term%, but please note that Full-Text indexes are only supported in MyISAM tables (InnoDB supported this syntax since MySQL 5.6).
However as your data grow bundled MySQL search speed might become an issue. In this case I would suggest to use external search engine like Solr or Sphinx.
If you decided to switch to Sphinx you may want to take a look on this guide http://astellar.com/2011/12/replacing-mysql-full-text-search-with-sphinx/
Create an additional index holding a single column where you simply concatenate the values of all table columns you want to query during the search. This way you can use a single SELECT query with a LIKE clause to search through all columns at once.
This is often referred to as "full text search".
Using MySQL the most efficient way is to set up Full-text searching... http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html
I don't know whether its any more efficient than the other suggestions but you could also do
SELECT id, post_title FROM wp_posts WHERE FIND_IN_SET(post_title, '%term1%, %term2%, %term3%, %term4%') <> 0
I have a simple search feature that will search a table for a hit. Basically the one flaw of this search is that it tries to match exactly the string. I want it to take a string of say "large red bird" and search "large", "red" and "bird" separately against the table. here's my search query...
$result = mysql_query("SELECT * FROM files
WHERE (tags LIKE '%$search_ar%' OR
name LIKE '%$search_ar%' OR
company LIKE '%$search_ar%' OR
brand LIKE '%$search_ar%')
$str_thera $str_global $str_branded $str_medium $str_files");
any Ideas? thanks
Edit
OK here's my updated query but it doesnt return anything.
$result = mysql_query("SELECT * FROM files
WHERE MATCH(tags, name, company, brand)
AGAINST ('$seach_ar' IN boolean MODE)");
A fulltext search will indeed help, given you do have a lot of data to search by. A good reference for the full text can be found here: http://forge.mysql.com/w/images/c/c5/Fulltext.pdf
The reason I say a lot of data, or a fair amount, is that if the search results yields above a certain percentage of returned rows to total rows, nothing is returned.
If you want to continue using the LIKE method, you can. You just have to seperate the words (explode) and the join them in the sql query using AND:
...(tags LIKE '%$search_ar[0]%' AND tags LIKE '%$search_ar[1]%') OR ....
In a fashion like that. This method can get overly complicated especially if say, you want to return matches which has any of the words and not all of them. So yea, it will take some customization to do and to automate, but it is possible.
If, for whatever reason, you can't use full text searching (nice idea, Nick and Brad, but only MyISAM supports it, and from what I hear, it's not really all that good), it's not too hard to rig up basic searching with explode() and implode():
$search_arr = explode(' ',$search_str);
$search_str = implode('%',$search_arr);
//query where fields like $search_arr
Or:
$search_arr = explode(' ', $search_str);
$sql = 'select * from files where ';
foreach($search_arr = $term){
$sql .= "tags like '%{$term}%' or " //fill in rest of fields
}
As a side note: If you plan on growing your searching and can't use MySQL's full text search, I recommend checking out one of the search servers, such as Solr or Lucene.