How search for thousands of possible keywords in a string - php

Question

I have a database of thousands (about 10,000) keywords. When a user posts a blog on my site, I would like to automatically search for the keywords in the text, and tag the post with any direct matches.
So far, all I can think of is to pull the ENTIRE list of keywords, loop through it, and check for the presence of each tag in the post...which seems very inefficient (that's 10,000 loops).
Is there a more common way to do this? Should I maybe use a MySQL query to limit it down?
I imagine this is not a totally rare task.

Answer 1

No, just don't do that.
Instead of looping through 10000 elements, it is better to extract the words from the sentence or text, then add it to the SQL query and that way you will have all the needed records. This is surely more efficient than the solution you proposed.
You can do this in the following way using PHP:
$possible_keywords = preg_split('/\b/', $your_text, PREG_SPLIT_NO_EMPTY);
The above will split the text on the words' boundaries and will return no empty elements in the array.
Then you just can create the SQL query in a fashion similar to the following:
SELECT * FROM `keywords` WHERE `keywords`.`keyword` IN (...)
(just put the comma-separated list of extracted words in the bracket)
You should probably filter the $possible_keywords array before making the query (to include only the keywords with appropriate length and to exclude duplicates) plus make keyword column indexed.

Answer 2

I don't know what language you intend on using, but a standard trie (prefix tree) would solve this issue, if you were feeling up to it.

Answer 3

I guess you could build a regular expression dynamically which will enable you to match keywords inside a specific string. You can package all this in a class which does the grunt work.
class KeywordTagger {
static function getTags($body) {
if(preg_match_all(self::getRegex(), $body, $keywords)) {
return $keywords[0];
} else {
return null;
}
}
private static $regex;
private static function getRegex() {
if(self::$regex === null) {
// Load Keywords from DB here
$keywords = KeywordsTable::getAllKeywords();
// Let's escape
$keywords = array_map('KeywordTagger::pregQuoteWords', $keywords);
// Base Regex
$regex = '/\b(?:%s)\b/ui';
// Build Final
self::$regex = sprintf($regex, implode('|', $keywords));
}
return self::$regex;
}
private static function pregQuoteWords($word) {
return preg_quote($word, '/');
}
}
Then, all you have to do is, when a user writes a post, run it through the class:
$tags = KeywordTagger::getTags($_POST['messageBody']);
For a small speed up, you could cache the built regex using memcached, APC or a good-old file-based cache.

Answer 4

Well, I think that PHP's stripos is already quite optimized. If you want to optimize this search further, you would have to take advantage of similarities between your keywords (e.g. instead of looking for "foobar" and then for "foobaz", look for "fooba" and then check for each "fooba" if it's followed by a 'r', a 'z', or none). But this would require some sort of tree-representation of your keywords, like:
root (empty string)
|
fooba
/ \
foobar foobaz
Yes, that's a trie.