Keywords erroneous, extracting content from a website. OOP - php

Question

I have a problem when extracting the keywords from a website (wiki article), the keywords that are extracted, they are not exactly keywords, are words taken from the html, and not of the web site.
I use the following code:
include("Extkeys.php");
[...]
if (empty($keywords)){
$ekeywords = new KeyPer;
$keywords = $ekeywords->Keys($webhtml);
}
And the code of "Extkeys" is:
<?php
class Extkeys {
function Keys($webhtml) {
$webhtml = $this->clean($webhtml);
$blacklist='de,la,los,las,el,ella,nosotros,yo,tu,el,te,mi,del,ellos';
$sticklist='test';
$minlength = 3;
$count = 17;
$webhtml = preg_replace('/[\.;:|\'|\"|\`|\,|\(|\)|\-]/', ' ', $webhtml);
$webhtml = preg_replace('/¡/', '', $webhtml);
$webhtml = preg_replace('/¿/', '', $webhtml);
$keysArray = explode(" ", $webhtml);
$keysArray = array_count_values(array_map('strtolower', $keysArray));
$blackArray = explode(",", $blacklist);
foreach($blackArray as $blackWord){
if(isset($keysArray[trim($blackWord)]))
unset($keysArray[trim($blackWord)]);
}
arsort($keysArray);
$i = 1;
$keywords = "";
foreach($keysArray as $word => $instances){
if($i > $count) break;
if(strlen(trim($word)) >= $minlength && is_string($word)) {
$keywords .= $word . ", ";
$i++;
}
}
$keywords = rtrim($keywords, ", ");
return $keywords=$sticklist.''.$keywords;
}
function clean($webhtml) {
$regex = '/(([_A-Za-z0-9-]+)(\\.[_A-Za-z0-9-]+)*#([A-Za-z0-9-]+)(\\.[A-Za-z0-9-]+)*)/iex';
$desc = preg_replace($regex, '', $webhtml);
$webhtml = preg_replace( "''si", '', $webhtml );
$webhtml = preg_replace( '/]*>([^<]+)<\/a>/is', '\2 (\1)', $webhtml );
$webhtml = preg_replace( '//', '', $webhtml );
$webhtml = preg_replace( '/{.+?}/', '', $webhtml );
$webhtml = preg_replace( '/ /', ' ', $webhtml );
$webhtml = preg_replace( '/&/', ' ', $webhtml );
$webhtml = preg_replace( '/"/', ' ', $webhtml );
$webhtml = strip_tags( $webhtml );
$webhtml = htmlspecialchars($webhtml);
$webhtml = str_replace(array("\r\n", "\r", "\n", "\t"), " ", $webhtml);
while (strchr($webhtml," ")) {
$webhtml = str_replace(" ", "",$webhtml);
}
for ($cnt = 1;
$cnt < strlen($webhtml)-1; $cnt++) {
if (($webhtml{$cnt} == '.') || ($webhtml{$cnt} == ',')) {
if ($webhtml{$cnt+1} != ' ') {
$webhtml = substr_replace($webhtml, ' ', $cnt + 1, 0);
}
}
}
return $webhtml;
}
}
?>
This is an example of the keywords extracted:
testfalse, lang, {mw, loader, window, function, true, vector, user, gadget, mediawiki, legacy, options, usebetatoolbar, implement, resourceloader, default
Of the article:
http://en.wikipedia.org/wiki/Searchengine
The code "Extkeys", its a copy of a code from a tutorial, adapted for me to make it functional.
How i can make the code can extract the keywords of a website, and not of a html?
Best regards!

Answer 1

Assuming I understand your question, I think simply doing the following is the solution you're looking for.
This will read the HTML from a URL (e.g. http://www.whatever.com/page.html) and use that to generate the keys, rather than requiring the HTML as a parameter.
function Keys($url) {
$webhtml = file_get_contents($url);

Answer 2

You want to extract the content from the page first and then search for keywords. Meaning you want to find the actual content from the page and strip stuff as sidebars, footers etc.
Just google for HTML content extraction, there are numberous articles about this.
I did this once in java, there a library called boilerpipe i'm not sure if there's a PHP port/interface a quick google search didn't reveal anything. But i'm sure there are similar libraries for PHP.
The easiest way to just get rid of the HTML and not specifically search only the page content would be using a regex to strip all html, something like s/<[^>]+>//g. However for a search engine that's probably not the best approach since you end up with a lot of crap that could mess up your key extraction.
EDIT: Here is an article on content extraction with PHP.