Calculating the distance between two articles accurately - php

Question

I am writing a software to compare articles. I am looking for an efficient and accurate algorithm to calculate the difference (variation) between two articles. The variation should completely depend on words and not letters. I have tried levenshtein() but it has a time complexity of O(n*m) which is very expensive when performed on big texts like an article. I have also tried similar_text() which has a higher time complexity of O(n*m*3). Moreover, levenshtein() and similar_text() calculates the number of operations needed to transform one string to another which is not an accurate way to calculate the difference between two big articles.
What other options do I have?
EDIT:
I am trying to calculate the variation approximately from the point of view of a search engine (Google).

Answer 1

PostgreSQL uses a tsvector for it's full-text search feature. Maybe that's something that could get quite handy for you.

Answer 2

If you can define how to measure text similarity based on words, you are half way through. For example: You may count the occurence of each word for both article and then create a simple difference of the two lists. However, this does not work for similarity by meaning.
If you have a database, use their fulltext features. As mentioned before, PostGres offers such a feature. I work with MSSQL and you could simply call the FREETEXT function which will calculate a 'rank' indicating how similar two texts are.
I highly recommend using a mature product, instead of trying to write your own.

Answer 3

There is no way to compare two articles. levenshtein() and similar_text() designed to compare two words, not articles.
The simplest algorythm is to explode your articles by words, find word-by-word similaryty and do some math, depending on your task, like this:
// not tested!
function similar_articles($articleA, $articleB) {
$wordsA = array_unique(preg_split('#[\W]+#', $articleA));
$wordsB = array_unique(preg_split('#[\W]+#', $articleA));
$resultSimilarity = 0;
foreach($wordsA as $wordA) {
$wordSimilarity = 0;
foreach($wordsB as $wordB) {
similar_text($wordA, $wordB, $percent);
$wordSimilarity = max($wordSimilarity, $percent);
}
$resultSimilarity += $wordSimilarity;
}
return($resultSimilarity / count($wordsA));
}
Note: similar_articles($artileA, $articleB) != similar_articles($artileB, $articleA) because of similar_text($wordA, $wordB) != similar_text($wordB, $wordA).

Answer 4

A simple method for calculate a type of distance is to compare references. Another method is to select some key word in concordance to a dictionary and calculate the distance in order of social relevance.
Also, in order to use Levenshtein distance take a look on stringmetric.

Answer 5

In my case, I needed to calculate the variation between two articles. So, I found that very simple solution working for me very well. It works by simply calculating the similarity as the common words between the two articles divided by max(number of words in article A, number of words in article B). The variation then is calculated by subtracting the similarity from 100 to get the variation percentage. The code below explains it all.
function get_variation($article1,$article2){
$wordsA = array_unique(preg_split('#[\W]+#', $article1));
$wordsB = array_unique(preg_split('#[\W]+#', $article2));
$intersection = array_intersect($wordsA, $wordsB);
$similarity = (count($intersection)/ (max(count($wordsA),count($wordsB))) * 100);
$similarity = number_format($similarity, 2, '.', '');
$variation = 100-$similarity;
return $variation;
}