PHP find n-grams in an array - php

Question

I have a PHP array:
$excerpts = array(
'I love cheap red apples',
'Cheap red apples are what I love',
'Do you sell cheap red apples?',
'I want red apples',
'Give me my red apples',
'OK now where are my apples?'
);
I would like to find all the n-grams in these lines to get a result like this:
cheap red apples: 3
red apples: 5
apples: 6
I tried to implode the array and then parse it, but it's stupid because new n-grams can be found because of the concatenation of strings that have nothing to see between each other.
How would you proceed?

Answer 1

I want to find group of words without knowing them before although
with your function I need to provide them before anything
Try this:
mb_internal_encoding('UTF-8');
$joinedExcerpts = implode(".\n", $excerpts);
$sentences = preg_split('/[^\s|\pL]/umi', $joinedExcerpts, -1, PREG_SPLIT_NO_EMPTY);
$wordsSequencesCount = array();
foreach($sentences as $sentence) {
$words = array_map('mb_strtolower',
preg_split('/[^\pL+]/umi', $sentence, -1, PREG_SPLIT_NO_EMPTY));
foreach($words as $index => $word) {
$wordsSequence = '';
foreach(array_slice($words, $index) as $nextWord) {
$wordsSequence .= $wordsSequence ? (' ' . $nextWord) : $nextWord;
if( !isset($wordsSequencesCount[$wordsSequence]) ) {
$wordsSequencesCount[$wordsSequence] = 0;
}
++$wordsSequencesCount[$wordsSequence];
}
}
}
$ngramsCount = array_filter($wordsSequencesCount,
function($count) { return $count > 1; });
I'm assuming you only want repeated group of words.
The ouput of var_dump($ngramsCount); is:
array (size=11)
'i' => int 3
'i love' => int 2
'love' => int 2
'cheap' => int 3
'cheap red' => int 3
'cheap red apples' => int 3
'red' => int 5
'red apples' => int 5
'apples' => int 6
'are' => int 2
'my' => int 2
The code could be optimized to, for instance, use less memory.

Answer 2

The code provided by Pedro Amaral Couto above is very good.
Since I use it for French, I modified the regular expression as follows:
$sentences = preg_split('/[^\s|\pL-\'’]/umi', $joinedExcerpts, -1, PREG_SPLIT_NO_EMPTY);
This way, we can analyze the words containing hyphens and apostrophes ("est-ce que", "j'ai", etc.)

Answer 3

Try this (using the implode, since that's you've mentioned as an attempt):
$ngrams = array(
'cheap red apples',
'red apples',
'apples',
);
$joinedExcerpts = implode("\n", $excerpts);
$nGramsCount = array_fill_keys($ngrams, 0);
var_dump($ngrams, $joinedExcerpts);
foreach($ngrams as $ngram) {
$regex = '/(?:^|[^\pL])(' . preg_quote($ngram, '/') . ')(?:$|[^\pL])/umi';
$nGramsCount[$ngram] = preg_match_all($regex, $joinedExcerpts);
}

Answer 4

Assuming you just want to count the number of occurrences of a string:
$cheapRedAppleCount = 0;
$redAppleCount = 0;
$appleCount = 0;
for($i = 0; $i < count($excerpts); $i++)
{
$cheapRedAppleCount += preg_match_all('cheap red apples', $excerpts[$i]);
$redAppleCount += preg_match_all('red apples', $excerpts[$i]);
$appleCount += preg_match_all('apples', $excerpts[$i]);
}
preg_match_all returns the number of matches in a given string so you can just add the number of matches onto a counter.
preg_match_all for more information.
Apologies if I misunderstood.