Algorithm to reduce text based on word frequency - php

Question

How to reduce a text based on word frequency in PHP?
For example, if I have this text:
house house house house book book book
it should be reduced to something like this (or any similar form):
house house book
so this way the most used word is still house by 2 and book by 1.

Answer 1

The question is actually interesting. As I understand it, it is not about compression but word frequency - and this my friend, is the field of natural language processing.
My first thought was: Recommend using NLTK (and learning Python if required) since there is no real PHP equivalent to it (the closest library is probably NlpTools). However, it turned out Dan Cardin, an early NlpTools contributor, created a separate library that deals with your very problem: yooper/php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval
(IR) and Natural Language Processing (NLP) tasks using the PHP
language
Add PHP Text Analysis to your project
composer require yooper/php-text-analysis
Here is an example how to use it:
<?php
require_once('vendor/autoload.php');
$book = file_get_contents('pg74.txt'); // tom sawyer from the gutenberg project http://www.gutenberg.org/cache/epub/74/pg74.txt
// Create a tokenizer object to parse the book into a set of tokens
$tokenizer = new \TextAnalysis\Tokenizers\GeneralTokenizer();
$tokens = $tokenizer->tokenize($book);
$freqDist = new \TextAnalysis\Analysis\FreqDist($tokens);
//Get the top 10 most used words in Tom Sawyer
$top10 = array_splice($freqDist->getKeyValuesByFrequency(), 0, 10);
The call to freq_dist returns a FreqDist instance.
Then, you can then calculate the weights of words yourself (freq/numberOfAllTokens) or use the getKeyValuesByWeight() method.
$top10[0]/$freqDist->getTotalTokens();
$weights = $freqDist->getKeyValuesByWeight();
... or normalize the frequency of your selected words by the occurrence of your least frequent top word, e.g.
foreach ($top10 as $word => $freq) {
$relWeight[$word] = $freq/end($top10);
}
Depending on your input, you will find that your most frequent words are a, the, that, etc. This is why you want to remove stopwords. And we have only started..
Here are some more samples.

Answer 2

Compress & Uncompress a string in PHP: gzcompress,gzuncompress
Example:
$text = "house house house house book book book";
echo "Orignal text lenght : ". strlen($text)."<br>";
$compressed = gzcompress($text, 9);
echo "Compressed text: ".$compressed."<br>";
echo "Compress text length :". strlen($compressed);
echo "<br>";
echo "Uncompressed text :".$uncompressed = gzuncompress($compressed);
Output:
Orignal text length: 38
Compressed text: x���/-NU�� ����R
Compress text length: 22
Uncompressed text : house house house house book book book