Word Occurrences count in PHP - scaling and optimization - php

Question

You have a large collection of very short (<100 characters) to medium-long (~5k characters) text items in a Mysql table. You want a solid and fast algorithm to obtain an aggregated word occurrency count in all the items.
Items selected can be as few as one (or none) and as many as 1M+, so the size of the text that has to be analyzed can vary wildly.
The current solution involves:
reading text from the selected records into a text variable
preg_replacing out everything that is not a "word" getting a somewhat cleaned up, shorter text (hashtags, #mentions, http(s):// links, numeric sequences such as phone numbers, etc. are not "words" and are parsed out)
exploding what's left into a "buffer" array
taking out everything that's shorter than two words and dumping everything else in a "master" array
the resulting array is then transformed with array_count_values, sorted (arsort), spliced a first time to have a more manageable array size, then parsed against stopwords lists in several languages, processed and spliced some more and finally output in JSON form as the list of the 50 most frequent words.
By tracing timing on the various steps of the operation sequence, the apparent bottleneck is in the query at first but as the item count increases, it rapidly moves to the array_count_values function (everything after that is more or less immediate).
On a ~ 10k items test run the total time for execution is ~3s from beginning to end; with 20k items it takes ~7s.
A (rather extreme but not impossible) case with 1.3M items takes 1m for the mysql query, and is able to parse roughly ~75k items per minute (so 17 minutes-ish is the estimate).
The result should be visualized as a result of an AJAX call, so with this kind of timing the UX is evidently disrupted. I'm looking for ways to optimize everything as much as possible. A 30s load time may be acceptable (however little realistic), 10 minutes (or more) is not.
I've tried batch processing by array_count_values-ing the chunks and then adding the resulting count arrays to a master array by key, but that helps only so much - the sum of the parts is equal (or slightly larger) than the total in terms of timing.
I only need the 50 most frequent occurrences at the top of the list, so there's possibly some room for improvement by cutting a few corners.

Answer 1

dump the column(s) into a text file. Suggest SELECT ... INTO OUTFILE 'x.txt'...
(if on linux etc): tr -s '[:blank:]' '\n' <x.txt | sort | uniq -c
To get the top 50:
tr -s '[:blank:]' '\n' <x.txt | sort | uniq -c | sort -nbr | head -50
If you need to tweak the definition of "word", such as dealing with punctuation, see the documentation on tr. You may have trouble with contractions versus phrases in single-quotes.
Some of the cleansing of the text can (and should) be done in SQL:
WHERE col NOT LIKE 'http%'
Some can (and should) be done in the shell script.
For example, this will get rid of #variables and phone numbers and 0-, 1-, and 2-character "words":
egrep -v '^#|^([-0-9]+$|$|.$|..)$'
That can be in the pipe stream just before the first sort.
The only limitation is disk space. Each step in the script is quite fast.
Test case:
The table: 176K rows of mostly text (including some linefeeds)
$ wc x3.txt
3,428,398 lines 31,925,449 'words' 225,339,960 bytes
$ time tr -s '[:blank:]' '\n' <x.txt | egrep -v '^#|^([-0-9]+$|$|.$|..)$' |
sort | uniq -c | sort -nbr | head -50
658569 the
306135 and
194778 live
175529 rel="nofollow"
161684 you
156377 for
126378 that
121560 this
119729 with
...
real 2m16.926s
user 2m23.888s
sys 0m1.380s
Fast enough?
I was watching it via top. It seems that the slowest part is the first sort. The SELECT took under 2 seconds (however, the table was probably cached in RAM).