I am struggling with the decision to separate phone numbers stored in a MySQL database.
One school of thought is to break out the phone as:
area code (123)
prefix (123)
suffix (1234)
Another is to simply place the file in a single field with whatever formatting deemed appropriate:
123456789
(123) 123-4567
123-456-7890
My initial reason for thinking the first would be better is in terms of being able to quickly and easily gather statistical data based on the phone numbers collected from our members (X number of members have a 123 area code for example).
Is there really a 'right' way to do it? I do realize that paired with PHP I can retrieve and reformat any way I want but I'd like to use best practice.
Thanks for your advice
EDIT
I will only be storing North American phone numbers for the time being
I vote for one field, processing the data as you put it in so that it's in a known format. I've tried both ways, and the one-field approach seems to generate less code overall.
You want to store it in the most efficient way in the DB, precisely because it's so easy to reformat in PHP. Go for the all-numeric field, with no separators (1231231234) since that would be the best way. If you have international phone numbers, add the country code as well. Then in your code you can format it using regular expressions to look however you want it.
I would store phone numbers as strings, not numbers.
Phone numbers are identifiers that happen to use digits.
Phone numbers starting with zero are valid, but may be interpreted as octal by a programming language.
Strip the phone number to only digits and store the extension in a separate field.
This will allow for uniform formatting later.
For US, strip the prepending ’1′ digit (and determine formatting based on length of the string (10 digits for US)).
I'm in the process of building a callcenter application (it manages queues of contact information for a group of distributed callers to contact) and the architecture specified one field, no spaces, dashes, etc. After quite a bit of analysis, I agree it seems the best.
Based on the variability of entry for phone numbers (apostrophes, dots, dashes, and combinations of each) I built a simple function that deals with user entry, stripping down all but the numbers themselves, and also a "rebuilder" that reformats the raw number into something that's more visually appealing to the user.
Since they've been helpful to me, here's what I've written so far:
public static function cleanPhoneNumbers($input) {
return preg_replace("/[^0-9]/", "", $input);
}
public static function formatPhoneNumbers($phone_number) {
if(strlen($phone_number) == 7) {
return preg_replace("/([0-9]{3})([0-9]{4})/", "$1-$2", $phone_number);
} elseif(strlen($phone_number) == 10) {
return preg_replace("/([0-9]{3})([0-9]{3})([0-9]{4})/", "$1-$2-$3", $phone_number);
} else {
return $phone_number;
}
}
Some caveats: My app is not available for international customers right now (there's a voip application built into it that we don't want to allow to call outside of the US right now) so I've not taken the time to setup for international possibilities. Also, as this is in progress, I will likely return to refactor and bolster these functions later.
I've found one weakness so far that has been a bit of a pain for me. In my app, I have to disallow calls to be made by timezone based on the time of day (for instance, don't allow someone on the West Coast to be called at 6:00am when it's 9:00am in the East) To do that, I have to join a separate area code table to my table with the phone numbers by comparing 3 digit area codes to get the timezone. But I can't simply compare the zip code to my phone number field, because they'd never match. So, I have to deal with additional SQL to get just the first three digits of the number. Not a game-changer, but more work and confusion nonetheless.
Definitely store them in one field as a text string, and only store the numbers. Think of it this way; no matter what the numbers are, its all one telephone number. However, the segmenting of the numbers is dependent on a number of things (locality, how many numbers provided, even personal preference). Easier to store the one and change it later with text manipulation.
I think splitting the number in 3 fields is the best options if you want to use area codes as filters, otherwise, you should only use 1 field.
Remember to use ZEROFILL is you plan on storing them as numbers ;)
it really depends on a couple factors:
is it possible you will have international numbers?
how much area code/city code searching/manipulation will you be doing?
No matter what, I would only store numbers, it's easy enough to format either in MySQL or PHP and add parentheses and dashes.
Unless I was going to do a log of searching by area code, I would just put the entire phone number into a single field since I assume most of the time you would be retrieving the entire phone number anyway.
If it's possible that you will take international numbers in the future:
You might want to add a country field though, that way you won't have to guess what country they are from when dealing with the number.
What you use depends on how you plan to use the data, and where the program will be used.
If you want to efficiently search records by area code, then split out the area code; queries will perform much faster when they're doing simple string comparisons versus string manipulation of the full phone number to get the area code.
HOWEVER, be advised that phone numbers formatted XXX-XXX-XXXX are only found in the US, Canada, and other smaller Caribbean territories that are subject to the NANPA system. Various other world regions (EU, Africa, ASEAN) have very different numbering standards. In such cases, splitting out the equivalent of the "area code" may not make sense. Also, if all you want to do is display a phone number to the user, then just store it as a string.
Whether to store a number with a format or not is mostly personal preference. Storing the raw number allows the formatting to be changed easily; you could go from XXX-XXX-XXXX to (XXX) XXX-XXXX by changing a couple lines of code instead of reformatting the 10 million numbers you already have. Removing special characters from a phone number is also a relatively simple Regex. Storing without formatting will also save you a few bytes per number and allow you to use a fixed-length field (saving further data overhead inherent in varchars). This may be of use in a mobile app where storage is at a premium. However, that 5-terabyte distributed SQL cluster in your server room is probably not gonna notice much difference between a char(10) and a varchar(15). Storing them formatted also speeds up loading the data; you don't have to format it first, just yank it out of the DB and plaster it on the page.
Related
Background: I have a large database of people, and I want to look for duplicates, which is more difficult than it seems. I already do a lot of comparison between the names (which are often spelled in different ways), dates of birth and so on. When two profiles appear to be similar enough to the matching algorithm, they are presented to an operator who will judge.
Most profiles have more than one phone number attached, so I would like to use them to find duplicates. They can be entered as "001-555-123456", but also as "555-123456", "555-123456-7-8", "555-123456 call me in the evening" or anything you might imagine.
My first idea is to strip all non-numeric characters and get the "longest common substring".
There are a lot of algorithms around to find the longest common substring inside a set.
But whenever I compare two profiles A and B, I have two sets of phone numbers. I would like to find the longest common substring between a string in the set A and a string in a set B.
Can you please help me in finding such an algorithm?
I normally program in PHP, a SQL-only solution would be even better, but any other language would go.
As Voitcus said before, you have to clean your data first before you start comparing or looking for duplicates. A phone number should follow a strict pattern. For the numbers which do not match the pattern try to adjust them to it. Then you have the ability to look for duplicates.
Morevover you should do data-cleaning before persisting it, maybe in a seperate column. You then dont have to care for that when looking for duplicates ... just to avoid performance peaks.
Algorithms like levenshtein or similar_text() in php, doesnt fit to that use-case quite well.
In my opinion the best way is to strip all non-numeric characters from the texts containing phone numbers. You can do this in many ways, some regular expression would be the best, but see below.
Then, if it is possible, you can find the country direction code, if the user has its location country. If there is none, assume default and add to the string. The same would be probably with the cities. You can try to take a look also in place one lives, their zip code etc.
At the end of this you should have uniform phone numbers which can be easily compared.
The other way is to compare strings with the country (and city) code removed.
About searching "the longest common substring": The numbers thus filtered are the same, however you might need it eg. if someone typed "call me after 6 p.m.". If you're sure that the phone number is always at the beginning, so nobody typed something like 555-SUPERMAN which translates to 555-78737626, there is also possibility to remove everything after the last alphanumeric character (and this character, as well).
There is also a possibility to filter such data in the SQL statement. Consider something like a SELECT ..., [your trimming function(phone_number)] AS trimmed_phone WHERE (trimmed_phone is not numerical characters only) GROUP BY trimmed_phone. If trimming function would remove only whitespaces and special dividers like -, +, . (commonly in use in Germany), , perhaps etc., this query would leave you all phone numbers that are trimmed but contain characters not numeric -- take a look at the results, probably mostly digits and letters. How many of them are they? Maybe they have something common? Maybe some typical phrases you can filter out too?
If the result from such query would not be very much, maybe it's easier just to do it by hand?
This is something I'm working on and I'd like input from the intelligent people here on StackOverflow.
What I'm attempting is a function to repair text based on combining various bad versions of the same text page. Basically this can be used to combine different OCR results into one with greater accuracy than any of them individually.
I start with a dictionary of 600,000 English words, that's pretty much everything including legal and medical terms and common names. I have this already.
Then I have 4 versions of the text sample.
Something like this:
$text[0] = 'Fir5t text sample is thisline';
$text[1] = 'Fir5t text Smplee is this line.';
$text[2] = 'First te*t sample i this l1ne.';
$text[3] = 'F i r st text s ample is this line.';
I attempting to combine the above to get an output which looks like:
$text = 'First text sample is this line.';
Don't tell me it's impossible, because it is certainly not, just very difficult.
I would very much appreciate any ideas anyone has towards this.
Thank you!
My current thoughts:
Just checking the words against the dictionary will not work, since some of the spaces are in the wrong place and occasionally the word will not be in the dictionary.
The major concern is repairing broken spacings, once this is fixed then then the most commonly occurring dictionary word can be chosen if exists, or else the most commonly occurring non-dictionary word.
Have you tried using a longest common subsequence algorithm? These are commonly seen in the "diff" text comparison tools used in source control apps and some text editors. A diff algorithm helps identify changed and unchanged characters in two text samples.
http://en.wikipedia.org/wiki/Diff
Some years ago I worked on an OCR app similar to yours. Rather than applying multiple OCR engines to one image, I used one OCR engine to analyze multiple versions of the same image. Each of the processed images was the result of applying different denoising technique to the original image: one technique worked better for low contrast, another technique worked better when the characters were poorly formed. A "voting" scheme that compared OCR results on each image improved the read rate for arbitrary strings of text such as "BQCM10032". Other voting schemes are described in the academic literature for OCR.
On occasion you may need to match a word for which no combination of OCR results will yield all the letters. For example, a middle letter may be missing, as in either "w rd" or "c tch" (likely "word" and "catch"). In this case it can help to access your dictionary with any of three keys: initial letters, middle letters, and final letters (or letter combinations). Each key is associated with a list of words sorted by frequency of occurrence in the language. (I used this sort of multi-key lookup to improve the speed of a crossword generation app; there may well be better methods out there, but this one is easy to implement.)
To save on memory, you could apply the multi-key method only to the first few thousand common words in the language, and then have only one lookup technique for less common words.
There are several online lists of word frequency.
http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists
If you want to get fancy, you can also rely on prior frequency of occurrence in the text. For example, if "Byrd" appears multiple times, then it may be the better choice if the OCR engine(s) reports either "bird" or "bard" with a low confidence score. You might load a medical dictionary into memory only if there is a statistically unlikely occurrence of medical terms on the same page--otherwise leave medical terms out of your working dictionary, or at least assign them reasonable likelihoods. "Prosthetics" is a common word; "prostatitis" less so.
If you have experience with image processing techniques such as denoising and morphological operations, you can also try preprocessing the image before passing it to the OCR engine(s). Image processing could also be applied to select areas after your software identifies the words or regions where the OCR engine(s) fared poorly.
Certain letter/letter and letter/numeral substitutions are common. The numeral 0 (zero) can be confused with the letter O, C for O, 8 for B, E for F, P for R, and so on. If a word is found with low confidence, or if there are two common words that could match an incompletely read word, then ad hoc shape-matching rules could help. For example, "bcth" could match either "both" or "bath", but for many fonts (and contexts) "both" is the more likely match since "o" is more similar to "c" in shape. In a long string of words such as a a paragraph from a novel or magazine article, "bath" is a better match than "b8th."
Finally, you could probably write a plugin or script to pass the results into a spellcheck engine that checks for noun-verb agreement and other grammar checks. This may catch a few additional errors. Maybe you could try VBA for Word or whatever other script/app combo is popular these days.
Tackling complex algorithms like this by yourself will probably take longer and be more error prone than using a third party tool - unless you really need to program this yourself, you can check the Yahoo Spelling Suggestion API. They allow 5.000 requests per IP per day, I believe.
Others may offer something similar (I think there's a bing API, too).
UPDATE: Sorry, I just read that they've stopped this service in April 2011. They claim to offer a similar service called "Spelling Suggestion YQL table" now.
This is indeed a rather complicated problem.
When I do wonder how to spell a word, the direct way is to open a dictionary. But what if it is a small complex sentence that I'm trying to spell correctly ? One of my personal trick, which works most of the time, is to call Google. I place my sentence between quotes on Google and count the results. Here is an example : entering "your very smart" on Google gives 13'600k page. Entering "you're very smart" gives 20'000k pages. Then, likely, the correct spelling is "you're very smart". And... indeed it is ;)
Based on this concept, I guess you have samples which, for the most parts, are correctly misspelled (well, maybe not if your develop for a teens gaming site...). Can you try to divide the samples into sub pieces, not going up to the words, and matching these by frequency ? The most frequent piece is the most likely correctly spelled. Prior to this, you can already make a dictionary spellcheck with your 600'000 terms to increase the chance that small spelling mistakes will alredy be corrected. This should increase the frequency of correct sub pieces.
Dividing the sentences in pieces and finding the right "piece-size" is also tricky.
What concerns me a little too : how do you extract the samples and match them together to know the correctly spelled sentence is the same (or very close?). Your question seems to assume you have this, which also seems something very complex for me.
Well, what precedes is just a general tip based on my personal and human experience. Donno if this can help. This is obviously not a real answer and is not meant to be one.
You could try using google n-grams to achieve this.
If you need to get right string only by comparing other. Then Something like this maybe will help.
It not finished yet, but already gives some results.
$text[0] = 'Fir5t text sample is thisline';
$text[1] = 'Fir5t text Smplee is this line.';
$text[2] = 'First te*t sample i this l1ne.';
$text[3] = 'F i r st text s ample is this line.';
function getRight($arr){
$_final='';
$count=count($arr);
// Remove multi spaces AND get string lengths
for($i=0;$i<$count;$i++){
$arr[$i]=preg_replace('/\s\s+/', ' ',$arr[$i]);
$len[$i]=strlen($arr[$i]);
}
// Max length
$_max=max($len);
for($i=0;$i<$_max;$i++){
$_el=array();
for($j=0;$j<$count;$j++){
// Cheking letter counts
$_letter=$arr[$j][$i];
if(isset($_el[$_letter]))$_el[$_letter]++;
else$_el[$_letter]=1;
}
//Most probably count
list($mostProbably) = array_keys($_el, max($_el));
$_final.=$mostProbably;
// If probbaly example is not space
if($_el!=' '){
// THERE NEED TO BE CODE FOR REMOVING SPACE FROM LINES WHERE $text[$i] is space
}
}
return $_final;
}
echo getRight($text);
I have to format a telephone number list, and I'd wish to extract and separate the prefix from the number for better viewing.
I have a list of all possible prefixes, but there is no regular pattern.
I mean, I could have these numbers:
00 - 12345 (short prefix)
0000 - 12345 (long prefix)
How can I manage that? Numbers are plain, without any special char (ie without / \ - . , ecc ecc).
Prefixes are like that:
030
031
0321
0322
...
...
Most of the time I have the town of the customer (it's not required) so, usually i can get the prefix from there, but that's not a sure thing, since town and telephone couldn't be linked.
== EDIT ==
Prefix list is 231 entries long. Maybe I'll add somthing more, so take 300 as safe value
Moreover, prefixes come from a single country only (Italy)
I have to save plain numbers without any separator so users can search for it. Infact if they put separators they will never able to find again that.
More info
Prefix ALWAYS starts with a leading 0, its lenght ranges from 2-4
But the more i study this thing, the more i think i can't work it out :(
Because of the extremely varied telephone number formats used around the world, it's probably going to be tough to correctly parse any phone number that is put into your system.
I'm not certain if it would make your ask any easier, but I had the idea that parsing from Right-to-Left might be easier for you, since it's the Prefix length that's unknown
What a pain. I would use a logic funnel to narrow possible choices and finally take a best guess.
First, test if the first few numbers can match anything on your prefix list. For some, hopefully only one prefix can possibly be correct.
Then, perhaps you could use the city to eliminate prefixes from entirely different countries.
Finally, you could default to the most popular format for prefixes.
Without any other information, you can't do better than a good guess unless you want to default to no format at all.
I'm really confused. What do you mean, "extract and separate"? My guess is these phone numbers are in a MySQL database, and you get to use PHP. Are you trying to get the prefix from the numbers, and then insert them into a different field in the same row? Are you pulling these numbers from the database, and you would just like to print the prefixes to the screen?
Regardless of what you're trying to do, and taking for granted that you're using PHP and regexs, isn't this essentially what you're looking for?:
$telephone_number = '333-12345';
$matched = array();
preg_match('~^(\d+)-~', $telephone_number, $matched);
$matched[1] // Should be '333'
ok, I worked it out.
I saw that there aren't shor prefixes that share chars with longer one.
I mean:
02 -> there will never be a longer prefix as 021, 022 and so on
so things are pretty easy now:
I get first 4 numbers -> is that in my prefix table?
YES: stop here
NO: get first 3
and so on..
thanks for your help
What's the most efficient way to quickly check for telephone number format from a form input text?
easiest way is to strip out everything that is not a number and then count how many digits there are. That way you don't have to force a certain format on the user. As far as how many numbers you should check for...depends on if you are checking for just local numbers, US numbers, international numbers, etc...but for instance:
$number="(123) 123-1234";
if (strlen(preg_replace('~[^0-9]~','',$number)) == 10) {
// 10 digit number (US area code plus phone number)
}
That really depends on the location. There are different conventions from place-to-place as to the length of the area code and the length of the entire number. Being more permissive would be better.
A regular expression to ensure that it is an optional '+', followed by any number of digits would be the bare minimum. Allowing optional dashes ("-") or spaces separating groups of numbers could be a nice addition. Also, what about extension numbers?
In truth, it's probably best to just check that it includes some numbers.
If you are only dealing with U.S. phone numbers, you might follow the common UI design of using three textboxes = one for area code + second one for exchange + third one for number. That way, you can test that each one contains only digits, and that the correct number of digits is entered.
You can also use the common techniques of testing each keypress for numbers (ignoring keypresses that are not digits), and advancing to the next textbox after the required number of characters have been entered.
That second feature makes it easier for users to do the data entry.
Using separate textboxes also makes it a little easier for users to read their input, easier than, say, reading that they have entered ten digits in a row correctly.
It also avoids you having to deal with people who use different punctuation in their entries -- surrounding the area code with parentheses, using hyphens or dots between sections.
Edit: If you decide to stick with just one textbox, there are some excellent approaches to using regex in this SO question.
If you are only dealing with U.S. phone numbers, you might follow the common UI design of using three textboxes = one for area code + second one for exchange + third one for number. That way, you can test that each one contains only digits, and that the correct number of digits is entered.
If you are dealing with numbers worldwide, this breaks down because some countries don't use area codes at all and number lengths vary from country to country.
Subscriber numbers may be as short as 3 digits or as long as 11 or 12 digits. Area codes can range from 1 to 6 digits where used. The data will also need to be stored with the country code in order to have any chance of correctly formatting it for display.
Using PHP, how can I verify if a phone # is well formed?
It seems easiest to simply strip all non-numeric data, leaving only the numbers. Then to check if 10 digits exist.
Is this the best and easiest way?
The best? No. Issues I see with this approach:
Some area codes - like 000-###-#### - are not valid. See http://en.wikipedia.org/wiki/List_of_NANP_area_codes
Some exchanges - like ###-555-#### - are not valid. See http://en.wikipedia.org/wiki/555_%28telephone_number%29
Some people will enter a 1 before their number, i.e. 1-###-###-####.
Some people are only reachable at an extension, like ###-###-#### x####.
Some companies tack on extra digits, like 1-800-GO-FLOWERS. The additional digits are simply ignored by the phone system, but a user might expect to be able to enter the whole thing.
International phone numbers are not necessarily 10 digits, even if you discount the country codes.
Good enough? Quite possibly, but that's up to you and your app.
You can use a regex for it:
$pattern_phone = "|^[0-9\+][0-9\s+\-]*$|i";
if(!preg_match($pattern_phone,$phone)){
//Somethings wrong
}
Haven't tested the regex, so it may not be 100% correct.
Checking for 10 digits after stripping will check the syntax but won't check the validity. For that you'd need to determine what valid numbers are available in the region/country and probably write a regex to match the patterns.
The problem with validating/filtering data like this usually comes down the the answer to this question: "How strict do I want to be?" which then devolves into a series of "feature" questions
Are you going to accept international numbers?
Are you going to accept extensions?
Are you going to allow various formats i.e., (111) 222-3333 vs 111.222.3333
Depending on your business rules, the answers to these questions can vary. But to be the most flexible, I recommend 3 fields to take a phone number
Country Code (optional)
Phone Number
Extension (optional)
All 3 fields can be programmatically limited/filters for numeric values only. You can then combine them before storing into some parse-able format, or store each value individually.
Answering if something is "the best" thing to do, is nearly impossible (unless you're the one answering your own question).
The way you propose it, stripping all non-digits and then check if there are 10 digits, might result in unwanted behaviour for a string like:
George Washington (February 22, 1732 –
December 14, '99) was the commander
of the Continental Army in the
American Revolutionary War and served
as the first President of the United
States of America.
since stripping all non-digits will result in the string 2217321499 which is 10 fdigits long, but I highly doubt that the entire string should be considered as a valid phone number.
What format you need? You can use regular expressions to this.