How to strip subdomains to get valid root domain using PHP? - php

Question

Ok, here's what I'm looking for: from a list of links, I'm stripping everything but the domains. The result is a mixed list of domains and domain-names which represent subdomains.
stackoverflow.com
security.stackexchange.com
whoknows.test.co.uk
you.get.it.by.now.dont.you.com
What I want to do is to trim all list entries down to their VALID (=only existing) root domains like this:
stackoverflow.com
security.stackexchange.com
test.co.uk
-fail-
Currently I explode each line into an array and work my list from back to front, using curl to check each potential root domain for it's existance... as soon as curl throws back a HTTP code >= 200 and < 400, I regard the root domain to be found. When the end of each potential domain lookup is done and no valid domain has been found at all, the domain is considered to be non-existant.
input: stackoverflow.com
test: stackoverflow.com - succeeds and is the root domain
result: stackoverflow.com - valid root domain
input: whoknows.test.co.uk
test: co.uk - fails
test: test.co.uk - succeeds (theoretically) and is the root domain
result: test.co.uk - valid root domain
input: you.get.it.by.now.dont.you.com
test: you.com - fails
test: dont.you.com - fails
test: now.dont.you.com - fails
test: by.now.dont.you.com - fails
test: it.by.now.dont.you.com - fails
test: get.it.by.now.dont.you.com - fails
test: you.get.it.by.now.dont.you.com - fails
result: you.get.it.by.now.dont.you.com - invalid domain
Is there any alternative way to do this? I would like to stop heating up my webserver's CPU with 2 to X (=near to unlimited) curl look-ups for every domain on my 100.000+ list. Also, all these lookups take a bunch of time. Maybe - so I hope - there is a quicker solution to do this.
The catch? It has to work with PHP.

Answer 1

There are a bunch of shortcuts to acheive what you need.
For example, you already know that .co.uk and .com are TLDs, so checking these you can obviously skip.
The problem is with all the other crazy TLDs.
I suggest you take a look at the source for ruby-domain-name
They have done a lot of work using RFCs and known data, to try and process it the right way.

Answer 2

So...
I've been fiddling around with this for a long time now, looking for the potential bottlenecks and after a few days of back and forth I discovered that it's actually CURL (that's waiting for the individual servers to respond with a HTTP code) that's making things slower than needed.
In the end, I opted in for a different "gethostbyname" function that takes IP6 into account to solve my problem(s).
function my_gethostbyname($host, $try_a = FALSE)
{
$dns = gethostbynamel6($host, $try_a);
if ($dns == FALSE)
{
return FALSE;
}
else
{
return $dns[0];
}
}
function gethostbynamel6($host, $try_a = FALSE)
{
$dns = array();
$dns6 = #dns_get_record($host, DNS_AAAA);
if($dns6!== FALSE)
{
$dns = array_merge($dns, $dns6);
}
if ($try_a == TRUE)
{
$dns4 = #dns_get_record($host, DNS_A);
if($dns4!== FALSE)
{
$dns = array_merge($dns, $dns4);
}
}
else
{
$dns = $dns6;
}
$ip6 = array();
$ip4 = array();
foreach ($dns as $record)
{
if ($record["type"] == "A")
{
$ip4[] = $record["ip"];
}
if ($record["type"] == "AAAA")
{
$ip6[] = $record["ipv6"];
}
}
if (count($ip6) < 1)
{
if ($try_a == TRUE)
{
if (count($ip4) < 1)
{
return FALSE;
}
else
{
return $ip4;
}
}
else
{
return FALSE;
}
}
else
{
return $ip6;
}
}
As soon as the first domain-part actually resolves to an IP, (a) the domain exists and (b) is the root domain.
This spares me major time and the trick of this is that you're only as slow as your DNS resolution and some microsecs. The curl option I used before took around 3 seconds per call (sometimes up to the full timeout range I had set to 20 secs), depending on the target server's response time - if any.
The path I now chose is easy to understand: I end up with a list of resolving domains and - when needed - I can check those using curl "on demand" or using one or more CRON jobs "on interval".
I know that it's kind of a workaround, but splitting the problem into two tasks (1 = pre-check for root domain, 2 = check if domain returns expected HTTP code) makes the whole thing faster than trying to do the complete job at once using curl.
What I've learned from this...
When checking domains, try to resolve them first so you can spare yourself the timeout burden that comes with curl.
Curl is great for many tasks, but it's not the smartest way to try to do everything with it.
When you think you can't solve a problem more than you've tried to do, split the problem in two or more parts and check again. Chances are big that you'll discover a whole new world of options to enhance what you've got.
I hope this spares someone the burden of fiddling around with an alike problem for weeks. ;)

Answer 3

class DomainUtils {
function __construct(){
//only these super domains
$this->superDomains = array(
'.com',
'.gov',
'.org',
'.co.uk',
'.info',
'.co',
'.net',
'.me',
'.tv',
'.mobi'
);
}
//get super domain
public function getMainDomain($domain = null){
$domainChunks = explode('.', str_ireplace($this->superDomains, '', $domain));
if(sizeof($domainChunks) == 0){
return false;
}
foreach($domainChunks as $key => $domainChunk){
if($key < sizeof($domainChunks) - 1){
$domain = str_ireplace($domainChunk . '.', '', $domain);
}
}
return $domain;
}
}