PHP Scraping company data using Guzzle - php

Question

I'm trying to audit a vast amount of company data from companycheck.co.uk my current script appears to be looping the first 10 results from only the first page. I had the script gather more than 10 results at one point, but this caused a fatal error after around 600 results (not a timeout error, but a connection error of some sort), I need the script to be more reliable as I'm fetching over 40,000 results.
My code so far:
<?php
set_time_limit(0);
ini_set('max_execution_time', 0);
require 'vendor/autoload.php';
require "Guzzle/guzzle.phar";
// Add this to allow your app to use Guzzle and the Cookie Plugin.
use Guzzle\Http\Client as GuzzleClient;
use Guzzle\Plugin\Cookie\Cookie;
use Guzzle\Plugin\Cookie\CookiePlugin;
use Guzzle\Plugin\Cookie\CookieJar\ArrayCookieJar;
use Guzzle\Plugin\Cookie\CookieJar\CookieJarInterface;
$Pagesurl = 'http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&type=name';
$pagesData = json_decode(file_get_contents($Pagesurl), true);
$resultsFound = $pagesData["hits"]["found"];
$pages = ceil($resultsFound / 10);
//echo $pages;
echo "<br>";
for ($p = 0; $p < $pages; $p++) {
$url = 'http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&type=name&companyPage=' . $p . '';
$data = json_decode(file_get_contents($url), true);
for ($i = 0; $i < 11; $i++) {
$id = $data["hits"]["hit"][$i]["id"];
$TradingAddress = $data["hits"]["hit"][$i]["data"]["address"][0];
$companyName = $data["hits"]["hit"][$i]["data"]["companyname"][0];
$companyNumber = $data["hits"]["hit"][$i]["data"]["companynumber"][0];
$finalURL = "http://companycheck.co.uk/company/" . $id . "";
$httpClient = new GuzzleClient($finalURL);
$httpClient->setSslVerification(FALSE);
$cookieJar = new ArrayCookieJar();
// Create a new cookie plugin
$cookiePlugin = new CookiePlugin($cookieJar);
// Add the cookie plugin to the client
$httpClient->addSubscriber($cookiePlugin);
$httpClient->setUserAgent("Opera/9.23 (Windows NT 5.1; U; en-US)");
$request = $httpClient->get($finalURL);
$response = $request->send();
$body = $response->getBody(true);
$matches = array();
preg_match_all('/<table.*?>(.*?)<\/table>/si', $body, $table);
preg_match('/<meta name=\"keywords\" content=\"(.*?)\"\/>/si', $body, $metaName);
preg_match('/<p itemprop="streetAddress".*?>(.*?)<\/p>/si', $body, $regOffice);
echo "<table><tbody>";
echo "<tr><th>Company Name</th><td>";
echo $companyName;
echo "</td></tr>";
echo "<tr><th>Company Number</th><td>";
echo $companyNumber;
echo "</td></tr>";
echo "<tr><th>Registar Address</th><td>";
echo str_replace("<br>", " ", $regOffice[0]);
echo "</td></tr>
<tr><th>Trading Address</th><td>";
echo $TradingAddress;
echo "</td></tr>
<tr>
<th>Director Name</th>
<td>";
$name = explode(',', $metaName[1]);
echo $name[2];
echo "</td>
</tr></tbody></table>";
echo $table[0][1];
echo "<br><br><br>";
}
}
To get each page, I use http://companycheck.co.uk/search/UpdateSearchCompany?searchTerm=cars&type=name&companyPage=1 which returns json for each page from http://companycheck.co.uk/search/results?SearchCompaniesForm[name]=cars&yt1= and some data, but not all.
With this I can get the ID of each company to navigate to each link and scrape some data from the frontend of the site.
For example the first result is:
"hits":{"found":42842,"start":0,"hit":[{"id":"08958547","data":{"address":["THE ALEXANDER SUITE SILK POINT, QUEENS AVENUE, MACCLESFIELD, SK10 2BB"],"assets":[],"assetsnegative":[],"cashatbank":[],"cashatbanknegative":[],"companyname":["CAR2CARS LIMITED"],"companynumber":["08958547"],"dissolved":["0"],"liabilities":[],"liabilitiesnegative":[],"networth":[],"networthnegative":[],"postcode":["SK10 2BB"],"siccode":[]}}
So the first link is: http://companycheck.co.uk/company/08958547
Then from this I can pull table data such as:
Registered Office
THE ALEXANDER SUITE SILK POINT
QUEENS AVENUE
MACCLESFIELD
SK10 2BB
And information from the meta tags such as:
<meta name="keywords" content="CAR2CARS LIMITED, 08958547,INCWISE COMPANY SECRETARIES LIMITED,MR ROBERT CARTER"/>
An example of one of the results returned:
Company Name CAR2CARS LIMITED
Company Number 08958547
Registar Address
THE ALEXANDER SUITE SILK POINT QUEENS AVENUE MACCLESFIELD SK10 2BB
Trading Address THE ALEXANDER SUITE SILK POINT, QUEENS AVENUE, MACCLESFIELD, SK10 2BB
Director Name INCWISE COMPANY SECRETARIES LIMITED
Telephone No telephone number available.
Email Address No email address available.
Contact Person No contact person available.
Business Activity No Business Activity on record.
Each json page contains 10 company IDs to put into the URL to find the company, from each of these companies I need to scrape data from the full URL, then after these 10 move onto the next page and get the next 10 and loop this up until the last page.

Answer 1

It is almost certainly blocking you deliberately due to an excessive number of requests. Try putting a pause in between requests - that might help you fly under their radar.
The website you are intending to scrape appears to be a private company that is reformatting and republishing data from Companies House, the official record of company information in the UK. This company offers an API which allows 10K requests per month, and this is either free or costs GBP200/month, depending on what data you need. Since you want 40K results immediately, it is no wonder they operate IP blocks.
The rights and wrongs of scraping are complicated, but there is an important point to understand: by copying someone else's data, you are attempting to avoid the costs of collating the data yourself. By taking them from someone else's server, you are also adding to their operational costs without reimbursing them, an economic phenomenon known as an externality.
There are some cases where I am sympathetic to passing on costs in this way, such as where the scrape target is engaged in potential market abuse (e.g. monopolistic practices) and scraping has an alleviating effect. I have heard that some airline companies operate anti-scraping devices because they don't want price scrapers to bring prices down. Since bringing prices down is in the interest of the consumer one could argue that the externality can be justified (on moral, if not legal grounds).
In your case, I would suggest obtaining this data directly from Companies House, where it might be available for a much lower cost. In any case, if you republish valuable data obtained from a scrape, having dodged technical attempts to block you, you may find yourself in legal trouble anyway. If in doubt (and if there is no moral or public interest defence such as I outlined earlier) get in touch with the site operator and ask if what you want to do is OK.