it doesn't work. the website keeps showing blanks - php

Question

I'm unable to scrape data from few websites using curls.
I'm using CURL to scrape website from url's. It works great in 80% of the urls I use. But some url's don't seem "scrapeable". For example, when I try to scrape https://www.nextdoorhub.com/ and https://www.atknsn.com/, it doesn't work. the website keeps showing blanks and at the end it doesn't return a result.
This is my code:
<center>
<br/>
<form method="post" name="scrap_form" id="scrap_form" action="scrape_data.php">
<b>Enter Website URL To Scrape Data:</b>
<input type="input" name="website_url" id="website_url">
<input type="submit" name="submit" value="Submit" >
</form>
</center>
<?php
error_reporting(E_ALL ^ E_NOTICE );
$website_url = $_POST['website_url'];
$result = scrapeWebsiteData($website_url);
function scrapeWebsiteData($website_url){
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $website_url);
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_BINARYTRANSFER,1);
$result = curl_exec($curl);
curl_close($curl);
return $result;
}
$regextit = '<div id="case_textlist">(.*?)<\/div>/s';
preg_match_all($regextit, $result, $list);
/* echo "<pre>";
print_r($list[1]); die; */
$regex = '/[\'" >\t^]([^\'" \n\r\t]+\.(jpe?g|bmp|gif|png))[\'" <\n\r\t]/i';
preg_match_all($regex, $result, $url_matches);
$count = count($url_matches[1]);
// set the local path of image
$local_path = 'C:\udeytech\htdocs\tests\images\\';
for($i=0; $i<$count; $i++)
{
preg_match_all('!.*?/!', $url_matches[1][$i], $matches);
$last_part = end($matches[0]);
////match image name last part of anything .jpg|jpeg|gif|png
preg_match("!$last_part(.*?.(jpg|jpeg|gif|png))!", $url_matches[1][$i], $matche);
$secons_part = $matche[0];
$info = pathinfo($secons_part);
$image_name = $info['basename'];
//save image url in a variable
$image_url = $url_matches[1][$i];
$image_path = scrapeWebsiteData($image_url);
$file_open = fopen($local_path.$image_name, 'w');
fwrite($file_open, $image_path);
fclose($file_open);
}
?>

Answer 1

Have you tried to load either of these sites in your browser and look at the responses?
nextdoorhub is using angular and atknsn looks to be heavy on jQuery. Long story short, these sites need to run javascript to render the full HTML you're intending to scrape.
Using PHP + cURL alone won't cut it. Look at threads that discuss scraping angular and that will point you in the right direction. (Hint: you need to scrape these sites with node.js)