strange issue with file_get_content for some urls!! - php - php

Question

I have an url and could scrap it using file_get_contents() and collect the links inside :
http://www.140online.com/Class/450/%D8%B3%D9%88%D8%A8%D8%B1%20%D9%85%D8%A7%D8%B1%D9%83%D8%AA
this is the result array :
Array
(
[0] => /company/C79955/المصرية ماركت - امام منصور امام/
[1] => /company/C574901/مريم ماركت/
[2] => /company/C1131/سوبر ماركت اولاد رجب/
[3] => /company/C19699/البركه/
[4] => /company/C21707/سوبر ماركت بست باى/
[5] => /company/C23552/الشامل مى ماركت/
[6] => /company/C21714/سوبر ماركت التميمى/
[7] => /company/C171121/الجزيره/
[8] => /company/C172099/الحمد ماركت/
[9] => /company/C172325/الخشاب ماركت/
[10] => /company/C174805/الدكان ماركت/
[11] => /company/C183996/اسواق الشمس المركزيه/
[12] => /company/C184039/اسواق مرحبا/
[13] => /company/C189580/البحيرى ماركت/
[14] => /company/C216371/سوق زمزم/
[15] => /company/C216941/سيتى ارت/
[16] => /company/C226223/وجينات محمد سعيد محمد موسى/
[17] => /company/C297432/ماركت اولاد محمود/
[18] => /company/C326553/سوبر ماركت عمرو عبد السلام سالم محمد/
[19] => /company/C326566/سوبر ماركت عونى عزيز هنرى/
)
then I need to scrap each link to get some data but here is the problem !
to use any link I have to add the domain name before.
http://www.140online.com/company/C79955/المصرية ماركت - امام منصور امام/
but when I try to get it's contents it send me this error :
file_get_contents(http://www.140online.com/company/C79955/%D8%A7%D9%84%D9%85%D8%B5%D8%B1%D9%8A%D8%A9%20%D9%85%D8%A7%D8%B1%D9%83%D8%AA%20-%20%D8%A7%D9%85%D8%A7%D9%85%20%D9%85%D9%86%D8%B5%D9%88%D8%B1%20%D8%A7%D9%85%D8%A7%D9%85/): failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error
although I encode the Arabic part as it shown by explode:
$url_ex = explode("/", $itm_url[$i]);
$i_url = 'http://'.$url_ex[2].'/'.$url_ex[3].'/'.$url_ex[4].'/'.rawurlencode($url_ex[5]).'/';
the another strange thing is I can get contents for some links but not for all
please help

Answer 1

I had some problems with file_get_contents for web scraping. I would recommend to use CURL. This library returns HTTP error codes to know what kind of problem you have.
http://php.net/manual/en/book.curl.php
This is an example how to use CURL, I hope it works.
$ch1 = curl_init('string_url');
$options1 = array(CURLOPT_POST => true,
CURLOPT_HEADER => $request_headers,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_POSTFIELDS => (if u need to send params)
);
curl_setopt_array($ch1, $options1);
$page = curl_exec($ch1);
if(curl_exec($ch1) === false)
{
\print_r(curl_error($ch1));
//HTTP and many other errors if you have it
}
curl_close($ch1);
$dom = new \DOMDocument();
$dom->validateOnParse = true;
$dom->preserveWhiteSpace = false;
$dom->substituteEntities = false;
//Load Document Object Model
$dom->loadHTML($page);
$domxPath = new \DOMXPath($dom);
$urls = $domxPath->query("/html/body//div[#class='urls']");
//Example here searchs all divs which has urls class
//You need to specify which element of DOM contains URLS
//$urls variable must contains an array with urls that you need