how to loop through ann array with curl function? - php

I can't loop through an array, stil get Curl error "3"
i have checked and inside array are all links I wanted to get.
function to make a get request
<?php
function geturl($url){
$starttime = microtime(TRUE);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0)
Gecko/20100101 Firefox/47.0");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
curl_setopt($ch, CURLOPT_TIMEOUT, 8);
curl_setopt($ch, CURLOPT_HTTPHEADER,
[
'https://betsapi.com/',
'authority: betsapi.com',
'upgrade-insecure-requests: 1',
'accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,
application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site: cross-site' ,
'sec-fetch-mode: navigate',
'sec-fetch-user: ?1',
'sec-fetch-dest: document',
//'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Microsoft Edge";v="98"',
'sec-ch-ua-mobile: ?0',
'sec-ch-ua-platform: "Windows"',
'referer: https://betsapi.com/',
'accept-language: pl,en;q=0.9,en-GB;q=0.8,en-US;q=0.7',
]);
$response = curl_exec($ch);
here i check errors and request time
if(curl_errno($ch))
{
echo "Curl error no: ". curl_errno($ch);
exit;
}
curl_close($ch);
$endtime = microtime(TRUE);
echo "Request time"." ". $endtime - $starttime."\n";
return $response;
}
function to load html
function simplehtml($item){
include_once('simple_html_dom.php');
$html = new simple_html_dom();
$html->load($item);
return $html;
}
when i call for the first time the geturl function and simplehtml function it works and puts all links into array
$starturl= geturl('https://betsapi.com/cin/soccer');
$starturlhtml= simplehtml($starturl);
foreach($starturlhtml->find("tr[class='c_1'] td[class='text-center'] a")as $url){
$url="https://betsapi.com".$url->href."\n";
$urlarray[]=$url;
}
here i call it in loop and i get Curl error 3
foreach($urlarray as $urls){
$urlss= geturl($urls);
$urlshtml=simplehtml($urlss);
foreach($urlshtml->find("div[class='col-md-6 text-center'] p b")as $stadium){
echo $stadium;
sleep(2);
}
}

Related

The requested URL returned error: 400 Bad Request

I have been working on setting up the frase api. and created the following curl snippet.
<?php
$url = 'http://api.frase.io/api/v1/process_url';
//The data you want to send via POST
$fields = ['url' => 'https://firstsiteguide.com/best-gaming-blogs/', 'token' => "dd528796a9924dae9962bc5bd7ccdb20"];
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST, 1);
curl_setopt($ch,CURLOPT_POSTFIELDS,http_build_query($fields));
curl_setopt($ch,CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT ,3);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0');
curl_setopt($ch,CURLOPT_TIMEOUT, 20);
$response = curl_exec($ch);
if (curl_errno($ch)) {
$error_msg = curl_error($ch);
echo "<br/>CURL ERROR: ". $error_msg ."<br/>";
}else{
print "curl response is:" . $response ;
}
curl_close ($ch);
?>
I am not sure why, But I am receiving the following error for the same
The requested URL returned error: 400 Bad Request
Can help me identify what part of code I am missing or doing wrong. Thank you so much in advance.
You're passing the token as a body parameter instead of a header. The body parameter needs to be sent as a JSON encoded string as mentioned on the API documentation page. Also, you need to remove or at least increase the cURL timeout value as it takes time to fetch, process and return a value from the API end. Note that the API will return the response in JSON format.
So, the complete code should be as:
<?php
$url = 'http://api.frase.io/api/v1/process_url'; //The endpoint url you want to send data via POST
$headers = ['token: dd528796a9924dae9962bc5bd7ccdb20']; // add this line, headers
$fields = ['url' => 'https://firstsiteguide.com/best-gaming-blogs/']; // modify this line, body
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); // add this line
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($fields)); // modify this
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0');
$response = curl_exec($ch);
if (curl_errno($ch))
{
$error_msg = curl_error($ch);
echo "<br/>CURL ERROR: " . $error_msg . "<br/>";
}
else
{
print($response);
// $values = json_decode($response, true); // this is an array with all the values
}
curl_close($ch);
?>

Cannot extract the data from the website using PHP cURL

I am working on a project which needs to get the data from other webpage:
https://eth.ethfans.org/#/miner?0x2998850087633a4806191960c94ed535d97da598
I am trying to use the function cRUL:
<?php
$url = "https://eth.ethfans.org/#/miner?0x2998850087633a4806191960c94ed535d97da598";
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$contents = curl_exec($ch);
curl_close($ch);
echo $contents;
?>
However, I can only get the layout of the site, but I cannot get the data inside.
Can anyone help for this ?
Thanks in Advance.
Regards,
Alex
Use str_get_html to fetch the data from the layout:
$get_html = str_get_html($contents);
Example:
function check()
{
$url = "https://stackoverflow.com/questions/49248329/cannot-extract-the-data-from-the-website-using-php-curl";
$get_html = $this->get_curl($url);
#print_r($get_html); exit;
$get_html = str_get_html($get_html);
$fb = NULL;
foreach ($get_html->find('a') as $v) { // you can get what data from the layout
if(strpos($v->href, 'facebook'))
{
echo $fb = $v->href;
echo "\n";
break;
}
}
unset($get_html);
}
public function get_curl($url)
{
ob_start();
$ch = curl_init($url);
$headers = [
'Accept-Language: en-US,en;q=0.5',
'Cache-Control: no-cache',
'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/51.0',
];
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch,CURLOPT_URL, $url);
$response = curl_exec ($ch);
curl_close ($ch);
ob_end_flush();
return $response;
}
you're hitting the wrong url, the page you're hitting only contains the layout and the javascript required to fetch the actual data, then the javascript fetch the data from https://eth.ethfans.org/api/page/miner?value=2998850087633a4806191960c94ed535d97da598 , so, do as the javascript does, and fetch that url.

How follow meta redirect with curl and php?

$url='http://wtion';
$headers = array(
'GET '.$url.' HTTP/1.1',
'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3',
'Accept: text/html',
'Accept-Language: ru,en-us;',
'Accept-Charset: windows-1251,utf-8;',
'Connection: close');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);//Массив с HTTP заголовками для передачи на сервер
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Не выводить ответ в браузер. Пусть функция пишет все в переменную.
$site=curl_exec($ch); //В случае успеха - html тест запрошенной страницы. Иначе - false
curl_close($ch);
echo $site;
After running the code , I get this line
<meta http-equiv='Refresh' content='0; url=/animation.php'>
How can I follow the redirect and get the response of /animation.php?
Curl cannot follow a meta refresh. Use DOMXml to parse the curl response as long as it's valid, you can check for a refresh return, then process the refresh path appropriately.
$url='http://wtion';
$headers = array(
'GET '.$url.' HTTP/1.1',
'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3',
'Accept: text/html',
'Accept-Language: ru,en-us;',
'Accept-Charset: windows-1251,utf-8;',
'Connection: close');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);//Массив с HTTP заголовками для передачи на сервер
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Не выводить ответ в браузер. Пусть функция пишет все в переменную.
$site=curl_exec($ch); //В случае успеха - html тест запрошенной страницы. Иначе - false
curl_close($ch);
$xml = simplexml_load_file($site);
$result = $xml->xpath("//meta[#http-equiv='refresh']");
if (!empty($result)) {
... do stuff to get the final $site value....
}
echo $site
I found an implementation in a comment of get_meta_tags() documentation page.
function sendRequest($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
/*curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'GET '.$url.' HTTP/1.1', // Are you sure about this?
'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3',
'Accept: text/html',
'Accept-Language: ru,en-us;',
'Accept-Charset: windows-1251,utf-8;',
'Connection: close'
));*/
$contents = curl_exec($ch);
curl_close($ch);
return $contents;
}
function getUrlContents($url, $maximumRedirections = null, $currentRedirection = 0)
{
$result = false;
$contents = sendRequest($url);
// Check if we need to go somewhere else
if (isset($contents) && is_string($contents))
{
preg_match_all('/<[\s]*meta[\s]*http-equiv="?REFRESH"?' . '[\s]*content="?[0-9]*;[\s]*URL[\s]*=[\s]*([^>"]*)"?' . '[\s]*[\/]?[\s]*>/si', $contents, $match);
if (isset($match) && is_array($match) && count($match) == 2 && count($match[1]) == 1)
{
if (!isset($maximumRedirections) || $currentRedirection < $maximumRedirections)
{
return getUrlContents($match[1][0], $maximumRedirections, ++$currentRedirection);
}
$result = false;
}
else
{
$result = $contents;
}
}
return $contents;
}
echo getUrlContents('http://wtion');

php how add pause in script?

We are have a code:
<?php
function cURL_AutonavigatorRu($level = false, $model_id = false){
#http://www.autonavigator.ru
$ch = curl_init('http://www.autonavigator.ru/dispatcher.pl');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36");
$headers = array
(
'Accept: application/json',
'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'Accept-Encoding: gzip,deflate',
'Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7'
);
curl_setopt($ch, CURLOPT_HTTPHEADER,$headers);
curl_setopt($ch, CURLOPT_REFERER, "http://www.autonavigator.ru/my/offer_add/");
if($level == '1'){
curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=make&show_all=1&vehicle=car&type=used');
}
elseif($level == '2' && $model_id){
curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=model&show_all=1&vehicle=car&type=used&make_id='.$model_id);
}
elseif($level == '3' && $model_id){
curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=modif&show_all=1&model_id='.$model_id);
}
else{
curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=modif&show_all=1&model_id='.$model_id);
}
curl_setopt($ch, CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
curl_close($ch);
$json = json_decode(iconv("windows-1251","utf-8",$result), true);
return $json['list'];
}
$ArrAuto = cURL_AutonavigatorRu('1');
foreach($ArrAuto as $auto) {
echo $auto['value'].'<br>';
//sleep(2);
$AllModif1 = cURL_AutonavigatorRu('2',$auto["id"]);
var_dump($AllModif1);
echo '<br><br>----------------------------------<br><br>';
}
We are have problem - web site block curl and not give results for each $AllModif1 = cURL_AutonavigatorRu('2',$auto["id"]); in one time(in curl we get null).
Tell me please how make to pause the script that cycle foreach($ArrAuto as $auto) worked every 5 seconds?
P.S.: How make pause 4 secound between cycle?
P.P.S.: we are know about sleep() but i not get result with it see please prntscr.com/4ylm9y
Use sleep:
foreach($ArrAuto as $auto){
//Your amazing code here
sleep(4);
}
You should check the sleep() function
Add sleep(4); in your foreach loop.
just add sleep(5); in your foreach()
Add the following:
sleep(4);
More info here: http://php.net/manual/en/function.sleep.php

php curl not saving file to location

I have been searching for an answer for this all day, but with no luck!
I want to download/copy an image from the web to a location on my server, The code below doesn't seam to throw any errors other than the image is just not saving to the required or any directory.
As you can see I am using cURL to get the image and the variable $contents is returning true (1) so I am assuming the script works but I am actually missing something.
Many thanks in advance for your help. :-)
$dir = URL::base() . "/img/products/";
$imgSrc = "an image on the web";
$file = fopen($dir, "wb");
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
$ch = curl_init($imgSrc);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FILE, $file); // location to write to
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 60);
$contents = curl_exec($ch);
$curl_errno = curl_errno($ch);
$curl_error = curl_error($ch);
curl_close($ch);
fclose($lfile);
if ($curl_errno > 0)
{
Log::write("CURL", "cURL Error (".$curl_errno."): ".$curl_error);
}
else
{
Log::write("CURL", "Data received: " . $contents);
}
return;
Provide the file the writing access to PHP FILE using curl to store the contents. This can be done in three ways:
If you have the terminal access then use chmod to provide the writing access
If you have the CPanel access then use directory explorer then provide the writing access to the file by changing file properties.
You must have the access to FTP and change the file access attributes and provide the writing access.
Don't use curl.
If all you need to do is download an image, go for "file_get_contents" instead.
It's dead easy:
$fileContents = file_get_contents("https://www.google.com/images/srpr/logo4w.png");
File::put('where/to/store/the/image.jpg', $fileContents);
function saveImageToFile($image_url,$output_filename)
{
$ch = curl_init ($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
$raw=curl_exec($ch);
curl_close ($ch);
if(file_exists($saveto))
{
unlink($saveto); //Saves over files
}
$fp = fopen($saveto,'x');
fwrite($fp, $raw);
fclose($fp);
}
Your problem is quite simple, and I have no idea how everyone else ignored it. It is Laravel-specific. Your $dir variable returns an HTTP resource identifier. What you need is a filesystem identifier.
For laravel, change your URL::to() to path("public") to tell Laravel to stop using HTTP URIs and instead take the local path to the public folder (/your/laravel/setup/path/public/).
code
$dir = path("public") . "img/products/";
$imgSrc = "an image on the web";
$file = fopen($dir . substr($imgSrc,strrpos("/",$imgSrc)+1), "wb");
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
$ch = curl_init($imgSrc);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FILE, $file); // location to write to
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 60);
$contents = curl_exec($ch);
$curl_errno = curl_errno($ch);
$curl_error = curl_error($ch);
curl_close($ch);
if ($curl_errno > 0)
{
Log::write("CURL", "cURL Error (".$curl_errno."): ".$curl_error);
}
else
{
Log::write("CURL", "Data received: " . $contents);
fwrite($file,$contents);
fclose($file);
}
return;
OK, finally got it all working and here is the code if anyone else ever tries to do the same sort of thing!
I was missing these parts:
$dir = $_SERVER['DOCUMENT_ROOT'] . "/img/products/";
and
fwrite($file,$contents);
So here is my final code... credit to Sébastien for pointing me in the right direction. Thanks.
if($method == 'save')
{
$productId = Input::get('pId');
$removeProductImages = DB::table('product_ref_images')->where('product_id', '=', $productId)->delete();
$imagesData = Input::get('imageRefs');
$dir = $_SERVER['DOCUMENT_ROOT'] . "/img/products/";
$sortOrder = 0;
for ($i=0; $i < count($imagesData); $i++) {
$imgSrc = trim($imagesData[$i]['imgSrc']);
$imgId = trim($imagesData[$i]['imgId']);
$file = fopen($dir . basename($imgSrc), "wb");
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
$ch = curl_init($imgSrc);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FILE, $file);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 60);
$contents = curl_exec($ch);
$curl_errno = curl_errno($ch);
$curl_error = curl_error($ch);
curl_close($ch);
if ($curl_errno > 0)
{
Log::write("CURL", "cURL Error (".$curl_errno."): ".$curl_error);
break;
}
else
{
fwrite($file,$contents);
fclose($file);
$imageIds = DB::table('product_ref_images')->order_by('image_id', 'desc')->first();
if($imageIds == null)
{
$imageIds = 0;
}
else
{
$imageIds = $imageIds->image_id;
}
$updateImages = DB::table('product_ref_images')
->insert(array(
'image_id' => $imageIds + 1,
'product_id' => $productId,
'flickr_image_id' => $imgId,
'sort_order' => $sortOrder++,
'local_storage_url' => $dir . basename($imgSrc),
'created_at' => date("Y-m-d H:i:s"),
'updated_at' => date("Y-m-d H:i:s")
));
}
}
return Response::json('Complete');
}
Remove this line:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
You're storing the response to a file, not to the return variable. Otherwise, you have to save it yourself (like you did in the other solution).

Categories