i am trying to get data from a url using curl. i've made a recursive function for this. i get the data successfully , but the problem what i am facing is that when no result is found against curl call, then the page show me nothing, only a blank page is shown.. no error at all. i've used var_dump() too for testing the response. but found nothing.
here is my recursive function
function recursive_get_scrap($offset, $page_size, $urls, $original_array){
ini_set('max_execution_time', 1800);
$of_set = $offset;
$pg_size = $page_size;
$off_sets = 'offset='.$of_set.'&page_size='.$pg_size.'';
$url = $urls.$off_sets;
$last_correct_array = $original_array;
$ch1 = curl_init();
// Disable SSL verification
curl_setopt($ch1, CURLOPT_SSL_VERIFYPEER, false);
// Will return the response, if false it print the response
curl_setopt($ch1, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch1, CURLOPT_URL,$url);
// Execute
$result2 = curl_exec($ch1);
$info = curl_getinfo($ch1);
if(curl_errno($ch1))
{
echo 'error:' . curl_error($ch1);
//return $last_correct_array;
}
// Closing
curl_close($ch1);
if(!$result2 || strlen(trim($result2)) == 0 || $result2 == false){
echo 'no array';
}
if(isset($result2) && !empty($result2)){
echo 'in recursive function <br>';
$a1 = json_decode( $original_array, true );
$a2 = json_decode( $result2, true );
$temp_array = array_merge_recursive($a1, $a2 );
$last_correct_array = $temp_array;
$offset += 100;
$page_size = 100;
recursive_get_scrap($offset, $page_size, $urls, json_encode($last_correct_array));
}
}
now what i only want it that if noting is get against curl call then no array message should be displayed.
Use this option to curl_setopt():
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
This will make curl_exec return the data instead of outputting it.
To see if it was successful you can then check $result and also
curl_error().
Related
i have a very weird issue with curl and url defined inside an array.
I have an array of url and i want perform an http GET on those urls with curl
for ($i = 0, $n = count($array_station) ; $i < $n ; $i++)
{
$station= curl_init();
curl_setopt($station, CURLOPT_VERBOSE, true);
curl_setopt($station, CURLOPT_URL, $array_station[$i]);
curl_setopt($station, CURLOPT_RETURNTRANSFER, true);
curl_setopt($station, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($station);
curl_close($station);
}
If i define my $array_station in the way below
$array_station=array("http://www.example.com","http://www.example2.com");
the code above with curl working flawlassy,but since my $array_station is build in the way below (i perform a scan of directory searchin a specific filename, then i clean the url), the curl does not work, no error showed and nothing happens..
$di = new RecursiveDirectoryIterator(__DIR__,RecursiveDirectoryIterator::SKIP_DOTS);
$it = new RecursiveIteratorIterator($di);
$array_station=array();
$i=0;
foreach($it as $file) {
if (pathinfo($file, PATHINFO_FILENAME ) == "db_insert") {
$string = str_replace('/web/htdocs/', 'http://', $file.PHP_EOL);
$string2 = str_replace('/home','', $string);
$array_station[$i]=$string2;
$i++;
}
}
Doyou have some ideas? i'm giving up :-(
I'm on mobile right now so i cannot test it, but why are you adding a new line (PHP_EOL) to the url? Try to remove the new line or trim() the url at the end.
Add the lines of code below.
If there is a curl error it will report the error number.
If the request is made, it will show the HTTP request and response headers. The request is in $info and response header is in $head
for ($i = 0, $n = count($array_station) ; $i < $n ; $i++)
{
$station= curl_init();
curl_setopt($station, CURLOPT_VERBOSE, true);
curl_setopt($station, CURLOPT_URL, $array_station[$i]);
curl_setopt($station, CURLOPT_RETURNTRANSFER, true);
curl_setopt($station, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$response = curl_exec($station);
if (curl_errno($station)){
$response .= 'Retreive Base Page Error: ' . curl_error($station);
}
else {
$skip = intval(curl_getinfo($station, CURLINFO_HEADER_SIZE));
$head = substr($response ,0,$skip);
$response = substr($response ,$skip);
$info = var_export(curl_getinfo($station),true);
}
echo $head;
echo $info;
curl_close($station);
}
I am creating a web service. I have tried to write an URL but its throwing error. I am not getting where I am going wrong. I want to pass these variable values in url and depending on this i want to call the web service
<?php
if($_POST["occupation"] == '1'){
$occupation = 'Salaried';
}
else{
$occupation = 'Self+Employed';
}
$url = 'http://www.aaa.com/ajaxv2/getCompareResults.html?interestRateType='.$_POST["interestRateType"]'.&occupation='.$_POST["occupation"].'&offeringTypeId='.$_POST["offeringID"].'&city='.$_POST["city"].'&loanAmt='.$_POST["loanAmt"].'&age='.$_POST["age"];
echo $url;
// Initiate curl
$ch = curl_init();
// Disable SSL verification
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// Will return the response, if false it print the response
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Set the url
curl_setopt($ch, CURLOPT_URL,$url);
// Execute
$result=curl_exec($ch);
// Closing
curl_close($ch);
$json = json_decode($result, true);
//print_r($json);
//echo $json['resultList']['interestRateMin'];
$json_array = $json['resultList'];
print_r($json_array);
?>
Try below code. You have syntax error before &occupation
$url = 'http://www.aaa.com/ajaxv2/getCompareResults.html?interestRateType='.$_POST["interestRateType"].'&occupation='.$_POST["occupation"].'&offeringTypeId='.$_POST["offeringID"].'&city='.$_POST["city"].'&loanAmt='.$_POST["loanAmt"].'&age='.$_POST["age"];
Copy this because there is some ' and . error
$url = 'http://www.aaa.com/ajaxv2/getCompareResults.html?
interestRateType='.$_POST["interestRateType"].'&
occupation='.$_POST["occupation"].'&
offeringTypeId='.$_POST["offeringID"].'&
city='.$_POST["city"].'&
loanAmt='.$_POST["loanAmt"].'&
age='.$_POST["age"];
I am trying to get the latest commit from github using the api, but I encounter some errors and not sure what the problem is with the curl requests. The CURLINFO_HTTP_CODE gives me 000.
What does it mean if I got 000 and why is it not getting the contents of the url?
function get_json($url){
$base = "https://api.github.com";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $base . $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
//curl_setopt($curl, CONNECTTIMEOUT, 1);
$content = curl_exec($curl);
echo $http_status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
curl_close($curl);
return $content;
}
echo get_json("users/$user/repos");
function get_latest_repo($user) {
// Get the json from github for the repos
$json = json_decode(get_json("users/$user/repos"),true);
print_r($json);
// Sort the array returend by pushed_at time
function compare_pushed_at($b, $a){
return strnatcmp($a['pushed_at'], $b['pushed_at']);
}
usort($json, 'compare_pushed_at');
//Now just get the latest repo
$json = $json[0];
return $json;
}
function get_commits($repo, $user){
// Get the name of the repo that we'll use in the request url
$repoName = $repo["name"];
return json_decode(get_json("repos/$user/$repoName/commits"),true);
}
I use your code and it will work if you add an user agent on curl
curl_setopt($ch, CURLOPT_USERAGENT,'YOUR_INVENTED_APP_NAME');
I am trying to make a redirect php script, I want that script to check if the link exist and then redirect the user to the link, if it doesn't exist then it will get the next link and so on, but for some reason is not working, maybe you could give me some help on this.
<?php
$URL = 'http://www.site1.com';
$URL = 'http://www.site2.com';
$URL = 'http://www.site3.com';
$handlerr = curl_init($URL);
curl_setopt($handlerr, CURLOPT_RETURNTRANSFER, TRUE);
$resp = curl_exec($handlerr);
$ht = curl_getinfo($handlerr, CURLINFO_HTTP_CODE);
if ($ht == '404')
{ echo "Sorry the website is down atm, please come back later!";}
else { header('Location: '. $URL);}
?>
You are overwriting your $URL variable..
$URL = 'http://www.site1.com';
$URL = 'http://www.site2.com';
$URL = 'http://www.site3.com';
Put these urls in an array and go through it with a for each loop.
You have a few issues in your code. For 1, your $URL will overwrite itself, resulting in only 1 url in there. It needs to be an array:
array( 'http://www.site1.com', 'http://www.site2.com', 'http://www.site3.com' );
You can get many responses, not just a 404, so you should tell cURL to follow redirects. If the URL was a redirect itself, could get a 301 that redirects to a 200. So we want to follow that.
Try This:
<?php
function curlGet($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ( $httpcode == 200 ) {
return true;
}
return false;
}
$urlArray = array( 'http://www.site1.com', 'http://www.site2.com', 'http://www.site3.com' );
foreach ( $urlArray as $url ) {
if ( $result = curlGet($url) ) {
header('Location: ' . $url);
exit;
}
}
// if we made it here, we looped through every url
// and none of them worked
echo "No valid URLs found...";
http://php.net/manual/en/function.file-exists.php#74469
<?php
function url_exists($url) {
if (!$fp = curl_init($url)) return false;
return true;
}
?>
This will give you the url exists check.
to check multiple urls though, you need an array:
<?
$url_array = [];
$url_array[] = 'http://www.site1.com';
$url_array[] = 'http://www.site2.com';
$url_array[] = 'http://www.site3.com';
foreach ($url_array as $url) {
if url_exists($url){
// do what you need;
break;
}
}
?>
PS - this is completely untested, but should theoretically do what you need.
I'm trying to use curl to do a simple GET with one parameter called redirect_uri. The php file that gets called prints out a empty string for $_GET["redirect_uri"] it shows red= and it seems like nothing is being sent.
code to do the get
//Get code from login and display it
$ch = curl_init();
$url = 'http://www.besttechsolutions.biz/projects/facebook/testget.php';
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_GET,1);
curl_setopt($ch,CURLOPT_GETFIELDS,"redirect_uri=my return url");
//execute post
print "new reply 2 <br>";
$result = curl_exec($ch);
print $result;
// print "<br> <br>";
// print $fields_string;
die("hello");
the testget.php file
<?php
print "red-";
print $_GET["redirect_uri"];
?>
This is how I usually do get requests, hopefully it will help you:
// create curl resource
$ch = curl_init();
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Follow redirects
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
// Set maximum redirects
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
// Allow a max of 5 seconds.
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
// set url
if( count($params) > 0 ) {
$query = http_build_query($params);
curl_setopt($ch, CURLOPT_URL, "$url?$query");
} else {
curl_setopt($ch, CURLOPT_URL, $url);
}
// $output contains the output string
$output = curl_exec($ch);
// Check for errors and such.
$info = curl_getinfo($ch);
$errno = curl_errno($ch);
if( $output === false || $errno != 0 ) {
// Do error checking
} else if($info['http_code'] != 200) {
// Got a non-200 error code.
// Do more error checking
}
// close curl resource to free up system resources
curl_close($ch);
return $output;
In this code, the $params could be an array where the key is the name, and the value is the value.