Authenticated curl not working in php - php

I am trying to create a simple PHP script that calls two different REST APIs on two different domains. Both services are HTTPS and require authentication. When I do a curl from the terminal, I get the response in JSON for both domains and everything works beautifully:
curl --user “myuser:mypassword” https://www.example.com/rest/api/2/projects
Notice that it's a GET, not a POST.
The strange thing is that when I try the exact same curl commands from my PHP script neither of them works.
This what happens:
The first domain returns an empty JSON array with no errors. Just this: []
The second domain returns this error in JSON:
{"errors":[
{
"context":null,
"message":"You are not permitted to access this resource",
"exceptionName":"com.atlassian.stash.exception.AuthorisationException"
}
]}
Here's what's NOT happening:
No SSL certificate errors or warnings
No authentication errors.
Even if put in a bad username or password, both services will act exactly the same way.
To me what's suspicious is that both domains don't authenticate my calls which makes me think there's either a problem with my code or in the php curl library.
Here's my code:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $link3);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$encodedAuth = base64_encode($username.":".$password);
curl_setopt($curl, CURLOPT_HTTPHEADER, array("Authentication : Basic ".$encodedAuth));
curl_setopt($curl, CURLOPT_USERPWD, $username.":".$password);
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION,true);
curl_setopt($curl, CURLINFO_HEADER_OUT, true);
$status_code = curl_getinfo($curl, CURLINFO_HTTP_CODE); //get status code
I know some of it is redundant, but I wanted to try everything and nothing works. Any ideas?
My environment:
OS X Yosemite (10.10.2)
PHP 5.6.6 (I manually upgraded to the latest version as an attempt to make this work)

The current code mixes various approaches and does it in a conflicting way:
the authentication header is named Authorization: and not Authentication:
the CURLOPT_HTTPAUTH, CURLAUTH_ANY tries to negotiate with the server and it shouldn't (neither does the working cURL command line)
Just use:
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $link3);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_USERPWD, $username.":".$password);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION,true);
# for debugging/non-prod
#curl_setopt($curl, CURLOPT_VERBOSE, true);
#curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$result = curl_exec($curl);
curl_close($curl);
echo $result;

Related

set cookie throw curl request

I want to set cookie throw curl request. I used this code but the requested URL return You must enable Javascript and accept cookies. what Im doing wrong here?
cookie.txt file is 0644 permission
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $base);
curl_setopt ($curl, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($curl, CURLOPT_REFERER, $base);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$str = curl_exec($curl);
curl_close($curl);
The problem is not your code, it's that the page you're trying to access works only by executing javascript. CURL doesn't support that, it just downloads the HTML code of the page, but doesn't execute any javascript.
If you're in the need of retrieving information from a website that needs to execute javascript, you need to rely on solutions that provide headless browsers, like Selenium

Clear Splash Browser Cache

I'm trying to visualize a website speed analysis. As a headless browser I use Splash 3.2 ... unfortunately I have problems getting a correct har file.
The first request looks good, after the second request I get only requests that were not cached.
I tried to empty the cache with a post request to the _gc endpoint, unfortunately without success.
My curl requests:
$url = 'http://localhost:8050/render.har?url=' . esc_url( $url ) .'&response_body=1&wait=5&timeout=10';
$cache_url = 'http://localhost:8050/_gc';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $cache_url);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS,"cached_args_removed=1");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec($curl);
curl_reset($curl);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_REFERER, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
$data = curl_exec($curl);
curl_close($curl);
the result of _gc request:
{"cached_args_removed": 0, "pyobjects_collected": 1269, "status": "ok"}
Afterwards I tried to start Splash with --disable-browser-caches to get a correct output, but Splash does not cache anything and therefore makes many requests to the same files, if they occur several times.
Is there another way to flush the browser cache before rendering, or should I prefer to use another headless browser (recommendation)?
#Tobias
You mentioned you're using splash version 3.2
I'm the author of PR 821 which introduced --disable-browser-caches , and according to the changelog , this feature landed in splash version 3.3.
So please upgrade to splash version 3.3 and you should be able to use that feature.

Basic web service with authentication using php curl

Need to consume some data from a web service that requires a username/password for access.
The following returns NULL
$service_url = 'https://example.com/2365139.json';
$curl = curl_init($service_url);
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($curl, CURLOPT_USERPWD, "username:password");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_POSTFIELDS, $curl_post_data);
$curl_response = curl_exec($curl);
$response = json_decode($curl_response);
curl_close($curl);
var_dump($response);
When I hit https://example.com/2365139.json in a browser, it prompts for un/pw and when I enter them it displays the JSON, so the data is there but something I have written above isn't working.
Original code works fine, but because the resource is https it requires the following option;
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
The caveat to using this is "This basically causes cURL to blindly accept any server certificate, without doing any verification as to which CA signed it, and whether or not that CA is trusted. If you’re at all concerned about the data you’re passing to or receiving from the server, you’ll want to enable this peer verification properly." - Taken from http://unitstep.net/blog/2009/05/05/using-curl-in-php-to-access-https-ssltls-protected-sites/

cURL to get response from Google Safe Browsing Lookup

I'm trying to get response using the Google Safe Browsing Lookup API like this:
$url ="https://sb-ssl.google.com/safebrowsing/api/lookup?client=myappname&apikey=mykey&appver=1.0&pver=3.0&url=".urlencode($myurl);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
$body = curl_exec($ch);
$info = curl_getinfo($ch);
I know that my URL is correct since if I dump it and then paste it to the browser I get the expected result (for example 'malware').
So I am assuming it must be something with cURL. I'm working on localhost and extension=php_curl.dll is un-commented in my php.ini, Php version 5.4.4
The html_code is always 0
Simply set CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST to false.
Anything wrong with my cURL code (http status of 0)?
Also check http://code.google.com/p/twitter-api/issues/detail?id=1291 , it might help. It is different APIs, but with the same problem anyway.
The problem and its solution is very nicely described here:
http://richardwarrender.com/2007/05/the-secret-to-curl-in-php-on-windows/
Basically if you are not using a standalone version of cURL the chances are that the cURL functions do not include a certificate bundle which was needed in my case since I was trying to connect to secure host.
$ch = curl_init();
// Apply various settings
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_CAINFO, "C:/xampp/ca-bundle.crt"); //path to the CA-bundle
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec ($ch);
curl_close($ch);

Making GET requests to the Meetup API in PHP

I am using PHP's build in cURL library to make GET requests to the Meetup API. This is an example of a query I'm running to view every meetup group 25 miles from central park:
https://api.meetup.com/groups.json/?lat=40.75&lon=-73.98999786376953&order=members&page=200&offset=0&key=MY_API_KEY
This query works correctly when passed to the browser, it returns the excepted 200 largest groups.
When I run this in a PHP script I'm using cURL set with these options
curl_setopt($cURL, CURLOPT_URL, $groups_url);
curl_setopt($cURL, CURLOPT_HEADER, 1);
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, 1);
$json_string = curl_exec($cURL);
I am hoping to be able to get the cURL to execute and return a json string that I can parse, but for some reason I do not understand, the result of curl_exec is always NULL, I am not sure why an input that works in the browser will not work in a script, this could just be me being dumb. Thank you for your help in advance.
this is becuase its https [SSL].
so the quick fix is to add this line
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, 0);
here example of it all working
$cURL = curl_init();
curl_setopt($cURL, CURLOPT_URL, "https://api.meetup.com/groups.json/?lat=40.75&lon=-73.98999786376953&order=members&page=200&offset=0&key=MY_API_KEY");
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, 0);
$json_string = curl_exec($cURL);
echo $json_string;

Categories