How to get info on sent PHP curl request - php

I'm trying to debug a curl request to a webservice 'getToken' endpoint.
I'm not 100% confident that the URL and the auth info is getting written in to the curl handle correctly.
I'm trying to use curl_getinfo($ch, CURLINFO_HEADER_OUT); to capture the sent request, but it doesn't give me much info. Is there a way to get more in depth diagnostics about what the actual curl request looks like?
Here's the code:
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_HEADER, 1); // just getting header to see if we got an auth token
curl_setopt($ch, CURLOPT_FILE, $fh);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, 1); // capture the header info
curl_setopt($ch, CURLOPT_VERBOSE, 1); // turn verbose on
// execute the curl request
$rh = fopen("request.txt", "w"); // open request file handle
$verbose = fopen('php://temp', 'rw+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
curl_exec($ch); // execute request
$sent_request = curl_getinfo($ch, CURLINFO_HEADER_OUT);
fwrite($rh, $sent_request); // save the request info
fclose($rh);
!rewind($verbose);
$verboseLog = stream_get_contents($verbose);
echo "Verbose information:\n<pre>", htmlspecialchars($verboseLog), "</pre>\n";
This all works as far as it goes, but returns a 401 every time-- the API admin assures me that the username / pass I have is correct.
I was wondering if I'm somehow getting the URL value wrong, or not sending the right username / pass, but this info isn't printed in the request data saved:
HEAD /export/auth HTTP/1.1
Authorization: Basic Y2FpcmRzdW5mYTpENWlAaVM4cw==
Host: webservices.mycompany.com
Accept: */*
You can see that the username/pass is not recorded (I assume for security). The endpoint URL I think is the host value plus the start of the HEAD value, so webservices.mycompany.com/export/auth?
The "Verbose Information" statement prints nothing. Not sure why on this either!
Thanks for help.
EDIT: added verbose mode from Php - Debugging Curl thanks to commenter immulatin

If you set CURLINFO_HEADER_OUT to true, outgoing headers are available in the array returned by curl_getinfo(), under request_header key:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://foo.com/bar");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, "someusername:secretpassword");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_exec($ch);
$info = curl_getinfo($ch);
print_r($info['request_header']);
This will print:
GET /bar HTTP/1.1
Authorization: Basic c29tZXVzZXJuYW1lOnNlY3JldHBhc3N3b3Jk
Host: foo.com
Accept: */*
Note the auth details are base64-encoded:
echo base64_decode('c29tZXVzZXJuYW1lOnNlY3JldHBhc3N3b3Jk');
// prints: someusername:secretpassword
Also note that username and password need to be percent-encoded to escape any URL reserved characters (/, ?, &, : and so on) they might contain:
curl_setopt($ch, CURLOPT_USERPWD, urlencode($username).':'.urlencode($password));

You can also use a proxy tool like Charles to capture the outgoing request headers, data, etc. by passing the proxy details through CURLOPT_PROXY to your curl_setopt_array method.
For example:
$proxy = '127.0.0.1:8888';
$opt = array (
CURLOPT_URL => "http://www.example.com",
CURLOPT_PROXY => $proxy,
CURLOPT_POST => true,
CURLOPT_VERBOSE => true,
);
$ch = curl_init();
curl_setopt_array($ch, $opt);
curl_exec($ch);
curl_close($ch);

curl_getinfo() must be added before closing the curl handler
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://example.com/bar");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_USERPWD, "someusername:secretpassword");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_exec($ch);
$info = curl_getinfo($ch);
print_r($info['request_header']);
curl_close($ch);

The request is printed in a request.txt with details
$ch = curl_init();
$f = fopen('request.txt', 'w');
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_VERBOSE => 1,
CURLOPT_STDERR => $f,
));
$response = curl_exec($ch);
fclose($f);
curl_close($ch);
You can also use curl_getinfo() function.

Related

PHP - with curl, how to send username password in a popup of htpasswd?

I have to submit POST method using curl, but while opening $URL it has a popup window with username/password to submit (like with htpasswd).
For some reason the following code is not working i get from server
HTTP/1.1 401 Unauthorized and HTTP/1.1 404 Not Found ( https://www.httpwatch.com/httpgallery/authentication/ )
// For Debug server response details
$curlOptions = array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_VERBOSE => TRUE,
CURLOPT_STDERR => $verbose = fopen('php://temp', 'rw+'),
CURLOPT_FILETIME => TRUE,
);
// Real meat
$ch = curl_init();
curl_setopt_array($ch, $curlOptions);
curl_setopt($ch, CURLOPT_URL,$URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
// is this the right way here in POST method???
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");// is this correct way to enter username/password in the popup?
$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$result=curl_exec ($ch);
// Pretty print the debug logs
echo '<pre>',
!rewind($verbose),
stream_get_contents($verbose),
"\n",
'</pre>';
curl_close ($ch);
echo $result;
How can i make sure my POST method submission is really submitting the username/password? (debug log i do not see if i have submitted correctly, server is also not maintained by me, as its third-party service providers API)
EDIT:
TRY1: FAILED
// For Debug server response details
$curlOptions = array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_VERBOSE => TRUE,
CURLOPT_STDERR => $verbose = fopen('php://temp', 'rw+'),
CURLOPT_FILETIME => TRUE,
);
// Real meat
$ch = curl_init();
curl_setopt_array($ch, $curlOptions);
//
//
//
//
// STACK-OVERFLOW EXPERT SAID UPDATE THE $URL WITH USER:PASS# THIS
//
//
//
//
$URL = "http://$username:$password#site.com/postblabla";
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
//
//
//
//
// STACK-OVERFLOW EXPERT SAID REMOVE THIS
//
//
//
//
//curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$result=curl_exec ($ch);
// Pretty print the debug logs
echo '<pre>',
!rewind($verbose),
stream_get_contents($verbose),
"\n",
'</pre>';
curl_close ($ch);
echo $result;
With HTTP Auth, you send a username and a password as a part of the URL:
<schema>://<username>:<password>#<url>
Also see this SO question and answer.
However, the CURLOPT_USERPWD should have worked as well as both methods only set HTTP Authorization header. as mentioned here. This is also safer as your credentials will not be transferred as plaintext in URL and with potential use of SSL/TLS they will not be visible out in the open.
Also, setting HTTP Basic auth may help.
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

How to view PHP cURL request body (like CURLINFO_HEADER_OUT for headers)

I can view the headers of a request sent using php curl with the following:
curl_getinfo($ch, CURLINFO_HEADER_OUT);
I wish to see the body of what is being sent out as well but cannot for the life of me find any way to do so.
I was unable to find any such option after extensively searching the PHP cURL documentation.
My solution was to use the web proxy tool Charles
Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information).
router.php
<?=file_get_contents('php://input');
Start embedded server
php -S 127.0.0.0:8080 ./router.php
Test cURL:
<?php
$ch = curl_init("http://127.0.0.1:8080/");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
"Content-Type: application/json",
"Expect: 100-Continue"
]);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
$fields = ["bri" => 255];
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode((object)$fields));
$response = curl_exec($ch);
print curl_getinfo($ch, CURLINFO_HEADER_OUT);
print $response;
Setting CURLOPT_HEADER to true will return the headers with the body of the response, this can then be parsed out of the response and processed seperately. Something like the following should work to print both the in and out headers:
$url = "https://www.example.com";
$ch = curl_init();
// Configure cURL handle
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$x = curl_exec($ch);
print "\nHeaders:\n";
// Get the out headers, explode into an array, and remove any empty string entries
$outHeaders = explode("\n", curl_getinfo($ch, CURLINFO_HEADER_OUT));
$outHeaders = array_filter($outHeaders, function($value) { return $value !== '' && $value !== ' ' && strlen($value) != 1; });
print_r($outHeaders);
// Seperate in headers from body of response
list($inHeaders, $content) = explode("\r\n\r\n", $x, 2);
// Break in headers into array and print_r them
$inHeaders = explode("\n", $inHeaders);
print_r($inHeaders);
Try this function:
function url_get_contents ($Url) {
if (!function_exists('curl_init')){
die('CURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}

PHP Curl Call when executed outputs an HTML content where as JSON was expected

When i do a curl call using php which is as shown below
<?php
$username = "XXXX";
$password = "XXXX";
$url = "https://domainname/method";
$ch = curl_init();
$fullAddress="202 220 GEORGE ST";
$payload = json_encode( array( "payload"=>[array("fullAddress"=>$fullAddress)],
"sourceOfTruth"=>'AUPAF'));
//var_dump($payload);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:application/json','Accept: application/json'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload );
# Return response instead of printing.
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
# Send request.
$result = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
echo '<pre>', htmlentities($result), '</pre>';
$json = json_decode($result, true);
?>
it outputs an weird response saying that
<html><head><title>Apache Tomcat/7.0.52 (Ubuntu) - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:whi
I'm actually stuck up with it for several hours so any suggestions would be helpfull
I've post a comment, but I recon that it's not very well readable.
Here is my suggestion, to use for nested arrays:
json_encode( [ 'payload' => [
['fullAddress' => $fullAddress]
],
'sourceOfTruth' => 'AUPAF'
] );
In your example your arrays where further nested, maybe the server does not expact this.
A few other tips:
Make sure the URL to the API is correct.
The headers send to the server are as expected.
the credentials you entered, are they right? might this be a permission issue?
Shouldn't the password be base64_encode'd?
Also try just printing all content, and accept text/html for a bit. Just to see what the error message might be.

Error when doing cURL request

This code always returns user doesn't exist from the API:
$data2 = array('user'=>$vars['mcusername'],
'pwd'=>$vars['mcpassword'],
'group'=>$postfields['group'],
'action'=>'Save');
// Connect to dvb API
$configWebAddress = "http://192.168.0.12:4040/dvbapi.html?part=userconfig&";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $configWebAddress);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data2);
$data = curl_exec($ch);
curl_close($ch);
The string that works in a browser is this:
dvbapi.html?part=userconfig&user=PeterTest&pwd=obfuscated&group=1,2&disabled=0&action=Save
When you access the URL in the browser, you're performing a GET. In your cURL attempt, you're attempting to POST. This is likely the issue; the script may only accept GET.
Try using this cURL code instead:
// Gather up all the values to send to the script
$data2 = array('part' => 'userconfig',
'user' => $vars['mcusername'],
'pwd' => $vars['mcpassword'],
'group' => $postfields['group'],
'action' => 'Save');
// Generate the request URL
$configWebAddress = "http://192.168.0.12:4040/dvbapi.html?".http_build_query($data2);
// cURL the URL for a responce
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $configWebAddress);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
curl_close($ch);
// Show the responce
var_dump($data);
You can use http_build_query() to turn your array into a URL-encoded string to make the GET request.

How to get page content using cURL?

I would like to scrape the content of this Google search result page using curl.
I've been trying setting different user agents, and setting other options but I just can't seem to get the content of that page, as I often get redirected or I get a "page moved" error.
I believe it has something to do with the fact that the query string gets encoded somewhere but I'm really not sure how to get around that.
//$url is the same as the link above
$ch = curl_init();
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0'
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch,CURLOPT_CONNECTTIMEOUT,120);
curl_setopt ($ch,CURLOPT_TIMEOUT,120);
curl_setopt ($ch,CURLOPT_MAXREDIRS,10);
curl_setopt ($ch,CURLOPT_COOKIEFILE,"cookie.txt");
curl_setopt ($ch,CURLOPT_COOKIEJAR,"cookie.txt");
echo curl_exec ($ch);
What do I need to do to get my php code to show the exact content of the page as I would see it on my browser? What am I missing? Can anyone point me to the right direction?
I've seen similar questions on SO, but none with an answer that could help me.
EDIT:
I tried to just open the link using the Selenium WebDriver, that gives the same results as cURL. I am still thinking that this has to do with the fact that there are special characters in the query string which are getting messed up somewhere in the process.
this is how:
/**
* Get a web file (HTML, XHTML, XML, image, etc.) from a URL. Return an
* array containing the HTTP server response header fields and content.
*/
function get_web_page( $url )
{
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_CUSTOMREQUEST =>"GET", //set request type post or get
CURLOPT_POST =>false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE =>"cookie.txt", //set cookie file
CURLOPT_COOKIEJAR =>"cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
Example
//Read a web page and check for errors:
$result = get_web_page( $url );
if ( $result['errno'] != 0 )
... error: bad url, timeout, redirect loop ...
if ( $result['http_code'] != 200 )
... error: no page, no permissions, no service ...
$page = $result['content'];
For a realistic approach that emulates the most human behavior, you may want to add a referer in your curl options. You may also want to add a follow_location to your curl options. Trust me, whoever said that cURLING Google results is impossible, is a complete dolt and should throw his/her computer against the wall in hopes of never returning to the internetz again.
Everything that you can do "IRL" with your own browser can all be emulated using PHP cURL or libCURL in Python. You just need to do more cURLS to get buff. Then you will see what I mean. :)
$url = "http://www.google.com/search?q=".$strSearch."&hl=en&start=0&sa=N";
$ch = curl_init();
curl_setopt($ch, CURLOPT_REFERER, 'http://www.example.com/1');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($ch, CURLOPT_URL, urlencode($url));
$response = curl_exec($ch);
curl_close($ch);
Try This:
$url = "http://www.google.com/search?q=".$strSearch."&hl=en&start=0&sa=N";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($ch, CURLOPT_URL, urlencode($url));
$response = curl_exec($ch);
curl_close($ch);
I suppose that have you noticed that your link is actually an HTTPS link....
It seems that CURL parameters do not include any kind of SSH handling... maybe this could be your problem.
Why don't you try with a non-HTTPS link to see what happens (i.e Google Custom Search Engine)...?
Get content with Curl php
request server support Curl function, enable in httpd.conf in folder Apache
function UrlOpener($url)
global $output;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
If get content by google cache use Curl you can use this url: http://webcache.googleusercontent.com/search?q=cache:Put your url
Sample: http://urlopener.mixaz.net/

Categories