I'm really confused by this. My company recently switched this URL to https, and I can no longer make cURL requests against it. No one else in the company does PHP, so there's no one to help me debug.
I don't have much experience with https, so I read through a bunch of posts on here and found a few that addressed this same issue, namely, this one:
Can't connect to HTTPS site using cURL. Returns 0 length content instead. What can I do?
At first, I was getting all sorts of errors related to security certificates, but after adding curl.cainfo="c:\xampp\htdocs\login\cacert.pem" to my php.ini file, I believe I have resolved those as I am not getting errors anymore, but I am still not getting back a successful login response.
As part of the process, I need to pass a cookie to the server, which is present. echo $strCookie gives back the correct cookie data, so I don't think it's that.
$searchURL = "https://url.com/valid?";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $searchURL);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$strCookie = 'testcookie=' . $_COOKIE['testcookie'] . '; path=/';
curl_setopt( $ch, CURLOPT_COOKIE, $strCookie );
curl_setopt($ch, CURLOPT_CAPATH, "\cacert.pem");
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
if($answer = curl_exec($ch))
{
echo "<pre>";
print_r($answer);
echo "</pre>";
}
else
{
echo "<pre>";
echo 'Curl error: ' . curl_error($ch);
echo "</pre>";
}
If I put the url into my browser (url.com/valid?), I get back boolean = true, so I know I am logged in. However, this script is returning boolean = false and I cannot figure out why.
Here are the headers:
TTP/1.1 200 OK
Date: Mon, 09 Nov 2015 21:33:16 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Pragma: no-cache
Content-Length: 14
Content-Type: text/plain; charset=UTF-8
X-Powered-By: Servlet/2.5 JSP/2.1
Set-Cookie: amlbcookie=01; domain=.****.com; path=/
Vary: User-Agent,Accept-Encoding
I'm confused by the Set-Cookie: amlbcookie=01 line, as I'm not sure what that cookie is. It does not match the output I get from echo $strCookie.
Related
i have big problem with login to site fileshark.pl
I use something like this
function grab_page($site){
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start();
return curl_exec ($ch);
ob_end_clean();
curl_close ($ch);
}
function post_data($site,$data){
$datapost = curl_init();
$headers[] = "Content-type: application/json";
$headers[] = "X-Requested-With: XMLHttpRequest";
curl_setopt($datapost, CURLOPT_URL, $site);
curl_setopt($datapost, CURLOPT_TIMEOUT, 40000);
curl_setopt($datapost, CURLOPT_HEADER, TRUE);
curl_setopt($datapost, CURLOPT_HTTPHEADER, $headers);
curl_setopt($datapost, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($datapost, CURLOPT_POST, TRUE);
curl_setopt($datapost, CURLOPT_POSTFIELDS, $data);
curl_setopt($datapost, CURLOPT_COOKIEFILE, "cookie.txt");
ob_start();
return curl_exec ($datapost);
ob_end_clean();
curl_close ($datapost);
unset($datapost);
}
$token = "";
$data = grab_page("https://fileshark.pl/zaloguj");
$token = cut_str($data,'_csrf_token" value="','" />'); -> take token, i need this to login post data
post_data("https://fileshark.pl/login_check","_username=MYUSER&_password=MYPASSWORD&_csrf_token={$token}");
$data = grab_page("https://fileshark.pl/");
echo $data;
but when i go to normal website i am not logged ... every time not logged :/ this is post data from website...
https://fileshark.pl/login_check
POST /login_check HTTP/1.1
Host: fileshark.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0
Accept: */*
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: https://fileshark.pl/
Content-Length: 97
Cookie: hl=pl; _ga=GA1.2.816314079.1500843124; __gfp_64b=Ocm7N2W0C.LWu6Ga3q6MIsWU1devhoy.6XNO0Bth6q3.O7; _gid=GA1.2.234598376.1502038746; PHPSESSID=j1ch3848vrg4itpd8fgl4ul1j4; _gat=1
Connection: keep-alive
_username=MYUSER&_password=MYPASSWORD&_csrf_token=xBcS0AVCVvLnN5e2puEfxd8v_Ol6a-KG0uscwqlLU98
HTTP/1.1 200 OK
Server: nginx/1.10.3
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache
Date: Sun, 06 Aug 2017 19:41:55 GMT
Set-Cookie: PHPSESSID=nsvibi4o3u5kqdc6u5rbjbi444; path=/; HttpOnly
Set-Cookie: REMEMBERME=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; HttpOnly
Anyone can help me with this? When i try to other site like catshare all working but not for this site :)
first off, you ignore any and all setopt errors, fix that (use something like
function ecurl_setopt ( /*resource*/$ch , int $option , /*mixed*/ $value ){
if(curl_setopt($ch,$option,$value)!==true){
//option should be obvious by stack trace
throw new RuntimeException ( 'curl_setopt() failed. curl_errno: ' . curl_errno ($ch).'. curl_error: '.curl_error($ch) );
}
}
)
.
second, obviously the csrf token is tied to a cookie session, which you completely ignore (you make no attempt to store nor resend any cookies), and without a cookie session in the login request, your csrf token will always be invalid, fix that (check CURLOPT_COOKIEFILE to get curl to handle cookies automatically). third, you extracts the csrf token from the html with a regex, that regex will NOT decode html entities for you (like & for &, < for <, > for >, etc), so if/when the csrf token contains any such encoded characters, you will send the wrong token, fix that (see html_entity_decode, or better yet, use a proper html parser to extract it). fourth, you don't urlencode the username, nor the password, nor the token, so if any of those contains any characters with special meaning in urlencoded, you will send the wrong credentials, and won't get logged in, fix that (this includes &, =, spaces, all non-ascii characters, and even some ascii characters). fifth, you never stop the output buffer started in the grab_page function, you put the code to stop it after the return statement, thus that code will never be executed, fix that. if, after fixing all those mentioned issues, you still can't get log in, let me know, and i'll dig deeper
Ok, to understand the problem, first please visit
http://unblockproxy.nu/
Try to surf any website, let's say (http://www.example.com/samplepage.html) put it in the field then click "unblock" button
After sending the $_POST request, the site should redirect you to something like:
http://unblockproxy.nu/index.php?x=Mfv0KjYRb3J3JO50MgBNbplFn2sTMoqPUIu1Unqn0bqdUoq5VbA9OnO8%3D
Response Headers of the browser is like:
HTTP/1.1 302 Found
Date: Fri, 06 Mar 2015 12:49:30 GMT
Server: Apache/2.2.15
x-powered-by: PHP/5.3.3
Location: http://unblockproxy.nu/index.php?x=Mfv0KjYRb3J3JO50MgBNbplFn2sTMoqPUIu1Unqn0bqdUoq5VbA9OnO8%3D
Cache-Control: max-age=600, private, must-revalidate
Expires: Fri, 06 Mar 2015 12:59:30 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
HTTP/1.1 200 OK
Date: Fri, 06 Mar 2015 12:49:34 GMT
Server: Apache/2.2.15
X-Powered-By: PHP/5.3.3
Content-Disposition: inline; filename="samplepage.html"
Cache-Control: max-age=600, private, must-revalidate
Expires: Fri, 06 Mar 2015 12:59:34 GMT
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
That's easy, now you got the contents of the surfed page by using this web proxy.
Now, i want to do the same job by using curl
My problem is, i don't know how to let curl deal with Content-Disposition of the response header
Here is some codes to simulate my problem::
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://unblockproxy.nu/index.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('x' => 'http://www.example.com/samplepage.html'));
curl_setopt($ch, CURLOPT_COOKIESESSION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
$string = curl_exec($ch);
curl_close($ch);
echo $string;
This will return the contents of http://unblockproxy.nu/ and that is not what i want (http://www.example.com/samplepage.html which surfed by http://unblockproxy.nu/)
If you want to take a look into the script of this site (2 PHP files only), you can go here
Thank you.
Try this. This works for me just fine if I'm understanding your question correctly. I removed a lot of code that did nothing. Turns out, the problems was that you weren't setting the referer in the request headers.
Let me start from the beginning. Upon submitting the form via POST to view a given website with a proxy, a request is sent to http://unblockproxy.nu/index.php. As you mentioned in your question, index.php handles the form submission and generates an HTTP status code of 302 which essentially just redirects you to another page. Assuming that you send a properly formatted request to index.php, you can parse the response headers and get the value of the redirect URL. Follow the code below to get the redirect URL.
/**
* Submit the form via POST
* #param [site_url] The link to the page that you want to view
* eg: http://sitetoget.com/page.html
* #return A string containing the response headers
*/
function GetRedirect($site_url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://unblockproxy.nu/index.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('x' => $site_url));
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
/**
* Turn a header string into an associative array
* #param [response] The response headers from the form submission
* #return An array containing all of the headers
*/
function GetHeaders($response) {
$headers = [];
$text = substr($response, strpos($response, "\r\n\r\n"));
foreach(explode("\r\n", $text) as $i => $line) {
if($i === 0 || $i == 1) {
$headers['http_code'] = $line;
} else {
list($key, $value) = explode(': ', $line);
if($key != '' && $value != '') {
$headers[$key] = $value;
}
}
}
return $headers;
}
// Get the redirect URL
$redirect = GetRedirect('http://lancenewman.me/');
// Parse the response headers
$headers = GetHeaders($redirect);
// Save the redirect URL
$new_url = $headers['Location'];
Now that you have the URL that index.php redirects to, send a cURL request to it as follows. Strangely enough, almost all of the other request headers that I've tinkered with play no role in determining whether or not this solution works. The reason your code is getting the contents of http://unblockproxy.nu instead of the contents of the given site as viewed by http://unblockproxy.nu is because you're not following the redirections correctly and you're not setting the referer in request headers. The cookies, content-disposition and all of the other headers seem to play no role in solving this.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $new_url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, 'http://unblockproxy.nu');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$string = curl_exec($ch);
curl_close($ch);
echo $string;
It's important to note that some of the images, CSS and JS on some of the pages might not properly load because some use relative URLs instead of absolute ones. Just keep that in mind.
The problem is it requires two round-trips to the server to complete the request. Many sites use the method to reduce the number or requests by "bots". The first request creates a cookie (typically for a "session") which must be present in order for the form to be processed.
Perform the curl_exec() twice and see if you get the results you want. The first time the response will send a cookie which curl will save since you have enabled cookies. The second time you should get the results you want.
I am here with a lot of hope to solve my problem. I have this website i want to login to and access a member only page. so basically php code with cURL to login and then open a member only page. Below is the code i used. But each time i try to access member only page, it asks me to login. So it does not save in the session or cookie that i already logged in.
..........
set_time_limit(0);
$username = 'myemail#yahoo.com';
$password = 'mypassword';
$loginUrl = 'http://radaris.com/login/a.login';
$cookie_file_path = getcwd() . '/tmp/cookie.txt';
//init curl
$ch = curl_init();
//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $loginUrl);
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 0);
//Set the post parameters
curl_setopt($ch, CURLOPT_HEADER, 1); // set to 0 to eliminate header info from response
curl_setopt($ch, CURLOPT_POSTFIELDS, 'email='.$username.'&password='.$password."&remember_me=1");
//Handle cookies for the login
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute the request (the login)
$store = curl_exec($ch);
echo "<font color=red>Login</font><br>".$store."<br><br>";
$postfields = array();
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://radaris.com/my/');
curl_setopt($ch, CURLOPT_HEADER, 1); // set to 0 to eliminate header info from response
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, 1);
//Handle cookies for the login
// Edit: prior variable $postFields should be $postfields;
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$result = curl_exec($ch);
echo "<font color=red>Report</font><br>".$result."<br><br>";
..........
The Result i get is :
Login HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Jun 2014 14:21:01
GMT Content-Type: application/json Transfer-Encoding: chunked
Connection: keep-alive Keep-Alive: timeout=20 Cache-Control: no-store,
no-cache, must-revalidate Cache-Control: post-check=0, pre-check=0
Expires: Wed, 14 Jan 2009 19:00:00 GMT X-Robots-Tag: noindex
{"/Rdf.showError":[["Invalid email or password","login_form"],["Check
your email","login_email"],["Check your password","login_password"]]}
Report HTTP/1.1 302 Found Server: nginx Date: Thu, 12 Jun 2014
14:21:03 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding:
chunked Connection: keep-alive Keep-Alive: timeout=20 Location:
/login?backurl=%2Fmy%2F HTTP/1.1 200 OK Server: nginx Date: Thu, 12
Jun 2014 14:21:03 GMT Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked Connection: keep-alive Keep-Alive:
timeout=20 Vary: Accept-Encoding Vary: Accept-Encoding Set-Cookie:
PHPSESSID=qdg80d48dd17hl879h86v9ruo5; path=/; HttpOnly Expires: Thu,
19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache,
must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Login We
never share or disclose your information!
The above are the headers only. in the 2nd header i get the login form again to login meaning the successful login in the first bit of my code did not save the session that is why when i access member only page, it asks to login.
I am really desperate to get this done right. Thanks everyone in advance.
You're using two separate curl handles, and you never set up the cookie stuff in the second one. Each curl handle is completely independent of each other, so if you don't tell the second handle about the cookiejar/cookiefile you created in the first one, you never actually use the cookie you got from the login request.
Technically, you do NOT have to do one-request-per-handle. A single handle can do multiple requests. The only time it might be easier to start a new one is if you made a complicated custom request with many options set, and it's easier to just start fresh rather than reset those options for the next request.
So your code sequence should be:
init curl
set up cookiefile/cookiefar and other options
do login request
do next next request
close curl
If you don't want to rewrite, then at least add the cookiejar/cookiefile options to the second request.
I would like to perform a PUT operation on a webservice using CURL. Let's assume that:
webservice url: http://stageapi.myprepaid.co.za/api/ConsumerRegisterRequest/cac52674-1711-e311-b4a8-00155d4905d3
municipality= NMBM
sgc= 12345
I've written the code below, but it outputs this error message: "ExceptionMessage":"Object reference not set to an instance of an object.". Any help would be so much appreciated. Thanks!
<?php
function sendJSONRequest($url, $data)
{
$data_string = json_encode($data);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Accept: application/json',
'X-MP-Version: 10072013')
);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
ob_start();
$result = curl_exec($ch);
$info = curl_getinfo($ch);
if ($result === false || $info['http_code'] == 400) {
return $result;
} else {
return $result;
}
ob_end_clean();
curl_close($ch);
}
$mun = $_GET['municipality'];
$sgc = $_GET['sgc'];
$req = $_GET['req']; //cac52674-1711-e311-b4a8-00155d4905d3
//myPrepaid PUT URL
echo $mpurl = "http://stageapi.myprepaid.co.za/api/ConsumerRegisterRequest/$req";
// Set Variables
$data = array("Municipality" => "$mun", "SGC" => "$sgc");
//Get Response
echo $response = sendJSONRequest($mpurl, $data);
?>
I copied your code, but changed it so it pointed at a very basic HTTP server on my localhost. Your code is working correctly, and making the following request:
PUT /api/ConsumerRegisterRequest/cac52674-1711-e311-b4a8-00155d4905d3 HTTP/1.1
Host: localhost:9420
Content-Type: application/json
Accept: application/json
X-MP-Version: 10072013
Content-Length: 37
{"Municipality":"NMBM","SGC":"12345"}
The error message you're receiving is coming from the stageapi.myprepaid.co.za server. This is the full response when I point it back to them:
HTTP/1.1 500 Internal Server Error
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 30 Aug 2013 04:30:41 GMT
Connection: close
Content-Length: 867
{"Message":"An error has occurred.","ExceptionMessage":"Object reference not set to an instance of an object.","ExceptionType":"System.NullReferenceException","StackTrace":" at MyPrepaidApi.Controllers.ConsumerRegisterRequestController.Put(CrmRegisterRequest value) in c:\\Workspace\\MyPrepaid\\Prepaid Vending System\\PrepaidCloud\\WebApi\\Controllers\\ConsumerRegisterRequestController.cs:line 190\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass13.<GetExecutor>b__c(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)\r\n at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func`1 func, CancellationToken cancellationToken)"}
You may want to check out the API to make sure you're passing them the correct information. If you are, the problem could be on their end.
And while I realize this isn't part of your question and this is in development, please remember to sanitize any data from $_GET. :)
Try with:
curl_setopt($ch, CURLOPT_PUT, true);
I'm trying to retrieve the location URL of an image after a redirect, but I can't seem to get it to work. I've never used cURL before, but from the research I've done it looks like I need it to get an image from Facebook. Here's my code so far:
function getImage($url) {
$ch = curl_init();
// Only calling the head
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true); // header will be at output
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'HEAD'); // HTTP request is 'HEAD'
$content = curl_exec($ch);
curl_close($ch);
return $content;
}
$image = explode("Location: ", getImage("http://graph.facebook.com/553451657/picture?type=large"));
print_r($image);
But my response is:
HTTP/1.1 302 Found Access-Control-Allow-Origin: * Cache-Control: private, no-cache, no-store, must-revalidate Content-Type: image/jpeg Expires: Sat, 01 Jan 2000 00:00:00 GMT Location: http://profile.ak.fbcdn.net/hprofile-ak-snc4/161877_553451657_798840908_n.jpg Pragma: no-cache X-FB-Rev: 590433 X-FB-Debug: WH1uJvIjUqiLT8ezVPdude8VKYnXjAHRlFaP8gqF9fI= Date: Fri, 13 Jul 2012 17:56:09 GMT Connection: keep-alive Content-Length: 0 Array ( [0] => 1 )
How do I JUST get the Location part ("http://profile.ak.fbcdn.net/hprofile-ak-snc4/161877_553451657_798840908_n.jpg")?
You can do a function on the headers:
$headers = [];
curl_setopt($curl, CURLOPT_HEADERFUNCTION, function ($ch, $header) use (&$headers) {
$matches = array();
if ( preg_match('/^([^:]+)\s*:\s*([^\x0D\x0A]*)\x0D?\x0A?$/', $header, $matches) )
{
$headers[$matches[1]][] = $matches[2];
}
return strlen($header);
});
(seen here)
Then you get it stored in $headers["Location"][0].
I am also, like you, trying to get the fb page image this way, it seems no PHP SDK, App priviledges or access_token is needed for that, but there might be some limitations (anybody knows?)
How do I JUST get the Location part
Getting that Location header is not you’re primary goal, right? You want to get the image from the address that’s being redirected to, correct?
So just set the option CURLOPT_FOLLOWLOCATION, and cURL will follow that redirect automatically and fetch the real image for you.