I am trying to figure out why pasing a custom header is resulting in a 400 BAD REQUEST from the server.
$headers = array(
'API KEY: asdf',
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0" );
curl_setopt($ch, CURLOPT_URL, 'http://url');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1 );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'stuff');
curl_setopt($ch, CURLOPT_COOKIEFILE, './tmp/cookie.txt');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_PROXYPORT, '0000');
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_PROXY, '0.0.0.0');
$result = curl_exec($ch);
curl_close($ch);
I thought that using CURLOPT_HTTPHEADER would add a custom header to the request, but I'm now wondering whether it's simply overriding everything else I set?
There are more reasons a server will give a 400 response than just a header value. Without more information about the endpoint it's difficult to say what's causing the 400 response. With the exception of the extra "," in the headers array in the example the code looks okay. cURL Options
Related
I am Trying to Make A PHP Curl Request to a site that apparently seems to be protected agains Cross-Site Forgery
and have done all my best to edit the headers, parameters and cookies to resemble that of a browser but I still keep getting ERROR 400 (Bad Request) Please I would like to know how to make this work with php curl the code below works on site with and without the crsf protection but doesn't work in the one i am currently trying to access
$url="https://candidate.scholastica.ng/candidate/login";
$cookie_file_path = "cookie.txt";
// I first load the page to save the cookie and the request token
// Below are the required curl settings that works in other sites
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
//set the cookie the site has for certain features, this is optional
// curl_setopt($ch, CURLOPT_COOKIE, "cookiename=0");
curl_setopt($ch, CURLOPT_USERAGENT,"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
curl_setopt($ch, CURLOPT_ENCODING,'gzip, deflate');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// curl_setopt($ch, CURLOPT_REFERER, "http://www.google.com/bot.html");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$headers = array("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.8)
Gecko/20061025 Firefox/1.5.0.8","origin:http://www.google.com/bot.html","accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "accept-language:en-US,en;q=0.5","accept-encoding:gzip, deflate","connection:keep-alive","upgrade-insecure-requests:1","keep-alive:300","accept-charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_POST, 1);
// curl_setopt($ch, CURLOPT_POSTFIELDS, "");
$details = curl_exec($ch);// I load the Page containing the form
// I remove all newline and return statements from the form to make my regex simple for me
$pattern = '/\r\n*/m';
$replace = '';
$details = preg_replace( $pattern, $replace, $details);
// Regular Expression to grab request token from form page
$pattern = '/<input.+?name="__RequestVerificationToken".+?value="(.*?)".+?>/m';
preg_match($pattern, $details, $matches);
$token = $matches[1];
// I set up urlencoded post form values and get content length
$field1 = urlencode("__RequestVerificationToken") ."=". urlencode($token);
$field2 = urlencode("Username") ."=". urlencode("example#gmail.com");
$field3 = urlencode("Pwd") ."=". urlencode("examplepassword");
$fields = "{$field1}&{$field2}&{$field3}";
$content_length = strlen($fields);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
// $headers = array("Host:candidate.scholastica.ng",
// "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0",
//I Modify header to add content length
$headers = array("Host:candidate.scholastica.ng","User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0","Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","Accept-Language:en-US,en;q=0.5","Accept-Encoding:gzip, deflate, br","Connection:keep-alive","Upgrade-Insecure-Requests:1","Origin:https://candidate.scholastica.ng","Referer:https://candidate.scholastica.ng/candidate/login","content-type:application/x-www-form-urlencoded","TE:Trailers","content-length:{$content_length}");
// I assign the headers and set the post fields
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
I make request with existing curl thereby sending back the set cookie
$details = curl_exec($ch);
echo $details;
curl_close($ch);
This still returns a Bad Request
Your Response will be Greatly appreciated
I am trying to scrape data from one website. But the problem is when i send CURL request google recaptcha is shown every time.
How i can remove this security check.
Here is my code:
$url = 'https://www.zillow.com/browse/homes/ca/';
$data = array(
'secret' => "my-secret",
'response' => "my-response"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 30); //timeout after 30 seconds
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$result=curl_exec ($ch);
curl_close ($ch);
echo '<pre>';
print_r($result);
echo '</pre>';
exit;
This is not possible (at least not by ease)
The usecase for reCAPTCHA is to exactly prohibit what are you doing.
However some sites are only requesting reCAPTCHA when no user agent header was sent.
Try
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'
));
This will let the site think you are using Firefox 47 on Windows 7
I am getting curl http code as 403:
The user myuser was denied access to perform the operation on the object defect due to the following reasons: The permission setting for operation Create doesn't allow user to perform the required operation on the secured object.
Same code was working fine with 'HTTP', now it is 'HTTPS', so I updated code like added curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); and curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); these lines and it is giving 403 as HTTP CODE. Can someone help me where I am wrong.
$ch = curl_init(MY_BASE_URL);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13');
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_COOKIEFILE, MY_COOKIES);
curl_setopt($ch, CURLOPT_COOKIEJAR, MY_COOKIES);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/json; charset=utf-8","Accept:application/json, text/javascript, */*; q=0.01"));
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($myfields));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response=curl_exec($ch);
$code=curl_getinfo($ch, CURLINFO_HTTP_CODE);
if($code!=201)
{throw new Exception("Error creating defect: ".$response);}
else{echo 'Success';}
try this user agent(if not works then try with random user agents)
$useragent= "curl/7.39.0";
curl_setopt($ch,CURLOPT_USERAGENT, $useragent);
I am trying to download a pdf using cURL and am getting stuck on a "Server Error in '/' Application" page. My code:
$url = "https://some.domain.com/Reports/Report?ReportID=123456"
$ch = curl_init($url);
$header = array ('Host: some.domain.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, implode(':', $arrayCiphers));
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_COOKIE, "ASP.NET_SessionId=XXXXXXXX; __RequestVerificationToken_XXXXXXX=lots-of-alpha-numeric-characters");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://some.domain.com");
$output = curl_exec($ch);
echo $output;
curl_close($ch);
Is there something else I can try or some more debugging I can do?
[edit] Apparently it's caused by one of my parameters. There are several parameters in the URL. &Flag=True seems to be causing the error. If I change it to &Flag=False I get a blank page.
This error was being caused because my curl request header did not include cookies that were required to download the file. Adding those cookies in the header fixed it.
I'm trying to get an AJAX response by posting parameters through PHP
CURL POST request
I am in need to getting certain results from a website(data
crawling).
The website is using AJAX calls to get the populate results inside.
I had inspected the AJAX request made in the site through firebug and when I 'POST'ed the same request URL through RESTClient debugger,I got the results as a JSON.
Then I tried to POST this request URL through PHP CURL POST method, but I'm getting some invalid output.
My Code
<?php
$ch = curl_init();
$url = 'http://www.example.com/Hotel-Search?inpAjax=true&responsive=true';
$fields = array(
'endDate'=>'04/15/2014',
'hashParam'=>'b2c21a315f0a0fb3f0c39f60XXXXXX',
'startDate'=>'04/14/2014',
);
$headers = array(
'Accept: application/json, text/javascript,text/html'
);
$agent = ' Mozilla/5.0 (Windows NT 5.1; rv:28.0) Gecko/20100101 Firefox/28.0';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($fields));
$result = curl_exec($ch) or die(curl_error($ch));
var_dump($result);
?>
OUTPUT
**string(20) "�"**
EDIT
With reference to the suggestion below, I have added
curl_setopt($ch, CURLOPT_ENCODING, "");
Now I'm getting error as follows,
Couldn't resolve host 'www.example.com'
Looks like its gzip content to me. Use this header to handle this:
curl_setopt($ch, CURLOPT_ENCODING, "");
add this curl_setopt($ch, CURLOPT_HTTPHEADER, array('x-requested-with' => 'XMLHttpRequest'), # don't return headers