I do a php curl to this website
http://www.hoovers.com/company-information/company-search.html
But it returned 404. Looks like something encrypted or what.
Can you give some clue about this problem.
Thanks
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://www.hoovers.com/company-information/company-search.html',
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
Looks like their web server is rejecting requests based on HTTP headers. Or it might be on the application level as well. Try this
<?php
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HEADER=>1,
CURLOPT_URL => 'http://www.hoovers.com/company-information/company-search.html',
CURLOPT_HTTPHEADER=> array(
'User-Agent: Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0', 'Accept-Language: en-US,en;q=0.5'
)
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
//debug
print_r($resp);
?>
Related
I can't get the folling script to work:
I'm using an api called swiftdil. Their example is as follows:
Example request:
curl -X POST https://sandbox.swiftdil.com/v1/oauth2/token \
-H 'Content-Type: application/x-www-form-urlencoded' \
-u 'your_username:your_password'
Example output:
{
"access_token":"your_access_token",
"expires_in": 3600,
"refresh_expires_in": 1800,
"refresh_token": "your_refresh_token",
"token_type": "bearer",
"not-before-policy": 0,
"session_state": "your_session_state"
}
So the url I've to submit my credentials to is https://sandbox.swiftdil.com/v1/oauth2/token
I've tried the following code:
// Api Credentials
$url = 'https://sandbox.swiftdil.com/v1/oauth2/token';
$username = "my_username";
$password = "my_password";
// Set up api environment
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:
application/x-www-form-urlencoded'));
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" .
$password);
// Give back curl result
$output = curl_exec($ch);
$info = curl_getinfo($ch);
$curl_error = curl_error($ch);
curl_close($ch);
print_r($output);
print_r($info);
print_r($curl_error);
?>
The script is giving me back the following result:
HTTP/1.1 400 Bad Request Server: nginx/1.13.8 Date: Tue, 15 May 2018 09:17:26 GMT Content-Type: text/html Content-Length: 173 Connection: close
400 Bad Request.
Am I missing something? I do fullfill the needs of the example given above right? I do make a postcall, give all the credenatials as asked, but still can't get anything back.
I am not a PHP developer, I mostly do JavaScript. When I integrate with other REST services I tend to use Postman (https://www.getpostman.com/).
Try the following:
Attempt to successfully connect with the API using Postman (should be relatively straightforward).
When successful, Postman has the ability to generate PHP code automatically, which you can then copy and paste. Works like a charm with JavaScript, don't see why it will be any different with PHP.
I just filled in the details in postman based on what you provided:
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://sandbox.swiftdil.com/v1/oauth2/token",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_HTTPHEADER => array(
"Authorization: Basic bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=",
"Content-Type: application/x-www-form-urlencoded"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Please note, 'Authorization: Basic' can be used as basic authorization mechanism instead of 'Bearer' (it should work too). So replace 'bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ' with the base64 encoded string 'username:password' (use actual username and password).
You also need to set the curl post fields by setting the below option as per your data.
"curl_setopt($ch, CURLOPT_POSTFIELDS, array(
'code' => $code,
'client_id' => $client_id,
'client_secret' => $client_secret,
'redirect_uri' => $redirect_uri,
'grant_type' => 'authorization_code'
))";
If still not work, you can find the curl error as :
if(curl_error($ch))
{
echo 'error:' . curl_error($ch);
}
I want to fetch response of a GET request by CURL, but get the Error "HTTP/1.1 405 Method Not Allowed" and no content is returned (except the header).
The following is my code.
$url = "http://api.example.com/Q5PLCmwYzho=/7avEU7ptYyummfheg9!0KA==?aqPI=93566&aqIIO=false&aqIP=false&aqSK=0&aqTK=14&aqSO=date&aqCI=0"
$options = array(
CURLOPT_URL => $url,
CURLOPT_HTTPGET => true, //set request type post or get
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36',
CURLOPT_HTTPHEADER => array(
'Accept: application/json, text/javascript, */*; q=0.01',
'Accept-Encoding: gzip, deflate, sdch',
//'Content-Type: application/json; charset=utf-8',
//'Access-Control-Allow-Origin: http://localhost:14189',
//'Origin: http://www.example.com',
//'Host: api.example.com',
//'Access-Control-Allow-Credentials: true',
),
CURLOPT_COOKIE => 'Cookie:scarab.profile=%2293566%7C1465411024%22; scarab.mayAdd=%5B%7B%22i%22%3A%2293566%22%7D%2C%7B%22i%22%3A%22117313%22%7D%5D; _ceg.s=o8gwtw; _ceg.u=o8gwtw; _gat=1; _dc_gtm_UA-13212406-1=1; DK-Client=CWUU,09b6ab76-f662-48b3-a1e6-f08779519236; __auc=764bbe2e155314db7ac1a950571; scarab.visitor=%22566B1DCD74FCD3A5%22; _ga=GA1.2.412464545.1465411024'
);
$ch = curl_init();
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err; //no error!
$header['errmsg'] = $errmsg;
$header['content'] = json_decode($content); // no content!
echo $response['http_code']; //output: 405
I checked the corresponding ajax request in the webpage and every curl options seems to be set correctly. The details (captured from chrome Dev tools) are shown in the image.
![1]: http://i.stack.imgur.com/XW8k1.png
What am I missing?? I searched a lot but couldn't solve the problem! I know one of the causes of this error is using an unauthorized request method for curl but the corresponding AJAX call uses GET method and get the result.
What is wrong? Any help would be appreciated.
Update: I wrote an AJAX request to fetch the data in local server (localhost) and got this error : "Cross-Origin Request Blocked". It means that I can't make request to a different domain (api.example.com) from localhost. Are there any workarounds or hack to get the response content from that server?
I was getting this error in Postman. In my case it was because I was using GET when I should have used POST. Also I was using GET params instead of POST params. Checking the proper method (specified in my java code) and using the "code" button in postman to display the curl CLI helped me.
Try to send User-Agent in your request.
If page that you requested needs any authorization then you need to send cookies with auth data.
I'm triying to register on a website using PHP CURL. Everything is okay, but when I execute my code I get an error from the host:
HTTP/1.1 412 Precondition Failed
Date: Mon, 15 Feb 2016 20:54:58 GMT
Server: Varnish
X-Varnish: 317635174
Content-Length: 0
Array ( [header] => 1 [body] => [res] => 1 )
After doing some research on this website, I've found this:
If you look at RFC 2616 you'll see a number of request headers that
can be used to apply conditions to a request:
If-Match If-Modified-Since If-None-Match If-Range If-Unmodified-Since
These headers contain 'preconditions', allowing the client to tell the
server to only complete the request if certain conditions are met. For
example, you use a PUT request to update the state of a resource, but
you only want the PUT to be actioned if the resource has not been
modified by someone else since your most recent GET.
The response status code 412 (Precondition Failed) is typically used
when these preconditions fail.
(source: When is it appropriate to respond with a HTTP 412 error?)
So I've added these headers
<?php
function register() {
$curl = curl_init();
$post = "name=username&email=".urlencode("email#email.com")."&password=thepassword&repassword=thepassword&parrain=test";
$useragent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36';
curl_setopt($curl, CURLOPT_URL, '[the website]');
curl_setopt($curl, CURLOPT_POST, "5");
curl_setopt($curl, CURLOPT_POSTFIELDS, $post);
curl_setopt($curl, CURLOPT_USERAGENT, $useragent);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Connection: keep-alive",
"Content-Length: 43",
"Cache-Control: max-age=0",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Upgrade-Insecure-Requests: 1",
"Content-Type: application/x-www-form-urlencoded",
"Accept-Encoding: gzip, deflate",
"Accept-Language: fr,fr-FR;q=0.8,en;q=0.6,en-US;q=0.",
"If-Match: 1",
"If-Modified-Since: 1",
"If-None-Match: 1",
"If-Range: 1",
"If-Unmodified-Since: 1"
));
curl_setopt($curl, CURLOPT_HEADER, true);
$result = curl_exec($curl);
$header_size = curl_getinfo($curl, CURLINFO_HEADER_SIZE);
$header = substr($result, 0, $header_size);
$body = substr($result, $header_size);
curl_close($curl);
return array(
"header" => $header,
"body" => $body,
"res" => $result
);
}
print_r(register());
?>
but It doesn't work. How can I solve it?
Generally, if you're interacting with a website that does authentication, you will need the cookiejar parameters for cURL to save session info. If you don't send session info back to the host it will most likely cause problems with your registration.
Here's a class I use to authenticate users remotely via cURL.
/* Makes an HTTP request
* #param String $url - The URL to request
* #param Mixed $params - string or array to POST
* #param String - filename to download
*/
public static function request($url, $params = array(), $filename = "") {
// Initiate cURL
$ch = curl_init();
$curlOpts = array(
CURLOPT_URL => $url,
CURLOPT_USERAGENT =>
'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true
);
// Send the cookies if we're logged in
if (!empty(self::$cookiejar)) {
$curlOpts[CURLOPT_COOKIEJAR] = self::$cookiejar;
$curlOpts[CURLOPT_COOKIEFILE] = self::$cookiejar;
}
// If $filename exists, save content to file
if (!empty($filename)) {
$file2 = fopen($filename, 'w+') or die("Error[" . __FILE__ . ":" . __LINE__ . "] Could not open file: $filename");
$curlOpts[CURLOPT_FILE] = $file2;
}
// Send POST values if there are any
if (!empty($params)) {
$curlOpts[CURLOPT_POST] = true;
$curlOpts[CURLOPT_POSTFIELDS] = is_array($params) ?
http_build_query($params) : $params;
}
// Send the request
curl_setopt_array($ch, $curlOpts);
$answer = curl_exec($ch);
// Errors?
if (curl_error($ch)) die($url . " || " . curl_error($ch));
// Close connection and return response
curl_close($ch);
if(!empty($filename)) fclose($file2);
return $answer;
}
1: curl_setopt($ch, CURLOPT_HTTPHEADER, array());
needs $curl instead of $ch.
2: curl_setopt($curl, CURLOPT_POST, "5");
doesn't expect 5, it needs TRUE or FALSE, see curlopt_post (php.net).
2.1: CURLOPT_FOLLOWLOCATION also expects TRUE or FALSE.
Edit: My mistake, 1 and 0 are booleans too
I am trying to fetch a response from here (example url), and first, I thought I should use file_get_contents()
When I tried this, I got the following error:
Warning: file_get_contents(https://steamcommunity.com/market/pricehistory/?country=US¤cy=1&appid=730&market_hash_name=SG%20553%20|%20Damascus%20Steel%20(Factory%20New)): failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request
I know this is because it is converting & to &. I have tried numerous ways to counter this, however they have all failed and after a quick google I came to the conclusion that file_get_contents() converts & to & automatically.
My next step was to try curl. I tried the below code first:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://steamcommunity.com/market/pricehistory/?country=US¤cy=1&appid=730&market_hash_name='.$hash,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) ChromePlus/4.0.222.3 Chrome/4.0.222.3 Safari/532.2'
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
But this returned ‹ŠŽÿÿ)»L as the response. I wondered if this was to do with json encoding, so I tried putting it through json_decode() but it didn't work.
Next, I tried:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://steamcommunity.com/market/pricehistory/',
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) ChromePlus/4.0.222.3 Chrome/4.0.222.3 Safari/532.2',
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array(
country => "US",
currency => 1,
appid => 730,
market_hash_name => "SG%20553%20|%20Damascus%20Steel%20(Factory%20New)"
)
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
But again got the response ‹ŠŽÿÿ)»L.
What does this response mean, and can I parse it? If not, how should I correctly fetch this data? Furthermore, why didn't file_get_contents() work?
I'm pretty sure this is happening because you need some type of access token to access the steam web API.
See this answer on SO.
Essentially, Steam is returning an error with the "400 Bad Request" status. This error can be ignored, however, by doing this:
<?php
$url = "https://steamcommunity.com/market/pricehistory/?country=US¤cy=1&appid=730&market_hash_name=SG%20553%20%7C%20Damascus%20Steel%20(Factory%20New)";
$context = stream_context_create(array(
'http' => array(
'ignore_errors'=>true,
'method'=>'GET'
// for more options check http://www.php.net/manual/en/context.http.php
)
));
$response = file_get_contents($url, false, $context);
echo $response; // returns "[]"
?>
Make sure you take a look at this answer on SO.
May be your response is gzip, try to use CURLOPT_ENCODING.
curl_setopt($curl ,CURLOPT_ENCODING, '')
If you use https don't forget to disable CURLOPT_SSL_VERIFYPEER.
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false)
One thing, if I follow your link with my browser and open my debug console.
I see you request have a 400 Status code (Bad Request).
I cant say about your enpoint, but you can get around your Bad Request error by using urlencode():
$url = urlencode('https://steamcommunity.com/market/pricehistory/?country=US¤cy=1&appid=730&market_hash_name=SG%20553%20%7C%20Damascus%20Steel%20(Factory%20New))'
file_get_contencts($url);
I am trying to make a cURL request. The problem I am facing is that the page have different text depending on which country it is. So I would like the cURL request to have the language en_US (English). So it will get the English text on the website.
Currently I have this code, but its not getting the US text.
$url = 'http://testurl.com'; // Not the real URL
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_HTTPHEADER => array("Accept-Language: en-US;q=0.6,en;q=0.4"),
CURLOPT_USERAGENT => Array("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.15) Gecko/20080623 Firefox/2.0.0.15"),
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
echo htmlspecialchars($content);
So to make this simple, I would like the cURL request to send the request with the US language, if possible.
Right now it has the language 'dutch' I think this is because my hosting server is located in Netherlands. So therefore it is deutch. But I would like to change it to English.