I am trying to get the response code from the response header using cURL PHP.
When I send the request, this is the response header that is returned by MYOB AccountRight API:
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Request-Context
Cache-Control: must-revalidate, private
Content-Encoding: gzip
Content-Type: application/json;charset=utf-8
Date: Thu, 20 May 2021 01:07:56 GMT
ETag: "XXXXXXXXX"
Expires: -1
Request-Context: appId=cid-v1:a4936349-ef26-4f8a-9268-XXXXXXXXX
Server: Microsoft-IIS/10.0
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Mashery-Message-ID: 2fc6b494-54e8-43e2-8bc4-XXXXXXXXX
X-Mashery-Responder: prod-j-worker-ap-southeast-2b-33.mashery.com
x-myobapi-elapsed: 1370
x-myobapi-requestid: bb0764c8-f62d-4848-bcae-XXXXXXXXX
X-Powered-By: ASP.NET
Content-Length: 1205
Connection: keep-alive
I have tried the solution from Getting HTTP code in PHP using curl , but I will not get the http code.
This is my code to get the accounts data:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'https://ar1.api.myob.com/accountright/766d620e-a5eb-41c3-8343-XXXXXXXX/GeneralLedger/Account?$filter=Name%20eq%20\'Inventory\'%20or%20Name%20eq%20\'Cost%20Of%20Sales\'%20or%20Name%20eq%20\'Inventory%20Income\'',
CURLOPT_HEADER => true,
CURLOPT_NOBODY => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'x-myobapi-version: v2',
'Accept-Encoding: gzip,deflate',
'x-myobapi-key: '.$theAPIKey,
'x-myobapi-cftoken: '.$theCFToken,
'Authorization: Bearer '.$theAccessToken
)
));
$response = curl_exec($curl);
$theInfo = curl_getinfo($response);
$http_code = $theInfo['http_code'];
curl_close($curl);
echo 'http code: ' . $http_code . '<br />';
echo '<pre>';
echo $response;
echo '</pre>';
When I echo the http code, nothing will be printed.
I think you need to pass $curl to the curl_getinfo method, not the $response
$response = curl_exec($curl);
$theInfo = curl_getinfo($curl);
$http_code = $theInfo['http_code'];
You can see the doco here.. https://www.php.net/manual/en/function.curl-getinfo.php
Here's the URL: https://www.grammarly.com
I'm trying to fetch HTTP headers by using the native get_headers() function:
$headers = get_headers('https://www.grammarly.com')
The result is
HTTP/1.1 400 Bad Request
Date: Fri, 27 Apr 2018 12:32:34 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 52
Connection: close
But, if I do the same with the curl command line tool, the result will be different:
curl -sI https://www.grammarly.com/
HTTP/1.1 200 OK
Date: Fri, 27 Apr 2018 12:54:47 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 25130
Connection: keep-alive
What is the reason for this difference in responses? Is it some kind of poorly implemented security feature on Grammarly's server-side or something else?
It is because get_headers() uses the default stream context, which basically means that almost no HTTP headers are sent to the URL, which most remote servers will be fussy about. Usually the missing header most likely to cause issues is the User-Agent. You can set it manually before calling get_headers() using stream_context_set_default. Here's an example that works for me:
$headers = get_headers('https://www.grammarly.com');
print_r($headers);
// has [0] => HTTP/1.1 400 Bad Request
stream_context_set_default(
array(
'http' => array(
'user_agent'=>"php/testing"
),
)
);
$headers = get_headers('https://www.grammarly.com');
print_r($headers);
// has [0] => HTTP/1.1 200 OK
Just use php curl function for it:
function getMyHeaders($url)
{
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_USERAGENT => "spider",
CURLOPT_AUTOREFERER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_NOBODY => true
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
curl_close($ch);
return $content;
}
print_r(getMyHeaders('https://www.grammarly.com'));
i can't get it to connect my php to my spiders
$url = 'https://app.scrapinghub.com/api/jobs/list.json';
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_HEADER => 1,
CURLOPT_URL => $url,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array(
'project' => 'projecid',
'spider' => 'spiderid',
),
CURLOPT_USERPWD => "apikey")
);
$response = curl_exec($ch);
curl_close($ch);
echo '<pre>';
print $response;
echo '</pre>';
it returns:
HTTP/1.1 100 Continue
HTTP/1.1 400 method not allowed
Server: nginx/1.10.1
Date: Fri, 07 Jul 2017 07:24:02 GMT
Content-Type: application/json
Content-Length: 57
Connection: keep-alive
Vary: Cookie
Set-Cookie: csrftoken2=67xgaAQB9ytsIYNxseDFAIUzkCivPZMda74Hvg7UNMp6iD3zALRDWP6zhknxiEIP; Domain=.scrapinghub.com; expires=Fri, 06-Jul-2018 07:24:02 GMT; Max-Age=31449600; Path=/; Secure
X-Upstream: dash-master_apiv1
{"status": "badrequest", "message": "method not allowed"}
it says bad request means i am wrong with my url but in the docs it said it should be like these or i am wrong on sending my apikey.
please help me here thanks.
You are sending a POST request to scrapinghub which is the message of your badrequest status as method not allowed. instead you can try sending a GET request through PHP curl like so
$url = 'https://app.scrapinghub.com/api/jobs/list.json';
$ch = curl_init();
$curl_opts = array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_HEADER => 1,
CURLOPT_URL => $url."?project=projectid&spider=spiderid",
CURLOPT_USERPWD => "apikey"
);
curl_setopt_array($ch, $curl_opts);
$response = curl_exec($ch);
curl_close($ch);
echo '<pre>';
print $response;
echo '</pre>';
The following script returns "HTTP/1.1 401 Unauthorized" by requesting, but i am not sure why. I know, the request goes to a https, but i "denied" the option "CURLOPT_SSL_VERIFYPEER".. and i think, that's not the problem at all, is it?
<?php
error_reporting(E_ALL);
ini_set('display_errors', 1);
$html_brand = "https://example.com/api/test";
$ch = curl_init();
$options = array(
CURLOPT_URL => $html_brand,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_HEADER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPAUTH => CURLAUTH_DIGEST,
CURLOPT_USERPWD => "user:pass",
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json; charset=utf-8'
)
);
curl_setopt_array( $ch, $options );
$response = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ( $httpCode != 200 ){
echo "Return code is {$httpCode} \n"
.curl_error($ch);
echo "<pre>";
print_r($response);
} else {
echo "<pre>".htmlspecialchars($response)."</pre>";
}
curl_close($ch);
I think, there is just one more option for the curl missing..
Response:
HTTP/1.1 401 Unauthorized
Server: nginx
Date: Tue, 19 May 2015 18:52:26 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=5
Www-Authenticate: Digest realm="REST-API", domain="/", nonce="", opaque="", algorithm="MD5", qop="auth"
Cache-Control: nocache, private
Vary: Accept-Encoding
HTTP/1.1 400 Bad Request
Server: nginx
Date: Tue, 19 May 2015 18:52:26 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=5
Cache-Control: nocache, private
Vary: Accept-Encoding
May it's kind of stupid, but is it something about the auth algorithm - md5? The password is in plain-text and not encrypted by md5.
EDIT: It seems, that it's not about MD5 - got same response after coding password to md5.
ONE MORE Edit: Okay, same client works pretty well on HTTP Layer (and another INSTANCE!) instead HTTPS.. So something is broken on HTTPS?
I had the same problem, in my case server had a 301 redirect to url with double slash. In browser's address bar it was invisble, when I checked server response to my browser I realized that.
I'm running into a rather strange issue.
I'm trying to log into a remote moodle install using curl from PHP.
I have a curl command, which works perfectly in the Terminal.
When I translate the same thing into PHP, it works, but it just doesn't login. The exact same value which successfully login via terminal, somehow trips up the login system via PHP and it doesn't login. Instead, it returns the login page again.
My cURL command (data section ommitted as it has my username and password):
curl 'http://moodle.tsrs.org/login/index.php'
-H 'Pragma: no-cache'
-H 'Origin: http://moodle.tsrs.org'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: en-US,en;q=0.8'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: no-cache'
-H 'Referer: http://moodle.tsrs.org/login/index.php'
-H 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
-H 'Connection: keep-alive'
The corresponding PHP code:
function login() {
$username = $_POST['username'];
$password = $_POST['password'];
if(!isset($_POST['username']) || !isset($_POST['password'])) {
echo "No login data received";
return;
}
$creq = curl_init();
$data = array('username' => $username, 'password' => $password, 'testcookies'=> '1');
$headers = array('Pragma: no-cache', 'Origin: http://moodle.tsrs.org', 'Accept-Encoding: ', 'Accept-Language: en-US,en;q=0.8', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36', 'Content-Type: application/x-www-form-urlencoded', 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cache-Control: no-cache', 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'Connection: keep-alive' );
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_POSTFIELDS => $data,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false
));
$output = curl_exec($creq);
echo print_r(curl_getinfo($creq));
echo "\n" . $output . "\n";
}
And the output of curlinfo:
Array
(
[url] => http://moodle.tsrs.org/login/index.php
[content_type] => text/html; charset=utf-8
[http_code] => 200
[header_size] => 541
[request_size] => 945
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 1.462409
[namelookup_time] => 0.002776
[connect_time] => 0.330766
[pretransfer_time] => 0.330779
[size_upload] => 365
[size_download] => 8758
[speed_download] => 5988
[speed_upload] => 249
[download_content_length] => -1
[upload_content_length] => 365
[starttransfer_time] => 0.694866
[redirect_time] => 0
[certinfo] => Array
(
)
[primary_ip] => 125.22.33.149
[redirect_url] =>
[request_header] => POST /login/index.php HTTP/1.1
Host: moodle.tsrs.org
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 365
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=----------------------------83564ee60d56
)
Does anyone know any possible reason for this? I've tried swapping out the hard coded cookie with COOKIEFILE and COOKIEJAR, but it doesn't change anything.
This could have been debugged better by seeing everything that was actually done by cURL. This is done by adding the verbose flag to the command: -v.
$ curl localhost/login [...] -v
We can get the same output from PHP's curl by adding the CURLOPT_VERBOSE option. Note that by adding this line you are instructing cURL to output the same information to STDOUT - it will not be returned and content will not be sent to the browser, so this must be debugged in the terminal.
curl_setopt($curl, CURLOPT_VERBOSE, 1);
By doing it this way, you can get a consistent and comparable output of both HTTP requests, it should look sommthing like this:
POST / HTTP/1.1
Host: localhost:3000
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 250
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=------------------------b4d79f17a3887f2d
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
< Date: Thu, 22 Dec 2016 19:13:40 GMT
< Connection: keep-alive
Left: Command line cURL as provided in the question (with extra -v flag)
Right: PHP cURL as posted in the question (with CURLOUT_VERBOSE enabled)
As you can see, the headers aren't the same, and this makes that clear. The PHP invocation is missing Accept-Encoding and Referer headers.
If that didn't turn up anything, let's try changing some more cURL settings in PHP back to the original cURL defaults.
Internally, PHP opts to override some defaults in cURL without telling you. While these settings should be fine, let's change them back by explicitly reseting them back to cURL defaults:
curl_setopt($curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);
curl_setopt($curl, CURLOPT_MAXREDIRS, -1);
curl_setopt($curl, CURLOPT_NOSIGNAL, 0);
Use http_build_query on the $data array before passing to curl to avoid Content-Type: application/x-www-form-urlencoded; boundary=---. This also ensures to encode any special characters from the password.
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));
Reshape your curl requests as follows:
Make a GET request to the login page with pointing a cookie file at $cookies = '/tmp/some/dir/xyz.cookie.txt'. Make sure using full path for cookie name. And then close the curl handle. This will store the cookie in cookie file.
$creq = curl_init();
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_COOKIEJAR => $cookies // save cookie
));
$output = curl_exec($creq);
curl_close($creq);
Now make the POST request with second curl request. This time point the same cookie file with COOKIEFILE option.
$creq = curl_init();
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_POSTFIELDS => http_build_query ($data),
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_COOKIEJAR => $cookies, // save cookie
CURLOPT_COOKIEFILE => $cookies // load cookie
);
$output = curl_exec($creq);
curl_close($creq);
It can happen sometimes the server look for the cookie when a login request made (to ensure that the request came after visiting the login page).
Most likely your problem is related to HTTP header Expect: 100-continue that cURL sends by default for each POST request.
The Expect: 100-continue header is used in POST requests containing big data when client is not sure that server will accept such request. In this case client first sends request with only headers including Expect: 100-continue and, if the server's response is successful, send the same request with body (POST data).
The problem is that not all web servers handle this header correctly. In such cases sending this header is undesired.
The solution is manually remove Expect header from sending headers by passing array('Expect:') to CURLOPT_HTTPHEADER option.
In your case you can simply add 'Expect:' string to $headers array:
$headers[] = 'Expect:';
I solved the issue by setting a User-Agent
$headers = array(
'Accept: */*',
'User-Agent: curl/7.68.0',
'Accept-Encoding: deflate,gzip,br',
'Content-Type:application/json',
);
I suspect your first attempt using the curl command is using the GET method in the index.php file. I suggest you enable --trace-ascii on your first curl request in the command line and see whether a GET request is being made by the page or not. If yes, you should change your PHP script which is using the POST method. If you change the CURLOPT_POST to false, the PHP script should work.