I have installed a new ubuntu server v20 with PHP Version 7.4.3 to move a web application from an older ubuntu server v18 with PHP Version 7.0 and I am getting a 403 error on the new server when performing a CURL REST API GET. Bellow is the code with the error debug, portions of the license key have been modified for the post. I haven't been able to find anything related to this searching around existing posts. Thanks for the help in advance
ob_start();
$out = fopen('php://output', 'w');
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "http://license.cmadsystems.com?lic=P9xP1o7USRLkS591cFzBbLSmI9ZTtvR7xgfr86dtYiCZuhy",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0",
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $out,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Content-Type: application/json"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
fclose($out);
$debug = ob_get_clean();
echo $response;
echo $err;
echo $debug;
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<p>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
* Trying 216.250.121.144:80...
* TCP_NODELAY set
* Connected to license.cmadsystems.com (216.250.121.144) port 80 (#0)
> GET /?lic= P9xP1o7USRLkS591cFzBbLSmI9ZTtvR7xgfr86dtYiCZuhy HTTP/1.1
Host: license.cmadsystems.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: */*
Accept-Encoding: deflate, gzip, br
Content-Type: application/json
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=15
< Date: Wed, 20 May 2020 23:45:29 GMT
< Server: Apache
< Content-Encoding: gzip
<
* Connection #0 to host license.cmadsystems.com left intact
In my case it this was a DNS issue. license.cmadsystems.com was resolving to the wrong IP address.
echo file_get_contents(
"http://license.cmadsystems.com?lic=P9xP1o7USRLkS591cFzBbLSmI9ZTtvR7xgfr86dtYiCZuhy"
);
Returns
"status":200,
"status_message":"Licensed",
"data":{
"Lic_End":null,
"Lic_Device":0,
"Lic_MOH":0,
"Lic_Info":null
}
}
Related
I'm getting started trying to understand Guzzle but one of my requests keeps returning an error, even though the exact same request when done using CURL works just fine.
I have a refresh_token and want to get an access_token from WEB API.
The Guzzle request that results in an error:
$refresh_token = '<token>';
$client = new GuzzleHttp\Client(['headers' => ['Content-Type' => 'application/x-www-form-urlencoded']]);
$response = $client->request('POST', 'https://foo.bar/secure/token', [
'query' => ['grant_type' => 'refresh_token','refresh_token' => $refresh_token]
]);
echo $response->getStatusCode();
echo $response->getBody();
Fatal error: Uncaught exception 'GuzzleHttp\Exception\ClientException' with message 'Client error: resulted in a 400 Bad Request response' in vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
This CURL request works just fine:
$refresh_token = '<token>';
$params=['grant_type'=>'refresh_token',
'refresh_token'=>$refresh_token
];
$headers = [
'POST /secure/token HTTP/1.1',
'Content-Type: application/x-www-form-urlencoded'
];
$curlURL='https://foo.bar/secure/token';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$curlURL);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,http_build_query($params));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$curl_res = curl_exec($ch);
if($curl_res) {
$server_output = json_decode($curl_res);
}
var_dump($curl_res);
I hope for your help.
Here's the Guzzle debug that was printed out in the browser.
Request Method: GET
Status Code: 200 OK
Remote Address: 87.236.19.237:80
Referrer Policy: no-referrer-when-downgrade
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Wed, 21 Aug 2019 15:46:25 GMT
Keep-Alive: timeout=30
Server: nginx-reuseport/1.13.4
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Powered-By: PHP/5.6.38
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36
Your request becomes GET probably because of the "query" parameter.
Use form_params instead of query.
See the documentation.
http://docs.guzzlephp.org/en/stable/request-options.html#form-params
This is correct! Tnx Jonnix!
$response = $client->request('POST', 'https://sso.tinkoff.ru/secure/token', [
'form_params' => [
'grant_type' => 'refresh_token',
'refresh_token' => $refresh_token
]
]);
I am using Guzzle to login to a page, and then parse the DOM for download links.
However, I won't receive the full DOM after login. The HTML with the download links is just about to start in the DOM string and then cuts off.
Does someone have any idea what could be the cause of this?
The page is behind login and not public accessible.
Note: I cannot share the URLs nor Login data, so replicating the issue is most likely impossible.
This is the end of the DOM
</SCRIPT>
<TABLE ALIGN=LEFT CELLSPACING=0 CELLPADDING=1 style='WIDTH:99%;max-width:1000px;'>
(after that there is nothing, but should be, its just not in the response somehow)
PHP: 7.1.26
Guzzle: 6.3.3
Some code, if its helpful:
$response = self::$client->get(self::getConfig()['baseurl'] . '/' . parse_url($mainScreenUri)['path'], [
'query' => $query_params,
'sink' => date('Y.m.d_H-i-s') . '_sink_.txt',
'debug' => TRUE,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Host' => 'snip',
]
]
);
$x = $response->getBody()->__toString();
file_put_contents(date('Y.m.d_H-i-s') . '.txt', $x);
Both files created by this are cut and do not show the full body.
Response debug:
* Found bundle for host snip: 0x5625c0ab6100 [can pipeline]
* Re-using existing connection! (#0) with host snip
* Connected to snip port 443 (#0)
> GET snip HTTP/1.1
Host: snip
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Cookie: snip
< HTTP/1.1 200 OK
< Date: Tue, 25 Jun 2019 12:55:56 GMT
< Server: Apache/2.4.7 (Ubuntu)
< X-Powered-By: PHP/5.5.9-1ubuntu4.26
< X-Frame-Options: sameorigin
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html
<
* Curl_http_done: called premature == 0
* Connection #0 to host snip left intact
edit
Using streams to only fetch few bytes at a time I have the same problem.
/** #var \GuzzleHttp\Promise\Promise $promise */
$promise = self::$client->getAsync(self::getConfig()['baseurl'] . '/' . parse_url($mainScreenUri)['path'], [
'query' => $query_params,
'sink' => 'snip' . date('Y.m.d_H-i-s') . '_sink_.txt',
'debug' => $resource,
'stream' => TRUE,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Host' => 'snip',
// 'Referer' => 'snip/popup.php?user=' . self::getConfig()['username'] . '&pwi=' . $pwi . '&pwh=' . $hpw,
],
'allow_redirects' => [
'max' => 50,
]
]
);
/** #var \GuzzleHttp\Psr7\Response $response */
$response = $promise->wait();
/** #var \GuzzleHttp\Psr7\Stream $body */
$body = $response->getBody();
$dataRead = "";
while (!$body->eof()) {
$data = $body->read(1024);
$dataRead .= $data;
}
$dataRead is cutoff like everything else.
I found the issue. It was a parameter which was broken and the server decided to give back broken HTML instead of a error message or nothing at all.
I have this POST request to login to a website:
http://xxxx.net-kont.it/
POST / HTTP/1.1
Host: xxxx.net-kont.it
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: */*
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Referer: http://xxxx.net-kont.it/
Content-Length: 1904
Cookie: ASP.NET_SessionId=s44bymd3lm4dsykvymjljv5s
Connection: keep-alive
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
Set-Cookie: SSOAuth=EDCCFF8CD40064D70B3377CD0389FF7F807F0B774F2CE1CA6C015314911D3D69AB819EAB9938C14608842D25991D11D8F1A5A94090DB926BD7001C526B1920A51AC986182EB016C323983716720E8F345B54E02E44C65753E9183843D23F569EF3FE52C03FC8567E809A77387B8C; path=/; HttpOnly
X-Powered-By: ASP.NET
Date: Sun, 22 Oct 2017 12:26:40 GMT
Content-Length: 714
----------------------------------------------------------
http://xxxx.net-kont.it/aspx/Empty.aspx?ControllaRichieste=true&CheckCode=29a29a891a7d4d7773f480064e5c869929bcca40e7c84812111f9affbc3be4628a3b7defe8fb9b14f9911be9c6545e7cd31c2fc04b79a8d1e7280e0277264bdcec7428037a43961c3dda5bbd54a2e7ae&wsid=1a57f5e6-bf68-4f2f-9a71-c43e8e8bfbaf&wsnew=false
GET /aspx/Empty.aspx?ControllaRichieste=true&CheckCode=29a29a891a7d4d7773f480064e5c869929bcca40e7c84812111f9affbc3be4628a3b7defe8fb9b14f9911be9c6545e7cd31c2fc04b79a8d1e7280e0277264bdcec7428037a43961c3dda5bbd54a2e7ae&wsid=1a57f5e6-bf68-4f2f-9a71-c43e8e8bfbaf&wsnew=false HTTP/1.1
Host: xxxx.net-kont.it
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://xxxx.net-kont.it/
Cookie: ASP.NET_SessionId=s44bymd3lm4dsykvymjljv5s; SSOAuth=EDCCFF8CD40064D70B3377CD0389FF7F807F0B774F2CE1CA6C015314911D3D69AB819EAB9938C14608842D25991D11D8F1A5A94090DB926BD7001C526B1920A51AC986182EB016C323983716720E8F345B54E02E44C65753E9183843D23F569EF3FE52C03FC8567E809A77387B8C
Connection: keep-alive
Upgrade-Insecure-Requests: 1
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sun, 22 Oct 2017 12:26:40 GMT
Content-Length: 95935
----------------------------------------------------------
The post request header requires the following fields:
'__LASTFOCUS' => '',
'__EVENTTARGET' => '',
'__EVENTARGUMENT' => '',
'__VIEWSTATE' => $viewstate,
'__VIEWSTATEGENERATOR' => $viewstategenerator,
'ctl00$hwsid' => $hwsid,
'ctl00$PageSessionId' => $pagesessionid,
'ctl00$DefaultUrl' => $defaulturl,
'ctl00$GenericErrorUrl' => $genericerrorurl,
'ctl00$PopupElement' => '',
'ctl00$PollingTimeoutSecs' => $pollingtimeoutsecs,
'ctl00$bodyContent$txtUser' => $user,
'ctl00$bodyContent$txtPassword' => $password,
'__CALLBACKID' => '__Page',
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
'__EVENTVALIDATION' => $eventvalidation
From an analysis of the post request, you notice that by sending the first cookie obtained from the website "ASP.NET_SessionId=", you immediately get an additional authentication cookie "SSOAuth="
How can I get the second cookie "SSOAuth=" so that I can get access to the site? I tried this code:
$user = "xx";
$password = "xx";
$url = 'http://xxx.it/Default.aspx';
$contents = file_get_contents($url);
$dom = new DOMDocument;
$dom->loadHTML($contents);
$xpath = new DOMXpath($dom);
$eventvalidation = $xpath->query('//*[#name="__EVENTVALIDATION"]')->item(0)->getAttribute('value');
$viewstate = $xpath->query('//*[#name="__VIEWSTATE"]')->item(0)->getAttribute('value');
$viewstategenerator = $xpath->query('//*[#name="__VIEWSTATEGENERATOR"]')->item(0)->getAttribute('value');
$hwsid = $xpath->query('//*[#name="ctl00$hwsid"]')->item(0)->getAttribute('value');
$pagesessionid = $xpath->query('//*[#name="ctl00$PageSessionId"]')->item(0)->getAttribute('value');
$defaulturl = $xpath->query('//*[#name="ctl00$DefaultUrl"]')->item(0)->getAttribute('value');
$genericerrorurl = $xpath->query('//*[#name="ctl00$GenericErrorUrl"]')->item(0)->getAttribute('value');
$pollingtimeoutsecs = $xpath->query('//*[#name="ctl00$PollingTimeoutSecs"]')->item(0)->getAttribute('value');
$cookies = array_filter(
$http_response_header,
function($v) {return strpos($v, "Set-Cookie:") === 0;}
);
$headers = [
"Accept-language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0",
];
foreach ($cookies as $cookie) {
$headers[] = preg_replace("/^Set-/", "", $cookie);
}
$request = array(
'http' => array(
'method' => 'POST',
'timeout' => 0,
'header'=> $headers,
'content' => http_build_query(array(
'__LASTFOCUS' => '',
'__EVENTTARGET' => '',
'__EVENTARGUMENT' => '',
'__VIEWSTATE' => $viewstate,
'__VIEWSTATEGENERATOR' => $viewstategenerator,
'ctl00$hwsid' => $hwsid,
'ctl00$PageSessionId' => $pagesessionid,
'ctl00$DefaultUrl' => $defaulturl,
'ctl00$GenericErrorUrl' => $genericerrorurl,
'ctl00$PopupElement' => '',
'ctl00$PollingTimeoutSecs' => $pollingtimeoutsecs,
'ctl00$bodyContent$txtUser' => $user,
'ctl00$bodyContent$txtPassword' => $password,
'__CALLBACKID' => '__Page',
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
'__EVENTVALIDATION' => $eventvalidation,
'ctl00$bodyContent$btnLogin' => 'Conferma'
)),
)
);
echo "<hr/>";
$context = stream_context_create($request);
$data = file_get_contents($url, false, $context);
echo htmlentities($data);
But I get the following output of "Authentication failed":
<Notification><Error Code="" Alert="True" ClosePopup="True" Fatal="False" Message="Autenticazione fallita." /></Notification>
The session will be in the HTTP Headers and file_get_contents only get the HTTP Body so you are losing the "metadata" in which is send your cookie.
I've really recommend to use something a bit more advanced than that. #Tarun Lalwani recommended you curl. Curl which can achieve that, although I prefer to use something more intuitive as Guzzle http://docs.guzzlephp.org/en/stable/ .
Guzzle use the PSR-7 http://www.php-fig.org/psr/psr-7/
This is an Guzzle use example where you can see how easy is to access the headers:
$client = new GuzzleHttp\Client();
$res = $client->request('GET', 'https://api.github.com/user', [
'auth' => ['user', 'pass']
]);
echo $res->getStatusCode();
// "200"
echo $res->getHeader('content-type');
// 'application/json; charset=utf8'
echo $res->getBody();
// {"type":"User"...'
I have solved! was easier than expected....in this I simply had to delete the quotes " :
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
converted to:
'__CALLBACKPARAM' => 'hwsid='.$hwsid.'&PageSessionId='.$pagesessionid.'&DefaultUrl='.$defaulturl.'&GenericErrorUrl='.$genericerrorurl.'&PopupElement='.'&PollingTimeoutSecs='.$pollingtimeoutsecs.'&txtUser='.$user.'&txtPassword='.$password,
It looks like you are trying to parse data directly from a website, have you considered approaching the website owners about building an API? in any event, I recommend using phantomjs, so that the scraper code is simpler and the traffic and other JS countermeasures are solved in an easier manner.
I wrote this code for getting https pages content but i couldnt succesfull.
<?php
function bot($url)
{
$header ="Host: tr-tr.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0\r\n
Accept: */*
Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br";
$options = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_PORT => 443,
CURLOPT_SSL_VERIFYPEER => true,
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_CAINFO => "C:\\xampp\\htdocs\\curl-ca-bundle.crt",
CURLOPT_HTTPHEADER => explode("\r\n",$header)
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
echo bot("https://tr-tr.facebook.com");
?>
When i run that codes it is returning that. "��0#a�jȌ#�#.3�j�##u�.����/#cw#,�q=ߓ���K"<�˞#%#����t[�d�:|��s##$!� ��(��M��ߛ#w'#u" Where is my mistake? Why is returning these characters?
I dont want to use CURLOPT_SSL_VERIFYPEER = false. I want to https handshake with curl..
Maybe you should remove CURLOPT_RETURNTRANSFER or handle it properly.
From the manual
http://php.net/manual/en/function.curl-setopt.php
Do you really want this?!
> CURLOPT_RETURNTRANSFER
> TRUE to return the transfer as a string of the
> return value of curl_exec() instead of outputting it out directly.
I'm running into a rather strange issue.
I'm trying to log into a remote moodle install using curl from PHP.
I have a curl command, which works perfectly in the Terminal.
When I translate the same thing into PHP, it works, but it just doesn't login. The exact same value which successfully login via terminal, somehow trips up the login system via PHP and it doesn't login. Instead, it returns the login page again.
My cURL command (data section ommitted as it has my username and password):
curl 'http://moodle.tsrs.org/login/index.php'
-H 'Pragma: no-cache'
-H 'Origin: http://moodle.tsrs.org'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: en-US,en;q=0.8'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: no-cache'
-H 'Referer: http://moodle.tsrs.org/login/index.php'
-H 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)'
-H 'Connection: keep-alive'
The corresponding PHP code:
function login() {
$username = $_POST['username'];
$password = $_POST['password'];
if(!isset($_POST['username']) || !isset($_POST['password'])) {
echo "No login data received";
return;
}
$creq = curl_init();
$data = array('username' => $username, 'password' => $password, 'testcookies'=> '1');
$headers = array('Pragma: no-cache', 'Origin: http://moodle.tsrs.org', 'Accept-Encoding: ', 'Accept-Language: en-US,en;q=0.8', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36', 'Content-Type: application/x-www-form-urlencoded', 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cache-Control: no-cache', 'Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'Connection: keep-alive' );
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_POSTFIELDS => $data,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false
));
$output = curl_exec($creq);
echo print_r(curl_getinfo($creq));
echo "\n" . $output . "\n";
}
And the output of curlinfo:
Array
(
[url] => http://moodle.tsrs.org/login/index.php
[content_type] => text/html; charset=utf-8
[http_code] => 200
[header_size] => 541
[request_size] => 945
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 1.462409
[namelookup_time] => 0.002776
[connect_time] => 0.330766
[pretransfer_time] => 0.330779
[size_upload] => 365
[size_download] => 8758
[speed_download] => 5988
[speed_upload] => 249
[download_content_length] => -1
[upload_content_length] => 365
[starttransfer_time] => 0.694866
[redirect_time] => 0
[certinfo] => Array
(
)
[primary_ip] => 125.22.33.149
[redirect_url] =>
[request_header] => POST /login/index.php HTTP/1.1
Host: moodle.tsrs.org
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 365
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=----------------------------83564ee60d56
)
Does anyone know any possible reason for this? I've tried swapping out the hard coded cookie with COOKIEFILE and COOKIEJAR, but it doesn't change anything.
This could have been debugged better by seeing everything that was actually done by cURL. This is done by adding the verbose flag to the command: -v.
$ curl localhost/login [...] -v
We can get the same output from PHP's curl by adding the CURLOPT_VERBOSE option. Note that by adding this line you are instructing cURL to output the same information to STDOUT - it will not be returned and content will not be sent to the browser, so this must be debugged in the terminal.
curl_setopt($curl, CURLOPT_VERBOSE, 1);
By doing it this way, you can get a consistent and comparable output of both HTTP requests, it should look sommthing like this:
POST / HTTP/1.1
Host: localhost:3000
Pragma: no-cache
Origin: http://moodle.tsrs.org
Accept-Language: en-US,en;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Cache-Control: no-cache
Cookie: MoodleSession=ngcidh028m37gm8gbdfe07mvs7; MOODLEID_=%25F1%25CD%2519D%25B2k%25FE%251D%25EFH%25E5t%25B1%2503%258E; MoodleSessionTest=NhzaTNij6j; _ga=GA1.2.925953522.1416155774; _gat=1; __utmt=1; __utma=147409963.925953522.1416155774.1416642544.1416692798.3; __utmb=147409963.1.10.1416692798; __utmc=147409963; __utmz=147409963.1416155774.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Connection: keep-alive
Content-Length: 250
Expect: 100-continue
Content-Type: application/x-www-form-urlencoded; boundary=------------------------b4d79f17a3887f2d
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-mZFLkyvTelC5g8XnyQrpOw"
< Date: Thu, 22 Dec 2016 19:13:40 GMT
< Connection: keep-alive
Left: Command line cURL as provided in the question (with extra -v flag)
Right: PHP cURL as posted in the question (with CURLOUT_VERBOSE enabled)
As you can see, the headers aren't the same, and this makes that clear. The PHP invocation is missing Accept-Encoding and Referer headers.
If that didn't turn up anything, let's try changing some more cURL settings in PHP back to the original cURL defaults.
Internally, PHP opts to override some defaults in cURL without telling you. While these settings should be fine, let's change them back by explicitly reseting them back to cURL defaults:
curl_setopt($curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_DNS_USE_GLOBAL_CACHE, 0);
curl_setopt($curl, CURLOPT_MAXREDIRS, -1);
curl_setopt($curl, CURLOPT_NOSIGNAL, 0);
Use http_build_query on the $data array before passing to curl to avoid Content-Type: application/x-www-form-urlencoded; boundary=---. This also ensures to encode any special characters from the password.
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));
Reshape your curl requests as follows:
Make a GET request to the login page with pointing a cookie file at $cookies = '/tmp/some/dir/xyz.cookie.txt'. Make sure using full path for cookie name. And then close the curl handle. This will store the cookie in cookie file.
$creq = curl_init();
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_COOKIEJAR => $cookies // save cookie
));
$output = curl_exec($creq);
curl_close($creq);
Now make the POST request with second curl request. This time point the same cookie file with COOKIEFILE option.
$creq = curl_init();
curl_setopt_array($creq, array(
CURLOPT_URL => 'http://moodle.tsrs.org/login/index.php',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_ENCODING => '',
CURLINFO_HEADER_OUT => true,
CURLOPT_POSTFIELDS => http_build_query ($data),
CURLOPT_HTTPHEADER => $headers,
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_COOKIEJAR => $cookies, // save cookie
CURLOPT_COOKIEFILE => $cookies // load cookie
);
$output = curl_exec($creq);
curl_close($creq);
It can happen sometimes the server look for the cookie when a login request made (to ensure that the request came after visiting the login page).
Most likely your problem is related to HTTP header Expect: 100-continue that cURL sends by default for each POST request.
The Expect: 100-continue header is used in POST requests containing big data when client is not sure that server will accept such request. In this case client first sends request with only headers including Expect: 100-continue and, if the server's response is successful, send the same request with body (POST data).
The problem is that not all web servers handle this header correctly. In such cases sending this header is undesired.
The solution is manually remove Expect header from sending headers by passing array('Expect:') to CURLOPT_HTTPHEADER option.
In your case you can simply add 'Expect:' string to $headers array:
$headers[] = 'Expect:';
I solved the issue by setting a User-Agent
$headers = array(
'Accept: */*',
'User-Agent: curl/7.68.0',
'Accept-Encoding: deflate,gzip,br',
'Content-Type:application/json',
);
I suspect your first attempt using the curl command is using the GET method in the index.php file. I suggest you enable --trace-ascii on your first curl request in the command line and see whether a GET request is being made by the page or not. If yes, you should change your PHP script which is using the POST method. If you change the CURLOPT_POST to false, the PHP script should work.