I am sending request header using XMLHttpRequest :
liveXhr.open("GET", url, true);
liveXhr.setRequestHeader("keychain_id", signatureKEYCHAINID);
liveXhr.setRequestHeader("timestamp", signatureTS);
liveXhr.setRequestHeader("signature", signature);
liveXhr.send();
On my localhost everything is fine, when i log :
error_log(print_r(apache_request_headers(), true));
I have this :
[Host] => localhost
[Connection] => keep-alive
[Pragma] => no-cache
[Cache-Control] => no-cache
[timestamp] => 1478279032
[signature] => abcd
[keychain_id] => abcd
[User-Agent] => Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ( KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
[Content-Type] => text/plain
I can get the values for timestamp, signature, keychain_id
On my preprod server for the same process :
...
[Access-Control-Request-Headers] => keychain_id, signature, timestamp
...
I can't get the values of timestamp, signature, keychain_id.
Is it an apache setting concern or something else ?
Try:
//array
$requestHeaders = apache_request_headers();
//get array item by name
$keyChainId = $requestHeaders['keychain_id'];
$timestamp = $requestHeaders['timestamp'];
$signature = $requestHeaders['signature'];
//print items
echo "- keychain_id = $keyChainId";
echo "\n- timestamp = $timestamp";
echo "\n- signature = $signature";
[Edited]
Try:
//not underscore
liveXhr.setRequestHeader("keychainid", KEYCHAINID);
...
//prefix HTTP_
var_dump($_SERVER['HTTP_TIMESTAMP']);
var_dump($_SERVER['HTTP_SIGNATURE']);
var_dump($_SERVER['HTTP_KEYCHAINID']);
Related
I am using Guzzle to login to a page, and then parse the DOM for download links.
However, I won't receive the full DOM after login. The HTML with the download links is just about to start in the DOM string and then cuts off.
Does someone have any idea what could be the cause of this?
The page is behind login and not public accessible.
Note: I cannot share the URLs nor Login data, so replicating the issue is most likely impossible.
This is the end of the DOM
</SCRIPT>
<TABLE ALIGN=LEFT CELLSPACING=0 CELLPADDING=1 style='WIDTH:99%;max-width:1000px;'>
(after that there is nothing, but should be, its just not in the response somehow)
PHP: 7.1.26
Guzzle: 6.3.3
Some code, if its helpful:
$response = self::$client->get(self::getConfig()['baseurl'] . '/' . parse_url($mainScreenUri)['path'], [
'query' => $query_params,
'sink' => date('Y.m.d_H-i-s') . '_sink_.txt',
'debug' => TRUE,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Host' => 'snip',
]
]
);
$x = $response->getBody()->__toString();
file_put_contents(date('Y.m.d_H-i-s') . '.txt', $x);
Both files created by this are cut and do not show the full body.
Response debug:
* Found bundle for host snip: 0x5625c0ab6100 [can pipeline]
* Re-using existing connection! (#0) with host snip
* Connected to snip port 443 (#0)
> GET snip HTTP/1.1
Host: snip
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Cookie: snip
< HTTP/1.1 200 OK
< Date: Tue, 25 Jun 2019 12:55:56 GMT
< Server: Apache/2.4.7 (Ubuntu)
< X-Powered-By: PHP/5.5.9-1ubuntu4.26
< X-Frame-Options: sameorigin
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html
<
* Curl_http_done: called premature == 0
* Connection #0 to host snip left intact
edit
Using streams to only fetch few bytes at a time I have the same problem.
/** #var \GuzzleHttp\Promise\Promise $promise */
$promise = self::$client->getAsync(self::getConfig()['baseurl'] . '/' . parse_url($mainScreenUri)['path'], [
'query' => $query_params,
'sink' => 'snip' . date('Y.m.d_H-i-s') . '_sink_.txt',
'debug' => $resource,
'stream' => TRUE,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Host' => 'snip',
// 'Referer' => 'snip/popup.php?user=' . self::getConfig()['username'] . '&pwi=' . $pwi . '&pwh=' . $hpw,
],
'allow_redirects' => [
'max' => 50,
]
]
);
/** #var \GuzzleHttp\Psr7\Response $response */
$response = $promise->wait();
/** #var \GuzzleHttp\Psr7\Stream $body */
$body = $response->getBody();
$dataRead = "";
while (!$body->eof()) {
$data = $body->read(1024);
$dataRead .= $data;
}
$dataRead is cutoff like everything else.
I found the issue. It was a parameter which was broken and the server decided to give back broken HTML instead of a error message or nothing at all.
I am sending a custom header to an API that I control using an AJAX call made via AngularJS.
Client (AngularJS)
var authorization_token = 'qwerty';
var custom_token_value = 'abc123';
$http.get('http://api.mywebsite.com/endpoint',{ headers:{'Auth':authorization_token,'Custom_Header':custom_token_value} }).then(function(res){
console.log(res.data);
});
Server (PHP 5.6)
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: GET,POST,PUT,DELETE');
header("Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Auth, Custom_Header");
$headers = getallheaders();
print_r($headers);
Response
Array
(
[Accept-Language] => en-US,en;q=0.9
[Accept-Encoding] => gzip, deflate
[Referer] => http://localhost:8080/
[Auth] => qwerty
[User-Agent] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36
[Origin] => http://localhost:8080
[Accept] => application/json, text/plain, */*
[Connection] => close
[Host] => api.mywebsite.com
)
Why am I not able to see Custom_Header?
For anyone having this issue, after some trial and error I've discovered that the reason Custom_Header was not showing up is because it contained an _ (underscore) character. Underscore characters apparently get dropped from header keys.
More info here: Why underscores are forbidden in HTTP header names
I have this POST request to login to a website:
http://xxxx.net-kont.it/
POST / HTTP/1.1
Host: xxxx.net-kont.it
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: */*
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Referer: http://xxxx.net-kont.it/
Content-Length: 1904
Cookie: ASP.NET_SessionId=s44bymd3lm4dsykvymjljv5s
Connection: keep-alive
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
Set-Cookie: SSOAuth=EDCCFF8CD40064D70B3377CD0389FF7F807F0B774F2CE1CA6C015314911D3D69AB819EAB9938C14608842D25991D11D8F1A5A94090DB926BD7001C526B1920A51AC986182EB016C323983716720E8F345B54E02E44C65753E9183843D23F569EF3FE52C03FC8567E809A77387B8C; path=/; HttpOnly
X-Powered-By: ASP.NET
Date: Sun, 22 Oct 2017 12:26:40 GMT
Content-Length: 714
----------------------------------------------------------
http://xxxx.net-kont.it/aspx/Empty.aspx?ControllaRichieste=true&CheckCode=29a29a891a7d4d7773f480064e5c869929bcca40e7c84812111f9affbc3be4628a3b7defe8fb9b14f9911be9c6545e7cd31c2fc04b79a8d1e7280e0277264bdcec7428037a43961c3dda5bbd54a2e7ae&wsid=1a57f5e6-bf68-4f2f-9a71-c43e8e8bfbaf&wsnew=false
GET /aspx/Empty.aspx?ControllaRichieste=true&CheckCode=29a29a891a7d4d7773f480064e5c869929bcca40e7c84812111f9affbc3be4628a3b7defe8fb9b14f9911be9c6545e7cd31c2fc04b79a8d1e7280e0277264bdcec7428037a43961c3dda5bbd54a2e7ae&wsid=1a57f5e6-bf68-4f2f-9a71-c43e8e8bfbaf&wsnew=false HTTP/1.1
Host: xxxx.net-kont.it
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://xxxx.net-kont.it/
Cookie: ASP.NET_SessionId=s44bymd3lm4dsykvymjljv5s; SSOAuth=EDCCFF8CD40064D70B3377CD0389FF7F807F0B774F2CE1CA6C015314911D3D69AB819EAB9938C14608842D25991D11D8F1A5A94090DB926BD7001C526B1920A51AC986182EB016C323983716720E8F345B54E02E44C65753E9183843D23F569EF3FE52C03FC8567E809A77387B8C
Connection: keep-alive
Upgrade-Insecure-Requests: 1
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sun, 22 Oct 2017 12:26:40 GMT
Content-Length: 95935
----------------------------------------------------------
The post request header requires the following fields:
'__LASTFOCUS' => '',
'__EVENTTARGET' => '',
'__EVENTARGUMENT' => '',
'__VIEWSTATE' => $viewstate,
'__VIEWSTATEGENERATOR' => $viewstategenerator,
'ctl00$hwsid' => $hwsid,
'ctl00$PageSessionId' => $pagesessionid,
'ctl00$DefaultUrl' => $defaulturl,
'ctl00$GenericErrorUrl' => $genericerrorurl,
'ctl00$PopupElement' => '',
'ctl00$PollingTimeoutSecs' => $pollingtimeoutsecs,
'ctl00$bodyContent$txtUser' => $user,
'ctl00$bodyContent$txtPassword' => $password,
'__CALLBACKID' => '__Page',
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
'__EVENTVALIDATION' => $eventvalidation
From an analysis of the post request, you notice that by sending the first cookie obtained from the website "ASP.NET_SessionId=", you immediately get an additional authentication cookie "SSOAuth="
How can I get the second cookie "SSOAuth=" so that I can get access to the site? I tried this code:
$user = "xx";
$password = "xx";
$url = 'http://xxx.it/Default.aspx';
$contents = file_get_contents($url);
$dom = new DOMDocument;
$dom->loadHTML($contents);
$xpath = new DOMXpath($dom);
$eventvalidation = $xpath->query('//*[#name="__EVENTVALIDATION"]')->item(0)->getAttribute('value');
$viewstate = $xpath->query('//*[#name="__VIEWSTATE"]')->item(0)->getAttribute('value');
$viewstategenerator = $xpath->query('//*[#name="__VIEWSTATEGENERATOR"]')->item(0)->getAttribute('value');
$hwsid = $xpath->query('//*[#name="ctl00$hwsid"]')->item(0)->getAttribute('value');
$pagesessionid = $xpath->query('//*[#name="ctl00$PageSessionId"]')->item(0)->getAttribute('value');
$defaulturl = $xpath->query('//*[#name="ctl00$DefaultUrl"]')->item(0)->getAttribute('value');
$genericerrorurl = $xpath->query('//*[#name="ctl00$GenericErrorUrl"]')->item(0)->getAttribute('value');
$pollingtimeoutsecs = $xpath->query('//*[#name="ctl00$PollingTimeoutSecs"]')->item(0)->getAttribute('value');
$cookies = array_filter(
$http_response_header,
function($v) {return strpos($v, "Set-Cookie:") === 0;}
);
$headers = [
"Accept-language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0",
];
foreach ($cookies as $cookie) {
$headers[] = preg_replace("/^Set-/", "", $cookie);
}
$request = array(
'http' => array(
'method' => 'POST',
'timeout' => 0,
'header'=> $headers,
'content' => http_build_query(array(
'__LASTFOCUS' => '',
'__EVENTTARGET' => '',
'__EVENTARGUMENT' => '',
'__VIEWSTATE' => $viewstate,
'__VIEWSTATEGENERATOR' => $viewstategenerator,
'ctl00$hwsid' => $hwsid,
'ctl00$PageSessionId' => $pagesessionid,
'ctl00$DefaultUrl' => $defaulturl,
'ctl00$GenericErrorUrl' => $genericerrorurl,
'ctl00$PopupElement' => '',
'ctl00$PollingTimeoutSecs' => $pollingtimeoutsecs,
'ctl00$bodyContent$txtUser' => $user,
'ctl00$bodyContent$txtPassword' => $password,
'__CALLBACKID' => '__Page',
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
'__EVENTVALIDATION' => $eventvalidation,
'ctl00$bodyContent$btnLogin' => 'Conferma'
)),
)
);
echo "<hr/>";
$context = stream_context_create($request);
$data = file_get_contents($url, false, $context);
echo htmlentities($data);
But I get the following output of "Authentication failed":
<Notification><Error Code="" Alert="True" ClosePopup="True" Fatal="False" Message="Autenticazione fallita." /></Notification>
The session will be in the HTTP Headers and file_get_contents only get the HTTP Body so you are losing the "metadata" in which is send your cookie.
I've really recommend to use something a bit more advanced than that. #Tarun Lalwani recommended you curl. Curl which can achieve that, although I prefer to use something more intuitive as Guzzle http://docs.guzzlephp.org/en/stable/ .
Guzzle use the PSR-7 http://www.php-fig.org/psr/psr-7/
This is an Guzzle use example where you can see how easy is to access the headers:
$client = new GuzzleHttp\Client();
$res = $client->request('GET', 'https://api.github.com/user', [
'auth' => ['user', 'pass']
]);
echo $res->getStatusCode();
// "200"
echo $res->getHeader('content-type');
// 'application/json; charset=utf8'
echo $res->getBody();
// {"type":"User"...'
I have solved! was easier than expected....in this I simply had to delete the quotes " :
'__CALLBACKPARAM' => '"hwsid="'.$hwsid.'"&PageSessionId="'.$pagesessionid.'"&DefaultUrl="'.$defaulturl.'"&GenericErrorUrl="'.$genericerrorurl.'"&PopupElement="'.'"&PollingTimeoutSecs="'.$pollingtimeoutsecs.'"&txtUser="'.$user.'"&txtPassword="'.$password,
converted to:
'__CALLBACKPARAM' => 'hwsid='.$hwsid.'&PageSessionId='.$pagesessionid.'&DefaultUrl='.$defaulturl.'&GenericErrorUrl='.$genericerrorurl.'&PopupElement='.'&PollingTimeoutSecs='.$pollingtimeoutsecs.'&txtUser='.$user.'&txtPassword='.$password,
It looks like you are trying to parse data directly from a website, have you considered approaching the website owners about building an API? in any event, I recommend using phantomjs, so that the scraper code is simpler and the traffic and other JS countermeasures are solved in an easier manner.
I am sending customer header in AJAX call,
$.ajaxSetup( {
data: {csrf_token : csrf},
headers: {"Csrf_token" : csrf}
});
Below is content which i seen in Request Headers
Accept:*/*
Accept-Encoding:gzip, deflate
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:no-cache
Connection:keep-alive
Content-Length:325
Content-Type:application/x-www-form-urlencoded; charset=UTF-8
Cookie:ci_session=kd817592v16s0p5b2f502hg39rs7olnu; csrf_cookie=22a1c908f3f036c90c2d0bf0f9b19497
Csrf_token:22a1c908f3f036c90c2d0bf0f9b19497
Host:testurl.com
Origin:http://testurl.com
Pragma:no-cache
Referer:http://testurl.com/xxx
User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
X-Requested-With:XMLHttpRequest
But when i print this in PHP it only return below, i could not see my custom header "Csrf_token"
> Array ( [X-Forwarded-For] => 57.73.33.1 [Cookie] =>
> `ci_session=kd817592v16s0p5b2f502hg39rs7olnu;
> csrf_cookie=22a1c908f3f036c90c2d0bf0f9b19497 [Accept-Language] =>
> en-GB,en-US;q=0.8,en;q=0.6 [Accept-Encoding] => gzip, deflate
> [Referer] => http://testurl.com/xxx [X-Requested-With] =>
> XMLHttpRequest [Accept] => */* [Content-Type] =>
> application/x-www-form-urlencoded; charset=UTF-8 [User-Agent] =>
> Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like
> Gecko) Chrome/54.0.2840.99 Safari/537.36 [Origin] =>
> http://testurl.com[Cache-Control] => no-cache [Pragma] => no-cache
> [Content-Length] => 325 [Connection] => close [Host] => testurl.com )`
How can i get that customer header in PHP? I am using codeigniter,
I only got output by below,
$.ajaxSetup( {
data: {csrf_token : csrf},
headers: {"csrf-token" : csrf, "csrf_token1" : csrf}
});
It will not print values with _ name but can print with - names.
Csrf-Token : "22a1c908f3f036werc90c2d0bf0f9b19497"
Use this
$headers = $this->input->request_headers();
link : https://www.codeigniter.com/user_guide/libraries/input.html#CI_Input::get_request_header
I am sending a token within the headers of each request, when I do a print_r(apache_request_headers()); I get ...
[Host] => 192.168.100.100
[Connection] => keep-alive
[Authorization] =>d868cbf31f676130649fbfd7fff64a70cc071cd0fc8afa676b ...
[Origin] => null
[User-Agent] => Mozilla/5.0 (Linux; U; Android 4.0; en-us; GT-I9300 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
[Accept] => */*
[Accept-Encoding] => gzip, deflate, sdch [Accept-Language] => en-US,en;q=0.8 )
I have tried echo $_SERVER['Authorization']; but get a an error Undefined index: Authorization.
How do I get the token.
You can either use array dereferencing (PHP >= 5.4), e.g.
echo apache_request_headers()["Authorization"];
Or assign it to a variable an then access it, e.g.
$headers = apache_request_headers();
echo $headers["Authorization"];
Try this, it is Server Independent
$header = getallheaders();
echo $header['Authorization'];