I am trying to send a GET request to a python server with an Authorization Token in the header.
I do receive the Request on the server side but without the authorization token
Here is the php code that is generating the request :
<?php
$url = "http://localhost:7432/f.php";
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$headers = array(
"Authorization: sdfascvthsgdgdssgvsgscf",
);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
//for debug only!
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$resp = curl_exec($curl);
curl_close($curl);
var_dump($resp);
?>
When I generate the request from Reqbin, I do Receive the request with the Authorization token in the header , but when I try to generate the request from the php file then it does not send the Authorization token to the server
Here is the request received on server side when I use the Reqbin website to make the request:
127.0.0.1 - - [03/Sep/2021 13:27:10] "GET /f.php HTTP/1.1" 200 -
Host: localhost:7432
Connection: keep-alive
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="92", "Opera";v="78"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159
Safari/537.36 OPR/78.0.4093.184
sec-ch-ua-mobile: ?0
Authorization: sdfascvthsgdgdssgvsgscf
Accept: */*
Sec-Fetch-Site: none
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: _ga=GA1.1.1359401486.1622671713
And This is the received Request when I use the php code mentioned above :
127.0.0.1 - - [03/Sep/2021 13:27:20] "GET /f.php HTTP/1.1" 200 -
Host: localhost:7432Connection: keep-alive Cache-Control: max-age=0
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="92", "Opera";v="78"
sec-ch-ua-mobile: ?0 Upgrade-Insecure-Requests: 1 User-Agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/92.0.4515.159 Safari/537.36 OPR/78.0.4093.184
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1
Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9 Cookie:
_ga=GA1.1.1359401486.1622671713
What am I doing wrong at the php side ?
Related
I have a problem in my code and I want your help
I have this site that makes an IPTV account
http://thgss.com/
This site depends on three pages before it can be downloaded
On the second page "http://thgss.com/index.php?p=download2" and after entering the captcha code.
POST request is sent to 'http://thgss.com/index.php?p=download3'
With the following data : 'done=true&submit=Download+Now'
request header
POST /index.php?p=download3 HTTP/1.1
Host: thgss.com
Connection: keep-alive
Content-Length: 29
Cache-Control: max-age=0
Origin: http://thgss.com
Upgrade-Insecure-Requests: 1
DNT: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://thgss.com/index.php?p=download2
Accept-Encoding: gzip, deflate
Accept-Language: ar,en-US;q=0.9,en;q=0.8
Cookie: PHPSESSID=imbdp791tqal3fq8ifa80till1
done=true&submit=Download+Now
The m3u file link is in the reply header in location
HTTP/1.1 302 Found
Date: Tue, 18 Dec 2018 04:56:17 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.24
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
location: http://24.thgss.com:8000/get.php?username=37441545108977&password=37441545108977&type=m3u&output=mpegts
Content-Length: 2584
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
My code works well on the local server
But when you upload it to remote server show"bool(false)" error
I want you to help me check my code
my code
function getUserIP() {
if( array_key_exists('HTTP_X_FORWARDED_FOR', $_SERVER) && !empty($_SERVER['HTTP_X_FORWARDED_FOR']) ) {
if (strpos($_SERVER['HTTP_X_FORWARDED_FOR'], ',')>0) {
$addr = explode(",",$_SERVER['HTTP_X_FORWARDED_FOR']);
return trim($addr[0]);
} else {
return $_SERVER['HTTP_X_FORWARDED_FOR'];
}
}
else {
return $_SERVER['REMOTE_ADDR'];
}
}
$ip = getUserIP() ;
$data = 'done=true&submit=Download+Now';
$headers = array(
$data,
'Content-Type: application/x-www-form-urlencoded',
'Referer: http://thgss.com/index.php?p=download2',
'X-Forwarded-For: '. $ip
);
$curl_handle=curl_init();
curl_setopt($curl_handle, CURLOPT_URL,'http://thgss.com/index.php?p=download3');
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_HEADER, 1);
curl_setopt($curl_handle,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36');
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $data);
$response = curl_exec($curl_handle);
curl_close($curl_handle);
var_dump($response);
help me get location url from reply header
You need to get info from your Curl request, and get the header from there.
Here is your code:
$ip = getUserIP() ;
$data = 'done=true&submit=Download+Now';
$headers = array(
$data,
'Content-Type: application/x-www-form-urlencoded',
'Referer: http://thgss.com/index.php?p=download2',
'X-Forwarded-For: '. $ip
);
$curl_handle=curl_init();
curl_setopt($curl_handle, CURLOPT_URL,'http://thgss.com/index.php?p=download3');
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_HEADER, 1);
curl_setopt($curl_handle,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36');
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $data);
$response = curl_exec($curl_handle);
curl_close($curl_handle);
var_dump($response);
Before your curl_close, add the following:
$curl_info = curl_getinfo($curl_handle);
You can then use preg_match to find your location header:
$headers = substr($response, 0, $curl_info["header_size"]);
preg_match('#Location: (.*)#', $headers, $location);
You should then find your Location header in $location, probably $location[1].
On a side note, is that the user's Username and Password you're sending in the header?
Edit:
Since the original question has been changed since I answered this, I am now editing my answer.
To help you debug any errors with the actual curl request, before curl_close you can add:
$errs = curl_error($curl_handle);
$errs will now provide you with any reasons the curl request may be failing. An example error is:
"Peer certificate cannot be authenticated with known CA certificates"
You can read more in the docs:
http://php.net/manual/en/function.curl-error.php
Edit 2:
After you said you have a timeout issue,I now notice that you have used 2 seconds:
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);...
Try increasing your timeout:
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_handle, CURLOPT_TIMEOUT, 30);
You can also use 0 which is to wait indefinitely.
Is there any command in php curl to print response query string parameters
Below is my php code
<?php
session_start();
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://192.168.1.220/cgi-bin/handle_login.tcl");
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"user=admin&pw=admin&submit=Login&sid=' '");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: x-www-form-urlencoded'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$response = curl_exec($ch);
echo "$response";
curl_close ($ch);
?>
below is header response
Request URL:http://localhost:8080/cgi-bin/frameset.tcl?
sid=3514275041172666092
Request Method:GET
Status Code:404 Not Found
Remote Address:[::1]:8080
Referrer Policy:no-referrer-when-downgrade
Response Headers
Accept-Ranges:bytes
Connection:Keep-Alive
Content-Language:en
Content-Type:text/html; charset=utf-8
Date:Mon, 29 May 2017 06:54:33 GMT
Keep-Alive:timeout=5, max=99
Server:Apache/2.4.18 (Win32) OpenSSL/1.0.2e PHP/7.0.4
Transfer-Encoding:chunked
Vary:accept-language,accept-charset
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,
*/*;q=0.8
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:PHPSESSID=hj7kbp4hp0pjq16m1tk7j3goo3; _ga=GA1.1.981975196.1489736033; XDEBUG_TRACE=XDEBUG_ECLIPSE
DNT:1
Host:localhost:8080
Referer:http://localhost:8080/nias/automate.php
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Query String Parameters
sid=3514275041172666092 //need to print these value
Since output was in string format and the output remains constant eachtime..needed to find the position of the required character in the string and extract the required value..
Below is the code
$response= curl_exec($ch);//executes the curl
echo strpos("$response","variable_to_find")."<br/>";//know the position
$sid = substr($response,position,length);
I'm trying to POST some form data to a site using cURL. It's not a secure site, and doesn't require logging in. It's just a form used to get back some information.
The form is here, and the form action is to the same page (it's in Turkish). This is how I have been trying to send the POST request in PHP:
$headers = array(
"content-length: 138",
"accept-language: en-US,en;q=0.8",
"accept-encoding: gzip, deflate",
"referer: http://objektifsonuc.com/",
"content-type: application/x-www-form-urlencoded",
"user-agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36",
"origin: http://objektifsonuc.com",
"accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"cache-control: max-age=0",
"connection: keep-alive"
"host: objektifsonuc.com"
);
$query="il=1&ilce=1%3B18&okul=1%3B18%3B743729%3BTEOGS%2CSBS&sinav=100%3BTEOGS&sinif=8&ogrno=1941&ograd=BERKANT+%DDPEK&ogr=%D6%F0renci+Veli+Giri%FE";
$url = "http://objektifsonuc.com/index.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
echo curl_exec($ch);
Current result: The form page renders
Desired result: A different page containing some tables renders
Here are valid form inputs:
Şehir: ADANA
İlçe: SARIÇAM
Okul: Hacı Özcan Sinağ Ortaokulu
Sınav Türü: TEOGS | ADANA İL MİLLİ EĞİTİM OKDS
Sınıf: 8
Öğrenci No: 1941
Öğrenci Ad: BERKANT İPEK
Here is what the form looks like filled out:
Here are the form headers taken from Chrome on a successful POST request:
POST /index.php HTTP/1.1
Host: objektifsonuc.com
Connection: keep-alive
Content-Length: 138
Cache-Control: max-age=0
Origin: http://objektifsonuc.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: http://objektifsonuc.com/index.php
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
And here is the POST body:
il=1&ilce=1%3B18&okul=1%3B18%3B743729%3BTEOGS%2CSBS&sinav=100%3BTEOGS&sinif=8&ogrno=1941&ograd=BERKANT+%DDPEK&ogr=%D6%F0renci+Veli+Giri%FE
I'm stumped. What is causing the POST to fail? Can anyone make this POST succeed?
This should work for you:
//open connection
$url = 'http://objektifsonuc.com/index.php';
$string = 'il=1&ilce=1%3B18&okul=1%3B18%3B743729%3BTEOGS%2CSBS&sinav=100%3BTEOGS&sinif=8&ogrno=1941&ograd=BERKANT+%DDPEK&ogr=%D6%F0renci+Veli+Giri%FE';
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, count($string));
curl_setopt($ch,CURLOPT_POSTFIELDS, $string);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);
I am trying to cURL a base64_encoded xml string to a c# WebAPI that i dont have control over. I can cURL the string successfully but it is not being accepted by the API.
Logging the output shows that cURL is stripping + characters from the base64 string which i believe to be the problem.
The code i have is:
$username = "username";
$password = "$23hrlkbl";
$xml = "<Envelope><Header><User>".$username."</User><Password>".$password."</Password></Header></Envelope>";
$passThru = "https://api.domain.com/SignIn.aspx?passthruUrl=/Management/Api/DataEnrichment/GetAddresses/?buildingNumber=1%26streetName=Nightingale%20Road%26postcode=L12%200QN
$post_packet_data = 'XMLdataPacket='.urlencode(base64_encode($xml));
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $passThru);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_packet_data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$response = curl_exec ($ch);
curl_close($ch);
print_r($response);
Posting the following string (non urlencoded $post_packet_data) via an html form to the $passThru address works successfully
XMLdataPacket=PEVudmVsb3BlPjxIZWFkZXI+PFVzZXI+dXNlcm5hbWU8L1VzZXI+PFBhc3N3b3JkPiQyM2hybGtibDwvUGFzc3dvcmQ+PC9IZWFkZXI+PC9FbnZlbG9wZT4=
However when posting the same string via cURL the following is sent and not accepted
XMLdataPacket=PEVudmVsb3BlPjxIZWFkZXI PFVzZXI dXNlcm5hbWU8L1VzZXI PFBhc3N3b3JkPiQyM2hybGtibDwvUGFzc3dvcmQ PC9IZWFkZXI PC9FbnZlbG9wZT4=
UPDATE
After speaking to the developers of the API they have confirmed the requests are now coming in in the correct format but believe the calls are not being processed due to incorrect headers being sent. They have sent me the headers of a working call which they got using fiddler:
POST https://api.domain.com/SignIn.aspx?passthruUrl=/Management/Api/DataEnrichment/GetAddresses/?buildingNumber=1%26streetName=Nightingale%20Road%26postcode=L12%200QN HTTP/1.1
Host: test.domain.com
Connection: keep-alive
Content-Length: 164
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: null
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Cookie: ASP.NET_SessionId=uzwfjq3zojao3l5fw141l453
XMLdataPacket=PEVudmVsb3BlPjxIZWFkZXI%2BPFVzZXI+dXNlcm5hbWU8L1VzZXI%2BPFBhc3N3b3JkPiQyM2hybGtibDwvUGFzc3dvcmQ%2BPC9IZWFkZXI%2BPC9FbnZlbG9wZT4%3D
How would i amend the headers of the cURL call to replicate the above. Currently again using fiddler the cURL call headers appear as the following:
GET http://test.domain.com/test_call.php HTTP/1.1
Host: test.domain.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: CAKEPHP=g22d7kf7qs8b1v02uui2mnop30
Authorization: Basic bmV0aG91c2VwcmljZXMxOnRlbXAxODkx
Connection: keep-alive
Cache-Control: max-age=0
The data is receieved at the api end as POST but the curl headers are coming through as GET according to the above
Try using rawurlencode() instead of urlencode() - it encodes the space character using '%20' instead of using the '+' character.
I have this cURL code in php.
curl_setopt($ch, CURLOPT_URL, trim("http://stackoverflow.com/questions/tagged/java"));
curl_setopt($ch, CURLOPT_PORT, 80); //ignore explicit setting of port 80
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_HTTPHEADER, $v);
curl_setopt($ch, CURLOPT_VERBOSE, true);
The contents of HTTPHEADER are ;
Proxy-Connection: Close
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1017.2 Safari/535.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: __qca=blabla
Connection: Close
Each of them individual items in the array $v.
When I upload the file on my host and run the code, what I get is :
400 Bad request
Your browser sent an invalid request.
But when I run it on my system using command line PHP, what I get is this and the full page.
< HTTP/1.1 200 OK
< Vary: Accept-Encoding
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Content-Encoding: gzip
< Date: Sat, 03 Mar 2012 21:50:17 GMT
< Connection: close
< Set-Cookie: buncha cokkies; path=/; HttpOnly
< Content-Length: 22151
<
* Closing connection #0
.
It's not only on stackoverflow, this happens, it happens also on 4shared, but works on google and others.
Thanks for any help.
Your empty CURLOPT_ENCODING argument is causing the issue. If you don't want gzip/deflate, simply omit the header.
I also see you're defining encoding both in your curl_setopt() and in the HTTP_HEADER array.
You should use native curl_setopt() commands when possible. CURLOPT_USERAGENT is one you can move out of your HTTP_HEADER array.
But as Andrew Marshall mentioned, screen-scraping isn't something you should be doing; especially since they have an API.
EDIT
Here's the sample script I'm using:
<?php
$v = Array(
'Proxy-Connection: Close',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1017.2 Safari/535.19',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.8',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Cookie: __qca=blabla',
'Connection: Close'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, trim("http://stackoverflow.com/questions/tagged/java"));
//curl_setopt($ch, CURLOPT_PORT, 80); //ignore explicit setting of port 80
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
//curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_HTTPHEADER, $v);
curl_setopt($ch, CURLOPT_VERBOSE, true);
echo curl_exec($ch);
?>
Now I'm running this via command-line, but the net effect is the same. I removed the Accept-Encoding in the $v array simply so I could get un-compressed output.
The one thing we haven't established is your PHP and libcurl versions. For me, this is PHP 5.3.2 with libcurl 7.12.1. This can be important. You can find your libcurl version either by php -i | grep -i curl on the command line, or phpinfo() via a web-based script on your server.
It seems some header is breaking the expected request pattern on some sites. The easiest way to fix this would be to remove the headers one by one and test.
I think it should be the encoding one.
It seems the "Host" header is missing:
Host: stackoverflow.com