I need to setup some custom headers start with ":".
$option['headers'][] = ":authority: example.com"; //<-- Here is the problem
$option['headers'][] = "accept-encoding: gzip, deflate, br";
$option['post'] = json_encode(array("Domain"=>"example.com"));
$url = "https://www.google.com";
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36");
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch,CURLOPT_COOKIEFILE,"file.cookie");
curl_setopt($ch,CURLOPT_COOKIEJAR,"file.cookie");
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch,CURLOPT_HEADER,0);
curl_setopt($ch,CURLOPT_VERBOSE, true);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $option['post']);
curl_setopt($ch, CURLOPT_HTTPHEADER, $option['headers']);
$getdata = curl_exec($ch);
I try to replace the ":" with chr(58) but same problem. I get error 55 and from log "* Failed sending HTTP POST request". If I comment first line is working, but I really need that header. I'm stuck here. Any solutions?
:authority: looks like an HTTP/2 psuedo header, and you can't set them like this with curl. curl will however pass it on itself and it will use the same content as it would set for Host: to make it work the same way, independently of which HTTP version that is eventually used (it will also work with HTTP/3).
Related
I am calling GitHub API https://api.github.com/user/emails from a PHP script. Basically I need GitHub email address of GitHub account used for logging in.
Here is the response I am receiving from API:
{
"message": "Requires authentication",
"documentation_url": "https://developer.github.com/v3/users/emails/#list-email-addresses-for-a-user"
}
Even though I am passing access token with header for authorization, I am still receiving requires authentication message.
Here is my code:
$token = $_GET['token'];
echo $token;
$url = "https://api.github.com/user/emails";
$auth = "Authorization: token ".$token;
echo $auth;
$accept = "Accept: application/vnd.github.v3+json";
$useragent = "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 YaBrowser/16.3.0.7146 Yowser/2.5 Safari/537.36";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array($auth));
curl_setopt($ch, CURLOPT_HTTPHEADER, array($accept));
curl_setopt($ch, CURLOPT_HTTPHEADER, array($useragent));
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$res = curl_exec($ch);
echo ($res);
curl_close($ch);
Any idea what I am doing wrong here? I am open to any other suggestions for getting GitHub email address associated with logged in GitHub account.
Calling curl_setopt() multiple times for CURLOPT_HTTPHEADER won't work because it will overwrite the previous value specified for the CURLOPT_HTTPHEADER field.
You can check this yourself by enabling CURLOPT_VERBOSE, after which you can see what cURL does internally for your request.
You'll have to specify a single array which you supply to CURLOPT_HTTPHEADER:
$headers = array(
"Authorization: token ".$token,
"Accept: application/vnd.github.v3+json",
"User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 YaBrowser/16.3.0.7146 Yowser/2.5 Safari/537.36"
);
Now you can just use curl_setopt() to set CURLOPT_HTTPHEADER to include all the headers specified in $headers:
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
I'm trying to set up a proxy server to make a post request. Problem is when I make the request I am not seeing the payload.
One thing I notice is that curl seems to be adding an extra "boundary" to the content-type in the request.
Am I missing something?
The Code:
$contentType = $_SERVER["HTTP_CONTENT_TYPE"];
$post = http_build_query($_POST);
$ch = curl_init();
$header = array("Content-Type:" . $contentType,
"Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding:gzip, deflate, br",
"Accept-Language:en-US,en;q=0.8",
"Connection:keep-alive",
"User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
"Cache-Control:max-age=0",
"Upgrade-Insecure-Requests:1",
"Origin:<url here>");
echo "<b>POST</b><br>" . var_dump($_POST) . "<br><br>";
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, count($_POST));
curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookiejar.txt");
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HEADER, 1);
$result = curl_exec($ch);
$headerSent = curl_getinfo($ch, CURLINFO_HEADER_OUT );
echo "<b>Request Header</b><br>$headerSent<br><br>";
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($result, 0, $header_size);
$body = substr($result, $header_size);
echo "<b>Response Header</b><br>$header<br><br>";
echo "<b>Response Body</b><br>$body";
Response
$_POST = array(5) { ["formFields_Complaint_Type"]=> string(9) "1-GM2-226"
["formFields_Descriptor_1"]=> string(10) "1-GM3-3085"
["formFields_Descriptor_2"]=> string(9) "1-GM4-903"
["formFields_Date/Time_of_Occurrence"]=> string(0) "" ["_target1"]=> string(1) " " }
Request Header:
POST <relative address> HTTP/1.1 Host: <url>
Cookie:
JSESSIONID=mDMJZQdLV4bhvJQ6vPyQvxqHVTynGS3byBnYsTpjDvY1xBnB93R8!-759339305!-1867032216 Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36
Cache-Control:max-age=0 Upgrade-Insecure-Requests:1
Origin: <url>
Content-Length: 633
Expect: 100-continue
Content-Type:multipart/form-data; boundary=----
WebKitFormBoundarybdBepqnmjSF86t50; boundary=------------------------
f8e2ad22b9bb626c
best guess: your (biggest, code-breaking, but not only) problem is that the target server supports only application/x-www-form-urlencoded-encoded POST requests, but your curl code converts both application/x-www-form-urlencoded-encoded requests, and multipart/form-data requests to multipart/form-data, regardless of what the client used. (this is because PHP transparently translates both of them to an equal native PHP array called $_POST)
this will use multipart/form-data encoding:
curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
this will use application/x-www-form-urlencoded encoding:
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($_POST));
you must decide which encoding to use, based on $_SERVER["HTTP_CONTENT_TYPE"];
and if its neither of those (for example, if its application/json), you must add special code to handle each, and you should probably error out whenever $_SERVER["HTTP_CONTENT_TYPE"]; is not 1 of the types you have made a special case for (like raw $_POST for multipart, and http_build_query($_POST) x-www-form-urlencoded)
also you're not forwarding arbitrary http headers, you should probably add some code for that
and if you really need to support Upgrade-Insecure-Requests:1 header, you need to implement specific code to handle that at the proxy side (go read the http specs on the subject - https://www.w3.org/TR/upgrade-insecure-requests/ )
and you say to the target that you accept Accept-Encoding:gzip, deflate, br , but provide no code to decode any of them, so it will look like garbage binary data to the client if the target server decide to use any of them (curl can decode them for you though, using CURLOPT_ENCODING, if libcurl was compiled with gzip and deflate and br support. i've never seen a libcurl with br support, and i bet your curl doesn't have it. probably have gzip/deflate support compiled-in though)
I use file_get_contents in PHP. In the below code in first URL works fine but the second one isn't working.
$URL = "http://test6473.blogspot.com";
$domain = file_get_contents($URL);
print_r($domain);
$add_url= "http://adfoc.us/1575051";
$add_domain = file_get_contents($add_url);
echo $add_domain;
Any suggestions on why the second one doesn't work?
URL which is not retrieved by file_get_contents, because their server checks whether the request come from browser or any script. If they found request from script they simply disable page contents.
So that I have to make a request similar as browser request. So I have used following code to get 2nd url contents. It might be different for different web server. Because they might keep different checks.
Even though why dont you try to use following code! If you are lucky this might work for you!!
function getUrlContent($url) {
fopen("cookies.txt", "w");
$parts = parse_url($url);
$host = $parts['host'];
$ch = curl_init();
$header = array('GET /1575051 HTTP/1.1',
"Host: {$host}",
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Host:adfoc.us',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36',
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "http://adfoc.us/1575051";
$html = getUrlContent($url);
Thanks everyone for the guidance.
Unfortunately it looks like the second site blocks access from unrecognized browsers. Even using curl from the command line doesn't work:
curl -I http://adfoc.us/1575051
gives:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Fri, 28 Jun 2013 12:15:40 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.5.0
Set-Cookie: __cfduid=d7cd1bf18c136a288cc2b36065a3b31f01372421740; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.adfoc.us
CF-RAY: 85a4dc6829e06d0
but no content. Note it returns status 200 so if you check the returned string for boolean === false to see if it failed, it will actually appear as if it has worked.
If you need to spoof the useragent (and possibly other things) to try and get the url to accept your request, you'll need to take the plunge with the curl libraries and try different combinations to try and get it working. Experimenting to see what works with the curl command line first would also be a good way to reduce development time in investigating this.
Here's someone who has been through this before:
php curl: how can i emulate a get request exactly like a web browser?
looks like the second url answers too slow sometimes, maybe have redirects.
try to use curl and set bigger timeout.
also, turn errors on
error_reporting(-1);
ini_set('display_errors','On');
you can try this code also
<?php
function getUrlContent($url) {
$parts = parse_url($url);
$host = $parts['host'];
$ch = curl_init();
$header = array('GET /1575051 HTTP/1.1',
"Host: {$host}",
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Host:adfoc.us',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36',
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "https://news.google.com/rss/search?q=apple&hl=en-IN&gl=IN&ceid=IN:en";
$html = getUrlContent($url);
$xml = simplexml_load_string($html);
$json = json_encode($xml);
$array = json_decode($json,TRUE);
print_r($array);
?>
I am trying to download a source code of web pages using curl php code but its downloading only for few pages for rest pages file is empty.
I googled it but im not getting solution.
My source code is :-
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $strurl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch,CURLOPT_USERAGENT, 'CURL via PHP');
$out = curl_exec($ch);
$fp = fopen('f1.html', 'w');
fwrite($fp, $out);
fclose($fp);
curl_close($ch);
What options to add ? Where i am wrong ?
Pls help.
Try setting a user-agent that suggests you're a browser. Some servers will block curl/wget/etc.
For example: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
I can access a web page when I type its URL in my browser. However, while using curl to access the details of that web page, I get the message on screen
User information is disabled.
This operation cannot be accepted. User certification is invalid or date expired.
Update page.
I can access the details of my network printer(Canon IR3570) by typing in the IP of that printer in my browser. This opens up the remote UI. However, it doesn't seem to work with curl.
This is my code in PHP curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"URL");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1');
$result= curl_exec($ch);
echo $result;
curl_close ($ch);
What could be the reason for such a message?
Where's UserAgent set? They may filtrate requests.
try adding:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION ,1);
curl_setopt($ch, CURLOPT_USERAGENT, "User-Agent Mozilla/5.0");
curl_setopt($ch, CURLOPT_HEADER ,1); // DO NOT RETURN HTTP HEADERS
This isn't CURL but it works in Visual Basic 2012 - one of the headers solved it for me
Sub Main()
Dim web_client As New System.Net.WebClient
Dim baseDate As DateTime = New DateTime(1970, 1, 1)
Dim diff As TimeSpan = DateTime.Now - baseDate
web_client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
web_client.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3")
web_client.Headers.Add("Accept-Encoding", "gzip, deflate, sdch")
web_client.Headers.Add("Accept-Language", "en-US,en;q=0.8")
web_client.Headers.Add("Cookie", "iR = 1711753823")
web_client.Headers.Add("Host", "172.23.100.14")
web_client.Headers.Add("Referer", "http://172.23.100.14/jlp.cgi?Flag=Html_Data&LogType=0&Dummy=" & diff.Milliseconds)
web_client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31")
web_client.DownloadFile("http://172.23.100.14/pprint.csv?Flag=Csv_Data&LogType=0&Dummy=" & diff.Milliseconds, "P:\Transfer\mstavers\printlogs\" & Format(Now, "yyyy-MM-dd-hh-mm-ss") & ".csv")
End Sub