I had been using PHP curl to get the contents of a file, hosted on a different server. The file can easily be opened on a browser like Chrome etc., but with cURL, it always returns a blank page.
The file is hosted on an Nginx server and even miniproxy.php fails to get contents. Instead, it returns 406 not acceptable. I tried using the HTTP spy extension to monitor the request sent and found the following header:
Upgrade-Insecure-Requests:1
I tried sending the same header along With other headers, but in vain. Still, I couldn't rectify my mistake. On the Internet, I found the zalmos proxy which was able to get the contents of the file. The curl code I wrote is attached below.
$url = "http://smumcdnems01.cdnsrv.jio.com/jiotv.live.cdn.jio.com/" . $ch . "/" . $ch . "_" . $q . ".m3u8" . $tok;
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$headers = array(
"User-Agent: agent",
"lbcookie: 300",
"devicetype: 1",
"os: android",
"appkey: 1111111",
"deviceId: device id",
"uniqueId: unique id",
"ssotoken: any token",
"Upgrade-Insecure-Requests: 1",
"Host: example.com",
"Connection: keep-alive",
"X-Chrome-offline: persist=0 reason=reload",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"Accept-Encoding: gzip, deflate, sdch",
"Accept-Language: en-GB,en-US;q=0.9,en;q=0.8",
"subscriberId: any id",
);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
//for debug only!
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$resp = curl_exec($curl);
curl_close($curl);
echo $url;
echo $resp;
I believe that any part is missing in my code which is posing a problem. How can this be rectified?
Check your URL. Curl must give you the response. If it's hit the target URL, either the target URL is not responding to anything when sending the request.
You may be trying to access a websocket. Try to simulate with Postman to get more information.
Related
What I'm trying to achieve:
Get request to an API Endpoint, retrieving an XML and subsequently parse the results.
I am sending a file_get_contents request to achieve this.
Issues:
`file_get_Contents` fails, error:
Warning: file_get_contents(https://api.twitter.com/1.1/statuses/mentions_timeline.json):
failed to open stream:
A connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed because
connected host has failed to respond.
Update 17/08
To consolidate my current understanding:
1. PHP FAILS:
1.a it fails via php (timeout)
1.b it fails via command line (curl -G http://api.eve-central.com/api/quicklook?typeid=34)
1.c file_get_contents
1.d file_get_contents w/ create_stream_context
2. What WORKS:
2.a Pasting the url in a chrome tab
2.b via postman
What has been attempted:
- Check Headers in Postman ,and try to replicate them via php
Postman Headers sent back by eve-central:
Access-Control-Allow-Origin → *
Connection → Keep-Alive
Content-Encoding → gzip
Content-Type → text/xml; charset=UTF-8
Date → Wed, 17 Aug 2016 10:40:24 GMT
Proxy-Connection → Keep-Alive
Server → nginx
Transfer-Encoding → chunked
Vary → Accept-Encoding
Via → HTTP/1.1 proxy10014
Corresponding Code:
$headers = array(
'method' => 'GET',
'header' => 'Connection: Keep-Alive',
'header' => 'Content-Encoding: gzip',
'header' => 'Content-Type: text/xml',
'header' => 'Proxy-Connection: Keep-Alive',
'header' => 'Server: nginx',
'header' => 'Transfer-Encoding: chunked',
'header' => 'Vary: Accept-Encoding',
'header' => 'Via: HTTP/1.1 proxy10014');
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true );
curl_setopt($curl, CURLOPT_PORT , 8080); // Attempt at changing port in the event it was blocked.
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_POST, false );
curl_setopt($curl, CURLOPT_URL, $url );
$resp = curl_exec($curl);
if(curl_error($curl))
{
echo 'error:' . curl_error($curl);
}
Use Wireshark to capture the GET request to see if changing the port helped
Run cUrl via command line
I'm out of ideas and option.
So the questions are:
If it works in a browser, and in Postman, why does it not work via PHP ?
How can I modify my code so that it mimics what Postman does? ?
Previous Attempts
What I have tried:
Various cURL options from other threads, such as
function curl_get_contents($url) {
$ch = curl_init();
if (!$ch)
{
die("Couldn't initialize a cURL handle");
} else
echo "Curl Handle initialized ";
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
$data = curl_exec($ch);
// Check if any error occurred
if (!curl_errno($ch))
{
$info = curl_getinfo($ch);
echo 'Took ', $info['total_time'], ' seconds to send a request to ', $info['url'], "";
displayData($info);
} else
echo "Failed Curl, reason: ".curl_error($ch)." ";
curl_close($ch);
return $data;
}
result: nothing, no data returned.
- Checked php.ini options:
- allow_fopen is On
- allow_url_include = on
- relevant ssl extensions are enabled
- Raised the timeout window
- both via php.ini
- also via explicit declaration within the php file.
- Tried with a different url
- same error, so it doesn't really depends on my particular endpoint
- for example, both twitter/wikipedia/google return the specific error
- tried with:
- file_get_contents on a local xml file (https://msdn.microsoft.com/en-us/library/ms762271(v=vs.85).aspx) --> works
- file_get_contents on a remote xml file (http://www.xmlfiles.com/examples/note.xml) --> fails same error
- Overall, the following is true, so far:
- Curl fails, timeout
- file_get_Contents fails, timeout
- Open XML file url in a browser works
- Make a GET request via Postman, works
Obviously, in all cases where the file_get_contents fails via php, I can easily access the file via any browser.
Tried to work around the issue.
Attempt 1:
Use nitrous.io, create a LAMP stack, perform the deed via the platform
results: file_get_contents works, however, due to the large number of xml files to be retrieved, the operation times-out.
Tentative solution:
- Download XML files from source
- Zip them
- Download xml_file
- Locally parse said xml files
Later on, write a small php scripts that, when invoked, performs the bits above, sends the data to the local directory, which then unpacks it and performs additional work on it.
Another attempt would be to use Google Sheets, with a user function that pulls the data into the sheet, and just dump the excel file / values into mysql.
For my purposes, while an awfully ignorant solution, it does the trick.
Code used for avoiding timeout issue on shared host:
function downloadUrlToFile2($url, $outFileName)
{
//file_put_contents($xmlFileName, fopen($link, 'r'));
//copy($link, $xmlFileName); // download xml file
;
echo "Passing $url into $outFileName ";
// $outFileName = touch();
$fp = fopen($outFileName, "w");
if(is_file($url))
{
copy($url, $outFileName); // download xml file
} else
{
$ch = curl_init();
$options = array(
CURLOPT_TIMEOUT => 28800, // set this to 8 hours so we dont timeout on big files
CURLOPT_URL => $url
);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt_array($ch, $options);
$contents = curl_exec($ch);
fwrite($fp, $contents);
curl_close($ch);
}
}
I have also added this on top of the ini script:
ignore_user_abort(true);
set_time_limit(0);
ini_set('memory_limit', '2048M');
I see some issue with HTTPS url request, for fix issue you have to add below lines in your CURL request
function curl_get_contents($url) {
$ch = curl_init();
$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";
curl_setopt( $ch, CURLOPT_HTTPHEADER, $header );
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
// I have added below two lines
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
In PHP, I'm trying to retrieve the url for a specific page in DocuSign that constantly refreshes. The POST to retrieve this url is in the form:
POST http://demo.docusign.net/restapi/{apiVersion}/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
This should return a json file in the form:
{
"url": "example.example.com"
}
However, I am extremely new to using PHP and POST methods and don't believe I'm doing this correctly. The API explorer for this method in particular is here. I am using cURL methods to make this request. Here is my code ($recipient,$account_id,$access_token are found accurately within another file):
$url = "http://demo.docusign.net/restapi/v2/accounts/$account_id
/envelopes/$envelope_id/views/recipient";
$body = array("returnUrl" => "http://www.docusign.com/devcenter",
"authenticationMethod" => "None",
"email" => "$recipient",
"userName" => "$recipient");
$body_string = json_encode($body);
$header = array(
'Accept: application/json',
'Content-Type: application/json',
'Content-Length: '.strlen($body_string),
);
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $body_string);
$json_response = curl_exec($curl);
$response = json_decode($json_response, true);
var_dump($response);
I am able to get the correct return on the API explorer, but not when making the request with PHP. I believe this is due to the fact that I am not incorporating the $header or $body correctly, but at this point I am just not sure.
ADDED: This is the raw output for the request when correctly running the method on the API Explorer:
Accept: application/json
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,fa;q=0.6,sv;q=0.4
Cache-Control: no-cache
Origin: https://apiexplorer.docusign.com
Referer: https://apiexplorer.docusign.com/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Authorization: Bearer fGehcK7fkRvFguyu/7NGh01UUFs=
Content-Length:
Content-Type: application/json
This is the JSON request being formed in my code:
{
"returnUrl":"http:\/\/www.docusign.com\/devcenter",
"authenticationMethod":"Password",
"email":"example#example.com",
"userName":"example#example.com",
"clientUserId":"4c6228f4-fcfe-47f9-bee1-c9d5e6ab6a41",
"userId":"example#example.com"
}
You are not hitting a valid DocuSign URL in your cURL code. Right now you are sending requests to:
http://demo.docusign.net/apiVersion/v2/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
Instead of "apiVersion" it should be "restApi" like this:
http://demo.docusign.net/restapi/v2/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
We can't send post fields, because we want to send JSON, not pretend to be a form (the merits of an API which accepts POST requests with data in form-format is an interesting debate). Instead, we create the correct JSON data, set that as the body of the POST request, and also set the headers correctly so that the server that receives this request will understand what we sent:
$data = array("name" => "Hagrid", "age" => "36");
$data_string = json_encode($data);
$ch = curl_init('http://api.local/rest/users');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($data_string))
);
$result = curl_exec($ch);
All these settings are pretty well explained on the curl_setopt() page, but basically the idea is to set the request to be a POST request, set the json-encoded data to be the body, and then set the correct headers to describe that post body. The CURLOPT_RETURNTRANSFER is purely so that the response from the remote server gets placed in $result rather than echoed. If you're sending JSON data with PHP, I hope this might help!
I know this question was asked more than 3 years ago, but this may help someone who finds this question because they are having the same problem. I do not see a cURL option that will decode the response in your code. I have found that I need to use the cURL option CURLOPT_ENCODING like this: curl_setopt($ch,CURLOPT_ENCODING,""); According to the PHP manual online, it says, 'CURLOPT_ENCODING - The contents of the "Accept-Encoding: " header. This enables decoding of the response. Supported encodings are "identity", "deflate", and "gzip". If an empty string, "", is set, a header containing all supported encoding types is sent.' You can find this option at https://www.php.net/manual/en/function.curl-setopt.php. I hope this helps save someone from having a headache.
I'm making request to LinkedIn page and receiving "HTTP/1.1 999 Request denied" response.
I use AWS/EC-2 and get this response.
On localhost everything works fine.
This is sample of my code to get html-code of the page.
<?php
error_reporting(E_ALL);
$url= 'https://www.linkedin.com/pulse/5-essential-strategies-digital-michelle';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
var_dump($response);
var_dump($info);
I don't need whole page content, just meta-tags (title, og-tags).
Note that the error 999 don't exist in W3C Hypertext Transfer Protocol - HTTP/1.1, probably this error is customized (sounds like a joke)
LinkedIn don't allow direct access, the probable reason of them blocking any "url" from others webservers access should be to:
Prevent unauthorized copying of information
Prevent invasions
Prevent abuse of requests.
Force use API
Some IP addresses of servers are blocked, as the "IP" from "domestic ISP" are not blocked and that when you access the LinkedIn with web-browser you use the IP of your internet provider.
The only way to access the data is to use their APIs. See:
Accessing LinkedIn public pages using Python
Heroku requests return 999
Note: The search engines like Google and Bing probably have their IPs in a "whitelist".
<?php
header("Content-Type: text/plain");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.linkedin.com/company/technistone-a-s-");
$header = array();
$header[] = "Host: www.linkedin.com";
$header[] = "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0";
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$header[] = "Accept-Language: en-US,en;q=0.5";
$header[] = "Accept-Encoding: gzip, deflate, br";
$header[] = "Connection: keep-alive";
$header[] = "Upgrade-Insecure-Requests: 1";
curl_setopt($ch,CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_HTTPHEADER , $header);
$my_var = curl_exec($ch);
echo $my_var;
LinkedIn is not supporting the default encoding 'identity' , so if you set the header
'Accept-Encoding': 'gzip, deflate'
you should get the response , but you would have to decompress it.
I ran into this while doing local web development and using the LinkedIn badge feature (profile.js). I was only getting the 999 Request denied in Chrome, so I just cleared my browser cache and localStorage and it started to work again.
UPDATE - Clearing cache was just a coincidence and the issue came back. LinkedIn is having issues with their badge functionality.
I submitted a help thread to their forums.
https://www.linkedin.com/help/linkedin/forum/question/714971
I'm trying to upload a file to google drive using http header and CURL post, and I get a "not found" returned from google error.
I think it's because of the way Im uploading the file via CURL, Because I never Did it.
Here's My Code :
$file = file_get_contents("./ima.jpg");
$length = strlen($file);
test($file,$length);
function test($file,$length){
$url2="https://www.googleapis.com/upload/drive/v2/filesuploadType=media";
$header = array(
"Content-Type: image/jpeg",
"Content-Length:$length ",
"Authorization: Bearer $token",
);
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url2);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,false);
curl_setopt ($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch,CURLOPT_POST,true);
curl_setopt($ch,CURLOPT_POSTFIELDS,$file);
$data2 = curl_exec($ch);
echo $data2;
curl_close($ch);
}
The Token is set in a variable token, and it is a valid token because it works with listing files from google drive, Thank you !
uploadType is a parameter to the URL and needs to be separated using ?, which means that in your case the URL should most likely be;
$url2="https://www.googleapis.com/upload/drive/v2/files?uploadType=media";
See here for more detailed documentation.
<?php
if(isset($_GET['token']))
{
$url="http://www.google.com/calendar/feeds/default/allcalendars/full";
$useragent="PHP 5.2";
$header=array( "GET /accounts/AuthSubSessionToken HTTP/1.1",
"Content-Type: application/x-www-form-urlencoded",
"Authorization: AuthSub token=".$_GET['token'],
"User-Agent: PHP/5.2",
"Host: https://www.google.com",
"Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"Connection: keep-alive"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
print_r($data);
}
?>
The result is page not found. However, I call http://www.google.com/calendar/feeds/default/allcalendars/full from firefox , it's return XML file. So, I think, my code may wrong. But I can't find the error. :(
That is because you are accessing Google Calendar via your personal port. Whenever you access that specific URL, Google checks to see if you are logged in. If not, it sends a 404. If you are, it outputs the calendar based on the settings you provided. That URL does not specify a specific calendar that it's supposed to pull from the site, and it cannot use the cookies stored on the user's computer because it is being fetched from your server, which will not have any cookies for a calendar. When I try to access that page without logging on, I get a 401 Authorization Required error, which I bet is what PHP is getting and you just don't realize it.
You need to go into your Google Calendar settings and find the embedding options to find a URL that is specific to your account so that it will always fetch an XML feed for your calendar.
Read more about the Google 'Calendar Address' here: http://www.google.com/support/calendar/bin/answer.py?answer=34578
View from other applications: http://www.google.com/support/calendar/bin/answer.py?hl=en&answer=37648
I think that you may be overriding the URL with this line in the header:
GET /accounts/AuthSubSessionToken HTTP/1.1
I think that will point CURL to http://www.google.com/accounts/AuthSubSessionToken
What happens when you remove it?
I got it.... I changed like this
<?php
function make_api_call($url, $token)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$curlheader[0] = sprintf("Authorization: AuthSub token=\"%s\"/n", $token);
curl_setopt($ch, CURLOPT_HTTPHEADER, $curlheader);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
function get_session_token($onetimetoken) {
$output = make_api_call("https://www.google.com/accounts/AuthSubSessionToken", $onetimetoken);
if (preg_match("/Token=(.*)/", $output, $matches))
{
$sessiontoken = $matches[1];
} else {
echo "Error authenticating with Google.";
exit;
}
return $sessiontoken;
}
if(isset($_GET['token']))
{
$sessiontoken=get_session_token($_GET['token']);
$accountxml = make_api_call("http://www.google.com/m8/feeds/contacts/yourmail#gmail.com/full", $sessiontoken);
print_r($accountxml);
}
else
{
$next=urlencode("http://www.mysteryzillion.org/gdata/index.php");
$scope=urlencode("http://www.google.com/m8/feeds/contacts/yourmail#gmail.com/full");
?>
Click here to authenticate through Google.
<?
}
?>