Related
I wrote a PHP code that sends 3 GET Requests, one after the other
The purpose of the code is to get the content from the m3u8 file,
But in the last GET Request I get an error.
PHP:
<?php
//1. Create a proper token for the m3u8 to work
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Referer: http://www.hotstar.com",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "http://www.hotstar.com/get_cdn_token.php";
$data = file_get_contents($url, false, $context);
$values = json_decode($data, true);
$url = $values['token'];
//2. Send another GET request along with the token, to pull the master m3u8
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Referer: http://www.hotstar.com",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "https://secure-getcdn.hotstar.com/AVS/besc?hotstarauth=$url&action=GetCDN&appVersion=5.0.40&asJson=Y&channel=TABLET&id=1000055355&type=VOD";
$data = file_get_contents($url, false, $context);
$values = json_decode($data, true);
$link = $values['resultObj']['src'];
//3. Get the m3u8 content
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "$link";
$data = file_get_contents($url, false, $context);
echo $data;
Through Inspect Element I accept the result:
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=241000,RESOLUTION=320x180,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_1_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=461000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_2_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=861000,RESOLUTION=640x360,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_3_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1360000,RESOLUTION=720x404,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_4_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2060000,RESOLUTION=1280x720,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_5_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3060000,RESOLUTION=1600x900,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_6_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4562000,RESOLUTION=1920x1080,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_7_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
Through PHP I get the error:
Warning: file_get_contents(https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/master.m3u8?hdnea=st=1515937603~exp=1515938203~acl=/*~hmac=c5f9294a198233a9751edbca51631c9cb12db63a08a69499c20d1208bd07aca8): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden in **** on line 37
How do I arrange this please?
Your file needs a login to be accessed, for that you need a cookie mechanism on your file. This can be done with php-curl, or with curl alone in the shell.
Here is a php-curl snippet that can help you on this case.
There is more to do with curl. Anyway, the system needs an additional php package for this to works.
sudo apt install php-curl
$handle = curl_init();
$url = "https//lalala.com/file/files/oups.m3u";
$domain = preg_replace("(^https?://)", "", $url );
$header = array('Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3');
curl_setopt($handle, CURLOPT_URL, $url);
curl_setopt($handle, CURLINFO_HEADER_OUT, 1);
curl_setopt($handle, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0.1');
curl_setopt($handle, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($handle, CURLOPT_NOSIGNAL, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, false);
curl_setopt($handle, CURLOPT_HTTPHEADER, $header);
curl_setopt($handle, CURLOPT_HEADER, false);
header('Content-Type: text/html');
header("Access-Control-Allow-Origin: *");
$result = curl_exec($handle);
var_dump($result);
See https://curl.haxx.se/libcurl/c/CURLOPT_USERAGENT.html for more details about php-curl.
See https://curl.haxx.se/libcurl/c/CURLOPT_COOKIE.html for how to setup cookies.
I want to retrieve a JSON data via REST API using PHP cURL.
Using the PHP code in this post, I was able to bypass the Spring Security page with the URL:
https://<host>/appserver/j_spring_security_check
However, the JSON exist inside this URL and I want to fetch it:
https://<host>/appserver/portal/api/1.0/apps
This is the PHP code I used to login automatically, I tried using file_get_contents(url) but it failed, only the below code worked to confirm login.
<?php
function login(){
$headers[] = "Accept: */*";
$headers[] = "Connection: Keep-Alive";
$headers[] = "Content-type: application/x-www-form-urlencoded;charset=UTF-8";
$data = "j_username=admin&j_password=demoserver";
$data = rtrim($data, "&");
global $sessionid;
$sessionid = dirname(__FILE__).'/cookie2.txt'
$ch = curl_init();
$options = array(
CURLOPT_REFERER => "https://<host>/appserver/portal/login",
CURLOPT_HTTPHEADER => $headers,
CURLOPT_COOKIEFILE => $sessionid,
CURLOPT_COOKIEJAR => $sessionid,
CURLOPT_URL => "https://<host>/appserver/j_spring_security_check",
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $data,
CURLOPT_HEADER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0');
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
curl_close($ch);
}
login();
$text = file_get_contents($sessionid);
preg_match('/\w{32}/u', $text, $match);
$jsession = implode($match);
$headers1[] = "Accept: */*";
$headers1[] = "Connection: Keep-Alive";
$headers1[] = "Content-type: application/json";
$headers1[] = "Cookie: JSESSIONID=".$jsession;
$url = 'https://<host>/appserver/portal/api/1.0/apps';
$ch2 = curl_init();
curl_setopt($ch2, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_HTTPHEADER, $headers1);
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch2, CURLOPT_COOKIEFILE, $sessionid);
curl_setopt($ch2, CURLOPT_COOKIEJAR, $sessionid);
curl_setopt($ch2, CURLOPT_HTTPGET, 1);
$result = curl_exec($ch2);
curl_close($ch2);
// var_dump(json_decode($result, true));
$output = json_decode ( $result );
This is the result after execution of PHP code:
HTTP/1.1 302 Found Date: Fri, 17 Feb 2017 03:59:00 GMT
Server: Apache/2.2.26 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e mod_jk/1.2.37
Set-Cookie: JSESSIONID=DB7139F7B8EE30F868A8392A4BF15523; Path=/appserver/; HttpOnly
Location: https://192.168.100.100:444/appserver/portal/welcome
Content-Length: 0
Access-Control-Allow-Origin: *
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain
How can I fetch the JSON given the above cURL options?
Why are you setting the cookie header manually when you already use cookiefile and cookiejar? That should not be needed.
It is also suspicious that you receive a setcookie header, probably the server is not accepting the session cookie you are sending.
Also, why are you not ignoring invalid SSL on the second request?
Finally, why are you not sending the same user-agent on the second request?
I guess that if you fix these problems you will see the expected behavior.
I am trying to download a file from this url in php: http://www.roblox.com/Asset/BodyColors.ashx?userId=36377783
The page returns a file your webbrowser automatically downloads.
I tried using cURL:
<?php
$uid = 36377783;
$xUrl = "http://www.roblox.com/Asset/BodyColors.ashx?userId=".$uid;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $xUrl);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
$xml = curl_exec($ch);
curl_close($ch);
echo $xml;
?>
But it redirects me to an error page.
How do I download the file the .ashx url returns?
(Setting CURLOPT_USERAGENT doesn't work.)
There is a redirection - i use file_get_contents() (but why not curl) and $http_response_header:
$uid = 36377783;
$xUrl = "http://www.roblox.com/Asset/BodyColors.ashx?userId=".$uid;
$opts = array(
'http'=>array(
'method'=>"GET",
'follow_location' => true,
'header'=>
"Host: www.roblox.com\r\n" .
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\r\n" .
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" .
"Accept-Encoding: gzip, deflate\r\n" .
"DNT: 1\r\n"
)
);
$context = stream_context_create($opts);
$xml = file_get_contents($xUrl, false, $context);
#print_r($http_response_header);
$url_redirect = str_replace('Location: ',"",$http_response_header[5]);
#print $url_redirect;
$xml = file_get_contents($url_redirect);
#print_r($xml);
$roblox_responses = new SimpleXMLElement($xml);
print_r($roblox_responses);
I use file_get_contents in PHP. In the below code in first URL works fine but the second one isn't working.
$URL = "http://test6473.blogspot.com";
$domain = file_get_contents($URL);
print_r($domain);
$add_url= "http://adfoc.us/1575051";
$add_domain = file_get_contents($add_url);
echo $add_domain;
Any suggestions on why the second one doesn't work?
URL which is not retrieved by file_get_contents, because their server checks whether the request come from browser or any script. If they found request from script they simply disable page contents.
So that I have to make a request similar as browser request. So I have used following code to get 2nd url contents. It might be different for different web server. Because they might keep different checks.
Even though why dont you try to use following code! If you are lucky this might work for you!!
function getUrlContent($url) {
fopen("cookies.txt", "w");
$parts = parse_url($url);
$host = $parts['host'];
$ch = curl_init();
$header = array('GET /1575051 HTTP/1.1',
"Host: {$host}",
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Host:adfoc.us',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36',
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "http://adfoc.us/1575051";
$html = getUrlContent($url);
Thanks everyone for the guidance.
Unfortunately it looks like the second site blocks access from unrecognized browsers. Even using curl from the command line doesn't work:
curl -I http://adfoc.us/1575051
gives:
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Fri, 28 Jun 2013 12:15:40 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.5.0
Set-Cookie: __cfduid=d7cd1bf18c136a288cc2b36065a3b31f01372421740; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.adfoc.us
CF-RAY: 85a4dc6829e06d0
but no content. Note it returns status 200 so if you check the returned string for boolean === false to see if it failed, it will actually appear as if it has worked.
If you need to spoof the useragent (and possibly other things) to try and get the url to accept your request, you'll need to take the plunge with the curl libraries and try different combinations to try and get it working. Experimenting to see what works with the curl command line first would also be a good way to reduce development time in investigating this.
Here's someone who has been through this before:
php curl: how can i emulate a get request exactly like a web browser?
looks like the second url answers too slow sometimes, maybe have redirects.
try to use curl and set bigger timeout.
also, turn errors on
error_reporting(-1);
ini_set('display_errors','On');
you can try this code also
<?php
function getUrlContent($url) {
$parts = parse_url($url);
$host = $parts['host'];
$ch = curl_init();
$header = array('GET /1575051 HTTP/1.1',
"Host: {$host}",
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Host:adfoc.us',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36',
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "https://news.google.com/rss/search?q=apple&hl=en-IN&gl=IN&ceid=IN:en";
$html = getUrlContent($url);
$xml = simplexml_load_string($html);
$json = json_encode($xml);
$array = json_decode($json,TRUE);
print_r($array);
?>
I've been working on writing a script which automatically logs me into my school's network, checks if the classes I'm trying to get into are no longer completely full, and if a spot has opened up, registers the class for me. However, I've hit a big snag in just the logging-in process.
Basically, I've been looking at the headers that are sent when I log in and try to replicate them. The problem is I keep getting an error saying "HTTP/1.1 400 Bad Request Content-Type: text/html Date: Sat, 23 Oct 2010 18:42:20 GMT Connection: close Content-Length: 42
Bad Request (Invalid Header Name)".
I'm guessing it has something to do with Host parameter I'm setting being different from what it really is (I set it so it is elion.psu.edu, but when looking at the headers from my script it has changed back to grantbachman.com, where the script is hosted). I guess it'll be best just to show you.
The beginning of the header I'm trying to create:
https://elion.psu.edu/cgi-bin/elion-student.exe/submit
POST /cgi-bin/elion-student.exe/submit HTTP/1.1
Host: elion.psu.edu
The beginning of the header which shows up when I run my script:
http://myDomain.com/myScriptName.php
GET /elionScript.php HTTP/1.1
Host: myDomain.com
Basically, the first line is different, the Host name is different, and it says I'm sending my info with a GET variable instead of a POST variable (even though I set curlopt_post to true). I'm basically looking for any help with altering this info such that the server accepts my script. I'm fresh out of ideas. Thanks.
Oh here's the code I'm using:
$data = array(
"$userIDName" => '********',
"$passName" => '********',
"$submitName" => 'Login+to+eLion',
'submitController' => '',
'forceUnicode' => '%D0%B4%D0%B0',
'sessionKey' => "$sessionValue",
'pageKey' => "$pageKeyValue",
'shopperID' => '');
$contentLength = strlen($userIDName . '=*********&' . $passName . '=********&' . $submitName .'=Login+to+eLion&submitController=&forceUnicode=%D0%B4%D0%B0&sessionKey=' . $sessionValue . '&pageKey=' . $pageKeyValue . '&shopperID=');
$ch = curl_init("https://elion.psu.edu/cgi-bin/elion-student.exe/submit");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_COOKIE,'sessionKey="$sessionValue";pageKey="$pageKeyValue";BIGipServerelion_prod_pool="$prodPoolValue"');
curl_setopt($ch,CURLOPT_HTTPHEADER,array(
'Host: elion.psu.edu',
'User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-us,en;q=0.5','Accept-Encoding: gzip,deflate',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Keep-Alive: 115',
'Referer: https://elion.psu.edu/cgi-bin/elion-student.exe/launch/ELionMainGUI/Student',
"Cookie: sessionKey=$sessionValue; pageKey=$pageKeyValue; BIGipServerelion_prod_pool=$prodPoolValue", 'Content-Type: application/x-www-form-urlencoded',
"Content-Length: $contentLength"));
$contents2 = curl_exec ($ch);
I't also probably important to note that when I run the script, none of the information below the 'Keep-Alive: 115' line is displayed when I view the header.
Seems it missing some code in your question but try this :
1- Save your certificate on your server
2- Try this code
$pg = curl_init();
// Set the form data for posting the login information
$postData = array();
$postData["username"] = $username;
$postData["password"] = $password;
$postText = "";
foreach( $postData as $key => $value ) {
$postText .= $key . "=" . $value . "&";
}
curl_setopt( $pg, CURLOPT_URL, $YOUR_URL );
curl_setopt( $pg, CURLOPT_POST, true );
curl_setopt( $pg, CURLOPT_POSTFIELDS, $postText );
curl_setopt( $pg, CURLOPT_SSL_VERIFYPEER, true );
curl_setopt( $pg, CURLOPT_SSL_VERIFYHOST, 2 );
curl_setopt( $pg, CURLOPT_CAINFO, getcwd() . '/web'); //web is the exported certificate
//curl_setopt( $pg, CURLOPT_VERBOSE, true ); // for debug
//curl_setopt( $pg, CURLOPT_RETURNTRANSFERT, true); // if you want a ouput
curl_setopt( $pg, CURLOPT_COOKIEJAR, "cookies.txt" );
curl_setopt( $pg, CURLOPT_COOKIEFILE, "cookies.txt" );
curl_setopt( $pg, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; MSIE 5.01; Windows NT 5.0)" );
curl_setopt( $pg, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $pg, CURLOPT_COOKIE, session_name() . '=' . session_id() );
if( ( $response = curl_exec( $pg ) ) === false ) {
echo '*Curl erro' . curl_error($pg) . "\n";
}
curl_close($pg)
$YOUR_URL:https://elion.psu.edu/cgi-bin/elion-student.exe/launch/ELionMainGUI/Student
The form using dynamic name so its not simple like "username" and "password". Check on the website to know a "good" one.
Do not forget to add others hidden field like you did in the postData array and update the cookie section too.