I wrote a PHP code that sends 3 GET Requests, one after the other
The purpose of the code is to get the content from the m3u8 file,
But in the last GET Request I get an error.
PHP:
<?php
//1. Create a proper token for the m3u8 to work
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Referer: http://www.hotstar.com",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "http://www.hotstar.com/get_cdn_token.php";
$data = file_get_contents($url, false, $context);
$values = json_decode($data, true);
$url = $values['token'];
//2. Send another GET request along with the token, to pull the master m3u8
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Referer: http://www.hotstar.com",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "https://secure-getcdn.hotstar.com/AVS/besc?hotstarauth=$url&action=GetCDN&appVersion=5.0.40&asJson=Y&channel=TABLET&id=1000055355&type=VOD";
$data = file_get_contents($url, false, $context);
$values = json_decode($data, true);
$link = $values['resultObj']['src'];
//3. Get the m3u8 content
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0"
));
$context = stream_context_create($opts);
$url = "$link";
$data = file_get_contents($url, false, $context);
echo $data;
Through Inspect Element I accept the result:
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=241000,RESOLUTION=320x180,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_1_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=461000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_2_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=861000,RESOLUTION=640x360,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_3_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1360000,RESOLUTION=720x404,CODECS="avc1.66.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_4_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2060000,RESOLUTION=1280x720,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_5_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3060000,RESOLUTION=1600x900,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_6_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4562000,RESOLUTION=1920x1080,CODECS="avc1.77.30, mp4a.40.2",CLOSED-CAPTIONS=NONE
https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/index_7_av.m3u8?null=0&id=AgC0lfI2aGb2DFFZW1pBPartIAq++S+ee++3UM8jU49rfzGeMpTl2IaWB4PCyZ0c2yGZOtSqAhal4g%3d%3d
Through PHP I get the error:
Warning: file_get_contents(https://staragvod1-vh.akamaihd.net/i/videos/plus/sns/1365/1000055355_,16,180,400,800,1300,2000,3000,4500,_STAR.mp4.csmil/master.m3u8?hdnea=st=1515937603~exp=1515938203~acl=/*~hmac=c5f9294a198233a9751edbca51631c9cb12db63a08a69499c20d1208bd07aca8): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden in **** on line 37
How do I arrange this please?
Your file needs a login to be accessed, for that you need a cookie mechanism on your file. This can be done with php-curl, or with curl alone in the shell.
Here is a php-curl snippet that can help you on this case.
There is more to do with curl. Anyway, the system needs an additional php package for this to works.
sudo apt install php-curl
$handle = curl_init();
$url = "https//lalala.com/file/files/oups.m3u";
$domain = preg_replace("(^https?://)", "", $url );
$header = array('Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3');
curl_setopt($handle, CURLOPT_URL, $url);
curl_setopt($handle, CURLINFO_HEADER_OUT, 1);
curl_setopt($handle, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0.1');
curl_setopt($handle, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($handle, CURLOPT_NOSIGNAL, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, false);
curl_setopt($handle, CURLOPT_HTTPHEADER, $header);
curl_setopt($handle, CURLOPT_HEADER, false);
header('Content-Type: text/html');
header("Access-Control-Allow-Origin: *");
$result = curl_exec($handle);
var_dump($result);
See https://curl.haxx.se/libcurl/c/CURLOPT_USERAGENT.html for more details about php-curl.
See https://curl.haxx.se/libcurl/c/CURLOPT_COOKIE.html for how to setup cookies.
Related
The below code scrapes the proxy from the website but what I want is the program to check if the proxy is alive or not one by one and then save that proxy in the file. Can someone help me out to do so
<?php
header('Content-Type:application/json');
$url = "https://www.my-proxy.com/free-proxy-list.html";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/'.rand(111,999).'.36 (KHTML, like Gecko) Chrome/88.0.'.rand(1111,9999).'.104 Safari/'.rand(111,999).'.36');
curl_setopt($ch, CURLOPT_URL, $url);
$proxies = array();
$firstcount = 1;
$endcound = 10;
for ($i = $firstcount; $i <= $endcound; $i++){
curl_setopt($ch, CURLOPT_URL, "https://www.my-proxy.com/free-proxy-list-$i.html");
$result =curl_exec($ch);
///Get Proxy
// >102.64.122.214:8085#U
preg_match_all("!\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}:.\d{2,4}!", $result, $matches);
$proxies = array_merge($proxies, $matches[0]);
}
curl_close($ch);
print_r($proxies);
?>
There are multiple ways to test, easiest one being an option in 'file_get_contents' request
$options = array(
'http'=>array(
'proxy' => 'tcp://' . $prox, //IP:PORT info. ie: 8.8.8.8:2222
'timeout' => 2,
'request_fulluri' => true,
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36\r\n"
)
);
$context = stream_context_create($options);
$base_url='http://lotsofrandomstuff.com/1.php'; //url that simply returns '1' each time
$web=#file_get_contents($base_url,false,$context);
if($web=='1')
{
echo "proxy is good";
}
else
{
echo "proxy is dead";
}
I need to setup some custom headers start with ":".
$option['headers'][] = ":authority: example.com"; //<-- Here is the problem
$option['headers'][] = "accept-encoding: gzip, deflate, br";
$option['post'] = json_encode(array("Domain"=>"example.com"));
$url = "https://www.google.com";
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36");
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch,CURLOPT_COOKIEFILE,"file.cookie");
curl_setopt($ch,CURLOPT_COOKIEJAR,"file.cookie");
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch,CURLOPT_HEADER,0);
curl_setopt($ch,CURLOPT_VERBOSE, true);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $option['post']);
curl_setopt($ch, CURLOPT_HTTPHEADER, $option['headers']);
$getdata = curl_exec($ch);
I try to replace the ":" with chr(58) but same problem. I get error 55 and from log "* Failed sending HTTP POST request". If I comment first line is working, but I really need that header. I'm stuck here. Any solutions?
:authority: looks like an HTTP/2 psuedo header, and you can't set them like this with curl. curl will however pass it on itself and it will use the same content as it would set for Host: to make it work the same way, independently of which HTTP version that is eventually used (it will also work with HTTP/3).
I am trying to download a file from this url in php: http://www.roblox.com/Asset/BodyColors.ashx?userId=36377783
The page returns a file your webbrowser automatically downloads.
I tried using cURL:
<?php
$uid = 36377783;
$xUrl = "http://www.roblox.com/Asset/BodyColors.ashx?userId=".$uid;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $xUrl);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
$xml = curl_exec($ch);
curl_close($ch);
echo $xml;
?>
But it redirects me to an error page.
How do I download the file the .ashx url returns?
(Setting CURLOPT_USERAGENT doesn't work.)
There is a redirection - i use file_get_contents() (but why not curl) and $http_response_header:
$uid = 36377783;
$xUrl = "http://www.roblox.com/Asset/BodyColors.ashx?userId=".$uid;
$opts = array(
'http'=>array(
'method'=>"GET",
'follow_location' => true,
'header'=>
"Host: www.roblox.com\r\n" .
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\r\n" .
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" .
"Accept-Encoding: gzip, deflate\r\n" .
"DNT: 1\r\n"
)
);
$context = stream_context_create($opts);
$xml = file_get_contents($xUrl, false, $context);
#print_r($http_response_header);
$url_redirect = str_replace('Location: ',"",$http_response_header[5]);
#print $url_redirect;
$xml = file_get_contents($url_redirect);
#print_r($xml);
$roblox_responses = new SimpleXMLElement($xml);
print_r($roblox_responses);
I am trying to send a POST query to a web-page, but unfortunately it does not work. Below is my code:
<?php
$url_ref = "http://www.wunschgrundstueck.de/immobilien/wohnung-suchen.html?sp_was=wohnungen_m&lses=1";
$url = "http://www.wunschgrundstueck.de/immobilien/suchergebnis/wohnungen.html";
$fields = array(
'id_land' => 'DEU',
'id_bundesland' => '7',
'id_landkreis'=> '186',
'id_gemeinde'=>'5234',
'su_anb' => '0',
'sp_was' => 'wohnungen_m',
'sp_wo' => 'Frankfurt am Main',
'sp_ort_land' => 'DEU',
'sp_ort_bula' => '7',
'sp_ort_lakr' => '186',
'sp_ort_geme' => '5234',
'sp_ort_teil' => '0',
'sp_ort_umkreis' => '0',
'miete' => 'on',
'su_wohn_preis_miete_art' => '1',
);
$fields_string;
//url-ify the data for the POST
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string,'&');
echo $fields_string;
//open connection
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_REFERER, $url_ref);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($curl,CURLOPT_POST,count($fields));
curl_setopt($curl,CURLOPT_POSTFIELDS,$fields_string);
$result = curl_exec($curl);
?>
Unfortunately, it seems that I do something wrong because the result that is returned is not the one I am expecting. What puzzles me a lot is that if I send the same request using the LiveHTTP headers plugin for firefox then I get the correct result. I then tried to send the request with WFetch, and then I receive a 301 Moved permanently error. This is the HTTP Post headers/fields that I tested with HTTPLive from firefox where it works correctly and with WFetch where I get the 301 error.
Host: www.wunschgrundstueck.de\r\n
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3\r\n
Accept-Encoding: gzip, deflate\r\n
Referer: http://www.wunschgrundstueck.de/immobilien/landkreis_frankfurt_am_main/wohnungen.html?s=1\r\n
Connection: keep-alive\r\n
Content-Type: application/x-www-form-urlencoded\r\n
\r\n
id_land=DEU&id_bundesland=7&id_landkreis=186&id_gemeinde=&id_ortsteil=&gemeid_bereich=&su_suchart=erweitert&su_anb=0&sp_was=wohnungen&sp_wo=&sp_ort_land=DEU&sp_ort_bula=7&sp_ort_lakr=186&sp_ort_geme=0&sp_ort_teil=0&sp_wo_ausl=&miete=on&su_wohnfl_wohn_von=&su_wohnfl_wohn_bis=&su_wohn_preis_von_kauf=&su_wohn_preis_bis_kauf=&su_wohn_preis_miete_art=1&su_wohn_preis_von_miete=&su_wohn_preis_bis_miete=&su_zimmer_wohn_anz_von=&su_zimmer_wohn_anz_bis=&su_filter_wohntyp=&su_wohn_etage_von=&su_wohn_etage_bis=&su_wohn_filter_anzbad=&su_wohn_filter_anzschlaf=&su_wohn_filter_baf_det=&su_wohn_filter_moebliert=&submit=Ergebnisse+zeigen+%BB\r\n
\r\n
I don't see you urlencode the key-value pairs you assembly $fields_string from. This might be the reason why you're not getting expected results as some fields have spaces that need to be urlencoded.
That said, you don't need to assembly the post data manually, curl will do that for you if provided an array:
curl_setopt($curl, CURLOPT_POSTFIELDS, $fields);
On a sidenote: 301 is not an error; it's a redirection code, which WFetch doesn't follow automatically, while your browser does.
This does not get gzipped content, but plain content. How to make file_get_contents send headers with https ?
$url = 'https://www.google.co.in/';
///Try to fetch compressed content using the file_get_contents function
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en-US,en;q=0.8\r\n" .
"Accept-Encoding: gzip,deflate,sdch\r\n" .
"Accept-Charset:UTF-8,*;q=0.5\r\n"
)
);
$context = stream_context_create($opts);
$zipped_content = file_get_contents($url ,false,$context);
echo $zipped_content;
print_r($http_response_header);
If the url is http://www.yahoo.co.in then the gzipped content is served (and to confirm, it appears like rubbish).
But when using "https://" it seems that file_get_contents does not send the headers specified.
Header are no OK... Add User-agent and it will be fine.
"User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0 FirePHP/0.4\r\n".
Why? Google decides.
Try this
$url = "https://www.google.co.in/";
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept-Encoding: gzip'));
$contents = curl_exec($ch);
curl_close($ch);