for some time I have been trying to write the ideal browser simulations, for this purpose I wrote a script that to some extent simulates the browser and works correctly on many pages with ssl. Recently during the test site pornhub.com and wikipedia.com encountered a strange error in my script, just for pornhub after a few page reloads shows the status of header "Loading .. Content-length: 1456" and the number of loaded data changes in real time on smaller and larger values. I have a question for already very experienced and professional programmers: Have you met with such a situation, if so you have any hints or corrections for my script ?.
I post my code (test for wikipedia). If you fire it on 3 browser tabs and you will refresh, you will get an error.
<?php
function curl($url)
{
$headers = [
'Accept-Language: pl,en-US;q=0.9,en;q=0.8',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
];
$cookie = 'cookie.txt';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_REFERER, 'https://www.wikipedia.org');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
if (!file_exists($cookie)){
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
}else{
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
}
$c = curl_exec($ch);
curl_close($ch);
return $c;
}
echo curl('https://www.wikipedia.org');
?>
Related
I´m new to php and trying to program a few things.
In the first step I want to show the following page: "https://www.mytischtennis.de/public/home" on my website. I am using Curl to grab the page. But every time I want to output the page I am getting a blank page.
My code looks like this:
<?php
function grab_page($site){
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start();
return curl_exec($ch);
ob_end_clean();
curl_close($ch);
}
echo grab_page("https://www.mytischtennis.de/public/home");
echo "hallo";
?>
With other pages this code works. Only for "https://www.mytischtennis.de/public/home" it wont work for me?
Can someone help me why i get only with this site a blank page?
Thank you :)
You need to set two more options in your curl request:
// Add some more headers here if you need to
$headers = [
"Accept-Encoding: gzip, deflate, br"
];
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
// Decode the response automatically when needed
curl_setopt($ch, CURLOPT_ENCODING, '');
After this you will get the page you want.
$ch2 = SSLCURL("https://www.tcpvpn.com/create-tcpvpn-account-server");
curl_setopt($ch2, CURLOPT_REFERER, "https://www.tcpvpn.com/free-vpn-server-continent-europe");
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS, "server=115");
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
$ex = curl_exec($ch2);
echo nl2br(str_replace("<","!!",$ex));
curl_close($ch2);
This is the code that I access to website. I handle cookies, SSL access, redirecting actions and UserAgent (Latest Chrome) on SSLCURL function.
The thing is when I access to that website over my browser or even with Glype (a proxy script written in PHP), I can reach to website without problem, but everytime I try to access over my script, I just get a meta redirection. How can I fix it?
edit: here comes the SSLCURL
function SSLCURL($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Accept-Language:tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4",
"Connection:keep-alive",
"Upgrade-Insecure-Requests:1"
));
return $ch;
}
I have script for video player
<?php
$id = 35719350987;
$url = 'http://ok.ru/dk?cmd=videoPlayerMetadata&mid='.$id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIE, 'viewport=1040; _flashVersion=1');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json','Accept: *'));
curl_setopt($ch, CURLOPT_USERAGENT,$_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_REFERER, 'ok.ru/videoembed/'.$id);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$html = curl_exec($ch);
$json = json_decode($html);
?>
The output of browser chrome is
callbackFunc([ {file:"........"video/mp4"}])
Is work
But when i use mobile is not work
When from mobile json need to take [{"name":"mobile","url":"....} , from
http://www.ok.ru/dk?cmd=videoPlayerMetadata&mid=35719350987 but is not take nothing
This line does not work curl_setopt($ch, CURLOPT_USERAGENT,$_SERVER['HTTP_USER_AGENT']); instead curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.3 Mobile'); but it does not work in mobile browsers
I am trying to download a pdf using cURL and am getting stuck on a "Server Error in '/' Application" page. My code:
$url = "https://some.domain.com/Reports/Report?ReportID=123456"
$ch = curl_init($url);
$header = array ('Host: some.domain.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, implode(':', $arrayCiphers));
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_COOKIE, "ASP.NET_SessionId=XXXXXXXX; __RequestVerificationToken_XXXXXXX=lots-of-alpha-numeric-characters");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://some.domain.com");
$output = curl_exec($ch);
echo $output;
curl_close($ch);
Is there something else I can try or some more debugging I can do?
[edit] Apparently it's caused by one of my parameters. There are several parameters in the URL. &Flag=True seems to be causing the error. If I change it to &Flag=False I get a blank page.
This error was being caused because my curl request header did not include cookies that were required to download the file. Adding those cookies in the header fixed it.
This script gets through the main login form, but does not post at the security question screen. Any help? The result just prints out the security question screen, even though I am trying to make it POST.
$username = 'XXXX';
$password = 'XXXX';
$loginUrl = 'https://www.dandh.ca/v4/dh';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $loginUrl);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'Login='.urlencode($username).'&PW='.urlencode($password).'&Request=Login&formName=Login&jsEnabled=0&queryString=&Platform=Full&btLogin='.urlencode('Log In'));
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36");
curl_exec($ch);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'securityAnswer=XXXX&Request=postForm&formName=loginChallengeValidation&btContinue=Continue');
echo curl_exec($ch);
This is because you don't have a cert file. You could set option CURLOPT_SSL_VERIFYPEER to FALSE.