I am trying to download a pdf using cURL and am getting stuck on a "Server Error in '/' Application" page. My code:
$url = "https://some.domain.com/Reports/Report?ReportID=123456"
$ch = curl_init($url);
$header = array ('Host: some.domain.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, implode(':', $arrayCiphers));
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_COOKIE, "ASP.NET_SessionId=XXXXXXXX; __RequestVerificationToken_XXXXXXX=lots-of-alpha-numeric-characters");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://some.domain.com");
$output = curl_exec($ch);
echo $output;
curl_close($ch);
Is there something else I can try or some more debugging I can do?
[edit] Apparently it's caused by one of my parameters. There are several parameters in the URL. &Flag=True seems to be causing the error. If I change it to &Flag=False I get a blank page.
This error was being caused because my curl request header did not include cookies that were required to download the file. Adding those cookies in the header fixed it.
Related
I'm currently trying to feed my PHP an URL which I know leads to downloading a file.
Unfortunately, it seems like the place where the file is being taken from performs some kind of check that won't let me download the file
After reviewing some similar questions I got this:
fopen('cookies.txt', "w");
$parts = parse_url($url);
$host = $parts['host'];
$ch = curl_init();
$header = array('GET /1575051 HTTP/1.1',
"Host: {$host}",
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive'
);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$result = curl_exec($ch);
curl_close($ch);
die(print_r($result)) ;
But unfortunately my Print_r still returns false. The url is this:
"https://www.yt-download.org/download/lGS9flfg8Vg/mp3/128/1625766046/246071ecaf23ce7f82782a54c1c44ea8343aa59de0723cee7a98bfd2c1865b59/0"
Anybody could spare a hand in understanding more about this?
for some time I have been trying to write the ideal browser simulations, for this purpose I wrote a script that to some extent simulates the browser and works correctly on many pages with ssl. Recently during the test site pornhub.com and wikipedia.com encountered a strange error in my script, just for pornhub after a few page reloads shows the status of header "Loading .. Content-length: 1456" and the number of loaded data changes in real time on smaller and larger values. I have a question for already very experienced and professional programmers: Have you met with such a situation, if so you have any hints or corrections for my script ?.
I post my code (test for wikipedia). If you fire it on 3 browser tabs and you will refresh, you will get an error.
<?php
function curl($url)
{
$headers = [
'Accept-Language: pl,en-US;q=0.9,en;q=0.8',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
];
$cookie = 'cookie.txt';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_REFERER, 'https://www.wikipedia.org');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
if (!file_exists($cookie)){
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
}else{
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
}
$c = curl_exec($ch);
curl_close($ch);
return $c;
}
echo curl('https://www.wikipedia.org');
?>
I am getting curl http code as 403:
The user myuser was denied access to perform the operation on the object defect due to the following reasons: The permission setting for operation Create doesn't allow user to perform the required operation on the secured object.
Same code was working fine with 'HTTP', now it is 'HTTPS', so I updated code like added curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); and curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); these lines and it is giving 403 as HTTP CODE. Can someone help me where I am wrong.
$ch = curl_init(MY_BASE_URL);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13');
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_COOKIEFILE, MY_COOKIES);
curl_setopt($ch, CURLOPT_COOKIEJAR, MY_COOKIES);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/json; charset=utf-8","Accept:application/json, text/javascript, */*; q=0.01"));
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($myfields));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response=curl_exec($ch);
$code=curl_getinfo($ch, CURLINFO_HTTP_CODE);
if($code!=201)
{throw new Exception("Error creating defect: ".$response);}
else{echo 'Success';}
try this user agent(if not works then try with random user agents)
$useragent= "curl/7.39.0";
curl_setopt($ch,CURLOPT_USERAGENT, $useragent);
I prefer if this could be done using wget otherwise update my curl code. I have searched a lot and tried many suggested options like cookies, referer url, user-agent.
Here is my code so far:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.symop.com/config/includes/common/dw.asp?PF=Publications&NF=symop_guide2013.pdf');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_REFERER, "http://www.symop.com/fr/Publications.asp");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
302 Object Moved
I'm trying to use cURL and PHP to download the HTML source (as it appears in the browser) of here. But instead of the actual source code, this is returned instead (a meta refresh link set to 0).
<html>
<head><title>Object moved</title></head>
<body>
<h2>Object moved to here.
</h2>
</body>
</html>
I'm trying to spoof the referral header to be the site, but it seems I'm doing it wrong. Code is below. Any suggestions? Thanks
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e");
$html = curl_exec($ch);
curl_close($ch);
Add the curl option to follow redirects:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
If it is a meta refresh and not an HTTP moved header, see:
PHP: Can CURL follow meta redirects
As mentioned by flesk, you may also need to store the cookies.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.windowsphone.com/en-US/apps/ea39f002-ac30-e011-854c-00237de2db9e');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_REFERER, "http://www.windowsphone.com");
$html = curl_exec($ch);
curl_close($ch);
echo $html;
The problem isn't the referrer but that you need to enable cookies for it to work.
Try something like this:
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
You have to query the page twice. First allow redirects to get the cookie from login.live.com, then query again with the cookie set.