I have a authorization to use an image from partiner website to qork on my own.
So I try download the image from they src, with curl:
ini_set('memory_limit', '-1');
error_reporting(E_ALL);
ini_set('display_errors', 1);
$ch = curl_init();
$url = 'https://imgs.kenlo.io/VWRCUkQ2Tnp3d1BJRDBJVe1szkhnWr9UfpZS9bJDwnbk9Kawbnev1nxMNm9yHFhIP-MQkSx9WYNv23hLzO9I+z0StvcEfe6ggnXCx6+I286Lt3WU3rLdkIrG9OL9c8jv4qJ6-arDAl+tq417XfgQOV8gR8xCbkPFTqGKY2BopgOv92LEFusYUk1n0h8jsUZu6AHoQYT1sl0yxA2UXuyB8nTQCaC5XwwWUawNEtEw7ldR7AL0SVpntREFocalqyCZPuL1JoCihipUBAow-56-X7-hkZaPK5RkVqk2dXwP61MDUbcUm+6GWehj7QEYoKGTVkPBwEKa07d3JM+tW4tMnqVWwgaVGrAa6B-pxYyVhaWpZk7UOk5Bqu-A7fCxS+CyRv7jOCq8j4pBrN-bbYYZKprkVzEVTw==.jpg';
//curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE );
curl_setopt($ch, CURLOPT_REFERER, "https://www.ricimoveis.com");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE );
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$resource_ext_img_destaque = curl_exec( $ch );
curl_close($ch);
file_put_contents('/var/www/html/magickscp/dev-misc/fdp12.jpg', $resource_ext_img_destaque);
The partiner use a regional platform that store images on imgs.kenlo.io server, apparently that server uses some security tricks to block this operation.
I note somethings like:
The url to img probably have a redirect becouse when i see the requests sometimes a i have '200' or '301'.
So the doubt is, what things i need to evaluate on the response header, to make this work?
Related
I'm trying to get a Captcha(old, the image one) image from a web page. But, I know it always changes and being regenerated on every HTTP request. But I can't get the image via cURL.
I've tried this with this code in PHP:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://example.com/login.aspx');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36");
curl_setopt($ch, CURLOPT_NETRC, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
$data = curl_exec($ch);
curl_close($ch);
Image just comes as empty. There is field like Captcha but nothing is written on it. I couldn't understand if there is a difference between browser request or cURL request.
for some time I have been trying to write the ideal browser simulations, for this purpose I wrote a script that to some extent simulates the browser and works correctly on many pages with ssl. Recently during the test site pornhub.com and wikipedia.com encountered a strange error in my script, just for pornhub after a few page reloads shows the status of header "Loading .. Content-length: 1456" and the number of loaded data changes in real time on smaller and larger values. I have a question for already very experienced and professional programmers: Have you met with such a situation, if so you have any hints or corrections for my script ?.
I post my code (test for wikipedia). If you fire it on 3 browser tabs and you will refresh, you will get an error.
<?php
function curl($url)
{
$headers = [
'Accept-Language: pl,en-US;q=0.9,en;q=0.8',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
];
$cookie = 'cookie.txt';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_REFERER, 'https://www.wikipedia.org');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
if (!file_exists($cookie)){
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
}else{
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
}
$c = curl_exec($ch);
curl_close($ch);
return $c;
}
echo curl('https://www.wikipedia.org');
?>
I have a question I can't understand....
I use CURL to this page....
https://propiedades.com/alvaro-obregon-df/departamentos-renta
My Code is this:
$ch = curl_init($uri); // Init cURL session
curl_setopt($ch, CURLOPT_URL, $uri);
curl_setopt($ch, CURLOPT_NOBODY, false);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
$uagent = 'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Firefox/22.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/36.0.1985.125 Chrome/36.0.1985.125 Safari/537.36';
curl_setopt($ch, CURLOPT_USERAGENT, $uagent);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
curl_setopt( $ch, CURLOPT_MAXREDIRS, 10);
if (!is_null($postFields))
{
$arr = explode("&", $postFields);
curl_setopt($ch, CURLOPT_POST, count(sizeof($arr)));
curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
}
$info = curl_exec($ch);
curl_close($ch);
In my Local host $info Always HAS all HTML content... in My Remote Server (where is the hosting of my web app)... $info always is Empty. Why???
Really I don't understand what is happening?
Some ideas?
Help me please!
PD. If I use single CURL or curl_multi_exec (MULTI-CURL with array of URLs goes bad always with the page propiedades.com .. in my LOCAL XAMP Server returns HTML Content... in my Remote Server return NOTHING).
$ch2 = SSLCURL("https://www.tcpvpn.com/create-tcpvpn-account-server");
curl_setopt($ch2, CURLOPT_REFERER, "https://www.tcpvpn.com/free-vpn-server-continent-europe");
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS, "server=115");
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
$ex = curl_exec($ch2);
echo nl2br(str_replace("<","!!",$ex));
curl_close($ch2);
This is the code that I access to website. I handle cookies, SSL access, redirecting actions and UserAgent (Latest Chrome) on SSLCURL function.
The thing is when I access to that website over my browser or even with Glype (a proxy script written in PHP), I can reach to website without problem, but everytime I try to access over my script, I just get a meta redirection. How can I fix it?
edit: here comes the SSLCURL
function SSLCURL($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Accept-Language:tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4",
"Connection:keep-alive",
"Upgrade-Insecure-Requests:1"
));
return $ch;
}
I am trying to download a pdf using cURL and am getting stuck on a "Server Error in '/' Application" page. My code:
$url = "https://some.domain.com/Reports/Report?ReportID=123456"
$ch = curl_init($url);
$header = array ('Host: some.domain.com');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, implode(':', $arrayCiphers));
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_COOKIE, "ASP.NET_SessionId=XXXXXXXX; __RequestVerificationToken_XXXXXXX=lots-of-alpha-numeric-characters");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_REFERER, "https://some.domain.com");
$output = curl_exec($ch);
echo $output;
curl_close($ch);
Is there something else I can try or some more debugging I can do?
[edit] Apparently it's caused by one of my parameters. There are several parameters in the URL. &Flag=True seems to be causing the error. If I change it to &Flag=False I get a blank page.
This error was being caused because my curl request header did not include cookies that were required to download the file. Adding those cookies in the header fixed it.