I'm trying to get a Captcha(old, the image one) image from a web page. But, I know it always changes and being regenerated on every HTTP request. But I can't get the image via cURL.
I've tried this with this code in PHP:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://example.com/login.aspx');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36");
curl_setopt($ch, CURLOPT_NETRC, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
$data = curl_exec($ch);
curl_close($ch);
Image just comes as empty. There is field like Captcha but nothing is written on it. I couldn't understand if there is a difference between browser request or cURL request.
Related
I have a authorization to use an image from partiner website to qork on my own.
So I try download the image from they src, with curl:
ini_set('memory_limit', '-1');
error_reporting(E_ALL);
ini_set('display_errors', 1);
$ch = curl_init();
$url = 'https://imgs.kenlo.io/VWRCUkQ2Tnp3d1BJRDBJVe1szkhnWr9UfpZS9bJDwnbk9Kawbnev1nxMNm9yHFhIP-MQkSx9WYNv23hLzO9I+z0StvcEfe6ggnXCx6+I286Lt3WU3rLdkIrG9OL9c8jv4qJ6-arDAl+tq417XfgQOV8gR8xCbkPFTqGKY2BopgOv92LEFusYUk1n0h8jsUZu6AHoQYT1sl0yxA2UXuyB8nTQCaC5XwwWUawNEtEw7ldR7AL0SVpntREFocalqyCZPuL1JoCihipUBAow-56-X7-hkZaPK5RkVqk2dXwP61MDUbcUm+6GWehj7QEYoKGTVkPBwEKa07d3JM+tW4tMnqVWwgaVGrAa6B-pxYyVhaWpZk7UOk5Bqu-A7fCxS+CyRv7jOCq8j4pBrN-bbYYZKprkVzEVTw==.jpg';
//curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE );
curl_setopt($ch, CURLOPT_REFERER, "https://www.ricimoveis.com");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt($ch, CURLOPT_URL, $url );
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE );
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$resource_ext_img_destaque = curl_exec( $ch );
curl_close($ch);
file_put_contents('/var/www/html/magickscp/dev-misc/fdp12.jpg', $resource_ext_img_destaque);
The partiner use a regional platform that store images on imgs.kenlo.io server, apparently that server uses some security tricks to block this operation.
I note somethings like:
The url to img probably have a redirect becouse when i see the requests sometimes a i have '200' or '301'.
So the doubt is, what things i need to evaluate on the response header, to make this work?
I want to get the html content from this link
https://store.nike.com/in/en_gb/pw/boys-shoes/7pvZoi3
and for this i have created the below curl request php script
$ua = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13';
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'https://store.nike.com/in/en_gb/pw/boys-shoes/7pvZoi3');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_USERAGENT, $ua);
curl_setopt($ch, CURLOPT_COOKIE, '<Pasted_cookie>');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 20);
$result = curl_exec($ch);
$last = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
print_r($result);
But the above script redirects me to the page that is showing me a screen to select region.
Please help me as what i need to change to make the script work.
Thanks.
To set location, there is always a network call that set your location in cookies or somewhere else, totally web dependent.
What you can do is, find that call, first mock the location call to set location then hit the main page with same cookies.
I currently learning to web scraping an asynchronous website. First, I need to get the cookie. I'm using the code below to save the cookie to a txt file. But it not save the cookie when I run it. When I access the file, it's empty. I don't know where my problem is, because you know I still a noob in this thing. Hope you guys can answer it. Thanks for your time.
$cookie_file_path = dirname(__FILE__) . "/cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_URL, url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36');
curl_setopt($ch, CURLOPT_TIMEOUT, 40);
curl_exec($ch);
curl_close($ch);
$ch2 = SSLCURL("https://www.tcpvpn.com/create-tcpvpn-account-server");
curl_setopt($ch2, CURLOPT_REFERER, "https://www.tcpvpn.com/free-vpn-server-continent-europe");
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS, "server=115");
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
$ex = curl_exec($ch2);
echo nl2br(str_replace("<","!!",$ex));
curl_close($ch2);
This is the code that I access to website. I handle cookies, SSL access, redirecting actions and UserAgent (Latest Chrome) on SSLCURL function.
The thing is when I access to that website over my browser or even with Glype (a proxy script written in PHP), I can reach to website without problem, but everytime I try to access over my script, I just get a meta redirection. How can I fix it?
edit: here comes the SSLCURL
function SSLCURL($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/jamjar.txt");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
"Accept-Language:tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4",
"Connection:keep-alive",
"Upgrade-Insecure-Requests:1"
));
return $ch;
}
I am trying to retrieve the HTML from a user profile on Instagram using cURL.
I am new to cURL so do not know the cause of this error.
Nothing happens when the cURL is executed , the page seems to refresh?
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.instagram.com/zohebchaudhry1/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookiess.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookiess.txt');
curl_setopt($ch ,CURLOPT_TIMEOUT , 10);
curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36" );
$html = curl_exec($ch);
curl_close($ch);
echo $html;
above is the PHP cURL code.
It appears that cURL is working, however you're unable to see the output because printing HTML may not be desired.
I suggest replacing echo $html; with echo htmlentities($html);
Read more: php.net/htmlentities