Curl in PHP like a Real Browser Still Detected as a Bot - php

So I'm trying to just get the HTML from a page. I have added any possible data into curl headers SSL anything. But they still know that its a CURL BOT. How can I bypass this or how they do it?
When I visit other pages from them I dont get Detected as a Bot only when I'm on search
$url = "https://suchen.mobile.de/fahrzeuge/search.html?damageUnrepaired=NO_DAMAGE_UNREPAIRED&isSearchRequest=true&maxPowerAsArray=PS&minPowerAsArray=PS&scopeId=C";
$data = curl($url);
echo $data;
function curl($url, $post = "") {
$cookie = "cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('authority: suchen.mobile.de', 'path: /fahrzeuge/search.html?damageUnrepaired=NO_DAMAGE_UNREPAIRED&isSearchRequest=true&maxPowerAsArray=PS&minPowerAsArray=PS&scopeId=C', 'scheme: https', 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'accept-encoding: gzip, deflate, br', 'accept-language: en-US,en;q=0.9', 'upgrade-insecure-requests: 1'));
$data = curl_exec ($ch);
if (curl_error($ch))
return "Bad";
if (curl_getinfo($ch)["http_code"] == 200)
return $data;
}

Related

php curl is not working while debug proxy server like fiddler is working

I am trying to scrape from a site which is behind cloudflare.
My code is::
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'https://targetsite');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, '{"current_bid_status":true}');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36',
'Accept: application/json',
'Accept-Language: en-US,en;q=0.5',
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: '.strlen($data)
]);
$result = curl_exec($ch);
$status = curl_getinfo ($ch);
The response header was 403, and the response body was error code: 1020
It looks like cloudflare is blocking the request.
But when i add fiddler proxy:
curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:8888');
It works nicely!
What can be the possible reason here? Is it something related with ssl certificate?
_

PHP curl ; symbol on url

I need to access this URL on php:
https://wmf.ok.ru/play;jsessionid=a-pt2O8FJKq_wzqod9LAJNtwgjNSjaNa-KVIGc1d1eRUSWhdAw9dlDo13fLzh57rGyKPzk2V0jMFrnKw8R4HjA.p162X6pZ_FG0kKMmKa6bkQ?client=flash&jsonp=&tid=40542951634095&ctx=my
But on my PHP code I got 404 error. I have done everything correctly. I think there is a mistake with ; symbol. We can open the link above on chrome, but not on php curl. Here is my code:
<?php
error_reporting(E_ALL);
ini_set('display_errors', 1);
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch,CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Accept: application/json, text/javascript, */*; q=0.01',
'Accept-Encoding: gzip, deflate, br',
'Accept-Language: en-US,en;q=0.9,az;q=0.8,tr;q=0.7,uz;q=0.6,ru;q=0.5',
'Referer: https://ok.ru/',
'Origin: https://ok.rus'
));
$data = curl_exec($ch);
if(curl_error($ch))
{
echo 'error:' . curl_error($ch);
}
echo curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $data;
}
$url = 'https://wmf.ok.ru/play;jsessionid=a-pt2O8FJKq_wzqod9LAJNtwgjNSjaNa-KVIGc1d1eRUSWhdAw9dlDo13fLzh57rGyKPzk2V0jMFrnKw8R4HjA.p162X6pZ_FG0kKMmKa6bkQ?client=flash&jsonp=&tid=40542951634095&ctx=my';
echo file_get_contents_curl($url);
?>
After executing this code, I got microsoft's server 404 error. How can I make Curl to open URLs like this?
Just add this to your function and it will work:
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
Here is the full working function:
<?php
error_reporting(E_ALL);
ini_set('display_errors', 1);
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch,CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Accept: application/json, text/javascript, */*; q=0.01',
'Accept-Encoding: gzip, deflate, br',
'Accept-Language: en-US,en;q=0.9,az;q=0.8,tr;q=0.7,uz;q=0.6,ru;q=0.5',
'Referer: https://ok.ru/',
'Origin: https://ok.rus'
));
$data = curl_exec($ch);
if(curl_error($ch))
{
echo 'error:' . curl_error($ch);
}
echo curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $data;
}
$url = 'https://wmf.ok.ru/play;jsessionid=a-pt2O8FJKq_wzqod9LAJNtwgjNSjaNa-KVIGc1d1eRUSWhdAw9dlDo13fLzh57rGyKPzk2V0jMFrnKw8R4HjA.p162X6pZ_FG0kKMmKa6bkQ?client=flash&jsonp=&tid=40542951634095&ctx=my';
echo file_get_contents_curl($url);
?>

file_get_contents not working in wayfair page

i am having a problem with PHP file_get_contents.i am trying to fetch inforamtion following url but is getting captcha page.
$link = 'http://www.wayfair.com/a/product_review_page/get_update_reviews_json?_format=json&product_sku=KUS1523&page_number=5&sort_order=relevance&filter_rating=&filter_tag=&item_per_page=5';
$Page_information = file_get_contents($link);
print_r($Page_information);
Also i am trying to get page information using php curl but same captcha page is display.
$cookie='cookie.txt';
if(!file_exists($cookie)){
$fh = fopen($cookie, "w");
fwrite($fh, "");
fclose($fh);
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_URL, "http://www.wayfair.com/a/product_review_page/get_update_reviews_json?_format=json&product_sku=KUS1523&page_number=5&sort_order=relevance&filter_rating=&filter_tag=&item_per_page=5");
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_COOKIE,1);
curl_setopt($ch, CURLOPT_COOKIEJAR,$cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE,$cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
$result11 = curl_exec($ch);
print_r($result11);
If you analyze the headers from a browser where cookies and javascript are disabled you should see the bare minimum sent - some, perhaps all might be required and are set with the context argument.
/* set the options for the stream context */
$args=array(
'http'=>array(
'method' => "GET",
'header' => array(
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Host: www.wayfair.com',
'Accept-Encoding: gzip, deflate'
)
)
);
/* create the context */
$context=stream_context_create( $args );
$link = 'http://www.wayfair.com/a/product_review_page/get_update_reviews_json?_format=json&product_sku=KUS1523&page_number=5&sort_order=relevance&filter_rating=&filter_tag=&item_per_page=5';
/* Get the response from remote url */
$res = file_get_contents( $link, FILE_TEXT, $context );
/* process the response */
print_r( $res );
$url = "http://www.wayfair.com/a/product_review_page/get_update_reviews_json?_format=json&product_sku=KUS1523&page_number=5&sort_order=relevance&filter_rating=&filter_tag=&item_per_page=5";
$cookie = getcwd().DIRECTORY_SEPARATOR.'cookie.txt';
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_COOKIE,1);
curl_setopt($ch, CURLOPT_COOKIEJAR,$cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE,$cookie);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
//added
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36");
$result11 = curl_exec($ch);
print_r($result11);
try this

JSON for mobile

I have script for video player
<?php
$id = 35719350987;
$url = 'http://ok.ru/dk?cmd=videoPlayerMetadata&mid='.$id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIE, 'viewport=1040; _flashVersion=1');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json','Accept: *'));
curl_setopt($ch, CURLOPT_USERAGENT,$_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_REFERER, 'ok.ru/videoembed/'.$id);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
#curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$html = curl_exec($ch);
$json = json_decode($html);
?>
The output of browser chrome is
callbackFunc([ {file:"........"video/mp4"}])
Is work
But when i use mobile is not work
When from mobile json need to take [{"name":"mobile","url":"....} , from
http://www.ok.ru/dk?cmd=videoPlayerMetadata&mid=35719350987 but is not take nothing
This line does not work curl_setopt($ch, CURLOPT_USERAGENT,$_SERVER['HTTP_USER_AGENT']); instead curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.3 Mobile'); but it does not work in mobile browsers

Paypal login via Curl (PHP)

There is a problem when I try to login to Paypal via Curl, I think I have all post fields correctly setup, but it doesn't give any output. I want to receive the page content after login, but there is no actual output.
Code:
$loginUrl = 'https://www.paypal.com/en/cgi-bin/webscr?cmd=_login-submit&dispatch=5885d80a13c0db1f8e263663d3faee8d66f31424b43e9a70645c907a6cbd8fb4';
//init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $loginUrl);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'User-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-language: nl,en-US;q=0.7,en;q=0.3',
'accept-encoding: gzip, deflate'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'login_email='.urlencode($username).'&login_password='.urlencode($password).'&submit=inloggen&browser_name=Firefox&browser_version=32&browser_version_full=32.0&operating_system=Windows&bp_mid='.urlencode('v=1;a1=na~a2=na~a3=na~a4=Mozilla~a5=Netscape~a6=5.0 (Windows)~a7=20100101~a8=na~a9=true~a10=Windows NT 6.3; WOW64~a11=true~a12=Win32~a13=na~a14=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0~a15=false~a16=nl~a17=na~a18=www.paypal.com~a19=na~a20=na~a21=na~a22=na~a23=1600~a24=900~a25=24~a26=860~a27=na~a28=Wed Oct 29 2014 17:56:23 GMT+0100~a29=1~a30=na~a31=yes~a32=na~a33=na~a34=no~a35=no~a36=yes~a37=no~a38=online~a39=no~a40=Windows NT 6.3; WOW64~a41=no~a42=no~&').'bpks1='. urlencode('v=1;l=8;Di0:141497Ui0:99Di1:182').'&bpks2=&bpks3=');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$content = curl_exec($ch);
exit(var_dump($content)); // OUTPUT: string(26) ▒▒";
I have tried this on other websites successfully, so I have no idea why this isn't working? Any advice would be appreciated.
try this
<?php
$ch = curl_init();
//Header
curl_setopt($ch, CURLOPT_URL, "https://www.paypal.com/pl/cgi-bin/webscr?cmd=_login-submit&dispatch=5885d80a13c0db1f8e263663d3faee8de62a88b92df045c56447d40d60b23a7c");
//curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // return server response
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch, CURLOPT_HTTPHEADER, array("REMOTE_ADDR: 127.0.0.1", "HTTP_X_FORWARDED_FOR: 127.0.0.1"));
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookieJar.txt'); // save cookie file
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_REFERER, 'https://www.paypal.com');
curl_setopt ($ch, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($ch, CURLOPT_POST, 1); // use post data
$post = array(
"login_cmd" => null,
"login_params" => null,
"login_email" => "test",
"login_password" => "test",
"submit.x" => "login",
//"auth" => "AOeCYVv0IxkugC2Pyz2AhTaW2P7hWuy5w9FoeuyB48gjjJZN3mTtuL79Tzs9dY.CF",
"form_charset" => "UTF-8",
"browser_name" => "Chrome",
"browser_version" => "537.36",
//"browser_version_full" => "40.0.2214.115",
//"operating_system" => "Windows",
);
$post = http_build_query($post);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$data = curl_exec($ch);
if(curl_errno($ch))
{
echo 'error:' . curl_error($ch);
}
curl_close($ch);
print_r($data);
?>

Categories