I am attempting to download the document of a certain webpage(the stuff you would see if you hit 'view page source'), which has cloudfare protection, then echo it. Here is my current code:
<?php
$ch = curl_init();
$curlConfig = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6',
CURLOPT_FOLLOWLOCATION => 0
);
curl_setopt_array($ch, $curlConfig);
echo(curl_exec($ch));
curl_close($ch);
?>
It redirects me to this url:
http://localhost/cdn-cgi/l/chk_jschl? jschl_vc=7df2b994a588cc3cb3b91ecc95acca7e&pass=1443965612.707-g3h0lQ531e&jschl_answer=377
Of course, I do not have chk_jschl on my localhost. How can I work around this?
Related
I am using the following code to run curl.But for one url, I am getting 502 Bad Gateway error.
<?php
//$proxy = '127.0.0.1:80';
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => '<requesturl>',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER=>false,
CURLOPT_SSL_VERIFYHOST=>false,
//CURLOPT_PROXY=>$proxy,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=e03338f51c56ada6870d530080127581'
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
print_r($response);
print_r($err);
curl_close($curl);
?>
I have removed https and put http and checked but for that also not working.
My PHP version is 5.6.
Thanks,
Rekha
I have found the answer.I have added user agent and it worked fine.
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
For dynamically get user agent in php,
$userAgent = $_SERVER['HTTP_USER_AGENT'];
This will help someone.
Thanks,
Rekha
I'm trying to download a xml file from url: https://www.bayzade.net/exportlink_181019170321.xml
But I cant download this with curl. Error: Connection Refused.
But I can see on url from browser. How can I fix this error?
Xampp PHP 7.
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
Connection Refused
you need to set CURLOPT_USERAGENT to make a completely request,
some sites serve completely different content or HTML to different OS and browser versions, this is done by detecting their user agent
just including the CURLOPT_USERAGENT to your CURLOPT:
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
My Curl-code works:
$ch = curl_init();
$header = array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Accept-Language: en-us;q=0.8,en;q=0.6'
);
$options = array(
CURLOPT_URL => $url,
CURLOPT_HEADER => 0,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13',
CURLOPT_HTTPHEADER => $header
);
curl_setopt_array($ch, $options);
$str = curl_exec($ch);
curl_close($ch);
But from one page only I get signs start so:
‹�������í}ëvâH²îïîµö;dk¦}™² ...
I detect with mb_detect_encoding that it is an utf8-string. I'm confused, why can't I read the source code like from other sites?
The site I want to curl is http://stores.ebay.com/artlines-2012
Thank you.
It looks like an encoding issue. Try adding CURLOPT_ENCODING.
Example:
curl_setopt($ch, CURLOPT_ENCODING, 'UTF-8');
How i can redirect user to another web site with cookie?
Im using this code
<?php
$fields_string = 'client_login=jadro&client_pass=jadro&client_remember=on&action=client_login';
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 FirePHP/0.3",
CURLOPT_AUTOREFERER => false,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init();
curl_setopt_array( $ch, $options );
curl_setopt($ch,CURLOPT_URL,'http://orion10.ru');
//curl_setopt($ch,CURLOPT_POST,count(explode('&',$fields)));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
//curl_setopt($ch,CURLOPT_COOKIEJAR, 'cooc.txt');
//curl_setopt($ch,CURLOPT_COOKIEFILE, 'cooc.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, getcwd()."/cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, getcwd()."/cookies.txt");
$result = curl_exec($ch);
echo $result;
//header("Location: http://orion10.ru".session_name().'='.session_id());
header('Refresh: 15; URL='.$url['http://orion10.ru']);
exit();
?>
I need authorize user to another site.
cURL is being executed on your server. Therefore, the website in question thinks that your server is the user. i.e. When you redirect the actual user to the website in question, it won't recognize them. Read this.
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>