I'm trying to download a xml file from url: https://www.bayzade.net/exportlink_181019170321.xml
But I cant download this with curl. Error: Connection Refused.
But I can see on url from browser. How can I fix this error?
Xampp PHP 7.
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
Connection Refused
you need to set CURLOPT_USERAGENT to make a completely request,
some sites serve completely different content or HTML to different OS and browser versions, this is done by detecting their user agent
just including the CURLOPT_USERAGENT to your CURLOPT:
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
Related
I have an URL like this one.
I'm trying to get location from header, but its not showing location as normal.
Here is my function to get header info.
function headerInfo ($service_url) {
$handle = curl_init($service_url);
curl_setopt_array($handle, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_NOBODY => 0,
CURLOPT_HEADER => 1
));
$curl_response = curl_exec($handle);
return $curl_response;
}
Response from PHP:
Response when trying from browser (chrome developer tool result):
Try setting a valid User-Agent.
Like so:
function headerInfo ($service_url) {
$handle = curl_init($service_url);
curl_setopt_array($handle, array(
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_NOBODY => 0,
CURLOPT_HEADER => 1
));
$curl_response = curl_exec($handle);
return $curl_response;
}
You can find a list of valid User-Agents here:
Valid User-Agents
I've already been able to do something similar on other websites, but now I'm having trouble authenticating using SSL certificate on a particular site. When I install the certificate in the browser, it works perfectly. But when I convert the .pfx file into .pem files and I try to use it with php + curl, it does not work.
My code is as follows:
<?php
$url = 'https://www2.agencianet.fazenda.df.gov.br/Inicio/Restrita/';
$pemfile = __DIR__ . '/pubKEY.pem';
$keyfile = __DIR__ . '/priKEY.pem';
$ch = curl_init($url);
$options = array(
CURLOPT_FRESH_CONNECT => true,
CURLOPT_HEADER => 1,
CURLOPT_RETURNTRANSFER =>true,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_HTTPHEADER => array(
"Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,de;q=0.2,es;q=0.2,gl;q=0.2,nl;q=0.2",
"Cache-Control:no-cache",
"Connection:keep-alive",
"Host:www2.agencianet.fazenda.df.gov.br",
"Pragma:no-cache",
"Referer:http://agnet.fazenda.df.gov.br/",
"Upgrade-Insecure-Requests:1",
"User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
),
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_SSLVERSION => 0,
CURLOPT_SSLCERT => $pemfile,
CURLOPT_SSLKEY => $keyfile,
CURLOPT_SSLCERTTYPE => 'PEM'
);
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
print_r($data);
print_r(curl_error($ch));
curl_close($ch);
The pem files are these:
drive.google.com/drive/folders/0BxSFCb666mqCbUJIMlhIMjh0aGM?usp=sharing
Some images:
my below code of PHP is not working. I have created test user id & password which is shared in the code. The file is not having any value. But I see with this code it is edited every time when I execute this code.
<?php
$username = '8632465';
$password = 'basade41';
$loginUrl = 'https://cabinet.instaforex.com/partner/login';
//$userAgent = 'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0';
$config['useragent'] = 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0';
//$userAgent = $_SERVER['HTTP_USER_AGENT'];
//init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $config['useragent']);
//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $loginUrl);
// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);
//Set the post parameters
curl_setopt($ch, CURLOPT_POSTFIELDS, 'user='.$username.'&pass='.$password);
//Handle cookies for the login
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute the request (the login)
$store = curl_exec($ch);
//the login is now done and you can continue to get the
//protected content.
//set the URL to the protected file
curl_setopt($ch, CURLOPT_URL, 'https://cabinet.instaforex.com/partner/aff_statistic');
//execute the request
$content = curl_exec($ch);
//save the data to disk
file_put_contents('c:/xampp/htdocs/upload/test.txt', $content);
?>
I think your code missing some settings. eg:CURLOPT_CAINFO...
This is my https code
<?php
//The CA certificate path
$caDir = '....../GeoTrustGlobalCA.crt';
//cookie file path
$cookieDir = '......';
//URL
$url = 'https://www.zhihu.com/......';
$defaultOpt = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36",
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_SSL_VERIFYPEER => true, // enabled SSL Cert checks
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_CAINFO => $caDir,
CURLOPT_COOKIEFILE => $cookieDir,
CURLOPT_COOKIEJAR => $cookieDir,
CURLOPT_URL => $url,
CURLOPT_HTTPGET => true //method is GET
);
$handle = curl_init();
curl_setopt_array($handle,$arrOpt);
$content = curl_exec($handle);
$err = curl_errno($handle);
$errmsg = curl_error($handle);
$header = curl_getinfo($handle);
curl_close($handle);
printf("Content is %s", $content);
?>
you can refer to this article php-curl-https (in (^__^)) OR this using-curl-in-php-to-access-https-ssltls-protected-sites
I am attempting to download the document of a certain webpage(the stuff you would see if you hit 'view page source'), which has cloudfare protection, then echo it. Here is my current code:
<?php
$ch = curl_init();
$curlConfig = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6',
CURLOPT_FOLLOWLOCATION => 0
);
curl_setopt_array($ch, $curlConfig);
echo(curl_exec($ch));
curl_close($ch);
?>
It redirects me to this url:
http://localhost/cdn-cgi/l/chk_jschl? jschl_vc=7df2b994a588cc3cb3b91ecc95acca7e&pass=1443965612.707-g3h0lQ531e&jschl_answer=377
Of course, I do not have chk_jschl on my localhost. How can I work around this?
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>