I have an URL like this one.
I'm trying to get location from header, but its not showing location as normal.
Here is my function to get header info.
function headerInfo ($service_url) {
$handle = curl_init($service_url);
curl_setopt_array($handle, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_NOBODY => 0,
CURLOPT_HEADER => 1
));
$curl_response = curl_exec($handle);
return $curl_response;
}
Response from PHP:
Response when trying from browser (chrome developer tool result):
Try setting a valid User-Agent.
Like so:
function headerInfo ($service_url) {
$handle = curl_init($service_url);
curl_setopt_array($handle, array(
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_NOBODY => 0,
CURLOPT_HEADER => 1
));
$curl_response = curl_exec($handle);
return $curl_response;
}
You can find a list of valid User-Agents here:
Valid User-Agents
Related
I'm trying to download a xml file from url: https://www.bayzade.net/exportlink_181019170321.xml
But I cant download this with curl. Error: Connection Refused.
But I can see on url from browser. How can I fix this error?
Xampp PHP 7.
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
Connection Refused
you need to set CURLOPT_USERAGENT to make a completely request,
some sites serve completely different content or HTML to different OS and browser versions, this is done by detecting their user agent
just including the CURLOPT_USERAGENT to your CURLOPT:
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
$instagram = "ugur2d";
$kaynak = file_get_contents("https://www.instagram.com/$instagram/");
preg_match('#<title>(.*?)</title>#si', $kaynak, $iglink);
echo $iglink[1];
Screen is empty. How can i run it?
After some checkings...
your url returns error 404 - file_get_contents will return false if 404 is returned by the server
(instagram returns a 404 error page if profile does not exist)
thats why you dont get any result!
anyway you should use curl instead!
Okey okey okey. Im fixed problem. Answer -> CURL :D
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => 'https://toosba.com/',
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => FALSE
]);
$source = curl_exec($ch);
curl_close($ch);
preg_match('/<title>(.*?)<\/title>/', $source, $title);
print_r($title);
echo "<hr>".$title[1];
Thank you :)
I've already been able to do something similar on other websites, but now I'm having trouble authenticating using SSL certificate on a particular site. When I install the certificate in the browser, it works perfectly. But when I convert the .pfx file into .pem files and I try to use it with php + curl, it does not work.
My code is as follows:
<?php
$url = 'https://www2.agencianet.fazenda.df.gov.br/Inicio/Restrita/';
$pemfile = __DIR__ . '/pubKEY.pem';
$keyfile = __DIR__ . '/priKEY.pem';
$ch = curl_init($url);
$options = array(
CURLOPT_FRESH_CONNECT => true,
CURLOPT_HEADER => 1,
CURLOPT_RETURNTRANSFER =>true,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_HTTPHEADER => array(
"Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4,de;q=0.2,es;q=0.2,gl;q=0.2,nl;q=0.2",
"Cache-Control:no-cache",
"Connection:keep-alive",
"Host:www2.agencianet.fazenda.df.gov.br",
"Pragma:no-cache",
"Referer:http://agnet.fazenda.df.gov.br/",
"Upgrade-Insecure-Requests:1",
"User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36"
),
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_SSLVERSION => 0,
CURLOPT_SSLCERT => $pemfile,
CURLOPT_SSLKEY => $keyfile,
CURLOPT_SSLCERTTYPE => 'PEM'
);
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
print_r($data);
print_r(curl_error($ch));
curl_close($ch);
The pem files are these:
drive.google.com/drive/folders/0BxSFCb666mqCbUJIMlhIMjh0aGM?usp=sharing
Some images:
I am using curl to hit a url and get contents from it and below is the code
$url = 'http://uber.com';
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \ Chrome/24.0.1304.0 Safari/537.16',
CURLOPT_CONNECTTIMEOUT => 180,
CURLOPT_TIMEOUT => 180 + 60,
CURLOPT_COOKIE => 'lang=en;'
));
$response = curl_exec($ch);
var_dump($response);
It gives me the content when I execute the script on local server but while executing this on staging and production server, situated in US, it give 403-forbidden error. I also tested the same from another server in US and it works fine.
Any help is appreciated.
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>