PHP curl and getting HTTPS site's html source code - php

I wrote this code for getting https pages content but i couldnt succesfull.
<?php
function bot($url)
{
$header ="Host: tr-tr.facebook.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0\r\n
Accept: */*
Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br";
$options = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_PORT => 443,
CURLOPT_SSL_VERIFYPEER => true,
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_CAINFO => "C:\\xampp\\htdocs\\curl-ca-bundle.crt",
CURLOPT_HTTPHEADER => explode("\r\n",$header)
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
echo bot("https://tr-tr.facebook.com");
?>
When i run that codes it is returning that. "��0#a�jȌ#�#.3�j�##u�.����/#cw#,�q=ߓ���K"<�˞#%#����t[�d�:|��s##$!� ��(��M��ߛ#w'#u" Where is my mistake? Why is returning these characters?
I dont want to use CURLOPT_SSL_VERIFYPEER = false. I want to https handshake with curl..

Maybe you should remove CURLOPT_RETURNTRANSFER or handle it properly.
From the manual
http://php.net/manual/en/function.curl-setopt.php
Do you really want this?!
> CURLOPT_RETURNTRANSFER
> TRUE to return the transfer as a string of the
> return value of curl_exec() instead of outputting it out directly.

Related

Why curl giving 1020 error while postman giving response [duplicate]

This question already has an answer here:
curl 1020 error when trying to scrape page using bash script
(1 answer)
Closed yesterday.
`I have sent a GET (Also POST) method to get data from api but the response is error 1020 and at the same time post man tried and its giving response then what is wrong here.
I am not able to share the api url due to security reasons.
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'url_with_parameter',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=f1r22a2srpgamnidkflgqsuem2; csrf_token=c31d6332c0066fa652b79948c43e5dd7_d5ec9637ea9c6914f3cebeab7fa4ad4d',
'Host' => "localhost",
'User-Agent' => 'PostmanRuntime/7.31.0',
'Accept' => '*/*',
'Accept-Encoding' => 'gzip, deflate, br',
'Connection' => 'keep-alive'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
When 1020 error is returned by curl, it means typically that the request was blocked by security settings, which can include challenges like CAPTCHAs or browser checks.
Try to add UA header into your curl request, like
$headers = [
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
];
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);

How to make PHP bot print the title

$instagram = "ugur2d";
$kaynak = file_get_contents("https://www.instagram.com/$instagram/");
preg_match('#<title>(.*?)</title>#si', $kaynak, $iglink);
echo $iglink[1];
Screen is empty. How can i run it?
After some checkings...
your url returns error 404 - file_get_contents will return false if 404 is returned by the server
(instagram returns a 404 error page if profile does not exist)
thats why you dont get any result!
anyway you should use curl instead!
Okey okey okey. Im fixed problem. Answer -> CURL :D
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => 'https://toosba.com/',
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => FALSE
]);
$source = curl_exec($ch);
curl_close($ch);
preg_match('/<title>(.*?)<\/title>/', $source, $title);
print_r($title);
echo "<hr>".$title[1];
Thank you :)

Request forbidden while accessing github api on php program

I'm reading the past of the question (Request forbidden while accessing github api on node.js program), but still does not solve the problem.
PHP code
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => "https://api.github.com/markdown/raw",
CURLOPT_HEADER => [
"Accept: application/vnd.github.v3+json",
"Content-Type: text/plain",
"User-Agent: mfmfneko"
],
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => "# Hello World!"
]);
$response = curl_exec($ch);
curl_close($ch);
print($response);
Response
HTTP/1.0 403 Forbidden
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Request forbidden by administrative rules. Please make sure your request has a User-Agent header (http://developer.github.com/v3/#user-agent-required). Check https://developer.github.com for other possible causes.
You made a mistake with the installation of headers.
CURLOPT_HEADER TRUE to include the header in the output.
CURLOPT_HTTPHEADER An array of HTTP header fields to set, in the
format array('Content-type: text/plain', 'Content-length: 100')
They should be reported using the CURLOPT_HTTPHEADER
Try this code:
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => "https://api.github.com/markdown/raw",
CURLOPT_HTTPHEADER => [
"Accept: application/vnd.github.v3+json",
"Content-Type: text/plain",
"User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 YaBrowser/16.3.0.7146 Yowser/2.5 Safari/537.36"
],
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => "# Hello World!"
]);
$response = curl_exec($ch);
curl_close($ch);
print($response);
Response:
<h1>
<a id="user-content-hello-world" class="anchor" href="#hello-world" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Hello World!</h1>
More details can be read here

PHP solution to JSON responses and parameters

I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>

curl_setopt_array throwing warning message

I am trying to login into a website through cURL and have it return to me the actual page. So far I can only get the footer of the page with the feedback link and this error:
Warning: curl_setopt_array() [function.curl-setopt-array]: Array keys
must be CURLOPT constants or equivalent integer values on line 18
Loading
line 18 is the ");" at the end of the curl array
So far I've been able to figure out that if I just put in the website name with the access_token in my url from my browser I'll automatically be logged in so I'm trying to get cURL to emulate that.
<?php
$ckfile =' __utma=173730677.1410450142.1370837396.1370843059.4; __utmz=173730677.1370843059.4.3.utmcsr=web.com|utmccn=(referral)|utmcmd=referral|utmcct=/ou; __utma=64278953.892306882.1370766510.1370838026.4; __utmz=64278953.4.3.utmcsr=.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; __utmc=64278953; __utmc=173730677; __utmb=173730677.5.10.1370843059; __utmb=64278953.1.10';
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com',
CURLOPT_COOKIEFILE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_GET=> true,
CURLOPT_REFERER => 'https://.com/signin',
CURLOPT_GETFIELDS=>'#access_token=',
CURLOPT_RETURNTRANSFER => true
)
);
$response = curl_exec($ch);
curl_close($ch);
echo '<pre>';
print_r($response);
?>
CURLOPT_GET and CURLOPT_GETFIELDS are not a valid options for cURL. The options may exist for POST, but they do not for GET. Just add the query string to the URL instead.
Also, make sure all the options are set to what cURL expects them to be set to. CURLOPT_COOKIEFILE should be set to a filename, not a string of cookie values. You want CURLOPT_COOKIE instead.
curl_setopt_array($ch, array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com?access_token=abc',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://web.com/signin',
CURLOPT_RETURNTRANSFER => true
));
Please see the PHP cURL docs for all the options: http://php.net/manual/en/function.curl-setopt.php

Categories