I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>
Related
This question already has an answer here:
curl 1020 error when trying to scrape page using bash script
(1 answer)
Closed yesterday.
`I have sent a GET (Also POST) method to get data from api but the response is error 1020 and at the same time post man tried and its giving response then what is wrong here.
I am not able to share the api url due to security reasons.
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'url_with_parameter',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=f1r22a2srpgamnidkflgqsuem2; csrf_token=c31d6332c0066fa652b79948c43e5dd7_d5ec9637ea9c6914f3cebeab7fa4ad4d',
'Host' => "localhost",
'User-Agent' => 'PostmanRuntime/7.31.0',
'Accept' => '*/*',
'Accept-Encoding' => 'gzip, deflate, br',
'Connection' => 'keep-alive'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
When 1020 error is returned by curl, it means typically that the request was blocked by security settings, which can include challenges like CAPTCHAs or browser checks.
Try to add UA header into your curl request, like
$headers = [
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
];
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);
I am using the following code to run curl.But for one url, I am getting 502 Bad Gateway error.
<?php
//$proxy = '127.0.0.1:80';
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => '<requesturl>',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER=>false,
CURLOPT_SSL_VERIFYHOST=>false,
//CURLOPT_PROXY=>$proxy,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=e03338f51c56ada6870d530080127581'
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
print_r($response);
print_r($err);
curl_close($curl);
?>
I have removed https and put http and checked but for that also not working.
My PHP version is 5.6.
Thanks,
Rekha
I have found the answer.I have added user agent and it worked fine.
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
For dynamically get user agent in php,
$userAgent = $_SERVER['HTTP_USER_AGENT'];
This will help someone.
Thanks,
Rekha
I am attempting to download the document of a certain webpage(the stuff you would see if you hit 'view page source'), which has cloudfare protection, then echo it. Here is my current code:
<?php
$ch = curl_init();
$curlConfig = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6',
CURLOPT_FOLLOWLOCATION => 0
);
curl_setopt_array($ch, $curlConfig);
echo(curl_exec($ch));
curl_close($ch);
?>
It redirects me to this url:
http://localhost/cdn-cgi/l/chk_jschl? jschl_vc=7df2b994a588cc3cb3b91ecc95acca7e&pass=1443965612.707-g3h0lQ531e&jschl_answer=377
Of course, I do not have chk_jschl on my localhost. How can I work around this?
My Curl-code works:
$ch = curl_init();
$header = array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Accept-Language: en-us;q=0.8,en;q=0.6'
);
$options = array(
CURLOPT_URL => $url,
CURLOPT_HEADER => 0,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13',
CURLOPT_HTTPHEADER => $header
);
curl_setopt_array($ch, $options);
$str = curl_exec($ch);
curl_close($ch);
But from one page only I get signs start so:
‹�������í}ëvâH²îïîµö;dk¦}™² ...
I detect with mb_detect_encoding that it is an utf8-string. I'm confused, why can't I read the source code like from other sites?
The site I want to curl is http://stores.ebay.com/artlines-2012
Thank you.
It looks like an encoding issue. Try adding CURLOPT_ENCODING.
Example:
curl_setopt($ch, CURLOPT_ENCODING, 'UTF-8');
I am trying to login into a website through cURL and have it return to me the actual page. So far I can only get the footer of the page with the feedback link and this error:
Warning: curl_setopt_array() [function.curl-setopt-array]: Array keys
must be CURLOPT constants or equivalent integer values on line 18
Loading
line 18 is the ");" at the end of the curl array
So far I've been able to figure out that if I just put in the website name with the access_token in my url from my browser I'll automatically be logged in so I'm trying to get cURL to emulate that.
<?php
$ckfile =' __utma=173730677.1410450142.1370837396.1370843059.4; __utmz=173730677.1370843059.4.3.utmcsr=web.com|utmccn=(referral)|utmcmd=referral|utmcct=/ou; __utma=64278953.892306882.1370766510.1370838026.4; __utmz=64278953.4.3.utmcsr=.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; __utmc=64278953; __utmc=173730677; __utmb=173730677.5.10.1370843059; __utmb=64278953.1.10';
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com',
CURLOPT_COOKIEFILE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_GET=> true,
CURLOPT_REFERER => 'https://.com/signin',
CURLOPT_GETFIELDS=>'#access_token=',
CURLOPT_RETURNTRANSFER => true
)
);
$response = curl_exec($ch);
curl_close($ch);
echo '<pre>';
print_r($response);
?>
CURLOPT_GET and CURLOPT_GETFIELDS are not a valid options for cURL. The options may exist for POST, but they do not for GET. Just add the query string to the URL instead.
Also, make sure all the options are set to what cURL expects them to be set to. CURLOPT_COOKIEFILE should be set to a filename, not a string of cookie values. You want CURLOPT_COOKIE instead.
curl_setopt_array($ch, array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com?access_token=abc',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://web.com/signin',
CURLOPT_RETURNTRANSFER => true
));
Please see the PHP cURL docs for all the options: http://php.net/manual/en/function.curl-setopt.php