I'm trying to get html as response but it only works in postman but not in php. In php I get different response like "Something went wrong..."
What I'm missing ?
Here is code:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.avnet.com/shop/SearchDisplay?searchTerm=LMK316BJ476ML-T&countryId=apac&deflangId=-1&storeId=715839038&catalogId=10001&langId=-1&sType=SimpleSearch&resultCatEntryType=2&searchSource=Q&searchType=100&avnSearchType=all",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
postman response
php response
Thanks in advance.
I played around the CURLOPT_HTTPHEADER and figured out the site is validating 2 headers which are User-Agent and Accept-Language. I sent it correctly and it worked. Please find the headers below
CURLOPT_HTTPHEADER => array(
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
"Accept-Language:en-US,en;q=0.5"
)
Enjoyed figuring out the issue.
I had the same problem. This might solve the problem:
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
Last night, I spent more than 3 hours to solve the similar problem and at the end, I noticed it was related with server configuration. My sysadmin solved the problem by updating configuration, as I understand, some server configuration doesn't let you send CURL request same server.
Check below ticket:
https://talk.plesk.com/threads/curl-dont-resolved-correctly-at-same-server.295173/
curl_setopt($ch, CURLOPT_HTTPHEADER, ["Accept: application/json","User-Agent: WooCommerce API Client-PHP/3.1.0"]);
After spending lots of time, above Header works for me.
remove this
CURLOPT_ENCODING => "",
Related
This question already has an answer here:
curl 1020 error when trying to scrape page using bash script
(1 answer)
Closed yesterday.
`I have sent a GET (Also POST) method to get data from api but the response is error 1020 and at the same time post man tried and its giving response then what is wrong here.
I am not able to share the api url due to security reasons.
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'url_with_parameter',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=f1r22a2srpgamnidkflgqsuem2; csrf_token=c31d6332c0066fa652b79948c43e5dd7_d5ec9637ea9c6914f3cebeab7fa4ad4d',
'Host' => "localhost",
'User-Agent' => 'PostmanRuntime/7.31.0',
'Accept' => '*/*',
'Accept-Encoding' => 'gzip, deflate, br',
'Connection' => 'keep-alive'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
When 1020 error is returned by curl, it means typically that the request was blocked by security settings, which can include challenges like CAPTCHAs or browser checks.
Try to add UA header into your curl request, like
$headers = [
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
];
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);
I have simple GET request on postman and it does working fine
https://www.instagram.com/p/CVhuRABqnAI/?__a=1&__d=dis
When Im using PHP CURL its did not respond me with Json data rather showing logo
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0",
CURLOPT_URL => "https://www.instagram.com/p/CVhuRABqnAI/?__a=1&__d=dis",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
));
$response = curl_exec($curl);
$error = curl_error($curl);
curl_close($curl);
if($error) {
echo "cURL Error:" . $error;
} else {
echo $response;
}
Can some one else me with this?
Finally I might know what to do as I just faced this problem.
Usually, there is just some kind of difference in between postman and curl. Or there is "something" else, because my solution was just to not send any headers at all...
I am using the following code to run curl.But for one url, I am getting 502 Bad Gateway error.
<?php
//$proxy = '127.0.0.1:80';
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => '<requesturl>',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER=>false,
CURLOPT_SSL_VERIFYHOST=>false,
//CURLOPT_PROXY=>$proxy,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=e03338f51c56ada6870d530080127581'
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
print_r($response);
print_r($err);
curl_close($curl);
?>
I have removed https and put http and checked but for that also not working.
My PHP version is 5.6.
Thanks,
Rekha
I have found the answer.I have added user agent and it worked fine.
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
For dynamically get user agent in php,
$userAgent = $_SERVER['HTTP_USER_AGENT'];
This will help someone.
Thanks,
Rekha
sorry could not get the html source code of this url:http://www.picbear.com/tag/ok;
i always get a blank page with a ["Hello Stranger"] message.
no problems if i use chrome's tools.
i've always used file_get_contents or php_simple_html DOM parser with no problems.
i can't access to the php.ini because of my provider restrictions but it should be ok.
help anyone? thanks.
$url = "http://www.picbear.com/tag/ok";
$html = file_get_contents($url);
echo $html;
Just add User-Agent header:
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "http://picbear.com/tag/ok",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" // Here we add the header
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>