I want to display some information on my website directly from Twitter. I know that there is their own API but what I need does not need all this since it is just information from the Twitter Home: https://twitter.com/explore
My code is just below I used to, but Twitter updated recently and now I can't.
$url = 'https://twitter.com/explore';
$options = [
CURLOPT_URL => $url,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HEADER => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
];
$ch = curl_init();
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
curl_close($ch);
This code shows me the Twitter page but says it was not found.
Sorry, that page doesn’t exist!
If I remove the User-Agent it shows me a page with the old version of Twitter but I need the new one what should I do?
Related
This question already has an answer here:
curl 1020 error when trying to scrape page using bash script
(1 answer)
Closed yesterday.
`I have sent a GET (Also POST) method to get data from api but the response is error 1020 and at the same time post man tried and its giving response then what is wrong here.
I am not able to share the api url due to security reasons.
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'url_with_parameter',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Cookie: PHPSESSID=f1r22a2srpgamnidkflgqsuem2; csrf_token=c31d6332c0066fa652b79948c43e5dd7_d5ec9637ea9c6914f3cebeab7fa4ad4d',
'Host' => "localhost",
'User-Agent' => 'PostmanRuntime/7.31.0',
'Accept' => '*/*',
'Accept-Encoding' => 'gzip, deflate, br',
'Connection' => 'keep-alive'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
When 1020 error is returned by curl, it means typically that the request was blocked by security settings, which can include challenges like CAPTCHAs or browser checks.
Try to add UA header into your curl request, like
$headers = [
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
];
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $headers);
I'm trying to download a xml file from url: https://www.bayzade.net/exportlink_181019170321.xml
But I cant download this with curl. Error: Connection Refused.
But I can see on url from browser. How can I fix this error?
Xampp PHP 7.
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
Connection Refused
you need to set CURLOPT_USERAGENT to make a completely request,
some sites serve completely different content or HTML to different OS and browser versions, this is done by detecting their user agent
just including the CURLOPT_USERAGENT to your CURLOPT:
$fp = fopen(dirname(__FILE__).'/errorlog.txt', 'a');
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://www.bayzade.net/exportlink_181019170321.xml',
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
));
$resp = curl_exec($ch);
curl_close($ch);
print_r($resp);
$instagram = "ugur2d";
$kaynak = file_get_contents("https://www.instagram.com/$instagram/");
preg_match('#<title>(.*?)</title>#si', $kaynak, $iglink);
echo $iglink[1];
Screen is empty. How can i run it?
After some checkings...
your url returns error 404 - file_get_contents will return false if 404 is returned by the server
(instagram returns a 404 error page if profile does not exist)
thats why you dont get any result!
anyway you should use curl instead!
Okey okey okey. Im fixed problem. Answer -> CURL :D
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_URL => 'https://toosba.com/',
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => FALSE
]);
$source = curl_exec($ch);
curl_close($ch);
preg_match('/<title>(.*?)<\/title>/', $source, $title);
print_r($title);
echo "<hr>".$title[1];
Thank you :)
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>
I am trying to login into a website through cURL and have it return to me the actual page. So far I can only get the footer of the page with the feedback link and this error:
Warning: curl_setopt_array() [function.curl-setopt-array]: Array keys
must be CURLOPT constants or equivalent integer values on line 18
Loading
line 18 is the ");" at the end of the curl array
So far I've been able to figure out that if I just put in the website name with the access_token in my url from my browser I'll automatically be logged in so I'm trying to get cURL to emulate that.
<?php
$ckfile =' __utma=173730677.1410450142.1370837396.1370843059.4; __utmz=173730677.1370843059.4.3.utmcsr=web.com|utmccn=(referral)|utmcmd=referral|utmcct=/ou; __utma=64278953.892306882.1370766510.1370838026.4; __utmz=64278953.4.3.utmcsr=.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; __utmc=64278953; __utmc=173730677; __utmb=173730677.5.10.1370843059; __utmb=64278953.1.10';
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com',
CURLOPT_COOKIEFILE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_GET=> true,
CURLOPT_REFERER => 'https://.com/signin',
CURLOPT_GETFIELDS=>'#access_token=',
CURLOPT_RETURNTRANSFER => true
)
);
$response = curl_exec($ch);
curl_close($ch);
echo '<pre>';
print_r($response);
?>
CURLOPT_GET and CURLOPT_GETFIELDS are not a valid options for cURL. The options may exist for POST, but they do not for GET. Just add the query string to the URL instead.
Also, make sure all the options are set to what cURL expects them to be set to. CURLOPT_COOKIEFILE should be set to a filename, not a string of cookie values. You want CURLOPT_COOKIE instead.
curl_setopt_array($ch, array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.com?access_token=abc',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://web.com/signin',
CURLOPT_RETURNTRANSFER => true
));
Please see the PHP cURL docs for all the options: http://php.net/manual/en/function.curl-setopt.php