I'm using cURL to generate HTTP Requests to login to my twitter and I'm trying to follow people using it, one of the problems I'm currently facing is that the follow button doesn't seem to be a form but actually just a button that sends the following request
GET https://twitter.com/i/user/follow
Status: HTTP/1.1 200 OK
Request Headers
Accept image/webp,*/*;q=0.8
Accept-Encoding gzip,deflate,sdch
Accept-Language en-US,en;q=0.8,ar;q=0.6
User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36
What I found weird was that no ID for the user was being passed on to be followed so I have no idea how is it being passed on, I thought maybe from the referer but I tried mimicking the request from my browser by visiting a profile then visiting that URL but I got a 405 error
Does anyone have any idea how does twitter send requests to follow people currently?
The thing is handled using cookie. Somehow you couldn't capture the Cookie from the request. Twitter is tracking the login using cookie. That's why the simple GET request works after login, and doesn't work(405 error) where you are not loggedin. The steps are:
Login using curl and store the cookie in a file. Just catch the login request as you caught for follow request.
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'user=username&pass=password&someparam=xyz');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'c:/temp/cookie.txt');
curl_close($ch); // don't forget to close curl
Now call the http GET for the follow request that you posted in description. This time use the cookie that you stored in previous call.
curl_setopt($ch, CURLOPT_COOKIEFILE, 'c:/temp/cookie.txt');
Related
There's a specific website I want to get the source code from with PHP cURL.
Visiting this website with a bowser from my computer works without any problems.
But when I want to access this website with my PHP script, the website recognizes that this is an automated request and shows an error message.
This is my PHP script:
<?php
$url = "https://www.example.com";
$user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.1 Safari/605.1.15";
$header = array('http' => array('user_agent' => $user_agent));
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
echo $data;
?>
The user agent is the same I'm also using with the browser. I'm using a local server with MAMP PRO. This means I'm using the same IP address for both, browser access and PHP script access.
I already tried my PHP script with many different headers and options but nothing worked.
There must be anything that makes a PHP script access look different than a browser access, for the web server I want so access the website from. But what? Do you have an idea?
EDIT
I found out that it's working with this cURL:
curl 'https://www.example.com/' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7'
If I type this in e.g. the Terminal, it's showing the correct source code.
I converted it to a PHP script as follows:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.example.com/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
$headers = array();
$headers[] = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36';
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3';
$headers[] = 'Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
?>
Unfortunately, this way it's still showing the error message.
This means, there must be anything that makes a command line access look different than a browser access, for the web server I want so access the website from. But what is it?
There is no difference between a cURL request and the request that a browser makes, apart from the HTTP headers it requests, and that a browser has JavaScript running on the client.
The only thing that identifies an HTTP client is its headers -- typically the user agent string -- and seeing as you have set the user agent to exactly the same as the browser, there must be other checks in place.
By default, cURL doesn't send any default Accept header, whereas browsers request pages with this header to show the capabilities of the browser. I expect the web server will be checking on something like this.
Take a look at the screenshot above of Chrome Developer Tools. It allows you to copy the whole request as a cURL request, including all the headers that were sent from Chrome, for testing in the terminal.
Try to match all the headers exactly from within your PHP, and I'm sure the web server will not be able to identify you as a script.
You should try to mimic a real browser by forging "real" http request. Add more headers than the User-Agent, like "Accept", "Accept-Language", "Accept-Encoding". Also, you probably need to accept (and handle correctly) cookies.
If your targeted website use javascript to detect a real browser, this is an other challenge.
How would I force HTTP (Not HTTPS), while getting the source code of: http://www.youtube.com/watch?v=2YqEDdzf-nY?
I've tried using get_file_contents, but it goes to HTTPS.
There is no way, because google forces you to use https. It will not accept longer unsecure connection.
They even start to downrank websites, which are not on SSL.
As for your Comment, i have done a little bit more research.
Maybe it is depended on the user-agent. I have no time to confirm this.
Try CURL with this User Agent:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101
I have PHP hosting with GoDaddy. Lately for last one hour, I am not able to load facebook content from my php scripts as it always says that
'You are using an Incompatible Web Browser'.
I know that it seems to be a browser issue, but i am sure that it is not because I have tried it with firefox+chrome+IE on two windows machine and I have tried with Firefox+safari browsers on a mac machine. Its getting the same error every time.
Could you please let me know what could be a possible reason for this?
[Try loading http://cabbie.apprumble.in/index.php?r=site/test]
In normal circumstances, This should load the facebook home page properly, instead of showing the error that You have incompatible browser.
[PS: I am loading the facebook page using php call file_get_contents("http://facebook.com") which was working perfectly fine until an hour back. Also, if I load the url from outside the browser, it works perfectly fine, but if its invoked from within the php using file_get_contents call, the said error appears.)]
Could someone please reply soon as I am stuck in my development due to this.
Thanks,
Kshitij
file_get_contents uses the user agent set in your php.ini file from the setting user_agent. You probably cannot change this as you are on godaddy hosting.
You will need to switch from file_get_contents to something that lets you control the user agent. You could use curl or sockets. Here is a curl example:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.facebook.com/");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13'); // set the user agent here
$data = curl_exec($ch);
echo $data; // this is the homepage
Facebook is attempting to block bots by not allowing certain user agents to request pages. You need to spoof the user agent to look like a normal browser.
How can I view the full request headers, including post data, using libcurl in php?
I am trying to simulate the post of a page, which when done from a browser and viewed in Live HTTP Headers looks like this:
https://###.com
POST /###/### HTTP/1.1
Host: ###.###.com
...snipped normal looking headers...
Content-Type: multipart/form-data; boundary=---------------------------28001808731060
Content-Length: 697
-----------------------------28001808731060
Content-Disposition: form-data; name="file_data"; filename="stats.csv"
Content-Type: text/csv
id,stats_id,scope_id,stat_begin,stat_end,value
61281,1,4,2011-01-01 00:00:00,2011-12-31 23:59:59,0
-----------------------------28001808731060
Content-Disposition: form-data; name="-save"
Submit
-----------------------------28001808731060--
So we nicely see the file I'm uploading, it's content, everything's there. But all my attempts at getting data out of cURL when I try to make the same post from php (using CURLOPT_VERBOSE, or CURLINFO_HEADER_OUT) show request headers that lack the post data, like so:
POST /###/### HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Host: ###.###.com
...snipped normal-looking headers...
Content-Length: 697
Content-Type: multipart/form-data; boundary=----------------------------e90de36c15f5
Based on the Content-Length here, it appears things are going well, but it would really help my debugging efforts to be able to see the complete request. I am also irked that it is difficult, I should be able to see the whole thing; I know I must be missing something.
--- EDIT ---
What I'm looking for is the equivalent of this:
curl --trace-ascii debugdump.txt http://www.example.com/
which seems to be available with the option CURLOPT_DEBUGFUNCTION in libcurl, but isn't implemented in php. Boo.
I had a need to do precisely this, but I needed to test communication with a bank.
It is extremely easy to use Fiddler2, enable HTTPS traffic decryption, and have cURL use Fiddler2 as a proxy for debugging in this situation:
$proxy = '127.0.0.1:8888';
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
You are sending multipart/formdata. cURL basically shows the HTTP header completely I guess. The "problem" is that multipart/formdata consist of multiple parts. This is beyond "first level HTTP headers" and part of the body of the "main HTTP body".
I don't know your environment, but you can debug using TCP traffic monitoring as well. For this, you can use Wireshark or tcpdump - Wireshark can as well show dump files created by tcpdump.
some sites are blocking #file_get_contents and the curl code also. I need code(PHP) that circumvents that problem. I only need to get the page contents so I can extract the title.
You probably need to set the user agent string to emulate a "real" browser:
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20110319 Firefox/4.0');