Curl Errors While Pulling from Twitter - php

Question

I'm trying to pull some data from twitter via PHP. I'm using the tmhOAuth plugin, which can be found here. https://github.com/themattharris/tmhOAuth/
I wrote my code based off the example file "streaming.php", which can also be found on the above github page. Here is my code:
require 'tmhOAuth.php';
$tmhOAuth = new tmhOAuth(array(
'consumer_key' => 'xxxhiddenxxx',
'consumer_secret' => 'xxxhiddenxxx',
'user_token' => 'xxxhiddenxxx',
'user_secret' => 'xxxhiddenxxx'
));
$method = 'http://stream.twitter.com/1/statuses/filter.json';
$params = array(
'follow' => '1307392917',
'count' => '5'
);
$tmhOAuth->streaming_request('POST', $method, $params, 'my_streaming_callback');
$tmhOAuth->pr($tmhOAuth);
That was not printing out any of the twitter data I wanted to pull, and was only showing the debug information that the pr() command writes.
While trying to debug why I wasn't getting any data, I went in and added a line to tmhOAuth.php so that I could see what error cURL was giving. I did this by using
echo curl_error($C);
The error that cURL outputed was :
transfer closed with outstanding read data remaining
I've done some research on that error, but I can't find anything that helps. There were a couple things that I found regarding content-length, but when I dug into the code I saw that the author of tmhOAuth had already addressed those issues (and commenting out his fixes didn't help).
Any help?
Update 1 Here is the response info gathered using curl_getinfo:
//Removed - an updated version is below
Update 2 Thanks to the comments below I realized that twitter was sending me data with transfer-encoding: chunked. I put this line into tmhOAuth.php to force out chunked data:
curl_setopt($c, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_0);
That worked, somewhat. I'm no longer getting any cURL errors, but my WRITEFUNCTION callback is still never getting called - so I'm never getting any actual data. Here's the output of my cURL object again:
[response] => Array
(
[content-length] => 0
[headers] => Array
(
[content_type] => text/html; charset=iso-8859-1
[server] => Jetty(6.1.25)
)
[code] => 416
[response] => 1
[info] => Array
(
[url] => http://stream.twitter.com/1/statuses/filter.json
[content_type] => text/html; charset=iso-8859-1
[http_code] => 416
[header_size] => 116
[request_size] => 532
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.118553
[namelookup_time] => 0.043927
[connect_time] => 0.070477
[pretransfer_time] => 0.07049
[size_upload] => 25
[size_download] => 0
[speed_download] => 0
[speed_upload] => 210
[download_content_length] => -1
[upload_content_length] => -1
[starttransfer_time] => 0.118384
[redirect_time] => 0
[request_header] => POST /1/statuses/filter.json HTTP/1.0
User-Agent: themattharris' HTTP Client
Host: stream.twitter.com
Accept: */*
Authorization: OAuth oauth_consumer_key="xxxhiddenxxx", oauth_nonce="xxxhidden", oauth_signature="xxxhidden", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1308226585", oauth_token="xxxhiddenxxx", oauth_version="1.0"
Content-Length: 25
Content-Type: application/x-www-form-urlencoded
)
)
)
Update 3: Couple things I've figured out so far... I removed the 'count' parameters from my POST request, and now the page seems to take forever. I figured this meant it was just downloading tons and tons of data, so I put a break into the streaming callback function, setup so that it kills the page after 5 loops.
I did this, and let it sit for quite awhile. After about 5 minutes, the page finished loading, and showed me what data I had gathered. It looked like I had gotten no data each time it ran through - only an end of line character. So, it's taking a minute for every piece of data I am downloading, and even then the only data that shows is an end of line character. Weird? Is this a twitter issue or a cURL issue?

Answer 1

I tried with the token api but never got something good, so this is the script I found here :
<?php
/**
* API Streaming for Twitter.
*
* #author Loïc Gerbaud <gerbaudloic#gmail.com>
* #version 0.1 "itjustworks"
*/
define('TWITTER_LOGIN','login'); //login twitter
define('TWITTER_PASSWORD','myp4ssw0rd'); //password twitter
$sTrackingList = 504443371;//read my account but could be keywords
// ?
while(1){
echo 'Connexion ';
read_the_stream($sTrackingList);
echo 'Deconnexion ';
}
/**read the stream
*
*/
function read_the_stream($sTrackingList){
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,'https://stream.twitter.com/1/statuses/filter.json');
curl_setopt($ch,CURLOPT_USERPWD,TWITTER_LOGIN.':'.TWITTER_PASSWORD);//Le couple login:password
curl_setopt($ch, CURLOPT_NOBODY, 0);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, '');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('X-Twitter-Client: ItsMe','X-Twitter-Client-Version: 0.1','X-Twitter-Client-URL: http://blog.loicg.net/'));
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,"follow=".$sTrackingList);//read the doc for your request
curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'write_callback');//function callback
curl_exec($ch);
curl_close($ch);
}
/** a demo with a writting log or put in MySQL
*/
function write_callback($ch, $data) {
if(strlen($data)>2){
$oData = json_decode($data);
if(isset($oData->text)){
file_put_contents('log',$oData->text."\n",FILE_APPEND);
}
}
return strlen($data);
}
?>
run this script in your browser (you can close it after), update your twitter account and check the .log

Answer 2

After about 5 minutes, the page finished loading
Are you running streaming.php in the browser? If so, you have to run it via ssl, otherwise it doesn't work. I have a server chron job pointing to the file but you can do it also with the terminal:
php /path/to/here/streaming.php
For view the data you are getting, you can store it into a database or log:
function my_streaming_callback($data, $length, $metrics) {
$ddf = fopen('/twitter/mydata.log','a');
fwrite($ddf,$data);
fclose($ddf);
}