After a lot of trial and error, I have finally managed to fetch tweets with Twitters new API (version 1.1). I'm using the PHP TwitterOauth library. Even though I'm able to fetch tweets, two things I do not understand.
The limit for statuses/user_timeline is 200 tweets. How do I loop through the results to fetch the maximum number of 3,200 tweets? I read something about making multiple GET requests, but I'm not sure how to do that.
It seems the number of tweets fetched varies randomly, and seldomly actually gets up to the number specified in the parameter 'count'. Why is that?
My application simply lets the visitor type in a username and fetch the tweets of that user. Here's my code.
if (isset($_GET['user'])) {
$user = $_GET['user'];
$content = $connection->get("statuses/user_timeline", array('count' => 200, 'exclude_replies' => true, 'screen_name' => $user));?>
$j = 0;
foreach ($content as $tweet) {
echo $j.' '.$tweet->text. '<br />';
$j++;
}
}
UPDATE: After trying out queremys suggestion below, I came up with a really ugly looking "solution" that has several major drawbacks. At least it shows the maximum amount of 3,200 tweets (and some duplicates). The result will look weird if the twitter account in question has less than 3,200 tweets. Anyways, just thought I'd share it if it can be of inspiration.
if (isset($_GET['user'])) {
$user = $_GET['user'];
$content = $connection->get('statuses/user_timeline', array(
'count' => 200, 'exclude_replies' => true, 'screen_name' => $user, 'include_rts' => 1
));
$x = 0;
while ($x < 15) {
$text = array();
foreach ($content as $tweet) {
$text[] = $tweet->id_str;
echo $tweet->text.'<br />';
}
$last_tweet = end($text);
$content = $connection->get('statuses/user_timeline', array(
'count' => 200, 'exclude_replies' => true, 'screen_name' => $user, 'include_rts' => 1, 'max_id' => $last_tweet
));
foreach ($content as $tweet) {
echo $tweet->text.'<br />';
}
$x++;
}
}
It could be a little bit pricy but i think possible (you can test something like that);
$contents = array();
$limit = 3200;
$max_id = null;
for ($count = 200; $count < $limit; $count += 200) {
if (null !== $max_id && $max_id == '') {
break;
}
$content = $connection->get('statuses/user_timeline', array(
'count' => $count, 'exclude_replies' => true, 'screen_name' => $user,
'max_id' => $max_id
));
$contents[] = $content;
// this indicates the last index of $content array
$max_id = $content[count($content) - 1]->id_str;
}
UPDATE!
You need to make $max_id to continue loop, and need to $max_id NULL to break loop.
// option 1, makes $max_id NULL silently
# $max_id = $content[count($content) - 1]->id_str;
// option 2, search for last index of array
if (count($content)) {
$last_tweet = end($content);
$max_id = $last_tweet->id_str;
} else $max_id = null;
Related
I use CodeIgniter, and when an insert_batch does not fully work (number of items inserted different from the number of items given), I have to do the inserts again, using insert ignore to maximize the number that goes through the process without having errors for existing ones.
When I use this method, the kind of data I'm inserting does not need strict compliance between the number of items given, and the number put in the database. Maximize is the way.
What would be the correct way of a) using insert_batch as much as possible b) when it fails, using a workaround, while minimizing the number of unnecessary requests?
Thanks
The Correct way of inserting data using insert_batch is :
CI_Controller :
public function add_monthly_record()
{
$date = $this->input->post('date');
$due_date = $this->input->post('due_date');
$billing_date = $this->input->post('billing_date');
$total_area = $this->input->post('total_area');
$comp_id = $this->input->post('comp_id');
$unit_id = $this->input->post('unit_id');
$percent = $this->input->post('percent');
$unit_consumed = $this->input->post('unit_consumed');
$per_unit = $this->input->post('per_unit');
$actual_amount = $this->input->post('actual_amount');
$subsidies_from_itb = $this->input->post('subsidies_from_itb');
$subsidies = $this->input->post('subsidies');
$data = array();
foreach ($unit_id as $id => $name) {
$data[] = array(
'date' => $date,
'comp_id' => $comp_id,
'due_date' => $due_date,
'billing_date' => $billing_date,
'total_area' => $total_area,
'unit_id' => $unit_id[$id],
'percent' =>$percent[$id],
'unit_consumed' => $unit_consumed[$id],
'per_unit' => $per_unit[$id],
'actual_amount' => $actual_amount[$id],
'subsidies_from_itb' => $subsidies_from_itb[$id],
'subsidies' => $subsidies[$id],
);
};
$result = $this->Companies_records->add_monthly_record($data);
//return from model
$total_affected_rows = $result[1];
$first_insert_id = $result[0];
//using last id
if ($total_affected_rows) {
$count = $total_affected_rows - 1;
for ($x = 0; $x <= $count; $x++) {
$id = $first_insert_id + $x;
$invoice = 'EBR' . date('m') . '/' . date('y') . '/' . str_pad($id, 6, '0', STR_PAD_LEFT);
$field = array(
'invoice_no' => $invoice,
);
$this->Companies_records->add_monthly_record_update($field,$id);
}
}
echo json_encode($result);
}
CI_Model :
public function add_monthly_record($data)
{
$this->db->insert_batch('monthly_record', $data);
$first_insert_id = $this->db->insert_id();
$total_affected_rows = $this->db->affected_rows();
return [$first_insert_id, $total_affected_rows];
}
AS #q81 mentioned, you would split the batches (as you see fit or depending on system resources) like this:
$insert_batch = array();
$maximum_items = 100;
$i = 1;
while ($condition == true) {
// code to add data into $insert_batch
// ...
// insert the batch every n items
if ($i == $maximum_items) {
$this->db->insert_batch('table', $insert_batch); // insert the batch
$insert_batch = array(); // empty batch array
$i = 0;
}
$i++;
}
// the last $insert_batch
if ($insert_batch) {
$this->db->insert_batch('table', $insert_batch);
}
Edit:
while insert batch already splits the batches, the reason why you have "number of items inserted different from the number of items given" might be because the allowed memory size is reached. this happened to me too many times.
I am working on facebook pagination, I have searched but didn't get relevant answer.
First I am fetching 10 result and after that onclick function I want to fetch next 10 results for this I am passing -
[paging] =>
Array
(
[previous] => https://graph.facebook.com/v2.9/....D&__previous=1
[next] => https://graph.facebook.com/........
)
as parameter,I also tried passing next URL as parameter but still it is not working, if I pass $feedEdge as associated I am getting response as null,below is my code
$response = self::$_FBINSTANCE->get('/me/feed?fields=id,message&limit=' . $_pagination->limit);
if(empty($_nextFeed)){
$feedEdge = $response->getGraphEdge();
$nextFeed = $response->getGraphEdge()->getMetaData();
}else{
$feedEdge=$response->next($_nextFeed);
$nextFeed = $response->getGraphEdge()->getMetaData();
}
$result = array();
foreach ($feedEdge as $status) {
$result[] = $status->asArray();
}
return array(
'result' => $result,
'totalRows' => $totalCount,
'nextFeed' => $nextFeed
);
using v2.9 version, what parameter I should pass for $response->next(); help me if I am wrong .
I found solution for this question...
If any one is trying for same try below code.
$result = array();
if(empty($_nextFeed)){
$response = $fb->get('/me/feed?fields=id,message&limit=' . $_pagination->limit);
$feedEdge = $response->getGraphEdge();
foreach ($feedEdge as $status) {
$result[] = $status->asArray();
}
$nextFeed = $response->getGraphEdge()->getMetaData();
}else{
//to get until time stamp form next url
$nextURl=parse_url($_nextFeed['paging']['next']);
parse_str($nextURl['query'], $URL);
$response = $fb->get('/me/feed?fields=id,message&limit='.(($_pagination->limit)+1).'&until='.$URL['until']);
$feedEdge = $response->getGraphEdge();
foreach ($feedEdge as $status) {
$result[] = $status->asArray();
}
//because result repeats last array of previous request
array_splice($result,0,1);
$nextFeed = $response->getGraphEdge()->getMetaData();
}
return array(
'result' => $result,
'totalRows' => $totalCount,
'nextFeed' => $nextFeed
);
It works :)
I'm workin on an app that would award its users points for each of their friends liking my page.
I need a way to check how many of user's friends like my page. I developed this piece of code to check each my friends for liking a particular page:
$friends = $facebook->api('/me/friends'); //list my friends
$time_start = microtime(true);
echo "<pre>";
$i = 0; //counter of likes
$b = 0; //counter of request in single batch API request
$batch_max = 50 // facebook allows max 50 requests in single batch API call
$page_id = '187941397895831'; //example page ID, i'm checking how many of my friends like this page
$batch = array(); //array used for creating batch API request
$data = array(); //array collecting all API requests results
foreach ($friends['data'] as $friend) {
$req = array(
'method' => 'GET',
'relative_url' => '/'.$friend['id'].'/likes'
);
$batch[] = json_encode($req);
$b++;
if ($b == 50) {
$params = array(
'batch' => '[' . implode(',',$batch) . ']'
);
$data[] = $facebook->api('/', 'POST', $params);
$b = 0;
$batch = array();
}
}
$params = array(
'batch' => '[' . implode(',',$batch) . ']'
);
$data[] = $facebook->api('/', 'POST', $params);
foreach ($data as $data_set) { //iterate through results of each api request
foreach ($data_set as $friend_likes) { //iterate through data of particular friend
$likes_array = json_decode($friend_likes['body'], true);
foreach ($likes_array['data'] as $single_like) { //iterate through all likes of a single friend
if ($single_like['id'] == $page_id) { //if page id found in user's likes increase counter
$i++;
}
}
}
}
$time_end = microtime(true);
$time = $time_end - $time_start;
echo $time."\n";
echo $i;
echo "</pre>";
exit;
In final version I will not be checking my friends but friends of each of my users. My code executes in 14 seconds, if I had to iterate through, let's say, 100 users it'd be 1400 seconds, which is waaaay to long. Is there better way to do that? I'm newbie to facebook API, so I could miss something obvious :)
Ok, so here's the working piece of code. You need to know access token for each of users being checked. It executes in ~0.6 seconds instead of 14 seconds for previous code.
$queryResults = array();
$facebook->setAccessToken($user->getAccessToken());
$fql = "SELECT '' FROM page_fan WHERE page_id = 'xxxxxxxxxxxx' AND uid IN (SELECT uid2 FROM friend WHERE uid1 = '".$user->getFbId()."')";
$fql = urlencode($fql);
try {
$queryResults = $facebook->api(
'/fql?q='.$fql
);
} catch (FacebookApiException $e) {
$log->setMessage(print_r($e,true));
$logMapper->save($log);
}
if (!empty($queryResults)) {
$user->setPoints(count($queryResults['data']));
$usersMapper->save($user);
}
foreach($friends_array as $user) {
$argstag = array('to' => $user);
$argstag['x'] = $locations_x[$i];
$argstag['y'] = $locations_y[$i];
$datatag = $facebook->api('/' . $photo_id . '/tags', 'post', $argstag);
$i++;
}
Instead of doing this in loop (multiple api requests), Is there a way to do it in a single api request?
below code works for me provided array $friends have less equal 50 friends:
$tags = array();
foreach ($friends as $friend)
{
$tag = array();
$tag['tag_uid'] = $friend;
$tag['x'] = rand() % 100;
$tag['y'] = rand() % 100;
$tags[] = $tag;
}
$argstag = array(
'tags' => $tags
);
$facebook->api("$photoId/tags","POST", $argstag);
If you're doing large amounts of operations at the same time, consider using the batch request API. It lets you perform multiple tasks (up to 50) at the same time.
Here's an example on how you could theoretically use it:
$batches = array();
$i = $b = 0;
foreach($friends_array as $user) {
$argstag = array('to' => $user);
$argstag['x'] = $locations_x[$i];
$argstag['y'] = $locations_y[$i];
// If we've reached the batch limit, create a new batch request.
if ($i == 50) {
$b++;
$i = 0;
}
// Single batch request.
$batches["$b"][] = array(
'method' => 'POST',
'relative_url' => '/' . $photo_id . '/tags',
'body' => 'to=' . $user . '&x=' . $locations_x[$i] . '&y=' . $locations_y[$i]
);
$i++;
}
if (!empty($batches)) {
foreach ($batches AS $key => $batch) {
$b = json_encode($batch);
$res = $facebook->api('?batch=' . urlencode($b), 'POST');
// Facebook populates $res with the response.
}
}
This isn't tested, but hopefully that's helpful. It may not be what you're looking for for that, but it's worth noting if you have a large number of requests you need to send to Facebook's servers.
I have a database field called "servers"
This field has a link in each row, this field content:
> http://www.rapidshare.com/download1
> http://www.rapidshare.com/download2
> http://www.rapidshare.com/download3
> http://www.megaupload.com/download1
> http://www.megaupload.com/download2
> http://www.megaupload.com/download3
> http://www.fileserve.com/download1
> http://www.fileserve.com/download2
> http://www.fileserve.com/download3
I want to create an array with all the server names, and create more array with links inside.
That's how it should be:
$servers = array(
'rapidshare' => array(
'link1' => 'http://www.rapidshare.com/download1',
'link2' => 'http://www.rapidshare.com/download2',
'link3' => 'http://www.rapidshare.com/download3'),
'megaupload' => array(
'link1' => 'http://www.megaupload.com/download1',
'link2' => 'http://www.megaupload.com/download2',
'link3' => 'http://www.megaupload.com/download3'),
'fileserve' => array(
'link1' => 'http://www.megaupload.com/download1',
'link2' => 'http://www.megaupload.com/download2',
'link3' => 'http://www.megaupload.com/download3')
);
This will do the trick: (make sure that domain is actually showing up in $domain variable though because it might be $matches[1]... I can't remember)
$newStructure = array();
foreach($links as $link) {
preg_match("/www\.([^\.])\.com/",$link,$matches);
$domain = $matches[0];
$currentLength = count($newStructure[$domain]);
if($currentLength) {
$newStructure[$domain]['link'.($currentLength+1)] = $link;
} else {
$newStructure[$domain] = array('link1'=>$link);
}
}
$server = array(
'http://www.rapidshare.com/download1',
'http://www.rapidshare.com/download2',
'http://www.rapidshare.com/download3',
'http://www.megaupload.com/download1',
'http://www.megaupload.com/download2',
'http://www.megaupload.com/download3',
'http://www.fileserve.com/download1',
'http://www.fileserve.com/download2',
'http://www.fileserve.com/download3'
);
$match = array();
$myarray = array();
foreach($server as $v) {
// grab server name
preg_match('/\.(.+)\./', $v, $match);
$serverName = $match[1];
// initialize new array if its the first link of that particular server
if (!isset($myarray[$serverName])) {
$myarray[$serverName] = array();
}
// count server array to check how many links are there, and make next link key
$linkKey = 'link' . (count($myarray[$serverName]) + 1);
// store value
$myarray[$serverName][$linkKey] = $v;
}
print_r($myarray);
Hey maybe this will help you. But i dont see the purpose of those names of the keys (link1,link2 etc..). This wont work on pagination thou.