I'm using PHP to display the most recent tweet from a user. This is in Wordpress. This works most of the time - but sometimes, I get this error:
file_get_contents(http://api.twitter.com/1/statuses/user_timeline/[username].json) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 400 Bad Request in [...]/twitter.php on line 47
I'm absolutely certain that I'm not going over the Twitter API limit, because even if my caching code is flawed, no one else can see this - it's hosted locally - and there's no way I viewed the page 150 times in an hour. I've tested that the username and database entries are indeed being retrieved. This is my code:
<?php
function twitter($username) {
$tweet = '';
echo $username;
if (!get_option('twitter_last_updated')) {
$format='json';
$tweet_raw=file_get_contents("http://api.twitter.com/1/statuses/user_timeline/{$username}.{$format}");
$tweet = json_decode($tweet_raw);
add_option('twitter_last_updated', time(), "", "yes");
add_option('twitter_last_updated_author', $username, "", "yes");
add_option('twitter_last_updated_data', $tweet_raw, "", "yes");
} elseif (time() - get_option('twitter_last_updated') > 30 || get_option('twitter_last_updated_author') != $username) {
$format='json';
$tweet_raw=file_get_contents("http://api.twitter.com/1/statuses/user_timeline/{$username}.{$format}");
$tweet = json_decode($tweet_raw);
update_option('twitter_last_updated', time());
update_option('twitter_last_updated_author', $username);
update_option('twitter_last_updated_data', $tweet_raw);
} else {
$tweet = json_decode(get_option('twitter_last_updated_data'));
} ?>
<!-- display the tweet -->
<?php } ?>
I would really appreciate some help with this. I feel totally stumped.
First, you should not be using file_get_contents to retrieve "files" over the network. You should use curl. It could be just system response delays, or twitter issuing a redirect for load balancing. file_get_contents doesn't follow redirects and basically times out immediately. Curl can be set to follow redirects and adheres to the network timeout (1 minute I think) if no time out is specified. Most importantly, curl can tell why it failed.
How often are you calling the function? If I remember correctly, twitter recently changed the maximum amount of calls per hour from 150~ to 75 per hour. You might want to cache the results, so as not to use up your allowance.
See this slashdot story: Twitter Throttling hits 3rd party apps
Why are you not using the WordPress HTTP API? This is exactly what it was designed for - a wrapper for working with HTTP using standard WordPress functions, regardless of platform or set-up.
I wrote a something like what you have and it keeps failing like every 3 requests, the solution was build up a little cache system and #'s on the file_get_contents to avoid php from throwing errors back to users.
When twitter fails, and it will fail a lot, you just fetch data from that previously built cache.
I also don't recommend you adding this onfly, it might slow down the whole page building due twitter issues.
Related
I'm running a website which monitors some servers and there are more servers added daily.
Sadly my current solution is very slow (about 10 seconds loadtime for 31 servers). I'm using fsockopen which checks the IP and port. Because I parse a XML with the entries (host and port) I had to create a function and use this function in the parser so the visitors of the website can see the online or offline status of the server.
My current "checkserver" function looks like this:
function checkServer($ip, $port)
{
$fsockopen = #fsockopen($ip, $port, $errorNo, $errorStr, 1);
if (!$fsockopen)
return false;
else return true;
}
And in the parser the "if" rule for the server status looks like this:
if (checkServer((string)$server->host, (string)$server->port))
{
echo "SERVER ONLINE!";
}
else
{
echo "SERVER OFFLINE!";
}
Where $server is every single listed server in the XML <serverlist></serverlist> tag.
I already tried to change the timeout of fsockopen from 1 to 0.1 but some servers appear offline then and the loadtime is still at 8-10 seconds.
How could I speedup the loadtime? Could someone please help me with this?.. the project is very important for me. Thank you! I really appreciate every helpful answer!
First of all I would suggest caching. I am not sure how many users will open the page, but if you have multiple users per second opening the page you will have a lot of traffic to handle which on long term could create issues.
You have 2 solutions :
Using asynchronous events can allow you to do what you wish.T here are some libraries outhere doing this that can help you. I have used none sofar so I can't say which is best.
Using a library that cheats and uses exec & command lines : https://github.com/oliverde8/PHP-AsynchronousJobs
Using pthread library, this isn't a library coded in php, so it binaries you need to add to your http://pthreads.org/. You can add another library ontop to make the usage easier
Finally using javascript, so you open your page, then some ajax calls your php individually for each server & ask status.
I managed to display the status of a server using FSOCKOPEN.
<?php
$ip = "0.0.0.0";
$port = "1337";
$online = '<font class="online" title="Server Online!">ONLINE</font>';
$offline = '<font class="offline" title="Server Offline!">OFFLINE</font>';
if (! $sock=#fsockopen($ip, $port, $num, $error, 0.25))
{echo "$offline";}
else
{echo "$online";}
?>
If the server with the IP "0.0.0.0" sends an error or does not respond within 1/4th of a second after the page has been loaded the Output will return OFFLINE, else it will return ONLINE.
The server status will update when the user refreshes the page.
Now I want to make it to update in real time (less than 7 seconds delay)
I know there is a very easy method to do that using <meta http-equiv=”refresh” content=”5" /> in the head tag.
But its pretty annoying that the COMPLETE page have to refresh and load everything again and its causing some unnecessary traffic.
Is there an easier way to do it?
You can use ajax and javascript in the client to open the PHP Script in the background and then update the element in the page with the result based on success/failure.
Take a look at jQuery and other library to do this easily.
Another option would be a simple <iframe /> to open another website as an element on the website.
Dont use ajax or any direct polling mechanism that will put more load on your server in terms of traffic.
What you can use is websockets to push data to a stream and read it in frontend
https://socket.io/
This can be done near realtime and your page will not be reuired to get updated every min.
Put PHP script in cron to run at required interval and end of it just push the output to sockets and catch them on frontend.
I wrote a script downloading a list of pages from a website. From time to time I receive the following error (the number of seconds is variable):
The bwshare module will refuse your requests for the next 7 seconds.
You have downloaded data too rapidly.
I found when using sleep(2) in the loop, it works much better, however the time delay is too expensive.
What's the best way how to deal with this module? Should I scrape it without any delay and if the response will be similar to the above message simply use sleep for the requested number of seconds?
It all depends on how many pages you can get before the error message.
Try and measure how many pages in average you can get.
4 pages before the bwshare message is the minimum.
If you are getting the error message before reaching 4 page downloads, then il would be faster to sleep(2) after each download.
try this way... it might help u.
$requestTime = 0.1; // s/connection
foreach(/* blah */) {
$start = microtime(true);
// Do your stuff to here.. get_file_content($url) and other processing .........
if($timeTaken = microtime(true)-$start < $requestTime) {
usleep(($requestTime-$timeTaken)*1000000);
}
}
if your problem is solved then try to post your answer so that other people may also be benefited
I have a simple script that makes redirection to mobile version of a website if it finds that user is browsing on mobile phone. It uses Tera-WURFL webservice to acomplish that and it will be placed on other hosting than Tera-WURFL itself. I want to protect it, in case of Tera-WURFL hosting downtime. In other words, if my script takes more than a second to run, then stop executing it and just redirect to regular website. How to do it effectively (so that the CPU would not be overly burdened by the script)?
EDIT: It looks that TeraWurflRemoteClient class have a timeout property. Read below. Now I need to find how to include it in my script, so that it would redirect to regular website in case of this timeout.
Here is the script:
// Instantiate a new TeraWurflRemoteClient object
$wurflObj = new TeraWurflRemoteClient('http://my-Tera-WURFL-install.pl/webservicep.php');
// Define which capabilities you want to test for. Full list: http://wurfl.sourceforge.net/help_doc.php#product_info
$capabilities = array("product_info");
// Define the response format (XML or JSON)
$data_format = TeraWurflRemoteClient::$FORMAT_JSON;
// Call the remote service (the first parameter is the User Agent - leave it as null to let TeraWurflRemoteClient find the user agent from the server global variable)
$wurflObj->getCapabilitiesFromAgent(null, $capabilities, $data_format);
// Use the results to serve the appropriate interface
if ($wurflObj->getDeviceCapability("is_tablet") || !$wurflObj->getDeviceCapability("is_wireless_device") || $_GET["ver"]=="desktop") {
header('Location: http://website.pl/'); //default index file
} else {
header('Location: http://m.website.pl/'); //where to go
}
?>
And here is source of TeraWurflRemoteClient.php that is being included. It has optional timeout argument as mentioned in documentation:
// The timeout in seconds to wait for the server to respond before giving up
$timeout = 1;
TeraWurflRemoteClient class have a timeout property. And it is 1 second by default, as I see in documentation.
So, this script won't be executed longer than a second.
Try achieving this by setting a very short timeout on the HTTP request to TeraWurfl inside their class, so that if the response doesn't come back in like 2-3 secs, consider the check to be false and show the full website.
The place to look for setting a shorter timeout might vary depending on the transport you use to make your HTTP request. Like in Curl you can set the timeout for the HTTP request.
After this do reset your HTTP request timeout back to what it was so that you don't affect any other code.
Also I found this while researching on it, you might want to give it a read, though I would say stay away from forking unless you are very well aware of how things work.
And just now Adelf posted that TeraWurflRemoteClient class has a timeout of 1 sec by default, so that solves your problem but I will post my answer anyway.
I have noticed a few websites such as hypem.com show a "You didnt get served" error message when the site is busy rather than just letting people wait, time out or refresh; aggravating what is probably a server load issue.
We are too loaded to process your request. Please click "back" in your
browser and try what you were doing again.
How is this achieved before the server becomes overloaded? It sounds like a really neat way to manage user expectation if a site happens to get overloaded whilst also giving the site time to recover.
Another options is this:
$load = sys_getloadavg();
if ($load[0] > 80) {
header('HTTP/1.1 503 Too busy, try again later');
die('Server too busy. Please try again later.');
}
I got it from php's site http://php.net/sys_getloadavg, altough I'm not sure what the values represent that the sys_getloadavg returns
You could simply create a 500.html file and have your webserver use that whenever a 50x error is thrown.
I.e. in your apache config:
ErrorDocument 500 /errors/500.html
Or use a php shutdown function to check if the request timeout (which defaults to 30s) has been reached and if so - redirect/render something static (so that rendering the error itself cannot cause problems).
Note that most sites where you'll see a "This site is taking too long to respond" message are effectively generating that message with javascript.
This may be to do with the database connection timing out, but that assumes that your server has a bigger DB load than CPU load when times get tough. If this is the case, you can make your DB connector show the message if no connection happens for 1 second.
You could also use a quick query to the logs table to find out how many hits/second there are and automatically not respond to any more after a certain point in order to preserve QOS for the others. In this case, you would have to set that level manually, based on server logs. An alternative method can be seen here in the Drupal throttle module.
Another alternative would be to use the Apache status page to get information on how many child processes are free and to throttle id there are none left as per #giltotherescue's answer to this question.
You can restrict the maximum connection in apache configuration too...
Refer
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients
http://www.howtoforge.com/configuring_apache_for_maximum_performance
This is not a strictly PHP solution, but you could do like Twitter, i.e.:
serve a mostly static HTML and Javascript app from a CDN or another server of yours
the calls to the actual heavy work server-side (PHP in your case) functions/APIs are actually done in AJAX from one of your static JS files
so you can set a timeout on your AJAX calls and return a "Seems like loading tweets may take longer than expected"-like notice.
You can use the php tick function to detect when a server isn't loading for a specified amount of time, then display an error messages. Basic usage:
<?php
$connection = false;
function checkConnection( $connectionWaitingTime = 3 )
{
// check connection & time
global $time,$connection;
if( ($t = (time() - $time)) >= $connectionWaitingTime && !$connection){
echo ("<p> Server not responding for <strong>$t</strong> seconds !! </p>");
die("Connection aborted");
}
}
register_tick_function("checkConnection");
$time = time();
declare (ticks=1)
{
require 'yourapp.php' // load your main app logic
$connection = true ;
}
The while(true) is just to simulate a loaded server.
To implement the script in your site, you need to remove the while statement and add your page logic E.G dispatch event or front controller action etc.
And the $connectionWaitingTime in the checkCOnnection function is set to timeout after 3 seconds, but you can change that to whatever you want