I wrote a script that pulls the current top Twitter trends using cURL and it works 100% of the time locally but when I FTP it up to my mediatemple server it seems to only work sometimes. Is this caused by Twitter? Mediatemple? Some error in my code?
EDIT: How can I cache content in a flat-file?
If the code works sometimes that suggests it is not a problem with your code so there are two logical areas for potential blame:
1) Web Server Load
This could be your server is to bogged down. If the server (not just your site - consider this if your on shared hosting) is experiencing a heavy load then it may take your server too long to complete the curl request. to combat this try and increase the timeout time on the request using the following:
CURLOPT_CONNECTTIMEOUT
2) Twitter Rate Limmit
Twitter limits the number of API calls you can make from one authorized account per hour (I believe the number is around 100ish - check their API Documentation) If you are hitting this limit you will be declined further calls until the 1 hour anniversary of the first call. To combat this have either a cron job run the curl at a set interval and cache the result in a text file or database or store the time of each request made and use an IF to only allow one request every 2 or 3 mins, cache the results and pull the results from the cache.
Making a call to the twitter API every time there is a page load is a waste of resources, bandwith and could reduce page load time.
Related
I have an App service in Azure: a php script that makes a migration from a database (server1) to a another database (azure db in a virtual machine).
This script makes a lot of queries and requests, so it takes a lot of time and the server (App service) returns:
"500 - The request timed out. The web server failed to respond within
the specified time."
I found that it's something about "idle timeout." I would like to know how to increase this time.
In my test, I have tried the following so far:
Add ini_set('max_execution_time', 300); at the top of my PHP script.
App settings on portal: SCM_COMMAND_IDLE_TIMEOUT = 3600.
But nothing seems to work.
After some searching, I found the post by David Ebbo, as he said:
There is a 230 second (i.e. a little less than 4 mins) timeout for
requests that are not sending any data back. After that, the client
gets the 500 you saw, even though in reality the request is allowed to
continue server side.
And the similar thread from SO, you can refer here.
The suggestion for migration is that you can leverage Web Jobs to run PHP scripts as background processes on App Service Web Apps.
For more details, you can refer to https://learn.microsoft.com/en-us/azure/app-service-web/web-sites-create-web-jobs.
We have a PHP backend which connects over an API to our Java backend for some heavy duty number crunching.
Unfortunately this number crunching sometimes takes longer than 1 minute, and the AWS load balancer times out.
Do you know of a way to prevent this?
I was thinking getting PHP to keep pinging, or JQuery to keep pinging, or increasing the timeout of the load balancer, but I haven't been able to any of those.
By default, the ELB will timeout if no data is received for 1 minute.
Ideally this would be designed as a job, and you would just send status reports with ajax. If you can't do that, there are a couple of other options.
Send data, even if its just empty spaces. Keep in mind that php may use output buffering, and may not send any data unless the packet is of at least a certian size.
Contact AWS support to have the timeout for your ELB increased.
I'm pretty new to server-side web development. What I want to do is to dynamically create a web page with php (no problem for that), but in this page there is an image refresh timer that update an image through AJAX.
My goal is to dynamically modify (server-side) the refresh rate written in the generated html page based on the current request rate. In other words, if there is only one client that requests refreshes, I want it to refresh once per second. But if there are two clients, I want them to refresh less often, and so, and so (it is a local application, so I don't expect a lot of clients).
I hope I can do that with php. I don't want to overload the server CPU with the computing of this rate. I would like to be able to get it quite easily.
My server uses Apache on Linux.
Someone has an idea? A suggestion? Thanks in advance!
EDIT 1: Maybe I could log a "hit" each time a request is done? But I read that putenv() will write environment var that will be valid only during the execution of the script...?
It looks like you are limiting the overall polling connections to the server. The client can be informed by the server response.
To calculate the server connection without tapping into system variables or induce much expensive I/O, you can use a heuristic: Set up memcache and increase a counter each time you get a request. Set the timeout to, say, 5 seconds. This allows you to limit the total connections within the 5-second window.
The server response then tells the client either the total count, or a simple yes/no to whether it has more connections to spare. Your client program can react accordingly.
Edit:
installing memcached and memcache extension on Ubuntu:
apt-get install php5-memcache
apt-get install memcached
Here's the documentation on how to use the memcache API.
My original strategy with a single variable won't work. You can set up multiple keys, say,
channel_0, channel_1, ..., channel_9, assuming there won't be too many channels because you're returning a video feed?
Then when a connection comes in, you look for a non-busy channel, and block that for a period of time:
$memcache_obj = memcache_connect('memcache_host', 11211);
$channel=null;
for ($i=0;$i<10;$i++){
if (memcache_get($memcache_obj,'channel_'.$channel)=='') {
$channel=$i;
memcache_set($memcache_obj, 'channel_'.$channel, '1', 0, 10); //block for 10 seconds
break;
}
}
if ($channel==null) // tell the client there's no more space
I'm currently working on Web Service which typically should handle 100 request at 1 minute and process all requests parallel. As per I know, the $_GET method only accept one request at a time and process it even if the client send multiple request at same instance of time. Until first request does not complete other request can not be executed.
For Example.. If suppose Client send the 10 request to the Web Service in one instance of time and consider that each request will take 10 secs to execute that means 10 requests will take 100 seconds to execute.
My question is; Can't we reduce the response time. I mean, If i execute all 10 request in parallel that means all request will execute within 10 Secs. I know this type of thing can be achieved in Java. Since I never created the web service in PHP. So please can anyone tell me how to achieve this in PHP.
Is there way to handle the requests concurrently or parallel in PHP. I searched many things regarding this on internet but unfortunately I didn't find appropriate results.
Thanks for replying on my post.. The number of concurrent will be changed once the web service is successfully serves the 100 request per minute.. My first target is to handle 100 request.. If this works fine then my next target will be 1000 per minute..
Although I tried to install pthread on my hosting space(On Godaddy) using pcntl. But unfortunately that installation failed..Also, I did not find proper documentation of PThread. Is it possible to install PThread on my local wamp?? If yes could share the the steps with me..If I successfully install PThread on local wamp then i can set my local ip over the internet so that web service can be accessed over the internet..
I am working on blog-aggregation project.
One of the main tasks is the fetching of RSS feeds of blogs and processing them. I have currently about 500 blogs, but the number will be increasing steadily with time (it should reach thousands soon).
Currently (still beta), I have cron job which periodically fetches all the RSS feeds once every day. But this puts all processing and network IO on only once per day.
Should I:
Keep the current situation (all at once)
Make hourly fetching of number_of_blogs / 24 (constant cron job timing)
Change cron periodicity to make constant number of RSS fetches (10 blogs every smaller time)
or there any other ideas?
I am on shared hosting, so reducing CPU and network IO is much appreciated :)
I have used a system that adapts the update frequency of the feed, described in this answer.
You can spare resources if you use conditional HTTP GET's to retrieve feeds that support it. Keep the values of the Last-Modified and ETag headers from the HTTP response. On the next try supply their values in the If-Modified-Since and If-None-Match request headers.
Now if you receive the HTTP 304 response code you know the feed hasn't changed. In this case the complete feed hasn't been send again, only the header telling you there are no new posts. This reduces the bandwidth and data processing.
I had similar situation, but not so many blogs :) I used to import them once in 24 hours but to save CPU load, I was using sleep() after every blog, like sleep(10); and it kept me safe.
I would consider using the Google App Engine to retrieve and process the 'raw' information and have it POST out the data in managable size packets to the web server. The GAE has its own cron job system and can run independantly 24/7.
Currently using a similar system to retrieve job information from several websites and compile it for another, brilliant way to offset the bandwidth & processing requirements as well.