I am doing a curl call to pass information to an API. The issue I am having is sometimes the API responds slowly. I need to immediately pass the data, but I don't want the user to be stuck on the processing page while the API tries to make the connection.
Is there a good alternative, kind of like multithreading or something that I could use to still query this API while moving the user onto the next page?
Thanks!
Use fire-and-forget.
I don't know if CURL can do this but make it so it wont attempt to read from the stream. Just send and close.
If the connection to the remote site is slow as well you'll need to do some proxying.
Fire-and-forget proxying is a poor man's solution to threading.
Related
I recently need to work on a project which involves having a chat. This chat must update in real-time and it is estimated to be used by more than 9000 users at the same time. I have done some researching on how to do that and came to a conclusion: Use ajax
While I researched on ajax, I found a problem:
Problem 1:
If there are a lot of users where the browser is constantly creating ajax call for a file to get the database chat content, wouldn't that put a lot of strain on the server and eventually won't it collapse?
There are a lot of libraries out there which maybe can fullfil my needs but I wanted to start from scratch, is it possible?
Take an example, whatsapp: if you open dev tools you don't see it making ajax calls but when I receive messages, it also doesn't makes the call. facebook on the other hand will get ajax call when users receive a message.
PS: I am not looking for the code, I just want a way to do it. I can code it myself. (I am using php with mysqli)
You'll need to utilize WebSockets: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
This allows the browser to keep a connection open with the server for constant communication, both to and from the server.
The alternative is polling, which is sending periodic ajax requests, as you described.
From the Mozilla page:
With this API, you can send messages to a server and receive
event-driven responses without having to poll the server for a reply.
I am using a script that fetch users infos from an api.
This is how it works I get the id of the user posted in php.
Then make a curl request to the api with the user id to get his infos.
The problem is that whenever the user refresh the page my script do the work again and fetch the infos to display them. But this api take much time to respond and it makes my script works slower.
Is there any solution to cache the api response for sometime (The server i request force cache : no cache in headers) ?
I am thinking about creating a directory with json file (the api response) for every user id . Or I can go with mysql solution .
What's the best solution for me ?
N.B : Im getting about 200 000 request/day.
Thanks
It sounds like you actually need to optimize this API, so that updates will be there as you expect them to be.
However, if you insist that caching is appropriate for your use, there's no need to reinvent the wheel. Simply put Nginx, Varnish, etc. in front of the API server. Then configure your application to request from this caching proxy rather than the API server directly.
See also:
Nginx Caching Configuration Documentation
I'm looking for the 'way to go' (i.e. the most efficient, most used, general accepted way) when it comes to the reloading of data from a web server to a front end. In the end application, I will have several output fields where data has to be written to, for example like this:
The data streams will be different from each other in the end application. The lines will have to be reloaded with fresh, up to date data from the server.
I have been thinking of using Ajax requests to update like every second, but there has to be an other way to do this. Ajax requests will cause a lot data traffic. Also, when using the Facebook chat, you don't have to wait every second, chats are received almost instantly. Yet I don't see any Ajax polling requests being made when I use the developer tools of Mozilla Firefox. This made me think if there would be a different way to do this.
I've looked into Node.js, but it appears that isn't possible with my host.
I have heard people talking about Ajax Push, is that what I should use? If so, can you give me a basic usage example?
If not, what would then be the way to go when having multiple data streams that have to be reloaded within a second?
Requirements are speed and low data traffic. It therefore wouldn't be an option to continuously poll the server, I think, because that would create an enormous overhead.
I don't think it's of any importance, but I'm using PHP5.3 in the back end and JavaScript with jQuery 1.9.1 in the front end.
This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:
What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
Using comet with PHP?
Apache with Comet Support
Server Scalability - HTML 5 websockets vs Comet
How to implement event listening in PHP
In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).
HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.
Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.
Also see the realtime web technology guide.
I think what you are looking for is generally called Comet. The was this technique is often made to work is as follows:
The client (web browser) makes a request to the server for new data. This is not reloading the page, but rather is done in JavaScript
The server responds to the request when it has some data for the client. Again, this doesn't impact the UI since it isn't the page itself that's getting reloaded: the loaindg of data is done "in background" so to speak, in JavaScript code.
On the serve side, the request waits for new data, and returns the new data when available, or returns nothing if a timeout interval (defined on the server) is reached. This timeout is usually set to be lower than the browser HTTP timeout. The reason for this is so that the server can know whether a particular client got a particular piece of data. If the request is allowed to time out on the client side, the original request might be responded to by the server after the client has timed out, and the client will not get the data, even though the server thinks that it did.
The data is indeed usually transferred as JSON, but you can choose whatever encoding you'd like. See here for one example of how to do this. Goosh is another example of this technique, and so is Interactive Python Shell. The code for all is available.
On the PHP side you will want to create a page that will respond to these "background" JavaScript Comet requests. It could be the same page as the one that user loads, but let's say it is different, for ease of explanation. So the user loads index.php and the JavaScript Comet code calls getNewData.php to retrieve new data.
In your getNewData.php you will want to wait for your event and return the data then. You don't want to use polling for this, but there are PHP libraries that allow one to use various interprocess communication strategies to wait on events, see this question for instance. The high-level pseudocode for your getNewData.php would look as follows:
parse JSON request
Enter an efficient wait state (with timeout), waiting for your "new data is available" event
Did previous step time out?
Yes: send response indicating no data
No: send response with new data
I'm looking for the 'way to go' (i.e. the most efficient, most used, general accepted way) when it comes to the reloading of data from a web server to a front end. In the end application, I will have several output fields where data has to be written to, for example like this:
The data streams will be different from each other in the end application. The lines will have to be reloaded with fresh, up to date data from the server.
I have been thinking of using Ajax requests to update like every second, but there has to be an other way to do this. Ajax requests will cause a lot data traffic. Also, when using the Facebook chat, you don't have to wait every second, chats are received almost instantly. Yet I don't see any Ajax polling requests being made when I use the developer tools of Mozilla Firefox. This made me think if there would be a different way to do this.
I've looked into Node.js, but it appears that isn't possible with my host.
I have heard people talking about Ajax Push, is that what I should use? If so, can you give me a basic usage example?
If not, what would then be the way to go when having multiple data streams that have to be reloaded within a second?
Requirements are speed and low data traffic. It therefore wouldn't be an option to continuously poll the server, I think, because that would create an enormous overhead.
I don't think it's of any importance, but I'm using PHP5.3 in the back end and JavaScript with jQuery 1.9.1 in the front end.
This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:
What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
Using comet with PHP?
Apache with Comet Support
Server Scalability - HTML 5 websockets vs Comet
How to implement event listening in PHP
In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).
HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.
Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.
Also see the realtime web technology guide.
I think what you are looking for is generally called Comet. The was this technique is often made to work is as follows:
The client (web browser) makes a request to the server for new data. This is not reloading the page, but rather is done in JavaScript
The server responds to the request when it has some data for the client. Again, this doesn't impact the UI since it isn't the page itself that's getting reloaded: the loaindg of data is done "in background" so to speak, in JavaScript code.
On the serve side, the request waits for new data, and returns the new data when available, or returns nothing if a timeout interval (defined on the server) is reached. This timeout is usually set to be lower than the browser HTTP timeout. The reason for this is so that the server can know whether a particular client got a particular piece of data. If the request is allowed to time out on the client side, the original request might be responded to by the server after the client has timed out, and the client will not get the data, even though the server thinks that it did.
The data is indeed usually transferred as JSON, but you can choose whatever encoding you'd like. See here for one example of how to do this. Goosh is another example of this technique, and so is Interactive Python Shell. The code for all is available.
On the PHP side you will want to create a page that will respond to these "background" JavaScript Comet requests. It could be the same page as the one that user loads, but let's say it is different, for ease of explanation. So the user loads index.php and the JavaScript Comet code calls getNewData.php to retrieve new data.
In your getNewData.php you will want to wait for your event and return the data then. You don't want to use polling for this, but there are PHP libraries that allow one to use various interprocess communication strategies to wait on events, see this question for instance. The high-level pseudocode for your getNewData.php would look as follows:
parse JSON request
Enter an efficient wait state (with timeout), waiting for your "new data is available" event
Did previous step time out?
Yes: send response indicating no data
No: send response with new data
I have created many websites that get all there data from a API.
(located on the same server on most cases).
The websites are operation really slow, of all the curl requests.
I first thought it was our mysql server (sepparate server) but now we implemented caching it's still slow.
Is there a good way to find out why it takes so long to do the curl requests.
And what would be a good way to go?
Could use a browser rest client to point to your API and use a profiling tool (xdebug/xhprof) to find the source of the bottleneck.
Possibly make sure the api calls resolve locally and dont go all the way out to the internet before coming back in (but may not shave off much time).
Would recommend starting with the APIs code.
The problem is not the cURL request.