I am looking to create a site that will allow users to create groups and then chat/post within those groups. However, when posts/chats are made within a group, I do not want users to have to reload the page to view these new posts/chats within that group. My question boils down to this: what is your best opinion of how to do this (languages, webservices, etc.)?
I know PHP, SQL, HTML, CSS well and know XML, Javascript, AJAX not so well (I have encountered them enough to read the code and know how they work, but I am by no means skilled or confident with them. I have feeling I will need to read a book on one/all of these to build the kind of site I described.)
Any and all input would be greatly appreciated.
The Web does a great job handling the typical request/response model where a client makes a request and a server responds with a resource. However, when it comes to applications where the server must send data to the client without that client requesting the data, this is where we must get creative.
There are a few different methods that one can use to facilitate a real time Web-based application.
Polling:
Polling involves the client periodically making requests to the server in order to receive updates. There are two main problems with this approach: First, there may not be any data for the server to push for a very long time. Hence, lots of bandwidth is potentially wasted continuing to poll the server for updates. Second, the polling rate determines how real-time the application feels. While a fast polling rate will make the updates appear sooner, it wastes bandwidth. Conversely, polling in longer intervals uses less bandwidth, but the downside is that updates don't appear as quickly.
In general, this is a very poor solution to use for a chat application in the year 2012.
Comet/Reverse AJAX:
Comet is a technique that has been successfully used in the last 5 years to take the concept of request/response and use a hack to simulate a real-time effect. The general idea behind Comet is that the client makes a request to the server, and the server holds the connection open indefinitely. The server waits until there is an update to send to the clients. Once the update is ready, the server sends the response, which simulates the server making the request to the client. Once the client receives the response, it opens a new connection, and the process repeats.
This technique has been shown to scale to over 20,000 simultaneous connections on some platforms when combined with Continuations, which ensure waiting threads are freed up for other tasks.
This not only saves bandwidth, but it makes the application feel extremely real time.
Websockets:
Websockets was introduced in HTML5 as a replacement for Comet, using the ws:// protocol instead of http. However, this has not yet been widely adopted by all browser vendors, and there may still be discussions regarding the specification for the protocol. It has many of the same benefits as Comet.
For more information on Comet, check out Comet and PHP and the challenges of Comet in PHP. For client side integration, check out the Dojo Cometd Library.
I would have said AJAX is the best method of doing this. Alternatively, create an iframe which reloads
Related
I am trying understand websockets.
I have seen 2 examples here in doc and also here.
Both examples are using endless loop cycling, listening for when a new client connects, when they do something interesting and when they are disconnected.
My question is: Is using websockets (with endless loop cycling) better than an ajax solution with http requests per x time ?
AJAX and WebSockets are vastly different. Asking if one is better than the other is like asking if a screwdriver is better than a hammer.
WebSockets are used for real time, interactive communication. Both sides of a WebSocket connection can send data and it will be received within milliseconds by the other end. The connection stays open, reducing latency due to connection negotiation.
However, it only sort of plays nicely with HTTP. That is, it plays nicely with proxies that are WebSocket aware, and with firewalls. WebSocket traffic is most definitely not HTTP traffic, except for the client's first packet, which requests switching from HTTP to the WebSocket protocol.
AJAX, on the other hand, is pure HTTP. The only difference between AJAX and a standard web request is that an AJAX request is initiated by client side scripts and the response is available to that same script rather than reloading the page.
In both AJAX and WebSockets, the client scripts can receive data and use it within that same script. That's where the similarities end.
WebSockets set up a permanent connection and both sides can send data at any time, or sit quietly at any time. With AJAX, the client makes a request and the server responds.
For instance, if you were to set up a new message notification system, if you were using WebSockets, then as soon as a new message is available, the server sends it straight to the browser. If there are no new messages, the server stays quiet. If you were using AJAX, the client would periodically send a request to the server, which would always respond, either saying there were no new messages, or delivering the notifications that are pending. There is no way for the server to initiate things on its end, it must wait for the AJAX request.
Server side, things diverge from the traditional PHP web development paradigms. A typical WebSocket server will be a stand alone, CLI application running as a daemon. (If that last sentence doesn't make sense, please spend a while taking the time to really understanding how to administer a server.)
This means that multiple clients will be connecting to the same script, and superglobal variables like $_GET and $_SESSION will be absolutely meaningless. It seems easy to conceptualize in a small use case, but remember that you will most likely want to get information from other parts of your site, which often means using libraries and frameworks that have absolutely no concept of accessing data outside of the HTTP request/response model.
Thus, for simplicity, you'll usually want to stick with AJAX requests and periodic polling, unless you have the means to rethink the network data and possibly re-implement things that your libraries automate, if you're looking to update standard web traffic.
As for the server's loop:
It is not a busy loop, it is an IO blocked loop.
If the server tries to read network data and none is available, the operating system will block (pause) the script and go off to do whatever else needs to be done. In my WS server, I block waiting for network traffic for at most 1 second at a time, before the script returns to check and see if anything else new happened that I should notify my clients of. Typically, this is barely a few milliseconds before the server goes right back to its IO blocked state waiting for new data on the wire. Some others have implemented my server using LibEv, which allows them to respond to events outside of the network IO without having to wait for the block to timeout.
This is the way nearly every server does things. This is why you can have Apache actively listening and serving web traffic without every server that runs Apache being pegged at 100% CPU usage even when there is no traffic.
In closing, WebSockets is a wonderful technology, but web libraries and frameworks are simply not built to use them. Thus, unless you're working in a system where waiting 3 seconds for a full AJAX request is far, far too long, it's probably best to use AJAX. If you're writing a multiplayer interactive game or a chat system, then you've found a perfect use for WebSockets.
I do heartily encourage everyone to learn WebSockets... but it's not a magic bullet, and few parts of the web are designed in ways where people can get real use out of it.
Yes, sockets are better in many cases.
It's not forever loop with 100% cpu utilizing, it's just liveloop, which exists in each daemon application.
Sync accept operation is where 99.99% of time we are.
Ajax heartbeat is more traffic, more server CPU and memory.
I too am in the learning phase. I have built a php-based websocket server and have it communicating with web pages. Perhaps my 2c perspective is useful.
Getting the websocket server (wss) working using available sources as a starting point is not that difficult, but what to do with it all next is.
The wss runs in CLI version of php. Late model browser loads a normal http or https page containing a request to the wss, along with anything else that page needs to do, a handshake occurs. Communication is then possible directly between browser and wss at the whim of either end. This is low overhead and hence fast and simple. Very cool. What is said over that link needs to be understood by both ends - subprotocol agreement. You may have to roll your own in php and in javascript. No more http headers, urls, etc etc.
The wss is a long-lived, stateful instance of php (very unlike apache etc which forget you on sending the page). An entire app can be run in the wss instance, keeping state for itself and each connected client. It used to be said that php was too leaky for long life but I don't hear that much any more. But I believe you still have to be careful with memory.
However, being a single php instance there is not the usual separation between client instances. For example statics in classes are shared with every class instance and hence every client. So for a single user style app sharing data with a heap of clients this is great. I can see that Ajax type calls can be replaced in this way, but if the app still had to rebuild state to service each client, and then release it to save resources, that seems to lessen the advantage.
Going a step further and keeping truly stateful instances for clients seems like a possible next step. Replicating the traditional session based system is one possibility, alternatively fork new php interpreters and look after communications between parent and children via sockets or suchlike. But this would require resources per client that would be severely limiting for any non-trivial app.
Or perhaps it is possible to put the bulk of the app in the parent and let the children just do the very client specific stuff. Or break the app design into small independent units that can communicate directly via sockets. Socket communication does seem to be catching on nowadays.
As Ghedpunk says in so many ways, the real world does not yet seem ready to realise the full potential of the web socket concept but it can certainly replace Ajax. The added advantage of the server sending without being asked opens up new possibilities previously too difficult to consider.
I have a web application driven primarily by javascript/ajax, somewhat similar to how google docs work; all people viewing a page will be seeing the same information in relative real-time. It's not crucial that the information is actually real-time, a second or so is fine.
Currently, the application is ajaxing the server every 5 seconds. I was researching server-sent events and they sound like exactly what I need... but this is my understanding: server-sent events essentially just move the polling to the server. The PHP script doing the server-sent events will check the database for changes every X seconds, and send an update to the application when it finds one.
Checking once per second would probably be adequate, but since I'm on shared hosting I want to avoid any unnecessary load possible. Is there way I can subscribe to updates to the database? Or is there a way I can notify the script from other PHP scripts that make changes to the database?
With PHP, polling the DB is the typical way to do this. You could also use TCP/IP sockets to connect to some kind of application server, that sits in front of your database, and knows about all writers and all consumers. I.e. when a write comes in, it both broadcasts it to all consumers and writes it to the DB. The consumers in that examples are the PHP scripts (one per SSE client).
If you use WebSockets, then you need exactly the same architecture, because PHP is single-threaded: each SSE connection is an independent PHP process.
If you switch to using, say, node.js, then that application server can be built-in. (Again, it would work the same way, whether SSE or WebSockets.)
But, you mention you intend to use shared hosting. SSE (and WebSockets, and comet technologies) hold a socket open, which interferes with the economics of shared hosting. So your sockets are likely to get closed regularly. My advice would be to stick with ajax (and therefore DB) polling every 5 seconds, instead of SSE, until your application is worth enough that the $10-$100/month for a real host is not an issue. Then consider using SSE to optimize the latency.
P.S. The decision between SSE and WebSockets is all about write frequency. My guideline is if your clients write data, on average, once a second or more frequently, web sockets are better, because it is keeping the write channel open. If once every 5+ seconds then web sockets does not bring much, compared to just using an Ajax post each time you have data to write. An SSE back-end is simpler to deal with than a WebSockets back-end. (Writes every 1-5 seconds is the grey area.)
What I would recommend is instead of polling the database for changes, you will know when there is going to be a database change because your application will be making that change. I would use web sockets (https://developer.mozilla.org/en-US/docs/WebSockets) and simply push an update to all active clients when any member makes a change.
Here is the difference between Server Send Events and Web Sockets. (In your case Web Sockets are the way to go)
Websockets and SSE (Server Sent Events) are both capable of pushing data to browsers, however they are not competing technologies.
Websockets connections can both send data to the browser and receive data from the browser. A good example of an application that could use websockets is a chat application.
SSE connections can only push data to the browser. Online stock quotes, or twitters updating timeline or feed are good examples of an application that could benefit from SSE.
In practice since everything that can be done with SSE can also be done with Websockets, Websockets is getting a lot more attention and love, and many more browsers support Websockets than SSE.
However, it can be overkill for some types of application, and the backend could be easier to implement with a protocol such as SSE.
I'm looking for the 'way to go' (i.e. the most efficient, most used, general accepted way) when it comes to the reloading of data from a web server to a front end. In the end application, I will have several output fields where data has to be written to, for example like this:
The data streams will be different from each other in the end application. The lines will have to be reloaded with fresh, up to date data from the server.
I have been thinking of using Ajax requests to update like every second, but there has to be an other way to do this. Ajax requests will cause a lot data traffic. Also, when using the Facebook chat, you don't have to wait every second, chats are received almost instantly. Yet I don't see any Ajax polling requests being made when I use the developer tools of Mozilla Firefox. This made me think if there would be a different way to do this.
I've looked into Node.js, but it appears that isn't possible with my host.
I have heard people talking about Ajax Push, is that what I should use? If so, can you give me a basic usage example?
If not, what would then be the way to go when having multiple data streams that have to be reloaded within a second?
Requirements are speed and low data traffic. It therefore wouldn't be an option to continuously poll the server, I think, because that would create an enormous overhead.
I don't think it's of any importance, but I'm using PHP5.3 in the back end and JavaScript with jQuery 1.9.1 in the front end.
This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:
What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
Using comet with PHP?
Apache with Comet Support
Server Scalability - HTML 5 websockets vs Comet
How to implement event listening in PHP
In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).
HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.
Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.
Also see the realtime web technology guide.
I think what you are looking for is generally called Comet. The was this technique is often made to work is as follows:
The client (web browser) makes a request to the server for new data. This is not reloading the page, but rather is done in JavaScript
The server responds to the request when it has some data for the client. Again, this doesn't impact the UI since it isn't the page itself that's getting reloaded: the loaindg of data is done "in background" so to speak, in JavaScript code.
On the serve side, the request waits for new data, and returns the new data when available, or returns nothing if a timeout interval (defined on the server) is reached. This timeout is usually set to be lower than the browser HTTP timeout. The reason for this is so that the server can know whether a particular client got a particular piece of data. If the request is allowed to time out on the client side, the original request might be responded to by the server after the client has timed out, and the client will not get the data, even though the server thinks that it did.
The data is indeed usually transferred as JSON, but you can choose whatever encoding you'd like. See here for one example of how to do this. Goosh is another example of this technique, and so is Interactive Python Shell. The code for all is available.
On the PHP side you will want to create a page that will respond to these "background" JavaScript Comet requests. It could be the same page as the one that user loads, but let's say it is different, for ease of explanation. So the user loads index.php and the JavaScript Comet code calls getNewData.php to retrieve new data.
In your getNewData.php you will want to wait for your event and return the data then. You don't want to use polling for this, but there are PHP libraries that allow one to use various interprocess communication strategies to wait on events, see this question for instance. The high-level pseudocode for your getNewData.php would look as follows:
parse JSON request
Enter an efficient wait state (with timeout), waiting for your "new data is available" event
Did previous step time out?
Yes: send response indicating no data
No: send response with new data
I'm looking for the 'way to go' (i.e. the most efficient, most used, general accepted way) when it comes to the reloading of data from a web server to a front end. In the end application, I will have several output fields where data has to be written to, for example like this:
The data streams will be different from each other in the end application. The lines will have to be reloaded with fresh, up to date data from the server.
I have been thinking of using Ajax requests to update like every second, but there has to be an other way to do this. Ajax requests will cause a lot data traffic. Also, when using the Facebook chat, you don't have to wait every second, chats are received almost instantly. Yet I don't see any Ajax polling requests being made when I use the developer tools of Mozilla Firefox. This made me think if there would be a different way to do this.
I've looked into Node.js, but it appears that isn't possible with my host.
I have heard people talking about Ajax Push, is that what I should use? If so, can you give me a basic usage example?
If not, what would then be the way to go when having multiple data streams that have to be reloaded within a second?
Requirements are speed and low data traffic. It therefore wouldn't be an option to continuously poll the server, I think, because that would create an enormous overhead.
I don't think it's of any importance, but I'm using PHP5.3 in the back end and JavaScript with jQuery 1.9.1 in the front end.
This question has been asked a number of times, but in a slightly different ways. Here are a few references that are worth a read:
What are Long-Polling, Websockets, Server-Sent Events (SSE) and Comet?
Using comet with PHP?
Apache with Comet Support
Server Scalability - HTML 5 websockets vs Comet
How to implement event listening in PHP
In summary: if you are looking at building your solution using PHP on Apache then holding open persistent connections (HTTP long-polling or streaming) is going to use up resources very quickly (is highly inefficient). So, you would be better using a hosted solution (*disclaimer - I work for a hosted solution).
HTTP-Long polling and HTTP Streaming are solutions which have been superseded by Server-Sent Events and WebSockets. So, where possible (where the web client provides support) you should use one of these solutions before falling back to an HTTP-based solution. A good realtime web technology will automatically handle this for you.
Since your diagram shows you are subscribing to multiple data streams you should also consider a Publish/Subscribe solution that naturally fits with this. Again, a good realtime web tech solution will provide you with this.
Also see the realtime web technology guide.
I think what you are looking for is generally called Comet. The was this technique is often made to work is as follows:
The client (web browser) makes a request to the server for new data. This is not reloading the page, but rather is done in JavaScript
The server responds to the request when it has some data for the client. Again, this doesn't impact the UI since it isn't the page itself that's getting reloaded: the loaindg of data is done "in background" so to speak, in JavaScript code.
On the serve side, the request waits for new data, and returns the new data when available, or returns nothing if a timeout interval (defined on the server) is reached. This timeout is usually set to be lower than the browser HTTP timeout. The reason for this is so that the server can know whether a particular client got a particular piece of data. If the request is allowed to time out on the client side, the original request might be responded to by the server after the client has timed out, and the client will not get the data, even though the server thinks that it did.
The data is indeed usually transferred as JSON, but you can choose whatever encoding you'd like. See here for one example of how to do this. Goosh is another example of this technique, and so is Interactive Python Shell. The code for all is available.
On the PHP side you will want to create a page that will respond to these "background" JavaScript Comet requests. It could be the same page as the one that user loads, but let's say it is different, for ease of explanation. So the user loads index.php and the JavaScript Comet code calls getNewData.php to retrieve new data.
In your getNewData.php you will want to wait for your event and return the data then. You don't want to use polling for this, but there are PHP libraries that allow one to use various interprocess communication strategies to wait on events, see this question for instance. The high-level pseudocode for your getNewData.php would look as follows:
parse JSON request
Enter an efficient wait state (with timeout), waiting for your "new data is available" event
Did previous step time out?
Yes: send response indicating no data
No: send response with new data
I need to implement real-time page data update with php and jquery.
(I found www.ape-project.org/ but it seems site is down)
Is any other solutions?
Very TNX!
You might want to check out Comet:
Comet is a web application model in
which a long-held HTTP request allows
a web server to push data to a
browser, without the browser
explicitly requesting it.[1][2] Comet
is an umbrella term, encompassing
multiple techniques for achieving this
interaction. All these methods rely on
features included by default in
browsers, such as JavaScript, rather
than on non-default plugins. The Comet
approach differs from the original
model of the web, in which a browser
requests a complete web page at a
time.
http://en.wikipedia.org/wiki/Comet_%28programming%29
If you want to do streaming (sending multiple messages over a single long lived, low latency connection), you probably need a comet server. Check out http://cometdaily.com/maturity.html for details on a variety of server implementations (I am the maintainer of one of them - Meteor).
If you are happy to reconnect after each message is received, you can do without complicated servers and transports and just use long polling - where you make an ajax request and the server simply sleeps until it has something to send back. But you will end up with LOTS of connections hanging off your web server, so if you're using a conventional web server like Apache, make sure it's configured to handle that. By default Apache doesn't like having more than a few hundred concurrent connections.
There is lots of solutions to do this...
Depending on what is your data, how your data are organized and stored (mysql ?).
Your question is too open to have a real answer.