Assume that I have a simple VPS setup with LAMP (so with PHP and MySQL on the same server and no other strings attached). And assume that I want to make a self-written ajax chat client on my website.
Obviously, each participant in the conversation would have to listen constantly for new things being said. Since it is very well possible that two or more participants say something in the very same second (and refreshing more than once per second would likely cause insane system load), it seems to me that I would need to store for each participant a list of things that happened since the last refresh.
Which would be the "best" way to do this (in terms of system load)? In the following, an "event" just 'any participant saying anything in the chat'. Clearly, this could be used for a more general as well.
(A) Use MySQL, connecting to the db every second and asking for events WHERE participant_id = $participant_id? (and then deleting all of these so they're only fetched once)
(B) Create a file $participant_id.php and append the events to it (in PHP format so that it can be included, and then empty or delete the file at the next refresh?
(C) Does anyone know any other useful alternatives?
An alternative would be to use a socket connection. Each person connected to the socket server daemon would be able to send a message to the daemon, the daemon would then send the message out to all or a partial list of subscribers which makes chat instantaneous with no need to save the data at all.
A good way to create socket connections from a client is socket IO. See below.
http://socket.io/
A good technology to use for creating a socket server daemon is node.js. This is a server side event driven javascript based library. Very efficient for things like this. See below.
http://nodejs.org/
On both A and B you are still effectively polling. You will either poll MySQL which really isn't too bad, or you can get notified on select() of a file change BUT you will still need to parse to see if the new data is the right stuff on the file-side.
For conceptual and support ease-of-use, it is really hard to beat a database as you won't have to worry about locking semantics. Debugging and message tracking are clean in this structure.
I however recommend you investigate the msg_send() and msg_receive() (of PHP) functions to put this data into an underlying message queue. Your problem seems to be a message queueing problem that should be solved by that mechanism.
Does anyone know any other useful alternatives?
If you search simple solutions on PHP, I can offer 2 ways:
Cache
It mean that you keep MySQL for store data, but install APC (this solution is simplest and fastest for small servers and applications) or Memcached (better for using width several servers). For each read-request you check APC/Memcached for you data and ask MySQL only if your cache is removed or updated. And on each write-request you inserting data in MySQL and update cache.
Other DB
In this case you change MySQL for one of memory-base DB (for example MongoDB). And you may not afraid hard disk usage.
Related
Is there any way you can push data to a page rather than checking for it periodically?
Obviously you can check for it periodically with ajax, but is there any way you can force the page to reload when a php script is executed?
Theoretically you can improve an ajax request's speed by having a table just for when the ajax function is supposed to execute (update a value in the table when the ajax function should retrieve new data from the database) but this still requires a sizable amount of memory and a mysql connection as well as still some waiting time while the query executes even when there isn't an update/you don't want to execute the ajax function that retrieves database data.
Is there any way to either make this even more efficient than querying a database and checking the table that stores the 'if updated' data OR tell the ajax function to execute from another page?
I guess node.js or HTML5 webSocket could be a viable solution as well?
Or you could store 'if updated' data in a text file? Any suggestions are welcome.
You're basically talking about notifying the client (i.e. browser) of server-side events. It really comes down to two things:
What web server are you using? (are you limited to a particular language?)
What browsers do you need to support?
Your best option is using WebSockets to do the job, anything beyond using web-sockets is a hack. Still, many "hacks" work just fine, I suggest you try Comet or AJAX long-polling.
There's a project called Atmosphere (and many more) that provide you with a solution suited towards the web server you are using and then will automatically pick the best option depending on the user's browser.
If you aren't limited by browsers and can pick your web stack then I suggest using SocketIO + nodejs. It's just my preference right now, WebSockets is still in it's infancy and things are going to get interesting once it starts to develop more. Sometimes my entire application isn't suited for nodejs, so I'll just offload the data operation to it alone.
Good luck.
Another possibility, if you can store the data in a simple format in a file, you update a file with the data and use the web server to check its timestamp.
Then the browser can poll, making HEAD requests, which will check the update times on the file to see if it needs an updated copy.
This avoids making a DB call for anything that doesn't change the data, but at the expense of keeping file system copies of important resources. It might be a good trade-off, though, if you can do this for active data, and roll them off after some time. You will need to ensure that you manage to change this on any call that updates the data.
It shares the synchronization risks of any systems with multiple copies of the same data, but it might be worth investigating if the enhanced responsiveness is worth the risks.
There was once a technology called "server push" that kept a Web server process sitting there waiting for more output from your script and forwarding it on to the client when it appeared. This was the hot new technology of 1995 and, while you can probably still do it, nobody does because it's a freakishly terrible idea.
So yeah, you can, but when you get there you'll most likely wish you hadn't.
Well you can (or will) with HTML5 Sockets.
This page has some great info about this technology:
http://www.html5rocks.com/en/tutorials/websockets/basics/
I'm working on a chat application which I would love to use a SQL db for.
My problem is, after a few google searches, i have people telling me from one site, that using a DB would be much slower then using a normal file (e.g Text or JSON file), but then on some other sites, people are saying the complete opposite. And I don't know about you guys, but when it comes to creating web apps for users, the users always come first.
So as much as I'd love to use a SQL DB as 1.) I have good experience with it and 2.) it allows me to make the application much more cooler (more features). but if it would slow things down on the users end (a noticeable lag), then its a no-no.
Either way, I will be "polling" the server continuously with AJAX and PHP to check the file/DB (for new messages, contact requests, ect ect).
Also, incase your wondering, the application wont be like a 1-to-1 chat, it will have "rooms" where multiple users can join and talk with all users joining in. The users will also be able to request a "private chat" with another user, where a 1-to-1 connection opens up.
So, MySQL Database OR a boring TEXT/JSON/OTHER file, in regards to performance?
Oh, one more thing, I don't want to use any third party libraries or APIs. Hate relying on other peoples work (been let down to many times).
If you're looking to implement an IRC clone, I think you've chosen all the wrong tools.
The best way to do this would be to write a custom HTTP server that handles everything in memory. No databases, no constant polling of files. When a message arrives, you simply loop through the correct in-memory list and dispatch the message to other users. For the browser to server connection, I suggest "Comet" (with web sockets for browsers that support them, if you're feeling up to it).
PHP likely isn't the language of choice for this, because pretty much all work done with PHP is based on traditional short, isolated requests. For a long-running process which serves multiple clients in real time, I'd suggest something like Python or Node.js.
You don't really want to be storing chats in files, that can create a management nightmare, I would recommend you go with MySQL and to make sure it works probably go with Sockets instead of AJAX polling, Sockets will scale really well.
However there isn't much around about how you can integrate socket based chats with MySQL.
I have done a few tests and have a basic example working here: https://github.com/andrefigueira/PHP-MySQL-Sockets-Chat
It makes use of Ratchet (http://socketo.me/) for the creation of the chat server in PHP.
And you can send chat messages to the DB by sending the server JSON with the information of who is chatting, (if of course you have user sessions)
I am building a "multiplayer world" with jQuery and PHP. Here is a bit how it works:
User's character's positions are taken from a database, user is plotted accordingly (the position values are CSS values - left and top)
User is able to move about using the arrow keys on the keyboard, making their character move using jQuery animations. While this is happening (on each arrow press) the user's position values are inserted into a database and updated.
In order to make this "global" (so users see each other) as you could say, the values need to be updated all at once for each user using AJAX
The problem I am having is I need to continuously call a JavaScript function I wrote which connects to the MySQL server and grabs values from a database table. And this function needs to be called constantly via setInterval(thisFunction, 1000); however my host just suspended me for overloading the server's resources and I think this was because of all my MySQL queries. And even after grabbing values from my database repeatedly, I had to insert values every few seconds as well so I can imagine that would cause a crash over time if enough clients were to login. How can I reduce the amount of queries I am using? Is there another way to do what I need to do? Thank you.
This is essentially the same thing as a chat system with regards to resource usage. Try a search and you'll find many different solution, including concepts like long polling and memcached. For example, check this thread: Scaling a chat app - short polling vs. long polling (AJAX, PHP)
You should look into long polling - http://en.wikipedia.org/wiki/Push_technology. This method allows you to establish a connection with your server, and then update it only when you need to. However by the sounds of it, you have a pretty intensive thing going on if you want to update every time, you may want to look into another way of storing this data, but if your wondering how big companies do it, they tend to have mass amounts of servers to handle it for them, but they will also use a technique similar to long polling.
You could store all the positions in memory using memcached See http://php.net/manual/fr/book.memcached.php and save them all at once every few seconds into the database (if needed).
You could use web sockets to overcome this problem. Check out this nettuts tutorial.
There is another way, it's to emulate or use actual sockets. Instead of constantly pulling the data (refreshing to check if there are new records), you can push the data over WebSockets which works in Chrome at the moment (at least to my knowledge, didn't try it in FF4) or you can use Node.js for leaner long pooling. That way, the communication between players will be bi-directional without the need of MySQL for storing positions.
Checkout Tornado
From their site:
Tornado is an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google's webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking infrastructure.
The framework is distinct from most mainstream web server frameworks (and certainly most Python frameworks) because it is non-blocking and reasonably fast. Because it is non-blocking and uses epoll, it can handle thousands of simultaneous standing connections, which means it is ideal for real-time web services. We built the web server specifically to handle FriendFeed's real-time features — every active user of FriendFeed maintains an open connection to the FriendFeed servers. (For more information on scaling servers to support thousands of clients, see The C10K problem.)
I'm using Flex 4. Have PHP backend and mysql database with one table consisting of multiple rows.
I take the raw data from the result event and decode it using JSON. I then dump the data into an ArrayCollection that I use as my datagrid's data provider.
My question is how can I tell when someone inserts a new row into the mysql table so that I can automatically refresh my ArrayCollection, thus seamlessly updating my datagrid one element at a time? Right now, it's just a one time call and the connection is closed. If someone inserts a new row into the database my program doesn't recognize, unless I restart it. I'd like to auto-update the AC whenever a single new row is inserted into the mysql database. Is there a way I can "listen" for this change?
Ah, you've stumbled upon the age old question of the web realm: Polling or Pushing?
Polling means that you ping the server every few seconds or minutes to check if there's any data that has changed. If there is, your server sends you the new changed data which up update appropriately on your front-end. The 'protocol' on how to interpret which piece of data needs to be updated is totally up to you since there's no real standard (since data in itself can be very different from system to system). Polling is still in use today in many systems that do not need crucial 'live' information and since it doesn't need a consistent connection, it's particularly good for iffy internet like mobile. Plus, everything is an HTTP request, so there's no enterprise firewall that can block it.
Pushing means that you have a constant connection between your front end and back end which normally goes over RTMPT (HTTP UDP protocol to circumvent enterprise firewalls, but not 100%). It's great if you need real time data (like say financial data) to be delivered to you quickly. However, the user needs a consistent internet connection and you need to have a server capable of dealing with the amount of connections and sessions management. Normally, most people end up using Java since there are many libraries to handle pushing (BlazeDS, GRaniteDS, Livecycle, Wowza, etc).
Since you're using PHP, you'll probably need to use polling as your solution, but need to implement it yourself. I'm sure there are libraries out there to help you out though.
No, there is no automatic way to do that. But you can regularly 'ping' your server and ask for new rows. Use
setInterval(myFunctionName, timeToWaitBetweenEachCallInMilliseconds);
to do that.
After searching the web for a good Comet and also and asking you guys what my best option is, I've chose to go with Orbited. The problem is if you need a good documentation about Comet you won't find. I've installed Orbited and It seems It works just fine.
Basically, I want to constantly check a database and see if there is a new data. If there is, I want to push it to my clients and update their home page but I can't find any good and clear doc explaining how constantly check the database and push the new info to Orbited and then to the clients. Have you guys implemented that?
Also, how many users can Orbited handle?
Any ideas?
You could add a database trigger that sends messages to your message queue when the database got changed. This is also suggested here. Or, if it is only your app talking to the database, you could handle this from within the app via a Subject/Observer pattern, notifying the queue whenever someone called an action changing something in the DB.
I don't know how good or bad Orbited scales.
Have a reference table that keeps track of the last updated time of the source table. Create a update/delete/insert trigger for the source table that updates the time in the reference table.
Your comet script should keep checking the reference table for any change in the time. If the change is noticed, you can read the updated source table and push the data to your client's home page. Checking the reference table in a loop is faster because the MySQL will serve the results from its cache if nothing has changed.
And sorry, I don't know much about Orbited.
I would use the STOMP protocol with Orbited to communicate and push data to clients. Just find a good STOMP client with PHP and get started.
Here is an example of a use case with STOMP, although the server side is written in Ruby:
http://fuglyatblogging.wordpress.com/2008/10/
I don't know if PHP with Apache (if that's what you are using) is the best suite for monitoring database changes. Read this article, under the section title "Orbited Server", for an explanation: http://thingsilearned.com/2009/06/09/starting-out-with-comet-orbited-part-1/
EDIT: If you want to go the route with PHP through a web server, you need to make one, and one only, request to a script that starts the monitoring and pushes out changes. And if that script times out or fails, you need to start a new one. A bit fugly :) A nicer, cleaner way would be, for example, to use twisted with python to start a monitoring process, completely separated from the web-server.