I have seen many questions similar to the one I asked.
A few of them said that its ok to poll, and that doesn't affect the database performance significantly, and some are still unanswered. What I want to know is this :
Can I do something similar to ASP.NET's sqldependency, which helps to find out when changes are made to the database WITHOUT polling.
I know polling is ok to do, might not affect database performance, but I need to know the update as soon as it is made, so I feel I need something better than just polling every 1 second.
As an add on point, the updates that are detected at the server then have to be pushed (Ajax Push - I'm using Ajax Push Engine) to all the clients who are subscribed to a particular channel.
Thanks & Regards,
Thothathri Srinivasan
Redis PubSub
I hoped triggers could help you out, but after a little examination I (highly) have my doubts.
You could achieve your goal using Redis pubsub. Redis is an extremely fast data structure server. Publish the database changes to a channel and let the interested clients detect the changes by subscribing to that same channel(in real-time).
Downloading/Installing Redis is really easy and as extra bonus you will also "get an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets."
Installing it is probably the best way, but if you can't, you could use the free http://redistogo.com plan to talk the Redis server with the limitation that you will only get 5MB to store your data and that it will not be persisted to disc.
You can use a server push with an implementation like websockets or comet.
I like Alfred's pubsub idea. I would think about the database as a subscriber to the insert events, not as the canonical version. Maybe this isn't possible with your existing codebase. You might also be able to monitor the binary logs on the database server to watch changes come in and push this information to the subscribers.
I think Q4M is exactly what you want
http://q4m.github.com/tutorial.html
Related
I'm building a chat widget using pusher service (http://pusher.com) and I need to save all the messages sent by users into database, so that it should be accessible after a period of time. I'm using mysql database and the only way that comes to my mind is that make a new insert each time the chat message event is triggered, but I'm afraid it will not be as fast as it should be.
What databases and techniques would you prefer in this case for saving chat messages?
One approach you can take is to use MySQL's INSERT DELAYED. When you do this, the client does not have to wait for the insert to complete. Rather, the server will queue the inserts and execute them as it can. So the process/thread thats handling the chat will not have to wait on them. But you'll still have a chat history stored in the database that can be retrieved. Assuming you are using pusher to share 'live' messages as they come in, and thus dont need immediate access to the chat history, this might do the trick for you.
Unless you're planning on creating a chat system including thousands of people I'm fairly sure that a properly structured mysql solution will handle this no problem. Where I work we have a intranet chat solution based on php and mysql and it is running flawlessly (for our medium sized team of ~100 some people).
One suggestion I do have is to make sure you understand mysql indexing. Make sure your chat system is truly utilizing the indexes to their fullest potential. You could greatly increase performance this way.
If concurrency really is becoming an issue for you I hear a lot of people are having good experiences with node.js. But if you're comfortable in php and mysql I'd say go that way first.
Happy coding!
(I know this is old but,) There is an interesting demo here (albeit in Rails) that does what you're talking about. It includes a possible schema.
https://github.com/dipth/pusher_demo
Assume that I have a simple VPS setup with LAMP (so with PHP and MySQL on the same server and no other strings attached). And assume that I want to make a self-written ajax chat client on my website.
Obviously, each participant in the conversation would have to listen constantly for new things being said. Since it is very well possible that two or more participants say something in the very same second (and refreshing more than once per second would likely cause insane system load), it seems to me that I would need to store for each participant a list of things that happened since the last refresh.
Which would be the "best" way to do this (in terms of system load)? In the following, an "event" just 'any participant saying anything in the chat'. Clearly, this could be used for a more general as well.
(A) Use MySQL, connecting to the db every second and asking for events WHERE participant_id = $participant_id? (and then deleting all of these so they're only fetched once)
(B) Create a file $participant_id.php and append the events to it (in PHP format so that it can be included, and then empty or delete the file at the next refresh?
(C) Does anyone know any other useful alternatives?
An alternative would be to use a socket connection. Each person connected to the socket server daemon would be able to send a message to the daemon, the daemon would then send the message out to all or a partial list of subscribers which makes chat instantaneous with no need to save the data at all.
A good way to create socket connections from a client is socket IO. See below.
http://socket.io/
A good technology to use for creating a socket server daemon is node.js. This is a server side event driven javascript based library. Very efficient for things like this. See below.
http://nodejs.org/
On both A and B you are still effectively polling. You will either poll MySQL which really isn't too bad, or you can get notified on select() of a file change BUT you will still need to parse to see if the new data is the right stuff on the file-side.
For conceptual and support ease-of-use, it is really hard to beat a database as you won't have to worry about locking semantics. Debugging and message tracking are clean in this structure.
I however recommend you investigate the msg_send() and msg_receive() (of PHP) functions to put this data into an underlying message queue. Your problem seems to be a message queueing problem that should be solved by that mechanism.
Does anyone know any other useful alternatives?
If you search simple solutions on PHP, I can offer 2 ways:
Cache
It mean that you keep MySQL for store data, but install APC (this solution is simplest and fastest for small servers and applications) or Memcached (better for using width several servers). For each read-request you check APC/Memcached for you data and ask MySQL only if your cache is removed or updated. And on each write-request you inserting data in MySQL and update cache.
Other DB
In this case you change MySQL for one of memory-base DB (for example MongoDB). And you may not afraid hard disk usage.
Why is HTTP-Polling so laggy?
What I have is a button, and whenever a user clicks it a MySQL database field gets updated and the value is displayed to the user. I'm polling every 800 milliseconds and it's very laggy/glitchy. Sometimes when clicking the button it doesn't register it. And I actually need to be polling quite a bit more frequent than every 800 milliseconds.
This is also with just 1 user on the website at a time... When in the end there is going to be many at once.
HTTP-streaming/Long-polling/Websockets instead of polling
When you need real-time information you should avoid polling(frequently). Below I would try to explain why this is wrong. You could compare it to a child in the back of your car screaming every second "are we there yet" while you are replying "we are not there yet" all the time.
Instead you would like to have something like long-polling/HTTP-streaming or websockets. You could compare this to a child in the back of your car telling you to let him know when "we are there" instead of asking us every second. You could imagine this is way more efficient then the previous example.
To be honest I don't think PHP is the right tool for this kind of applications(yet). Some options you have available are:
hosted solutions:
http://pusherapp.com:
Pusher is a hosted API for quickly,
easily and securely adding scalable
realtime functionality via WebSockets
to web and mobile apps.
Our free Sandbox plan includes up to
20 connections and 100,000 messages
per day. Simply upgrade to a paid plan
when you're ready.
http://beaconpush.com/
Beaconpush is a push service for
creating real-time web apps using
HTML5 WebSockets and Comet.
host yourself:
http://socket.io:
Socket.IO aims to make realtime apps
possible in every browser and mobile
device, blurring the differences
between the different transport
mechanisms
When becoming very big the "host yourself" solution is going to less expensive, but on the other hand using something like pusherapp will get you started easier(friendly API) and also is not that expensive. For example pusherapp's "Bootstrap" can have 100 concurrent connections and 200,000 messages per day for $19 per month(but when small beaconpush is cheaper => do the math :)). As a side-note this plan does not include SSL so can not be used for sensitive data. I guess having a dedicated machine(VPS) will cost you about the same amount of money(for a simple website) and you will also have to manage the streaming solution yourself, but when getting bigger this is probably way more attractive.
Memory instead of Disc
whenever a user clicks it a MySQL
database field gets updated and the
value is displayed to the user
When comparing disc I/O(MySQL in standard mode) to memory it is extremely slow. You should be using an in-memory database like for example redis(also has persistent snapshots) or memcached(completely in memory) to speed up the process. I myself really like redis for it's insane speed, simplicity and persistent snapshots. http://redistogo.com/ offers a free plan with 5MB of memory which will probably cover your needs. If not the mini plan of $5 a month will probably cover you, but when getting even bigger a VPS will be cheaper and in my opinion the prefered solution.
Best solution
The best solution(especially if you are getting big) is to host socket.io/redis yourself using a VPS(cost money). If really small I would use redistogo, if not I would host it myself. I would also start using something like beaconpush/pusherapp because of it's simplicity(getting started immediately). Hosting socket.io(advice to play with it on your own machine for when getting big) is pretty simple, but in my opinion more difficult than beaconpush/pusherapp.
Laggy/glitchy? Sounds like a client-side problem. As does the button thing. I'd get your JavaScript in order first.
As for polling, 0.8 sounds a bit time-critical. I don't know about most countries, but here in the third world simple network packets may get delayed for as long a few seconds. (Not to mention connection drops, packet losses and the speed of light.) Is your application ready to cope with all that?
As for an alternate approach, I agree with #Vern in that an interrupt-driven one would be much better. In HTTP terms, it translates to a long-standing HTTP request that does not receive a response until the server has some actual data to send, minimizing delay and bandwidth. (AFAIK) it's an older technique than AJAX, though has been named more recently. Search for "COMET" and you'll end up with both client- and server-side libraries.
there are many things that might cause the lag that you are experiencing. Your server might be able to process the requests fast enough, but if the connection between your client and the server is slow, then you'll see the obvious lag.
The first thing you should try is to ping the server and see what response time you're getting.
Secondly, rather than poll, you might want to consider an interrupt driven approach. This means that only when your server replies, will you send out your next request. This makes sense, so that many clients won't be flooding the server with requests till the point the server cannot cope. This is especially true, then the RTT (Round-Trip-Time) of your request is pretty long.
Hope it helps. Cheers!
A good place to start would be to use a tool like Firebug in Mozilla Firefox that will allow you to watch the requests being sent to the server and look for bottlenecks.
Firebug will break down each part of the request, so you can see if you are having trouble talking to the server or if it is simply taking a long time to come up with a response.
Along with #Vern's answer I would also say that if at all possible I would have the server cache the data ahead of time and then all of the clients will pull from that same cache and not need separate MySQL calls to reach the same data for every update. Then you just have your PHP update the cache whenever the actual DB data changes.
By cache I mean having php write to a file on the sever side, and then clients will simply look at the contents of that one file to see the most updated info. There might be better ways of caching, but being that I have never done this personally before, this is the first solution that popped into my mind.
This is THE MOST noobish question ever, I see it done so often.
I want a PHP page which is constantly runnng in the background (the backend) to occasionally query the frontend with updated data. YES, this is the way I want to do it.
But the only way I know of querying a page is to re-create that php, with XHR - so, I would XHR "index.php?data=newdata", but this would create a new process server-side. What do I do?
(Please ask for more info or correct me if there is a better way of doing it)
This is a great SO question/answer to look at:
Using comet with PHP?
The upshot is, you can do it with PHP...
Another way to do this is to create a bridge from your Apache set up and Node, if you read through the guides about Node you will see that it is:
Designed for high loads of networking
Only spawns new threads when it need's to do blocking tasks such as I/O
Extremely simple to use, based on Google V8 (Javascript Engine)
Can handle thousands of concurrent connections
With the above in mind my road plan would be to create a database for your PHP Application, and create 2 connections to that,
Connection used in the PHP Application
Connection used within Node.
The Node side of things would be simple:
Create a simple socket server (20~ lines)
Create an array
Listen for new connections, place the resource into the array.
Attach an for the database
When the event get's fired, pipe the new data to all the clients in the array.
All the clients will receive the data at pretty much the same time, this should be stable, it's extremely light weight solution as 1K Connections would use 1 process with a few I/O Threads, the ram used would be about 8~MB
Your first step's would be to set up node.js on your server, if you google around you will be able to find how to do that, a simple way under ubuntu is to do:
apt-get install nodejs
you should read the following resources:
http://nodejs.org/
http://www.youtube.com/watch?v=jo_B4LTHi3I
http://jeffkreeftmeijer.com/2010/experimenting-with-node-js/
http://www.slideshare.net/kompozer/nodejs-and-websockets-intro
http://remysharp.com/2010/02/14/slicehost-nodejs-websockets/
for more technical assistance you should connect to the #node.js irc server on freenode.net, those guys will really help you out over there!
Hope this helps.
COMET may be a way to go; however, having a finalized HTML page, and doing AJAX requests to get updates is the usual, and more robust way to do this. I would investigate ways to implement an Ajax based approached that is optimized for speed.
You say you are doing complex calculations, which you would have to repeat for each request when going the Ajax way. You may be able to help that e.g. by employing smart caching. If you write the results of whatever you do as JSON encoded data into a text file, you can fetch that with almost no overhead at all.
Any idea how to implement this (http://fluin.com/63) using MySQL+PHP+Javascript(mootools)?
In a nutshell, it's a realtime threaded conversational web app.
Update:
This uses http://www.ape-project.org/home.html
Any idea how to implement realtime stuff without AJAX push (ape)?
Install Firefox.
Install Web Development toolbar
Install Firebug
Install HttpFox
Read docs of above tools re how to use, what they can do.
Go to http://fluin.com/63. Use above tools to inspect.
Read up on Databases and data models, and MySQL.
Build your own.
Well, this depends on your definition of realtime, which, in its technical meaning, is simply impossible with public ip networks and traditional tcp stack, for you have no control over timing.
Closer to the topic though, to get any web page updated without direct user intervention, you'd have to use javascript to poll server for changes since the last successful poll, and do this over certain intervals of time. In calculating these intervals you'll have to consider both network/server load, and the delay that is comfortable for the user.
The server, of course, will have to store the new data and its timely status (creation timestamps are one way of doing it), to be able to distinguish between content already delivered to various clients.
As soon as the server reports new content, it is inserted into a dom page via javascript and the user sees the response.
This is a bit general, of course, but you should get the idea.
Isn't it like a shoutbox ? here an example of one
Doing this properly using PHP only is very hard. When you have 5 users you could use long-polling, but it will definitely not scale when you have let's say 1000 users.
Using comet with PHP?
The screencast(link) in my post shows how you could implement it, but it has a couple of flaws:
It touches the disc(disc is very slow compared to memory).
To make matters worse it also polls the disc frequently(filemtime()).
Maybe phet(PHP) is able to scale. You should try that out.
To make it scale I think you need at least:
a good implementation of long-polling(at least long-polling. You have better transports) that can handle load.
keep data in memory(much faster than dics) using something like redis or memcached.
I would use:
node.js with socket.io(video) module.
to keep data in memory I would use node_redis(video).