I'm building a chat widget using pusher service (http://pusher.com) and I need to save all the messages sent by users into database, so that it should be accessible after a period of time. I'm using mysql database and the only way that comes to my mind is that make a new insert each time the chat message event is triggered, but I'm afraid it will not be as fast as it should be.
What databases and techniques would you prefer in this case for saving chat messages?
One approach you can take is to use MySQL's INSERT DELAYED. When you do this, the client does not have to wait for the insert to complete. Rather, the server will queue the inserts and execute them as it can. So the process/thread thats handling the chat will not have to wait on them. But you'll still have a chat history stored in the database that can be retrieved. Assuming you are using pusher to share 'live' messages as they come in, and thus dont need immediate access to the chat history, this might do the trick for you.
Unless you're planning on creating a chat system including thousands of people I'm fairly sure that a properly structured mysql solution will handle this no problem. Where I work we have a intranet chat solution based on php and mysql and it is running flawlessly (for our medium sized team of ~100 some people).
One suggestion I do have is to make sure you understand mysql indexing. Make sure your chat system is truly utilizing the indexes to their fullest potential. You could greatly increase performance this way.
If concurrency really is becoming an issue for you I hear a lot of people are having good experiences with node.js. But if you're comfortable in php and mysql I'd say go that way first.
Happy coding!
(I know this is old but,) There is an interesting demo here (albeit in Rails) that does what you're talking about. It includes a possible schema.
https://github.com/dipth/pusher_demo
Related
Right now I'm using long-polling for my web chat app. The reason why I don't use sockets is because of legacy browser compatibility issues. I'm also refraining from NodeJS because my server don't allow long processes running (it's a shared server). However, I want to improve my chat response time without continuously sending AJAX signals to the database.
I thought maybe if we can make the mysql database trigger a response to the server, that would be great. Is it possible, or nah? Why/How? Thanks!
Unfortunately, not. The database is passive/reactive. It can perform actions, such as sending emails, but those are exceptional cases and not the norm.
If you want to make something happen when a specific thing is added to the database then the best place to add it is to the data layer which is inserting the data into the database.
Databases are not involved directly with long pulls, since they don't intercept the web request. Try moving your logic to the middle tier and having a caching mechanism there to interact with your clients.
Are you really seeing any performance problems that would warrant you wanting to change the polling mechanism? The database isn't able to contact a client, its job is to be a data store.
IMHO, applications not suffering from performance problems polling is a great approach. Anything else you are going to try is most likely going to add an extra layer to the application that you probably don't want to have to manage now.
Sorry for the "answer" but I gotta build the points to be able to comment :)
I have a PHP system, that does everything a social media platform does, i.e. add comments, upload images, add objects, logins, sessions etc. Storing all interactions in a MySQL database. So i've got a pretty good infrastructure to build on.
The next stage of my project is to develop the system so that notifications are sent to the "Networks" of "Contacts", which are associated with one and other. Such as the notifications system like Facebook. i.e. Chris has just commented on object N.
I'm looking at implementing this system for a lot of users: 10,000+, so it has to be reliable. I've researched the Facebook integration, resulting in techniques such as memcache, sockets & hashing.
Are they're any systems that can be easily adapted to this functionality, as I could do with a quick, reliable implementation.
P.s one thought I had was just querying the database every 5 seconds for e.g. "Select everything that has happened in the last 5 seconds" using jQuery, Ajax & PHP, but thats stupid, it would exhaust the server & database right?
I've seen this website & this article, can anyone reflect on this to tell me what is the best approach as I am hesitant about which path to follow.
Thanks
This is not possible with just pure vanilla PHP/MySQL. What you can do is set MySQL triggers (http://dev.mysql.com/doc/refman/5.0/en/triggers.html) on your data, and then use the UDF function sys_exec (Which you will need to install on your MySQL server) to run the notification php script. See this post: Invoking a PHP script from a MySQL trigger
If you can get this set up it should be pretty reliable and fast.
I am a programmer at an internet marketing company that primaraly makes tools. These tools have certian requirements:
They run in a browser and must work in all of them.
The user either uploads something (.csv) to process or they provide a URL and API calls are made to retrieve information about it.
They are moving around THOUSANDS of lines of data (think large databases). These tools literally run for hours, usually over night.
The user must be able to watch live as their information is processed and is presented to them.
Currently we are writing in PHP, MySQL and Ajax.
My question is how do I process LARGE quantities of data and provide a user experience as the tool is running. Currently I use a custom queue system that sends ajax calls and inserts rows into tables or data into divs.
This method is a huge pain in the ass and couldnt possibly be the correct method. Should I be using a templating system or is there a better way to refresh chunks of the page with A LOT of data. And I really mean a lot of data because we come close to maxing out PHP memory and is something we are always on the look for.
Also I would love to make it so these tools could run on the server by themselves. I mean upload a .csv and close the browser window and then have an email sent to the user when the tool is done.
Does anyone have any methods (programming standards) for me that are better than using .ajax calls? Thank you.
I wanted to update with some notes incase anyone has the same question. I am looking into the following to see which is the best solution:
SlickGrid / DataTables
GearMan
Web Socket
Ratchet
Node.js
These are in no particular order and the one I choose will be based on what works for my issue and what can be used by the rest of my department. I will update when I pick the golden framework.
First of all, you cannot handle big data via Ajax. To make users able to watch the processes live you can do this using web sockets. As you are experienced in PHP, I can suggest you Ratchet which is quite new.
On the other hand, to make calculations and store big data I would use NoSQL instead of MySQL
Since you're kind of pinched for time already, migrating to Node.js may not be time sensitive. It'll also help with the question of notifying users of when the results are ready as it can do browser notification push without polling. As it makes use of Javascript you might find some of your client-side code is reusable.
I think you can run what you need in the background with some kind of Queue manager. I use something similar with CakePHP and it lets me run time intensive processes in the background asynchronously, so the browser does not need to be open.
Another plus side for this is that it's scalable, as it's easy to increase the number of queue workers running.
Basically with PHP, you just need a cron job that runs every once in a while that starts a worker that checks a Queue database for pending tasks. If none are found it keeps running in a loop until one shows up.
I'm working on a chat application which I would love to use a SQL db for.
My problem is, after a few google searches, i have people telling me from one site, that using a DB would be much slower then using a normal file (e.g Text or JSON file), but then on some other sites, people are saying the complete opposite. And I don't know about you guys, but when it comes to creating web apps for users, the users always come first.
So as much as I'd love to use a SQL DB as 1.) I have good experience with it and 2.) it allows me to make the application much more cooler (more features). but if it would slow things down on the users end (a noticeable lag), then its a no-no.
Either way, I will be "polling" the server continuously with AJAX and PHP to check the file/DB (for new messages, contact requests, ect ect).
Also, incase your wondering, the application wont be like a 1-to-1 chat, it will have "rooms" where multiple users can join and talk with all users joining in. The users will also be able to request a "private chat" with another user, where a 1-to-1 connection opens up.
So, MySQL Database OR a boring TEXT/JSON/OTHER file, in regards to performance?
Oh, one more thing, I don't want to use any third party libraries or APIs. Hate relying on other peoples work (been let down to many times).
If you're looking to implement an IRC clone, I think you've chosen all the wrong tools.
The best way to do this would be to write a custom HTTP server that handles everything in memory. No databases, no constant polling of files. When a message arrives, you simply loop through the correct in-memory list and dispatch the message to other users. For the browser to server connection, I suggest "Comet" (with web sockets for browsers that support them, if you're feeling up to it).
PHP likely isn't the language of choice for this, because pretty much all work done with PHP is based on traditional short, isolated requests. For a long-running process which serves multiple clients in real time, I'd suggest something like Python or Node.js.
You don't really want to be storing chats in files, that can create a management nightmare, I would recommend you go with MySQL and to make sure it works probably go with Sockets instead of AJAX polling, Sockets will scale really well.
However there isn't much around about how you can integrate socket based chats with MySQL.
I have done a few tests and have a basic example working here: https://github.com/andrefigueira/PHP-MySQL-Sockets-Chat
It makes use of Ratchet (http://socketo.me/) for the creation of the chat server in PHP.
And you can send chat messages to the DB by sending the server JSON with the information of who is chatting, (if of course you have user sessions)
I have seen many questions similar to the one I asked.
A few of them said that its ok to poll, and that doesn't affect the database performance significantly, and some are still unanswered. What I want to know is this :
Can I do something similar to ASP.NET's sqldependency, which helps to find out when changes are made to the database WITHOUT polling.
I know polling is ok to do, might not affect database performance, but I need to know the update as soon as it is made, so I feel I need something better than just polling every 1 second.
As an add on point, the updates that are detected at the server then have to be pushed (Ajax Push - I'm using Ajax Push Engine) to all the clients who are subscribed to a particular channel.
Thanks & Regards,
Thothathri Srinivasan
Redis PubSub
I hoped triggers could help you out, but after a little examination I (highly) have my doubts.
You could achieve your goal using Redis pubsub. Redis is an extremely fast data structure server. Publish the database changes to a channel and let the interested clients detect the changes by subscribing to that same channel(in real-time).
Downloading/Installing Redis is really easy and as extra bonus you will also "get an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets."
Installing it is probably the best way, but if you can't, you could use the free http://redistogo.com plan to talk the Redis server with the limitation that you will only get 5MB to store your data and that it will not be persisted to disc.
You can use a server push with an implementation like websockets or comet.
I like Alfred's pubsub idea. I would think about the database as a subscriber to the insert events, not as the canonical version. Maybe this isn't possible with your existing codebase. You might also be able to monitor the binary logs on the database server to watch changes come in and push this information to the subscribers.
I think Q4M is exactly what you want
http://q4m.github.com/tutorial.html