I have this "crazy" project starting, the idea behind it is quite clear:
There is some software that writes to MySQL database. The interval between queries are 1 second.
Now I need and web interface which loads those database records, and continues to show new records, when they happen.
The technologies I am supposed to use are PHP and HTML5 WebSockets. I'v found this nice library Ratchet which I think fits my needs, however there's one problem, I am not sure how to notify PHP script, or send a message to running PHP WebSockets server when the MySQL query occurs.
Now I could use Comet and send request for database record every second, but then it beats the WebSokets which I am supposed to use.
So what I really need is MySQL Pub/Sub system.
I'v read of MySQL triggers but I see that it possess some security risks, and thought the security in this case isn't a real concern since the system will be isolated in a VPN and only few specific people will be using it, I still would like to address every possible issue and do everything in a right way.
Then there is MySQL proxy, of which I have no knowledge, but if it could help me achieve my goal, I would very much consider using it.
So in short the question is how can I notify or run PHP script when MySQL query occurs?
I would separate the issues a bit.
You definitely need some sort of pub/sub system. It's my understanding that Redis can act as one, but I've never used it for this and can't make a specific recommendation as to what system to use. In any case, I wouldn't attach it directly to your database. There are certainly database operations you need to do on your database for maintenance purposes and you don't want it flushing out a million rows to your clients because it's based on a trigger. Pick your pub/sub system independent of what you're doing client-side and what you're doing with your database. The deciding factors should be how it interacts with your server-side languages (PHP in this case).
Now that your pub/sub is out of the way, I would build an API server or ingest system of sorts that takes data in from wherever these transactions are coming from. It will handle these and publish messages as well as inserting them into the database at the same time.
Next, you need a way to get that to clients. Web Sockets are a good choice, as you have discovered. You can do this in PHP or anything really. I prefer Node.js with Socket.IO which provides a nice fallback to long-polling JSON (among others) for clients that don't suppot Web Sockets. Whatever you choose here needs to listen for messages on your pub/sub and send the right data for the client (likely stripping out some of the information that was published that isn't immediately needed client-side).
Related
I have added a chat capability to a site using jquery and PHP and it seems to generally work well, but I am worried about scalability. I wonder if anyone has some advice. The key area for me I think is efficiently managing awareness of who is onine.
detail:
I haven't implemented long-polling (yet) and I'm worried about the raw number of long-running processes in PHP (Apache) getting out of control.
My code runs a periodic jquery ajax poll (4secs), that first updates the db to say I am active and sets a timestamp.
Then there is a routine that checks the timestamp for all active users and sets those outside (10mins) to inactive.
This is fairly normal from my research so far. However, I am concenred that if I allow every active user to check every other active user and then everyone update the db to kick off inactive users, then I will get duplicated effort, record locks and unnecessary server load.
So I have implemented an idea of the role of a 'sweeper'. This is just one of the online users, who inherits the role of the person doing the cleanup. Everyone else just checks whether there is a 'sweeper' in existence (DB read) and carries on. If there is no sweeper when they check, they make themselves sweeper (DB write for their own record). If there are more than one, make yourself 'non-sweeper', sleep for a random period and check again.
My theory is that this way there is only one user regularly writing updates to several records on the relevant table and everyone else is either reading or just writing to their own record.
So it works OK, but the problem possibly is that the process requires a few DB reads and may actually be less efficient than just letting everyone do the cleanup as with other research as I mentioned.
I have had over 100 concurrent users running OK so far, but the client wants to scale up to several 100's, even over 1,000 and I have no idea of knowing at this stage whether this idea is good or not.
Does anyone know whether this is a good approach or not, whether it is scalable to hundreds of active users, or whether you can recommend a different approach?
AS an aside, long polling / comet for the actual chat messages seems simple and I have found a good resource for the code, but there are several blog comments that suggest it's dangerous with PHP and apache specifically. active threads etc. Impact minimsed with usleep and session_write_close.
Again does anyone have any practical experience of a PHP long polling set up for hundreds of active users, maybe you can put my mind at ease ! Do I really ahve to look to migrate this to node.js (no experience) ?
Thank you in advance
Tony
My advice would be to do this with meteor framework, which should be pretty trivial to do, even if you are not an expert, and then simply load such chat into your PHP website via iframe.
It will be scalable, won't consume much resources, and it will get only better in the future, I presume.
And it sure beats both PHP comet solutions and jquery & ajax timeout based calls to server.
I even believe you could find on github more or less a completed solution that just requires tweaking.
But of course, do read the docs before you implement it.
If you worry about security issues, read security with meteor
Long polling is indeed pretty disastrous for PHP. PHP is always runs with limited concurrent processes, and it will scale great as long as you optimize for handling each request as quickly as possible.
Long polling and similar solutions will quickly fill up your pipe.
It could be argued that PHP is simply not the right technology for this type of stuff, with the current tools out there. If you insist on using PHP you could try ReactPHP, which is a framework for PHP quite similar to how NodeJS is built. The implication with React is also that it's expected to run as a separate deamon, and not within a webserver such as apache. I have no experience on the stability of this, and how well it scales, so you will have to do the testing yourself.
NodeJS is not hard to get into, if you know javascript well. NodeJS + socket.io make it really easy to write the chat-server and client with websockets. This would be my recommendations. When I started with this is, I had something nice up and running within several hours.
If you want to keep your application stack using PHP, you want the chat application running in your actual web app (not an iframe) and your concerned about scaling your realtime infrastructure then I'd recommend you look at a hosted service for the realtime updates, such as Pusher who I work for. This way the hosted service handles the scaling of the realtime infrastructure for you and lets you concentrate on building your application functionality.
This way you only need to handle the chat message requests - sanitize/verify the content - and then push the information through Pusher to the 1000's of connected clients.
The quick start guide is available here:
http://pusher.com/docs/quickstart
I've a full list of hosted services on my realtime web tech guide.
I'm working on feeding the client realtime events using event-streams and HTML5 SSEs client-side.
But some of my events will actually come from form submissions by other clients.
What's the best method for detecting these form submissions (so as to append them to the event-stream script) ASAP (after they occur)?
So essentially, I need realtime cross-script messaging between multiple instances of different scripts instantiated by different clients, analagous to X-doc messaging in JS, but for PHP.
The best I can come up with is to repeatedly poll a subdir of /tmp for notification files, which is a terrible solution.
Often you can use MYSQL to play the role of the tmp dir you were talking about. This is more portable because they don't have to be on the same server to do this and the data is separate. However the scripts will have to manually check the mysql location to see if the other one has taken care of this. The other option is to open sockets and write back and forth in real time or to use some prebuilt tool for just this purpose which I'm pretty sure might exist.
If you want the events to be triggered near to realtime, then you need to handle them synchronously - which means running a daemon. And the simplest way to implement a daemon which can synchronize data across client connections is to use an event based server. There's a nice implementation of the latter using php here - there are plenty of examples of how to daemonize a PHP process on the interent. Then just open a blocking connection to the server from your PHP code / access this via comet.
I have been in search of making live websites by using PHP. (COMET) I have been searching for a very long time already. (around a month) I have even checked some PHP chat scripts and used on my webserver, but I had some problems on most of them. (will be explained)
So far, most of the people tells the same thing: PHP isn't the best language could be used with COMET. The chat/comet examples are hacky at best.
I am asking this because, I want to have some features on my websites, like allow logged in people to chat with each other. That is where I need an alive connection to PHP pages. I am also planning on making a browser based game, and an alive connection will still be a must!
AJAX was a rare thing 2 years ago, shined with Facebook. Now pretty much everyone uses it, and it became a standard on web development. Now, the COMET based websites are growing. Youtube, Google+, Facebook, EA's Battlelog and such. I believe I should learn how to use it.
Okay, here are my questions. (Some of the information at below are the ones I gathered while searching on Google. Not sure if they're correct or not.)
Some languages like Python have special web servers designed for this job. I believe one of them is called Tornado Web Server. Developed and configured to simulate thousands of alive connections. I believe there is no such option in Appserv, but some people told NGINX can handle it with a decent configuration. Is it true? What configurations should be made? Is there any PHP web servers for this job?
Some of the most suggested things are:
a. Using APE.
b. Using Socket.IO
c. Node.js
Is there any way we can use them with PHP? Which one is the most promising? Could you give some guidance on them? Is there anything else than these?
I have used a comet chat script. Basically, you kept querying database and output the result with flush() to browser by sleeping it with usleep() in a loop. (This one became hard to understand so I will provide a quick code)
while(true)
{
// query database
// output the result
// flush the browser
// sleep for few seconds to lower cpu load
}
usleep() function basically destroyed my web server on Windows based operating systems. Is it normal to rely on usleep() on comet applications which runs on windows based OS? I mean, is there even a way to "sleep" PHP scripts? No matter what I do, CPU load goes to %100 on both WIN and UNIX servers.
Is PHP "really" that weak on this area? Should I give up with PHP and focus on other languages? If so, which language would you suggest? (That language should be promising. For example, there is no much use of AS3 after HTML5 addition, and AS3 is more likely to die soon while JS shines.)
What is WebSync? Can it be used with PHP?
Please bear in mind that I need to use COMET to make following applications:
A chat script, where logged in players will be able to chat eachother.
A browser based game. I already use JSON/AJAX and things like that when coding, but to receive opponents steps, I should pull the data, so an alive connection is needed. (Don't imagine advanced games, I am speaking about games like chess at best.)
I would appreciate if you can give me a short guidance. After all, I have been getting confused day by day since there are SO MANY random articles on internet. People even call setInterval() as COMET, so it is a complete mess.
There needs to be some clarification here. What you're looking for is not language specific per se. If you wanted to achieve the comet functionality with PHP you'd be relying on the Web Server (e.g Apache) to do the HTTP streaming. Also you need to understand that this is not supported in HTTP versions prior to HTTP/1.1. With that said, if you want to have a non-blocking event based web server, you're better off looking at Tornado and Node as suggested.
Comet is a programming technique that enables web servers to send data to the client without having any need for the client to request it This technique will produce more responsive applications than classic AJAX The user must create a request (for example by clicking on a link) or a periodic AJAX request must happen in order to get new data fro the server.
but it's create lots of traffic on your web server. If you want to build chat application in PHP use pusher which is a third party service and easy to use.
here is a link for pusher https://pusher.com/tutorials/realtime_chat_widget
the second suggestion is use ratchet for creating a chat application.
here is link for ratchet http://socketo.me/docs/hello-world
i hope it will help you
I want to create a live, checkers-like app, which will work like this: There will be multiple icons/avatars displayed on this checkerboard like surface. I want to have a command prompt beneath this board, or some other sort of interface, that will allow them to control a certain avatar, and get it to preform actions. Multiple users will be using it at one time, and I will all be able to view the other user's changes/actions to the checkerboard.
What I'm wondering is: what's the best way to do this? I've got my HTML, CSS, and JS approach down, but not my data storage method. I know that, using PHP, I've got the choices to use either: file-based storage, MYSQL, or some other method. I need to know which is better, because I don't want to have server-lag, poor-response time, or some other issue, especially in this case since actions will be preformed every other second 2 or so, by these multiple users.
I've done similar stuff before, but I'm wanting to hear how others would handle it (advice, etc.) from more experienced programmers.
Sounds like a great project for node.js!
To clarify, node.js is a server-side implementation of javascript. What you'll want is a comet based application (a web-based client application that receives server side pushes instead of the client constantly polling the server), which is exactly what node.js is good at.
Traditional ajax calls for your clients to poll the server for data. This creates enormous overhead for both the client and the server. Allowing the server to push requests directly to the client without the client repeatedly asking solves the overhead issue and creates a more responsive interface. This is accomplished by holding asynchronous client connections on the server and only returning when the server has something to respond with. Once the server responds with data, another connection is immediately created and held by the server again until data is ready to be sent.
You may be able to accomplish the same thing with PHP, but I'm not that familiar with PHP and Comet type applications.
Number of users and hosting costs will play into your file vs DB options. If you're planning on more than a couple of users, I'd stick to the database. There are some NoSQL options available out there, but in my experience MySQL is much faster and more reliable than those options.
Good luck with your project!
http://en.wikipedia.org/wiki/Comet_%28programming%29
http://www.nodejs.org/
http://zenmachine.wordpress.com/2010/01/31/node-js-and-comet/
http://socket.io/ - abstracts away the communication layer with your clients based on their capability (LongPolling, WebSockets, etc.)
MySQL and XCache !!!!
Make sure you use predefined statements so MySQL does not need to compile the SQL again. Also memtables could be used to use memory storage
Of course make use of indexes appropriately.
If the 'gamestate' is not that important you can even store everything in XCache.
Remember that XCache does not store data persistently (after Apache restart)
I am developing a non-real time browser RPG game (think Kingdom of Loathing) which would be played from within a Flash app. At first I just wanted to make the communication with server using simply URLLoader to tell PHP what I am doing, and using $_SESSION to store data needed in-between request.
I wonder if it wouldn't be better to base it on a socket connection, an app residing on a server written in Java or Python. The problem is I have never ever written such an app so I have no idea how much I'd have to "shift" my thoughts from simple responding do request (like PHP) to continuously working application. I won't hide I am also concerned about the memory and CPU usage of such Server app, when for example there would be hundreds of users connected. I've done some research.
I have tried to do some research, but thanks to my nil knowledge on the sockets subject I haven't found anything helpful. So, considering the fact I don't need real time data exchange, will it be wise to develop the server side part as socket server, not in plain ol' PHP?
Since your game isn't something that's working in realtime you probably don't need to go down the socket route, though it's certainly a viable option. The nice thing about sockets is that updates would be instant without requiring page refresh (or server poll), so you're right to at least consider it.
If you do want to do a more real-time server setup, you might consider using something like Electroserver - this abstracts out much of the setup for you so you don't have to write your own server from scratch, plus it's free up to a certain number of concurrent users if I recall correctly.
Finally, a third option you have is a modified POST approach using AMF. Look into AMFPHP, it lets you call methods on a PHP back-end directly from your flash application. A little bit faster and easier than simply using POST stuff, but not quite as seamless as a socket connection or a specifically built gaming server.
Lots of options out there, it sounds like you are aware of this and kudos for trying to come up with the best approach rather than just rolling with what you know! I hope this helps, let me know if you have any questions.
Here's a link to Electroserver - http://www.electro-server.com/