I have a PHP system, that does everything a social media platform does, i.e. add comments, upload images, add objects, logins, sessions etc. Storing all interactions in a MySQL database. So i've got a pretty good infrastructure to build on.
The next stage of my project is to develop the system so that notifications are sent to the "Networks" of "Contacts", which are associated with one and other. Such as the notifications system like Facebook. i.e. Chris has just commented on object N.
I'm looking at implementing this system for a lot of users: 10,000+, so it has to be reliable. I've researched the Facebook integration, resulting in techniques such as memcache, sockets & hashing.
Are they're any systems that can be easily adapted to this functionality, as I could do with a quick, reliable implementation.
P.s one thought I had was just querying the database every 5 seconds for e.g. "Select everything that has happened in the last 5 seconds" using jQuery, Ajax & PHP, but thats stupid, it would exhaust the server & database right?
I've seen this website & this article, can anyone reflect on this to tell me what is the best approach as I am hesitant about which path to follow.
Thanks
This is not possible with just pure vanilla PHP/MySQL. What you can do is set MySQL triggers (http://dev.mysql.com/doc/refman/5.0/en/triggers.html) on your data, and then use the UDF function sys_exec (Which you will need to install on your MySQL server) to run the notification php script. See this post: Invoking a PHP script from a MySQL trigger
If you can get this set up it should be pretty reliable and fast.
Related
I am struggling with the logistics of getting a multi-device synchonisation to work.
Lets say I have an off-line web page (via app-cache) used as an address book - how would I go about structuring a robust multi-device synchronisation system.
I was thinking about the following but it doesn't seem robust:
Each device has 2 tables:
'synchronised (copy of server table at last update time) - along with the servers time-stamp at the last update point.
'awaiting synchronisation' - changes on that device waiting to be synchronised with the server. - this could have insert, delete and update items.
Then when the device connects to the server it:-
Uploads all changes to the server (via a mini API) and delete them from the 'awaiting synchronisation' table.
Sends the server the last synchonisation time.
Then download all updates from that time onwards and add them to the 'synchronised' table on the device.
Update the synchronization time. (using server time to ensure consistency across devices)
Would that work or would it fall over?
Like I said never tried a multi-device synchronisation and seems to be a hard thing to find topics on.
Finally are there any frameworks designed for this?
Anyone who has done similar with an App maybe as off-line web pages are rare - just need the logical order and any 'gotchyas' I need to be aware of.
Thanks in advance.
It can be tough to do that, in the end I realised that what I needed to do is create a version control system, similar to GIT... in JavaScript. Once you have the versions you can make reasoned decisions based on a change version number and the change that it was based upon.
I don't know if the project I started will be useful for you, it uses these concepts, but even if it is not I feel you should read through the documentation as it will help you think about how to tackle the wider problem. The project is stalled right now as I am looking for work (which is a lot of work) but I will be picking it up again soon.
Matt Forrester
I am a programmer at an internet marketing company that primaraly makes tools. These tools have certian requirements:
They run in a browser and must work in all of them.
The user either uploads something (.csv) to process or they provide a URL and API calls are made to retrieve information about it.
They are moving around THOUSANDS of lines of data (think large databases). These tools literally run for hours, usually over night.
The user must be able to watch live as their information is processed and is presented to them.
Currently we are writing in PHP, MySQL and Ajax.
My question is how do I process LARGE quantities of data and provide a user experience as the tool is running. Currently I use a custom queue system that sends ajax calls and inserts rows into tables or data into divs.
This method is a huge pain in the ass and couldnt possibly be the correct method. Should I be using a templating system or is there a better way to refresh chunks of the page with A LOT of data. And I really mean a lot of data because we come close to maxing out PHP memory and is something we are always on the look for.
Also I would love to make it so these tools could run on the server by themselves. I mean upload a .csv and close the browser window and then have an email sent to the user when the tool is done.
Does anyone have any methods (programming standards) for me that are better than using .ajax calls? Thank you.
I wanted to update with some notes incase anyone has the same question. I am looking into the following to see which is the best solution:
SlickGrid / DataTables
GearMan
Web Socket
Ratchet
Node.js
These are in no particular order and the one I choose will be based on what works for my issue and what can be used by the rest of my department. I will update when I pick the golden framework.
First of all, you cannot handle big data via Ajax. To make users able to watch the processes live you can do this using web sockets. As you are experienced in PHP, I can suggest you Ratchet which is quite new.
On the other hand, to make calculations and store big data I would use NoSQL instead of MySQL
Since you're kind of pinched for time already, migrating to Node.js may not be time sensitive. It'll also help with the question of notifying users of when the results are ready as it can do browser notification push without polling. As it makes use of Javascript you might find some of your client-side code is reusable.
I think you can run what you need in the background with some kind of Queue manager. I use something similar with CakePHP and it lets me run time intensive processes in the background asynchronously, so the browser does not need to be open.
Another plus side for this is that it's scalable, as it's easy to increase the number of queue workers running.
Basically with PHP, you just need a cron job that runs every once in a while that starts a worker that checks a Queue database for pending tasks. If none are found it keeps running in a loop until one shows up.
I'm building a chat widget using pusher service (http://pusher.com) and I need to save all the messages sent by users into database, so that it should be accessible after a period of time. I'm using mysql database and the only way that comes to my mind is that make a new insert each time the chat message event is triggered, but I'm afraid it will not be as fast as it should be.
What databases and techniques would you prefer in this case for saving chat messages?
One approach you can take is to use MySQL's INSERT DELAYED. When you do this, the client does not have to wait for the insert to complete. Rather, the server will queue the inserts and execute them as it can. So the process/thread thats handling the chat will not have to wait on them. But you'll still have a chat history stored in the database that can be retrieved. Assuming you are using pusher to share 'live' messages as they come in, and thus dont need immediate access to the chat history, this might do the trick for you.
Unless you're planning on creating a chat system including thousands of people I'm fairly sure that a properly structured mysql solution will handle this no problem. Where I work we have a intranet chat solution based on php and mysql and it is running flawlessly (for our medium sized team of ~100 some people).
One suggestion I do have is to make sure you understand mysql indexing. Make sure your chat system is truly utilizing the indexes to their fullest potential. You could greatly increase performance this way.
If concurrency really is becoming an issue for you I hear a lot of people are having good experiences with node.js. But if you're comfortable in php and mysql I'd say go that way first.
Happy coding!
(I know this is old but,) There is an interesting demo here (albeit in Rails) that does what you're talking about. It includes a possible schema.
https://github.com/dipth/pusher_demo
I have a WordPress plugin, which checks for an updated version of itself every hour with my website. On my website, I have a script running which listens for such update requests and responds with data.
What I want to implement is some basic analytics for this script, which can give me information like no of requests per day, no of unique requests per day/week/month etc.
What is the best way to go about this?
Use some existing analytics script which can do the job for me
Log this information in a file on the server and process that file on my computer to get the information out
Log this information in a database on the server and use queries to fetch the information
Also there will be about 4000 to 5000 requests every hour, so whatever approach I take should not be too heavy on the server.
I know this is a very open ended question, but I couldn't find anything useful that can get me started in a particular direction.
Wow. I'm surprised this doesn't have any answers yet. Anyways, here goes:
1. Using an existing script / framework
Obviously, Google analytics won't work for you since it is javascript based. I'm sure there exists PHP analytical frameworks out there. Whether you use them or not is really a matter of your personal choice. Do these existing frameworks record everything you need? If not, do they lend themselves to be easily modified? You could use a good existing framework and choose not to reinvent the wheel. Personally, I would write my own just for the learning experience.
I don't know any such frameworks off the top of my head because I've never needed one. I could do a Google search and paste the first few results here, but then so could you.
2. Log in a file or MySQL
There is absolutely NO GOOD REASON to log to a file. You'd first log it to a file. Then write a script to parse this file.Tomorrow you decide you want to capture some additional information. You now need to modify your parsing script. This will get messy. What I'm getting at is - you do not need to use a file as an intermediate store before the database. 4-5k write requests an hour (I don't think there will be a lot of read requests apart from when you query the DB) is a breeze for MySQL. Furthermore, since this DB won't be used to serve up data to users, you don't care if it is slightly un-optimized. As I see it, you're the only one who'll be querying the database.
EDIT:
When you talked about using a file, I assumed you meant to use it as a temporary store only until you process the file and transfer the contents to a DB. If you did not mean that, and instead meant to store the information permanently in files - that would be a nightmare. Imagine trying to query for certain information that is scattered across files. Not only would you have to write a script that can parse the files, you'd have to right a non-trivial script that can query them without loading all the contents into memory. That would get nasty very, very fast and tremendously impair your abilities to spot trends in data etc.
Once again - 4-5K might seem like a lot of requests, but a well optimized DB can handle it. Querying a reasonably optimized DB will be magnitudes upon magnitudes of orders faster than parsing and querying numerous files.
I would recommend to use an existing script or framework. It is always a good idea to use a specialized tool in which people invested a lot of time and ideas. Since you are using a php Piwik seems to be one way to go. From the webpage:
Piwik is a downloadable, Free/Libre (GPLv3 licensed) real time web analytics software program. It provides you with detailed reports on your website visitors: the search engines and keywords they used, the language they speak, your popular pages…
Piwik provides a Tracking API and you can track custom Variables. The DB schema seems highly optimized, have a look on their testimonials page.
I'm working on a chat application which I would love to use a SQL db for.
My problem is, after a few google searches, i have people telling me from one site, that using a DB would be much slower then using a normal file (e.g Text or JSON file), but then on some other sites, people are saying the complete opposite. And I don't know about you guys, but when it comes to creating web apps for users, the users always come first.
So as much as I'd love to use a SQL DB as 1.) I have good experience with it and 2.) it allows me to make the application much more cooler (more features). but if it would slow things down on the users end (a noticeable lag), then its a no-no.
Either way, I will be "polling" the server continuously with AJAX and PHP to check the file/DB (for new messages, contact requests, ect ect).
Also, incase your wondering, the application wont be like a 1-to-1 chat, it will have "rooms" where multiple users can join and talk with all users joining in. The users will also be able to request a "private chat" with another user, where a 1-to-1 connection opens up.
So, MySQL Database OR a boring TEXT/JSON/OTHER file, in regards to performance?
Oh, one more thing, I don't want to use any third party libraries or APIs. Hate relying on other peoples work (been let down to many times).
If you're looking to implement an IRC clone, I think you've chosen all the wrong tools.
The best way to do this would be to write a custom HTTP server that handles everything in memory. No databases, no constant polling of files. When a message arrives, you simply loop through the correct in-memory list and dispatch the message to other users. For the browser to server connection, I suggest "Comet" (with web sockets for browsers that support them, if you're feeling up to it).
PHP likely isn't the language of choice for this, because pretty much all work done with PHP is based on traditional short, isolated requests. For a long-running process which serves multiple clients in real time, I'd suggest something like Python or Node.js.
You don't really want to be storing chats in files, that can create a management nightmare, I would recommend you go with MySQL and to make sure it works probably go with Sockets instead of AJAX polling, Sockets will scale really well.
However there isn't much around about how you can integrate socket based chats with MySQL.
I have done a few tests and have a basic example working here: https://github.com/andrefigueira/PHP-MySQL-Sockets-Chat
It makes use of Ratchet (http://socketo.me/) for the creation of the chat server in PHP.
And you can send chat messages to the DB by sending the server JSON with the information of who is chatting, (if of course you have user sessions)