I am building an social media application where a user is required to post something and this posted content is then propogated to all the members within his/her 4 circles. Meaning the query goes into a loop. Its like a family tree. The logic works perfectly fine. But now when number of members in each circle keep growing, the the exceution is exceeding the max execution time which is currently set to 90 and which is fairly good enough.
We dont want to increase the time limit as this is not a permanently solution. So, we have though about implmenting this using bath processing concept. Like a user posts something on the web, and then the id and text is handed over to the batch script. The response is generated imediatly to the web user and the batch script continues to work behind the scene.
Any idea or though on how this can be implemented. Thanks in advance.
I suppose you're using a Relational DB for this (like mysql). In my experience, while it can work, it is surely not the best tool to model social interactions because it is not scalable/efficient.
You should probably explore a NoSQL database with graph support for this kind of problem, like OrientDB http://orientdb.com/orientdb/ .
Not exactly what you suggested but I think it will save you from many headache in the end.
What you are looking for is named a Message queuing service.
Basicly, in your app you send a message saying for exemple to dispatch the content to members of circles of the current user. Then this message will be consumed asynchronously by a consumer (an other running PHP instance of your app that is able to handle those messages)
Have a look at RabbitMQ or Beanstalkd, with PHP.
Related
Currently I'm making a chat application where only admin and users chat, no user-to-user chat . The design is: every chat is stored in database and each 2 seconds user and admin make an AJAX request (to a php file) to see if there is a new chat dialogue, and if there is, pull the data into the textbox. It all seems normal and working good.
Problem is as more user is talking to admin at the same time the AJAX request is becoming a lot, and by testing, the web performance already decreased with only 5 users chatting at the same time. And the input is slow too, every time user press enter they got to enter the data into database first before the admin can read it (and vice versa).
I have been told that using JSON is a recommended way, but I have no idea how to do it, can someone please at least tell me how's the design or flow is going to be if use JSON? Or is there a better way to make it? (by the way, using node.js is currently impossible for my current hosting, so don't put it in suggestion lists, sucks I know).
You should change the AJAX responder phps output to JSON. (you can use the json_encode php function for example.) And you should parse(eval) this in javascript.
I am a bit sceptic. It think It could reduce the network usage by more than 50%.
Maybe you can try a message queue, like 0mq or rabbitmq.
There are a lot of chat examples around.
I am a programmer at an internet marketing company that primaraly makes tools. These tools have certian requirements:
They run in a browser and must work in all of them.
The user either uploads something (.csv) to process or they provide a URL and API calls are made to retrieve information about it.
They are moving around THOUSANDS of lines of data (think large databases). These tools literally run for hours, usually over night.
The user must be able to watch live as their information is processed and is presented to them.
Currently we are writing in PHP, MySQL and Ajax.
My question is how do I process LARGE quantities of data and provide a user experience as the tool is running. Currently I use a custom queue system that sends ajax calls and inserts rows into tables or data into divs.
This method is a huge pain in the ass and couldnt possibly be the correct method. Should I be using a templating system or is there a better way to refresh chunks of the page with A LOT of data. And I really mean a lot of data because we come close to maxing out PHP memory and is something we are always on the look for.
Also I would love to make it so these tools could run on the server by themselves. I mean upload a .csv and close the browser window and then have an email sent to the user when the tool is done.
Does anyone have any methods (programming standards) for me that are better than using .ajax calls? Thank you.
I wanted to update with some notes incase anyone has the same question. I am looking into the following to see which is the best solution:
SlickGrid / DataTables
GearMan
Web Socket
Ratchet
Node.js
These are in no particular order and the one I choose will be based on what works for my issue and what can be used by the rest of my department. I will update when I pick the golden framework.
First of all, you cannot handle big data via Ajax. To make users able to watch the processes live you can do this using web sockets. As you are experienced in PHP, I can suggest you Ratchet which is quite new.
On the other hand, to make calculations and store big data I would use NoSQL instead of MySQL
Since you're kind of pinched for time already, migrating to Node.js may not be time sensitive. It'll also help with the question of notifying users of when the results are ready as it can do browser notification push without polling. As it makes use of Javascript you might find some of your client-side code is reusable.
I think you can run what you need in the background with some kind of Queue manager. I use something similar with CakePHP and it lets me run time intensive processes in the background asynchronously, so the browser does not need to be open.
Another plus side for this is that it's scalable, as it's easy to increase the number of queue workers running.
Basically with PHP, you just need a cron job that runs every once in a while that starts a worker that checks a Queue database for pending tasks. If none are found it keeps running in a loop until one shows up.
I am a PHP developer and the title basically says it all. However I was hoping on some more in-depth information as I am starting to get confused about how the flow for the project I work on should go.
For an (web) application I need to implement a feature like Facebook does it with notifying users about replies/comments and instantly showing these.
I figured I could use long-polling with ajax requests but this does not seem to be a nice solution as the notifications never really are instant and it is resource heavy.
So I should use some form of sockets if I understand correctly, and Node.Js would be a good choice. So based on the last assumption I now get confused about the work flow.
I thought about two possible solutions:
1) It seems to me, that if I would use Node.Js I could skip using PHP at all and base the application on Node.js only.
2) Or I could use PHP as a base and only use Node.js for notifying users and instantly showing messages but saving the data using PHP and Mysql.
These two possibilities confuse me and I can't make up my mind about what would be the "best" and cleanest way.
I do not have much experience in Node.js, played with it for a while. But managing and saving data seems to be hard in Node.js so that is why I came up with option 2.
I know Facebook is build on PHP so I am assuming that they save the data via PHP and notify / instantly show replies and comments via Node.
Could someone help me out on this?
Thanks in advance!
EDIT:
I just noticed, Stackoverflow does something similar. I get a notification in the upper left, and below my question a box with "new answer to this question". I am really interested in the technologie(s) used.
Well you could use node.js for the notifications and PHP for your app.
By googling I found this about real-time-notifications.
You could also just use node.js with socket.io, but this means that you have to learn new technologies as you mention that you have no experience with node.
I haven't used it but you could check this project, for websockets in PHP.
When you have an update that you want to notify users you can use the publish subscriber pattern to notify the intrested in this update.
Take a look in Gearman too.
Personally, I've built a notification system using the pubsub mechanism of redis, with node.js+socket.io. Everytime that there is an update on a record then there is a publish on the appropriate channel. If the channel has listeners then they will be notified. I also store the last 20 notifications in a Redis list.
The appplication is built in PHP. The notification system is built in node.js. They are different applications that see the same data. The communication occurs via redis. For example in the Facebook context:
1) A user updates his status.
2) PHP stores this to the database and Redis
3) Redis knows that this update must publish to the status channel of the specific user and it does.
4) All the friends of the specific user are listening to his status channel (here comes node.js)
5) Node.js pushes the notification in the browser with socket.io
As for facebook, I have read in an article that is using long polling for supporting older browsers. Not sure for this though, needs citation...
AFAIK It would be via two simple methods :
First one that could be very simple is adding a Boolean column to each record that determines if it has been notified or not.
The second method is creating a table to insert all notifications.
However, I'm not sure if there are alternative methods for better performance, But first method is what I do commonly myself. But I think Facebook is using 2nd method, because it has to notify each one to a lot of users.
Your question maybe dublicate of:
Facebook like notifications tracking (DB Design)
Database design to store notifications to users
You could use Server Side Events it involves a bit of JavaScript but nothing overly complicated I think.
The main bulk of this method is PHP though, so you would just use the PHP to query your DB for notifications and SSE will push them to the user.
It does have some limitations though, most notably it's not supported by IE (huge surprise) thought i'd mention it anyway to let you know of other possibilities.
Hope this helps
I am developing an iOS application from which users can post various kinds of events in a server and view the events created from all the users. The question i have , does not have a programming nature. I would like to show here , how i would design my servers database and tell me your opinion about it and how i could improve it.
The application has a very simple interface where the user when he wants to post an event , simply writes a title , some comments and attaches a photo if he wants . Then this information is sent through XML to my server and stored to the database.
The problem is that because some people are immature , they would try to post inappropriate words or even photos. So i would like to have some control on my users. What i am thinking is , at the first time that the application runs on the mobile phone and connects to the server , the server would send a user id back to the phone. Then every time the users sends an xml file , he attaches his user id with the xml (programatically attached). Also i would keep another database with all the user ids that have been created over time. So if i notice an inappropriate event , i could delete the user id from my database , and the next time the user tries to send something, the server would understand that this id is not in the database and so not allow the posting. Of course if someone decides to uninstall and reinstall the application and get a new user id , he could again post but thats ok with me.
Would it be an easier way to prevent immature behaviors or this one sounds ok?
Sure what you describe is possible. But there is one big problem with it: that interface can easily be used by a robot (a script). So if someone really wants to missuse your service, he can flood you with whatever he likes in a second. Or he can try again and again, until you have to give up removing his posts from the service.
I suggest you take a look at one of the existing frameworks instead. This way you do not have to reinvent the wheel (which has already been done 19562394792 times, and counting) and don't have to learn from your mistakes (which you certainly will make).
This is about the only workable solution with images since you can't easily perform any recognition on an image to decide if it is appropriate. The comments they post would be more easily scanned for inappropriate items.
If you are will to do this kind of moderation then your solution sounds like the right one. As with any open system you can't really stop people from creating new accounts as you mentioned. You could log and ban IP addresses, but that is not a very good solution anymore as most IP addresses could really be shared gateway addresses and those addresses rotate frequently between users.
Create an ID. Watch for behavior. Ban the ID. And encourage community involvment in alerting you of bad posts with some kind of a Flag button.
There are many ways to design a database and not just for use with an iOS app. However when I'm building a mobile app (iOS, Android or any other) I want to make sure that the amount of data being sent and received is as small as possible; this is why I use JSON instead of XML... smaller footprint.
Because I use JSON I like to use an object database like MongoDB (my favorite) or CouchDB, because I 1) don't need to worry about the structure of my data and 2) the database stores the objects in JSON format.
I then use Node.JS for my application server so now I have JSON database -> JSON objects in my server application code -> outputting JSON... seamless with no Mappers or serialisation required. FTW.
i'm running a social network with a messaging and notification feature. each time a user sends a message or places a notification for another user, a row is inserted into a table news_updates with the details about the message or notification and all his friends are inserted into the news_seen table. (once the message is read, or the item related to the notification is opened, seen is set to 1, i'm doing this at the end of my callback function for my ajax request - i'm gathering all the newsitem_ids from all the news items, that are currently open and then i'm doing a big insert with all the newsitem_ids in it).
news_seen:
newsitem_id bigint,
user_id big int,
seen int DEFAULT '0'
at the moment, i'm running an ajax request every 3 seconds, to check the news_updates JOIN news_seen for news.
this turns out to be a huge server load now that i'm getting more and more users. i've been reading a lot about xmpp and the like and i think a push notification service would be great for my site.
the only thing is, i can't really decide on which way to go, since there are so many options.
also, i thought about creating my own system. i'm planning to do it like this:
create an xml file for each user on initial registration (and run a batch for the already registered users)
once a user sends out a news update (i have my own php function for writing them into the db), i include a small command to manipulate the xml file for the respective friends
instead of doing my 3sec ajax request, i'd establish a long connection to the xml file using jquery stream and in case changes were made since the last request, i'd do my usual ajax request that polls the data from the db.
instead of running my check_seen inside the ajax request, i'd insert all the new items into a global array, to be used by an intervaled function that tests if any item in the list is currently being viewed.
do you think this is a good idea?
To be honest I do not think I would implement your specification.
For example I would use a lighter data-model then XML. I would use JSON instead.
I would avoid touching the DISC(database) as much as possible(slow).
Doing two requests instead of one(long-polling/polling). I would try to avoid this.
I would probably try to avoid wasting CPU-time by not using interval functions, but only calling function when needed. I would probably use something like REDIS's pubsub.
Polling / Long-polling is (most of the times) a bad idea in PHP because of blocking IO. I found an interesting project named REACT which I believe does non-blocking IO(expensive). I have not tested this(the performance) myself, but this could be an option.
For XMPP you will have to install additional software. I for instance liked Prosody for it's easy installation/usage. Then you would need to have a look at BOSH. For your bosh client I would use strophe.js or JSJaC.
I would probably use something like socket.io, Faye or maybe vertx.io instead, because it would scale a lot better.