I am trying to create a web application which have a similar functionality with Google Alerts. (by similar I mean, the user can provide their email address for the alert to be sent to, daily or hourly) The only limitation is that it only gives alerts to user based on a certain keyword or hashtag. I think that I have found the fundamental API needed for this web application.
https://dev.twitter.com/docs/api/1/get/search
The problem is I still don't know all the web technologies needed for this application to work properly. For example, Do I have to store all of the searched keywords in database? Do I have to keep pooling ajax request all the time in order to keep my database updated? What if the keyword the user provided is very popular right now that might have thousands of tweets just in an hour (not to mention, there might be several emails that request several trending topics)?
By the way, I am trying to build this application using PHP. So please let me know, what kind of techniques I need to learn for such web app (and some references maybe)? Any kind of help will be appreciated. Thanks in advance :)
Regards,
Felix Perdana
I guess you should store user's e-mails and search keywords (or whatever) in the database.
Then your app should make API queries (so it should be run by a server) to get some relevant data. Then you have to send data to the all users.
To understand here is the algorithm:
User adds his request to the page like http://www.google.ru/alerts
You store his e-mail and keyword in the database.
Then your server runs script (you can loop it or use cron) which makes queries to the Twitter to get some data.
Your script process all the data and send it to the user's e-mails.
Related
I've been working on an e-commerce project built on Symfony2 (for the backend) and AngularJS for the frontend. Currently the Symfony part is used only as an API, which has three different user levels (guest, customer & admin). Different actions that can be done within the system (like add/remove data) are secured by:
Symfony2 firewall with user roles/access control
JMS security extra (#PreAuthorize expressions)
For the parts that are secure everything works as intended and I'm very happy with the way things work.
Problem:
There are parts of the API which are public (like retrieving product information, categories, etc.). I'm retrieving such data in Angular with Ajax calls to my API that returns the data in JSON format. One example would be:
/api/product/get-all/?page=1&count=10&sorting[id]=asc
The problem is that anyone could look at the requests in browser and copy the path and have access to all the data (such as all the products) and could just download a JSON of all the information. Although this data is "public", I don't want to give others such an easy way of "stealing" my data.
Ideas & possible solutions:
I was looking at the JWT (Json Web Token) standard to try and secure the public calls to my API and implement it in such a way that I generate a token for "real" users that are on the website, and such limit direct access to public API links.
What do you think? Would this be a possible solution?
I was also reading in some other question on StackOverflow that I could check the HTTP_X_REQUESTED_WITH header from the request, but we all know this can be easily spoofed by an attacker.
Finally, I read a similar approach to "solution" 1) here : http://engineering.talis.com/articles/elegant-api-auth-angular-js/ but I'm not entirely sure that this fits my purpose.
Additional notes:
I don't want to make this bullet-proof, but I also don't want to give people the option to click 2 buttons and get all my data. I know that eventually all the information can be "stolen" (e.g.: by using a web scraper ), but "securing" the system in such a way that people would have to make a bit of an effort is what I have in mind.
I can't really re-model my API too much at this stage, but any ideas would be appreciated
Thanks for taking the time to read my question and I'm looking forward for any feedback.
You can limit the abuse of your system in a number of ways, including:
Limit the total number of requests that API will return before requiring CAPTCHA or some other validation method. This can be limited by IP, browser fingerprint, authentication token, etc.
Make it difficult for abuser to guess IDs of products, categories, etc. by using GUIDs or other randomly generated IDs.
Use API management proxy such as Azure API Management for more enterprise level management of the APIs (http://justazure.com/azure-api-management-part-one-introduction/)
You could try something like:
To access the site anonymous users first need to fill in the captcha to get temporary token.
Add referrer check on.
Limit amount of data anonymous users can view. For instance, first 50 products.
This way everyone who wants to steal your data first need to get anonymous temporary token by filling in the captcha and change referrer.
Try with DunglasAngularCsrfBundle
I am a PHP developer and the title basically says it all. However I was hoping on some more in-depth information as I am starting to get confused about how the flow for the project I work on should go.
For an (web) application I need to implement a feature like Facebook does it with notifying users about replies/comments and instantly showing these.
I figured I could use long-polling with ajax requests but this does not seem to be a nice solution as the notifications never really are instant and it is resource heavy.
So I should use some form of sockets if I understand correctly, and Node.Js would be a good choice. So based on the last assumption I now get confused about the work flow.
I thought about two possible solutions:
1) It seems to me, that if I would use Node.Js I could skip using PHP at all and base the application on Node.js only.
2) Or I could use PHP as a base and only use Node.js for notifying users and instantly showing messages but saving the data using PHP and Mysql.
These two possibilities confuse me and I can't make up my mind about what would be the "best" and cleanest way.
I do not have much experience in Node.js, played with it for a while. But managing and saving data seems to be hard in Node.js so that is why I came up with option 2.
I know Facebook is build on PHP so I am assuming that they save the data via PHP and notify / instantly show replies and comments via Node.
Could someone help me out on this?
Thanks in advance!
EDIT:
I just noticed, Stackoverflow does something similar. I get a notification in the upper left, and below my question a box with "new answer to this question". I am really interested in the technologie(s) used.
Well you could use node.js for the notifications and PHP for your app.
By googling I found this about real-time-notifications.
You could also just use node.js with socket.io, but this means that you have to learn new technologies as you mention that you have no experience with node.
I haven't used it but you could check this project, for websockets in PHP.
When you have an update that you want to notify users you can use the publish subscriber pattern to notify the intrested in this update.
Take a look in Gearman too.
Personally, I've built a notification system using the pubsub mechanism of redis, with node.js+socket.io. Everytime that there is an update on a record then there is a publish on the appropriate channel. If the channel has listeners then they will be notified. I also store the last 20 notifications in a Redis list.
The appplication is built in PHP. The notification system is built in node.js. They are different applications that see the same data. The communication occurs via redis. For example in the Facebook context:
1) A user updates his status.
2) PHP stores this to the database and Redis
3) Redis knows that this update must publish to the status channel of the specific user and it does.
4) All the friends of the specific user are listening to his status channel (here comes node.js)
5) Node.js pushes the notification in the browser with socket.io
As for facebook, I have read in an article that is using long polling for supporting older browsers. Not sure for this though, needs citation...
AFAIK It would be via two simple methods :
First one that could be very simple is adding a Boolean column to each record that determines if it has been notified or not.
The second method is creating a table to insert all notifications.
However, I'm not sure if there are alternative methods for better performance, But first method is what I do commonly myself. But I think Facebook is using 2nd method, because it has to notify each one to a lot of users.
Your question maybe dublicate of:
Facebook like notifications tracking (DB Design)
Database design to store notifications to users
You could use Server Side Events it involves a bit of JavaScript but nothing overly complicated I think.
The main bulk of this method is PHP though, so you would just use the PHP to query your DB for notifications and SSE will push them to the user.
It does have some limitations though, most notably it's not supported by IE (huge surprise) thought i'd mention it anyway to let you know of other possibilities.
Hope this helps
I have been playing around with Node.js for two days now, I am slowly understanding how it works. I have checked multiple threads and posts now but I seem to either misunderstanding them or the way I am thinking about this application is completely wrong.
My application is mainly based on PHP and uses Node.js as a notifications system.
I first wanted to this solely in Node.js but I am more familiar with PHP so that is why I only want to use Node.js as a notifications system.
I do not have any real code to show as I have been mainly playing around and see all what Node can do and so far it seems to be the thing I need, there is one thing I just can't figure out or seem to mis understand. So far I figured out how to send data between the user and the server and used socket.io for this.
So, what if I have a user, which is registered and logs-in on my application. He then has a socket id from socket.io, but when the user leaves my application and comes back the next day his socket ID is changed because it seems to change on every connection. I need to have my users somehow always have the same socket ID or something else which tells my node.js server that it should only send data to one specific user or multiple users. Also, as the socketid seems to change on every request it is even changed when the user visits a different page so I don't ever seem to know which user is what.
I am a little confused and the flow of working with both PHP and Node.js is still a little mystery to me so I hope my question is clear. I dont want to be depending on many modules as I find all these different modules kind of confusing for a beginner like me.
As long as PHP-Node.js are using sessions stored somewhere else other than flag file sessions let's say a cache service or a database mysql or nosql ..
you can use the "same flat file" sessions thought cache or database could be make your application "more"of course there are additional practises of allowing authenticated users to try to connect by controlling when to render the javascript code that holds the information to connect to socket.io server, where an additional list is stored in memory of all connected having information like username/log/timestamps/session variables/etc..
I am developing an iOS application from which users can post various kinds of events in a server and view the events created from all the users. The question i have , does not have a programming nature. I would like to show here , how i would design my servers database and tell me your opinion about it and how i could improve it.
The application has a very simple interface where the user when he wants to post an event , simply writes a title , some comments and attaches a photo if he wants . Then this information is sent through XML to my server and stored to the database.
The problem is that because some people are immature , they would try to post inappropriate words or even photos. So i would like to have some control on my users. What i am thinking is , at the first time that the application runs on the mobile phone and connects to the server , the server would send a user id back to the phone. Then every time the users sends an xml file , he attaches his user id with the xml (programatically attached). Also i would keep another database with all the user ids that have been created over time. So if i notice an inappropriate event , i could delete the user id from my database , and the next time the user tries to send something, the server would understand that this id is not in the database and so not allow the posting. Of course if someone decides to uninstall and reinstall the application and get a new user id , he could again post but thats ok with me.
Would it be an easier way to prevent immature behaviors or this one sounds ok?
Sure what you describe is possible. But there is one big problem with it: that interface can easily be used by a robot (a script). So if someone really wants to missuse your service, he can flood you with whatever he likes in a second. Or he can try again and again, until you have to give up removing his posts from the service.
I suggest you take a look at one of the existing frameworks instead. This way you do not have to reinvent the wheel (which has already been done 19562394792 times, and counting) and don't have to learn from your mistakes (which you certainly will make).
This is about the only workable solution with images since you can't easily perform any recognition on an image to decide if it is appropriate. The comments they post would be more easily scanned for inappropriate items.
If you are will to do this kind of moderation then your solution sounds like the right one. As with any open system you can't really stop people from creating new accounts as you mentioned. You could log and ban IP addresses, but that is not a very good solution anymore as most IP addresses could really be shared gateway addresses and those addresses rotate frequently between users.
Create an ID. Watch for behavior. Ban the ID. And encourage community involvment in alerting you of bad posts with some kind of a Flag button.
There are many ways to design a database and not just for use with an iOS app. However when I'm building a mobile app (iOS, Android or any other) I want to make sure that the amount of data being sent and received is as small as possible; this is why I use JSON instead of XML... smaller footprint.
Because I use JSON I like to use an object database like MongoDB (my favorite) or CouchDB, because I 1) don't need to worry about the structure of my data and 2) the database stores the objects in JSON format.
I then use Node.JS for my application server so now I have JSON database -> JSON objects in my server application code -> outputting JSON... seamless with no Mappers or serialisation required. FTW.
hii.. i am designing a chat application. i have tried using php + MySQL + JavaScript and developed the web based chat app. But the application is quite slow. What all i do is, just store the messages from the clients in a table, and each client retrieves the table at a lag of 2 seconds.... I manage this by ajax.
But the app response becomes very bad when many user joins.
Is their any other technique to build the chat app ??? or what else i can do to make my app better.
help plzz ..
thanks in advance :)
Don't know if you've touched upon it, but I'd recomend using an IRC server as backend.
If you have the ability to install nodejs(basicly server side javascript) on your server you should take a look at socket.io which is a plugin for nodejs. This gives you almost an instant response time, even for quite large masses of users.
There are almost no libraries/projects for real-time web applications in PHP. But if you really want to use it you can take a look at a technique called long-polling. This can still be quite heavy on your server though.
Maybe it is slow because each client retrieves full table of chat contents - try appending only the new messages.
I've done the same thing last year.
I suggest retrieving only so many chats when the user first logs in, e.g. the last 30, and displaying them. Then on each subsequent AJAX call only retrieve any new chat messages, rather than everything else again. You can do this by storing the latest chat id (a unique id) when you first retrieve all the chat msgs and then only retrieve anything with a chat id greater than that.
How often do you check for new msgs? You could also increase that time.