Basicaly, I have an PHP webservice which will be available from:
website
mobile phone clients - (android/iphone)
Data is retreived in JSON format. Request is sent using GET method over HTTPS protocol. Data in database is kept in mongo database.
Here comes my main questions:
I want to get the statistics for each user - how many calls he is doing per-minute/hour - how to save this data? Will it be okay to save everything into mongo database? (1 day statistics = 43 millions rows) Is there a solution maybe which will keep data only for X days and then auto-trunc everything automatically?
I also want to get statistics for each IP - how many calls were made from it per-minute/hour/day - how to save this data? Will it be okay to save everything into mongo database? Will it not become too large?
Is it always possible to see an IP of an user who is making a call to a webservice? what about IPv6?
These 3 questions are the most interesting to me currently. I am planning on letting users use basic services without the need of loggin-in! My main concern is the database/file system performance.
Next comes the description about - what measures I am planning to use. How did I come to this solution and why these 3 questions above are essential. Feel free to ignore the text below if you are not interested in details :)
I want to protect my webservice against crawling I.e. somebody can pass parameters (which are not hard to guess) to get entire data off my databse :)
Here is an usage example: https://mydomain/api.php?action=map&long=1.23&lat=2.45
As you can see - I am already using a secure https protocol, in order to prevent accidental catching entire GET request. It also protects agains 'man in the middle' attacks. However, it doesn't stop attackers from getting into website and going through JS AJAX calls to get an actual request structure. Or decompiling entire android's .APK file.
After reading a lot of questions through out the internet - I came to the conclusion that there is no way of protecting my data entirely, but I think I have found an approach of making the life of a crawlers a lot harder!
And I need your advice on either - if this whole thing is worth implementing and what technologies shall be used in my case (see next).
Next comes the security measures against non-website (mobile device) use of a service for users which are not logged-in.
<?php
/*
Case: "App first started". User: "not logged in"
0. generate UNIQUE_APP_ID for APP when first started
1. send UNIQUE_APP_ID to server (or request new UNIQUE_APP_ID from server to write it on device)
1.1. Verify how many UNIQUE_APP_IDs were created from this IP
if more than X unique values in last Y minutes ->
ban IP temporary
ban all UNIQUE_APP_IDs created by IP during Y minutes (use delay to link them together).
but exclude UNIQUE_APP_IDs from ban if these are not showing strange behaviour ( mainly - no calls to API)
force affected users to log-in to continue to use the service or ask to retry later when ban wears off
else register UNIQUE_APP_ID on server
Note: this is a 'hard' case, as the IP might belong to some public Wi-Fi AP. Precautions:
* temporary instead of permanent ban
* activity check for each UNIQUE_APP_ID belonging to IP. There could be legit users who use the service from long time thus will not be affected by this check. Created Z time ago from this IP.
* users will be not be ever banned - rather forced to log-in, where more restrictive actions will be performed individually!
Now that the application is registered and all validations are passed:
2. Case: "call to API is made". User: "not logged-in"
2.1. get IP from which call is made
2.2. get unique app ID of client
2.3. verity ID against DB on server
if not exists -> reject call
2.4. check how many calls this particular ID did in the last X minutes
if more than X calls -> ban only this unique ID
2.5 check how many Ids were banned from this IP in the last X minutes. If more than Y then ban new calls for whole IP temporary
check if all banned IDs were created from the same IP, if yes then also ban that IP, if it is different from new IP
*/
?>
As you can see - my whole solution is based on the idea that I can store the data about webservice and retrieve it for analysis easily.. for each single webservice call.. Or maybe each X'th call. I have no idea about - what kind of database shall be used. I was thinking that mongo might not be the best choice. Maybe MySQL? Keeping data safe from wrong users is one reason. Another reason is that abusal usage of database will result in a huge load on a database.(DDos?) So, i think this might be a good idea to count webservice calls.
On the other side. A bit of calculations.
If there are 1000 users working simultaniusly. Each generating 30 calls to a webservice per minute. So it's 30000 disc writes in a minute. In hour it's 60 times that i.e. 1.800.000 disc writes in an hour. If I am planning to keep statistics about daily usage then it's 24 times that i.e. in
average there will be 43.200.000 records for tracking purposes kept on a server.
Each record contains information about: time + IP + user unique ID
I was also thinking about not storing any data at all. Use redis instead. I know that there is some kind of counter exists. For for each individual IP I can create a separate key and start counting calls. In this case everything will be kept in server's RAM. There is also an expire date parameter which is possible to set for each key. And for separate users I can store their IDs instead of network IP. This solution only came to my mind after I finished writing this whole essay, so let me hear your ideas about questions above.
Related
I'm using Laravel 5.3 and I've built an application of which we can think of as sort of like an e-commerce app. I essentially want to allow the user to leave a review for their order, inside the application (not by email), but have them wait X amount of days after their purchase. Like eBay almost...
But what is the usual convention for creating a waiting period before a user can perform an action like this? I've been thinking whether I should set up cron jobs to send the user a request after X amount of days or perhaps by logging this in the database somehow, though I imagine that to be database heavy when having lots of users.
Any thoughts would be appreciated.
Update: The idea is to force a wait time upon the user. They can't leave a review until after the waiting period.
As I understand your use case, you need a persistent storage method to remember how many days have passed since the purchase of the item. There are, in general, 4 methods of persistent storage:
1. Cookies:
Storage: Client Side (but secure since Laravel encrypts and signs all cookies)
Specific: Unique to user machine and domain
Persistence Time: Can last forever unless the user specifically deletes the cookie
Common Use Cases: Saved preferences
2. Session
Storage: Server Side (depends on driver)
Specific: Specific to user (but depends on driver)
Persistence Time: Usually expires in a couple of hours, but can be extended
Common Use Cases: Persistence layer from one request to another (like shopping cart, popup notifications after action triggers)
3. Caching
Storage: Server Side
Specific: Generally application specific (but can be user specific)
Persistence Time: Generally an hour to a couple of days
Common Use Cases: Application specific storage use cases (e.g. total number of hits, most popular pages, database query caching, view caching)
4. Database
Storage: Server Side
Specific: Can be user or app specific
Persistence Time: Forever until deleted
Common Use Cases: Longer term data persistence layer (e.g. user details)
Now, if you want to send an automated reminder to the users after a couple of days of the purchase, the persistence layer has to be server side. That rules out cookies and also caching would not be a good method (since caching is best used for app specific data, not user specific data). But if you're looking for the review to be triggered by a user, I would suggest using cookies.
Needless to say, automated reminders need to be triggered by cron jobs.
In Summary:
If you want automated reminders: use database + cron jobs
If you don't want automated reminders: use cookies
first you should create service for have commenting logic (for example just allow user comment his order after n day) ...
after it you can use queue for handle this action by this way :
after submit order you add a job to queue for N day later.
it that job u can email user and say to him that he can leave comment for his order...
adding job to queue can be an Event or can be an listener on order model ...
for more information :
https://laravel.com/docs/5.4/providers
https://laravel.com/docs/5.4/queues
I'd suggest a different design. You cannot know, really, when they've received their order, when they've opened it, used it, or enjoyed it. Maybe it was delivered next day and they used it immediately. Maybe it was delivered 7 days later and not touched for a week. You really need to accept both ends of this spectrum as legitimate. But I get that you don't want people reviewing orders they've not legitimately used. So here is my alternate suggestion.
Allow the user to submit a review at any time after purchase. They might submit a review 5 days, 5 hours, or even 5 minutes after ordering. What you limit, instead, is when the review becomes visible on your site. So you set a policy that says reviews are accepted at any time, but only visible 72 hours after package delivery. (Or any number of hours; you might even scale the number. Trusted users can see their review in 24 hours, but new users require 72.)
This approach simplifies the code you have to write: all you need to track is review submission date time, and limit the display of reviews based on said submission date time.
I am creating a web application named Online Exam using PHP+MySQL and AngularJS. Now I am getting some trouble on project like changing the user looged in status. Let us take this condition as a example:
Suppose a authorized user/student successfully logged in online exam section(After successfully logged current time will be inserted in the db in exam_start_time column as unix timestamp format and exam_status will be set as 'ACTIVE`.
1hr(60 min) countdown timer is initialize for him/her as per the inserted exam_start_time in db.
Now suppose after 15 min the system shuts down automatically, then if user logged in again(In same system or other) then the countdown timer should be set for 45 minutes only.
Previously I was updating the last_activity_time in our table in every 10 sec(using ajax calls). but now I want to change this way, Is there any way like(socket or network programming using PHP) to update the column.
Here is my table structure which is managing user logged in status
Please give me some suggestions on it.
A Php socket server programming tutorial : http://www.christophh.net/2012/07/24/php-socket-programming/
Sockets, as Pascal Le Merrer mentioned, is IMO your best option. But beware of Apache! Every WebSocket holds one Apache thread, and Apache wasn't designed to do that. when too many (and by too many I mean few dozen) clients connect to your site, it will crash. (I've been there while trying to implement long polling/comet, ended up using NodeJS. If you're using nginx, it is more likely that it will become low on resources and effective, but there are also other ways. Take a look here:
Using WebSocket on Apache server
If you find this uncomfortable/hard to learn, try also another idea:
try to add hidden iFrame to your exam page, pointing to prepared site that updates database row. Use javascript to refresh this page every 10-15 seconds. Every refresh causes update of specific row in DB, using current date and time. It should work (not tested, but give it a try).
I'm writing up an online examination site for an educational institution. Time limits can be set for each assessment, and once a user begins an exam, a new timestamp is created on the server. The problem is that our school computers often lock up and students are forced to restart, losing exam time.
I figured out I could store the timer as a cookie, but that could easily be compromised. Any suggestions? Thanks.
You could store a timestamp in a database that is linked to their user (I would assume they have a login if they are doing exams) and perform an ajax call every few seconds to check if they are still online.
In this ajax check you can update the database with a new timestamp that indicates when they were last online, so if the computer locks up and they have to restart you can figure out how much time was lost.
You could generate unique hash to store it in the cookie like
$cookie = md5( $student_name . $test_start_timestamp . $computer_ip );
and store all the needed information in the database like:
INSERT INTO examination
SET student_hash = '$cookie',
test_start = '$test_start_timestamp',
computer_ip = '$computer_ip'
and later (if computer is restarted) you can get all the information from the database:
SELECT * FROM examination
WHERE student_hash = '$cookie'
this way the student won't be able to change the timestamp
While I realise that this can at times be difficult the only really reliable solution to your dilema is a reliable network.
But lacking that what about another communication media or pathway that confirms the client state? Can you confirm that a domain controller can see the client on the LAN? If the PC is present but the web client isn't it is more likely a user manuipulated situation. Can something (domain controller, other network location?) note the appearance or disappearance of a client from the network?
Any method is open to abuse somehow. I think the only real assurance will come from human monitoring of frequence of and user experiencing restarts.
I am trying to create a PBBG (persistent browser based game) like that of OGame, Space4k, and others.
My problem is with the always-updating resource collection and with the building times, as in a time is set when the building, ship, research, and etc completes building and updates the user's profile even if the user is offline. What and/or where should I learn to make this? Should it be a constantly running script in the background
Note that I wish to only use PHP, HTML, CSS, JavaScript, and Mysql but will learn something new if needed.
Cron jobs or the Windows equivalent seem to be the way, but it doesn't seem right or best to me.c
Do you have to query your db for many users properties, like "show me all users who already have a ship of the galaxy class"?
If you do not need this you could just check the build queue if someone requests the profile.
If this is not an option you could add an "finished_at" column to you database and include "WHERE finished_at>= SYSDATE()" in your query. In that case all resources (future and present) are in the same table.
Always keep in mind: what use is there to having "live" data if no one is requesting it?
My problem is with the always-updating
resource collection and with the
building times, as in a time is set
when the building, ship, research, and
etc completes building and updates the
user's profile even if the user is
offline
I think the best way to do this is to install message queue(But you need to be have install/compile it) like beanstalkd to do offline processing. Let's say it takes 30 seconds to build a ship. With pheanstalk client(I like pheanstalk) you first put message to the queue using:
$pheanstalk->put($data, $pri, $delay, $ttr);
You could see protocol for meaning of all arguments.
But with $delay=30. When a worker process does a reserve() it can process the message after 30 seconds.
$job = $pheanstalk->reserve();
Streaming data to user in real-time
Also you could look into XMPP over BOSH to stream the new data to all users in real-time.
http://www.ibm.com/developerworks/xml/tutorials/x-realtimeXMPPtut/index.html
http://abhinavsingh.com/blog/2010/08/php-code-setup-and-demo-of-jaxl-boshchat-application/
i need to show the number of online visitors, but there is a problem with selecting algoritm to do it!
maybe i must create a table in DB, where i'll store ip addresses of visitors and time of visit! by so i can show the count of ip addresses, which's time >= NOW() - 10 minutes, for example...("now()-10 minutes" is just to show the logic, i know that this is not a function:)
is this goog way to go?
please give me an idea.
Thanks
This is good tutorial. Note that mysql (i believe youll use it) online users table should be typed as MEMORY.
I'm not really sure how you would use AJAX to store the data...
I personally use the database solution.
I store user_id, last_seen, IP and location in the site (but that's not necessary to just get the count).
When the user requests a page refresh the last_seen column and delete all the entries with NOW()-last_seen greater than x minutes.
Keeping track of "visitors" (as opposed to raw page requests, which the web server should track on its own) is a complex art.
You could store IP addresses, as you described, but what about a visitor who's using a proxy that rotates their IP as frequently as every page load? What about a set of visitors all using the same proxy that uses the same IP for all of them?
My recommendation: don't bother doing any of it yourself, and use Google's free Analytics service. It tracks visitors, browsers, traffic sources, and just about anything else you could possibly want to know about who's looking at your site.
Yes, the algorithm is o.k. in general but with some corrections
May be you want to delete all outdated records first, and then just count the rest.
10 minutes is too much. 1-3 is the average time user spend on the page.
AJAX project is funny, but it has nothing to do with storage. You need more education of terms of client-server application. AJAX is the transport, not storage.
If you expect the website will have a lot of visitors, the query could make the page for one user every 10 minutes pretty slow...
If so, I would suggest to write an CLI script that will clean the old entries and run it in a cronjob. That way the user wouldn't notice any delay, as the parsetime will be spent on the CLI.