On my website, users can currently compare multiple company for a prestation.
For each company, I calculate the price of the prestation.
To calculate this price I have a very big SQL request in order to filter companies based on the user's previous input and get every parameter I need.
Once the query end, I loop througt the company list and calculate the price for each of them and for each additional services the company offer. Then I display those value to the user in HTML.
The user can then add or remove an option and order the company.
Then, when a user choose a price, I send the company_id along with the user's otpion (the different services he chose) to the server and get the price previously calculated from the user SESSION.
Prices are stored in user session in order to avoid the calculation process but, I have like ~6 prices by company and usually ~30 companies for one user request. Which mean that I store in session an array with around 180 different prices for one user.
I found it quite wastefull to store this many variables in session and I was wondering, is there a better way to store those variables ? Should I store them in database ?
By the way, server side, i'm using PHP along with Mysql for the database.
What you're effectively doing is a very primitive form of caching. Sadly, the session is not the best place to do so, for a variety of reasons:
The session can never be shared between users. Some values cached may be the same for every user. It's good to have a cache that allows you to go the "unique" or "shared" routes at will.
Is your session cached data used on every page? If it is, then forget this point. If, however, it isn't, on every page, you're still incurring the cost of fetching (which, depending on your server configuration, may involve a few fs calls, or network calls, or a combination) and deserializing the data, on every request. This, if your session payload is large, can make a significant different to load times.
Another point to consider is the simple fact that, if you are running a straight-out-of-the-box LAMP stack and have not configured a shared session driver, you're going to find a very nasty surprise when you scale out :-)
Before we go any further, ask yourself these questions:
Do the values in the session change on a user-by-user basis? And if they do, is it by a fixed amount of percentage?
Do the values in the session change often?
How are the values calculated?
If #1 is "No"
You are better off caching one copy and use it for every user.
If #2 is "No"
You are better off denormalizing (i.e. pre-calculating) the values in the database
If #3 is a complicated formula
See #2
In every case
MySQL is a very poor cache driver, and should be avoided if you can. Most people replace it with redis, but that's also not a very good way due to its inability to scale by itself (you need nutcracker/twemproxy to make it properly shard). Either way, if your data to be cached is relatively small, MySQL will do. However, plan for the future.
If you do plan on going the proper cache way, consider a key-value store such as Riak, or a document storage driver such as Aerospike.
Related
I have a webshop, user will buy something
When the user visit the my web, their data will be saved temporarly in a variable $user as an array
It will be checked when the user buy something, if their balance is more or equal to the things that they want to buy, but the problem arise when there are some users that try to use two browsers to buy things at the same times, their balance just cut once (it should be twice, since they buy it twice using two browsers)
I know I can just update the $user variable before checking, but I will have to run another query to MySQL, and there is many orders ...
Is there any SQL syntax that can be used to prevent this kind of attack?
for checking their balance and make sure it's correct
Based on your current setup (ie using a variable):
Someone using two browser on the same site trying to use up their balance is going to be fairly rare, with most cases someone trying to game your system.
Just finally check their balance at the point of processing the order and if ok allow it, otherwise don't. For those doing it by accident (which is rare), they'll soon realise the error.
An alternative:
It'd be better to check the real data rather than a variable which isn't reliable and has to be forced to be persistent, and as you know not available in different sessions.
I think a better way would be to use some fast centralised persistent storage like Redis (fairly easy to learn, essentially it's an array stored in memory). You can then store their username (or whatever uniquely IDs them) and while they may have different keys across the two browsers, there will be a common unique ID and you can update their credit value in both (all) sessions by searching for the unique ID.
Then whatever browser that user is logged in to will be updated same as other browsers.
Maybe a better idea:
Unless your application needs it, don't let people log in to different browsers/devices. When they try to login, state "already logged in on another place, want to log that one our and log in here?" etc
I have a web system that I built that is an online ordering portal for our customers. We store their stock and they place orders for it through this portal.
We do a duplication check on the customer reference number so that the same order cannot come through twice, however we have been experiencing some issues whereby if a customer sends the order to our API multiple times (within milliseconds, if that, of eachother), our system doesn't have enough time to mark the order as received and as such, the system is allowing duplicates.
I am trying to decide on ways to combat this. I don't want to use database constraints for this as I find this an application issue rather than a database issue and don't believe this is a good solution.
Any design ideas on how to combat this? One solution I thought of was to use a mutex with the reference number so that if a mutex is locked for that reference number, then it might retry in a second etc? My understanding is that Mutex's are almost fool proof as they are enforced by the filesystem?
Any ideas would be appreciated
You could try employing a nonce strategy. The idea is to set random number to a hidden form field and store it in the session. Verify the id on post. The user has to deliberately refresh the page to obtain a new id and be able to post a second time.
update
So since you are using API service, then I would have to say you could use a batch system. Where an order comes in and gets stored in a holding area. A chron job runs through the batch and does the necessary pruning operations.
the users registered in a web site will have the possibility to send invitations to the friends. I want to add a daily limit for the number of invitations that a user may send.
Initially I've just added a limit (40) in the php file, but then I thought it would be better to give to the administrators the possibility to change this limit, so I've added this limit in the database. But now every time a user want to send invitations the database will be used. Would this affect the performance?
How would you configure this feature?
TL;DR: just put it in the database. :)
Complete story: It should not be a performance hit. Everything (the user itself, the usernames of the recipient, loads of stuff from your page) will be coming from your database. you shouldn't care.
If you have a REALLY big userbase, and it becomes an issue, I'm sure there are other places to do performance updates (like use memcached for all sorts of stuff). But if you want to "cache" it, I guess you could retrieve it once while loggin in and put it in the session. Use this value to substract and check etc. then ALSO check once against the database (in the background) to make sure there isn't any sort of freakish thing going on for this user. But this can be async, and does not have as big of an impact for the user-experience.
In the rare case the session says it's ok, but the database says it isn't, just send the user an error. The other way around, might need the user to re-login. But it will be rare or even impossible if you implement it correctly :)
It depends a bit on hardware but a 400,000 row table (10,000 * 40) isn't that huge in MySQL standards. I think you'll be fine.
Just make sure that you've built it sensibly and from how you've described it that there's an index on the column that stores the unique invite code.
I am thinking about using a noSQL (mongoDB) paired with memcached to store sessions with in my webapp. The idea is that upon each page load, the user data is compared to the data in the memcache and if something has changed, the data would be written to both memcached and mySQL. This way the reads would be greatly reduced and memcached utilized to do what it does best.
However I am a bit concerned about using a non-ACID database for session storage especially with the memcached layer. Let's say something goes wrong while updating the session to the DB and our users got instant headache wondering why their product that they put in the cart doesn't show up...
What's an appropriate approach to this? Should we go for a mySQL session storage or is it fine to keep a non-acid supportive database for sessions?
Thanks!
I'm using MongoDB as session storage currently. It is possible to avoid race conditions mentioned by pilif. I found a class that implements a session handler for MongoDB (http://www.jqueryin.com/projects/mongo-session/) and forked it on github to suit my needs (http://github.com/halfdan/MongoSession).
If you don't want to lose your data, stick with ACID tested databases.
What's the payoff you're looking for?
If you want a secure system, you can't trust anything from the user, save for perhaps selected integers, so letting them store the information is typically a really bad idea.
I don't see the payoff for storing sessions outside of your MySQL database. You can cron cleanup on the tables if that's your concern, but why bother? Some users will shop on a site and then get distracted for a while. They would then come back a day or two later.
If you use cookies or something really temporary to store their session info, there is a really good chance their shopping time was wasted. Users really value their time... so if you stored their session info in the database, you can write something sexy to manage that data.
Plus, the nice side effect of this is that you'll generate a lot of residual information about what people like on your website that wouldn't perhaps be available to you later on. Like you could even consider some of it to be like a poll or something where the items people are adding to their cart could impact how you manage your business, order inventory or focus your marketing.
If you go with something really temporary then you lose out on getting residual benefits.
Without any locking on the session, be really, really careful of what you are storing. Never ever store anything that is dependent on what you have read before as the data might change between you reading and writing - especially in case of ajax where multiple requests can go out at once.
An example what you must not store in a non-locked session would be a shopping cart as, to add a product, you have to read, unserialize, add the product and then serialize again. If any other request does the same thing between the first requests read and write, you lose the second request's data.
Have a look at this article for detail: http://thwartedefforts.org/2006/11/11/race-conditions-with-ajax-and-php-sessions/
Keep Sessions on your filesystem (where PHP locks them for you), in your database (where you have to do manual locking) or never, ever, write anything of value to your session if that value is derived of a previous read.
While using memcached as a cache for database, it is the user who have to ensure the data consistency between database and cache. If you'll want to scale up and add more servers there is a probability to be out of sync with database even if everything seems ok.
Instead you may consider Hazelcast. As of 1.9 it also supports memcache protocol. Compared to memcached Hazelcast wants you to implement Map Persister and only itself updates the database for the updated entries. This way you don't have to handle "check cache, if data changed update database" kind of stuff.
If you write your app so that the user stores all session information client side, then you just verify that information as needed, you won't need to worry about sessions on the server side. This is one of the principles in REST style architecture. For instance, if the user is requesting adding an item to their shopping cart, just store the itemID list and count on the client side. When you hit the cart page, you can easily look up the item information from the list of itemIDs they are telling you are in their cart.
During checkout, go directly against the database with transactions to ensure you aren't getting any race conditions, and check your live inventory. If inventory isn't there when they go to check out, just say, "sorry, we just sold out". Of course, at that point you should go update any caches you have out there that are telling people you have inventory.
I would look at how much the user costs to acquire and then ask what is the cost for implementing a really good system. Keep in mind that users are a biological retry method. "I'm bored... press reload again..." While, this isn't the most perfect solution, it is sometimes acceptable vs the cost comparsion for "not lose anything - ever".
If you want additional security, you can have your sessions cached to a separate set of memcache servers so there are no accidental flushes. :)
There are a number of other systems membase.org, and some other persistent memcache solutions (java implementations) that will persist storage to disk. If you want to modify your client somewhat, or how you access memcache, you can do your own replication of memcache session objects.
-daniel
I'm at a crossroads, not exactly sure what is better to use. Right now I'm using sessions arrays to store calculations on the fly, but I need to switch to objects because I would like have some functions. But I was also considering using ajax and insert the data on the fly back and forth to the database but I'm worried it might be to many calculations under heavy trafic.
What i'm trying to do is have a cart were items are added to, all of the items in the cart will need to be recalculated if a change to a quantity field is made so I was wondering which is the better solution, to use objects and sessions or update a large table in the db with multiple users manipulating the data.
I'm using mysql db if that helps in the decision..
The session should be much faster, but it will not persist if the user closes their browser. I use sessions for that sort of thing (shopping carts, last accessed searches, etc) most of the time. On a site with a medium-large number of users, you'll need to do maintenance on a table to clean out old session information. That's too much of a headache for me unless I really need the data to be there days later.
Well keep in mind that sessions themselves can be stored in the database. So regardless of what you choose, you shouldn't try to roll your own database-based sessions.
Keep in mind though that standard PHP sessions store the session data on disc. If you have many users on your site with many sessions over it's possible that you could bog down your box with too much disc i/o. This would be less of a problem if you had an SSD drive, however.
Another option I've used before is to simply have a MySQL database on the same box as the server, and store the data in a memory only table. I think this is the best of both worlds:
It's faster than a traditional myisam
or innodb database because it doesn't
ever get dumped to disk (this mostly
only affects session writes)
Assuming your database is on a
different box than your web server,
it's faster because your session db
is on the same box.
You can perform some sql wizardry on
your session store if you need (can't
do this with files)
I create a cart table which stores a SESSION ID and the corresponding item id.
The session id would be used to associate the browsing user. Calculating the price would be as easy as retrieving all rows with the users session id and summing the price from the item table (for example). Any changes to the cart involve a simple insert or delete for each item.
For doing many different calculations, I'd go with session... This really is the most bandwidth friendly way. If you would like to persist their cart without using cookies, then you can store their cart in the database when they leave and reload it when they come back (store the cart based on userID obviously).