This question already has answers here:
Caching strategy, when does caching become pointless?
(3 answers)
Closed 9 years ago.
I am kinda new in the application development. I am developing one application at the moment and I want to know your valueable inputs about a problem that I am facing as a newb.
When it comes to caching, which parts of the application I should add in cache? Let me give you an example of what I am doing:
I am developing a picture sharing application for my users and I have created a cache mechanism to cache every user's data, his albums & his pictures. I currently have 1 query per page (only in case that it is not already stored in Cache) for example:
if ( !$ItemNotInCache )
{
$MyData = $DB->Query('blah');
Cache::CacheIt($MyData);
}
This is about every result I am getting, I am trying not to call too many queries every time a user requests a page, is this efficient? The problem is that as you can understand I had to build this huge mechanism to manipulate all this data in arrays, etc..
When a user deletes an image from DB, i have to remove the index from the cached data of his album as well.
When a user adds an image to DB, I use array_merge to insert this item to his cached data and so on.
A friend of mine told me that I shouldn't abuse the cache since my data is dynamic and it will change very often and he also told me this is why MySQL is for. My question is I should use the cache in that way or no? As you can understand the data is dynamic and it should be refreshed often. Is this efficient or i should just stick with MySQL queries?
I think what you are doing is perfectly fine. Retrieving from the cache uses less resources in almost all the cases as opposed to doing a query each time.
However, I wouldn't try to "merge" the added image, just call the database again. Just increment the cache when the album modified. I don't think editing the cache when the album is modified is the right approach. Databases are good at storing and building the data so it's easy to be retrieved. Why do the building yourself?
Related
Hi I am building a online quiz system and i am trying to figure out the best way to optimize the performance. Lets say there are total of 100 of quiz in the question bank and randomly pull 50 of them into a set of quiz. I am using Codeigniter framework.
What I am planning to do now was query the 50 question from database and store the query question with answer into temp_data(). User answer will be store in the temp-data as well. When user finished the quiz, all the answer and question will be inserted in one shot to the database. Session is destroyed.
So, is there any other more efficient way or this is it? Since cookies allowed 4kb so it is impossible to store it. i am thinking to use another table to store the answer but it involve query for each page and question load. But session kind of eat up memory as well. Any recommendation?
Try to use CI cache library you can use different cache methods file,redis or memcache with this library, as you said you are selecting random 50 question for each user than create unique cache file save all questions and answers in that file, just save filename in session or cookie. on each question answered you can save directly to database or save in same cache file later save all answered question at once. have look on cache library
https://www.codeigniter.com/user_guide/libraries/caching.html
This question already has answers here:
PHP - Visitors Online Counter
(5 answers)
Closed 8 years ago.
I am working on a very simple and small visitor counter and I am wondering myself if it is not too heavy to on the server resources to open a MySQL database every time a visitor enters on a page of the website.
Some people store the visits on a plain-text file and maybe I could store the number in a session (in an array with a key for each page), and when the session is closed, I copy it in the database in one time?
What is the lightest way to do this?
In most robust web applications, the database is queried on every page load anyway for some reason or another, so unless you have serious resource limits you're not going to break the bank with your counter query or save much load time by avoid it.
One consideration might be to increase the value of the database update so that one update can be queried for multiple uses. In your case, you could have a view log, like :
INSERT INTO view_log
VALUES (user_name, ip_address, visit_timestamp, page_name)
Which could be used for reporting on popularity of specific pages, tracking user activity, debugging, etc. And the hit count would simply be:
SELECT COUNT(1) FROM view_log
If your site has a database already, use it!
Connections are most likely pooled between opens and take very little effort.
If you write to a file the site requires write access to it and you risk concurrency problems during multiple user connections.
Only persisting when session closes is also a risk if the server is closed abruptly..
Most sites open a MySQL connection anyway, so if you do, you won't have to open it twice. Writing to disk also takes resources (although probably less), and additionally you might wear out a small part of your disk very fast if you have a file based counter on a busy website. And those file writes will have to wait for each other, while MySQL handles multiple requests more flexible.
Anyway, I would write a simple counter interface that abstracts this, so you can switch to file, MySQL or any other storage easily without having to modify your website too much.
I am running application (build on PHP & MySql) on VPS. I have article table which have millions of records in it. Whenever user login i am displaying last 50 records for each section.
So every-time use login or refresh page it is executing sql query to get those records. now there are lots of users on website due to that my page speed has dropped significantly.
I done some research on caching and found that i can read mysql data based on section, no. articles e.g (section - 1 and no. of articles - 50). store it in disk file cache/md5(section no.).
then in future when i get request for that section just get the data from cache/md5(section no).
Above solution looks great. But before i go ahead i really would like to clarify few below doubts from experts .
Will it really speed up my application (i know disk io faster than mysql query but dont know how much..)
i am currently using pagination on my page like display first 5 articles and when user click on "display more" then display next 5 articles etc... this can be easily don in mysql query. I have no idea how i should do it in if i store all records(50) in cache file. If someone could share some info that would be great.
any alternative solution if you believe above will not work.
Any opensource application if you know. (PHP)
Thank you in advance
Regards,
Raj
I ran into the same issue where every page load results in 2+ queries being run. Thankfully they're very similar queries being run over and over so caching (like your situation) is very helpful.
You have a couple options:
offload the database to a separate VPS on the same network to scale it up and down as needed
cache the data from each query and try to retrieve from the cache before hitting the database
In the end we chose both, installing Memecached and its php extension for query caching purposes. Memecached is a key-value store (much like PHP's associative array) with a set expiration time measured in seconds for each value stored. Since it stores everything in RAM, the tradeoff for volatile cache data is extremely fast read/write times, much better than the filesystem.
Our implementation was basically to run every query through a filter; if it's a select statement, cache it by setting the memecached key to "namespace_[md5 of query]" and the value to a serialized version of an array with all resulting rows. Caching for 120 seconds (3 minutes) should be more than enough to help with the server load.
If Memecached isn't a viable solution, store all 50 articles for each section as an RSS feed. You can pull all articles at once, grabbing the content of each article with SimpleXML and wrapping it in your site's article template HTML, as per the site design. Once the data is there, use CSS styling to only display X articles, using JavaScript for pagination.
Since two processes modifying the same file at the same time would be a bad idea, have adding a new story to a section trigger an event, which would add the story to a message queue. That message queue would be processed by a worker which does two consecutive things, also using SimpleXML:
Remove the oldest story at the end of the XML file
Add a newer story given from the message queue to the top of the XML file
If you'd like, RSS feeds according to section can be a publicly facing feature.
This question already has answers here:
Is it unreasonable to assign a MySQL database to each user on my site?
(10 answers)
Closed 8 years ago.
I have a small file hosting website that I am trying to store the files that users upload in a database. The issue that I am having is that I cannot decide which method would be better:
To store all users in one table
create a new table for each user.
I understand that the second method will slow performance but by how much? I am planning on having 1000+ users eventually. The issue with the first method is listing the files back to the user. What method should I go with and which one would be the most efficient?
Short answer:
No. Use the simplest thing that works: A single table.
Long answer:
You'll know what kind of scaling problems when you have a production system under production loads, and then you can analyze where your bottlenecks are and develop a sharding strategy based on real-world use cases and not hypotheticals.
Right now you're just guessing, and you'll probably guess wrong. Then you're stuck with an awful database structure you'll find impossible to undo.
Try not to store actual files in the MySQL database, this almost always leads to horrible disaster, but instead store them on the filesystem and keep references to them in the database. If you're going to be managing a lot of files, heaps and tons of them, you may want to look at document store database like Riak to help with that.
I suggest creating a table for each entity and having a correct relationship between them.
For example:
Users table will have user_id, user_name, etc.
Files table will have id, url, user_id
In this case, the relationship is created by having the same user_id. So when you upload a file, you attach the user_id to the image.
That means - no, don't create a separate table for each user. Go with method 1 and make sure each important entitiy has its own table.
Down the road, you will probably have more entities and more tables such as Permission, Products, etc etc. By using SQL queries you will be able to get all the data you want.
Hope this helps!
Having 1000 ish users is not a problem for MySQL. But tadman is rigth, save the files on the filesystems instead of the database.
If you know that you will endup with millions of users, I suggested that you read on how Facebook or others big users related sites handle this scalling problems.
Well this is kind of a question of how to design a website which uses less resources than normal websites. Mobile optimized as well.
Here it goes: I was about to display a specific overview of e.g. 5 posts (from e.g. a blog). Then if I'd click for example on the first post, I'd load this post in a new window. But instead of connecting to the Database again and getting this specific post with the specific id, I'd just look up that post (in PHP) in my array of 5 posts, that I've created earlier, when I fetched the website for the first time.
Would it save data to download? Because PHP works server-side as well, so that's why I'm not sure.
Ok, I'll explain again:
Method 1:
User connects to my website
5 Posts become displayed & saved to an array (with all its data)
User clicks on the first Post and expects more Information about this post.
My program looks up the post in my array and displays it.
Method 2:
User connects to my website
5 Posts become displayed
User clicks on the first Post and expects more Information about this post.
My program connects to MySQL again and fetches the post from the server.
First off, this sounds like a case of premature optimization. I would not start caching anything outside of the database until measurements prove that it's a wise thing to do. Caching takes your focus away from the core task at hand, and introduces complexity.
If you do want to keep DB results in memory, just using an array allocated in a PHP-processed HTTP request will not be sufficient. Once the page is processed, memory allocated at that scope is no longer available.
You could certainly put the results in SESSION scope. The advantage of saving some DB results in the SESSION is that you avoid DB round trips. Disadvantages include the increased complexity to program the solution, use of memory in the web server for data that may never be accessed, and increased initial load in the DB to retrieve the extra pages that may or may not every be requested by the user.
If DB performance, after measurement, really is causing you to miss your performance objectives you can use a well-proven caching system such as memcached to keep frequently accessed data in the web server's (or dedicated cache server's) memory.
Final note: You say
PHP works server-side as well
That's not accurate. PHP works server-side only.
Have you think in saving the posts in divs, and only make it visible when the user click somewhere? Here how to do that.
Put some sort of cache between your code and the database.
So your code will look like
if(isPostInCache()) {
loadPostFromCache();
} else {
loadPostFromDatabase();
}
Go for some caching system, the web is full of them. You can use memcached or a static caching you can made by yourself (i.e. save post in txt files on the server)
To me, this is a little more inefficient than making a 2nd call to the database and here is why.
The first query should only be pulling the fields you want like: title, author, date. The content of the post maybe a heavy query, so I'd exclude that (you can pull a teaser if you'd like).
Then if the user wants the details of the post, i would then query for the content with an indexed key column.
That way you're not pulling content for 5 posts that may never been seen.
If your PHP code is constantly re-connecting to the database you've configured it wrong and aren't using connection pooling properly. The execution time of a query should be a few milliseconds at most if you've got your stack properly tuned. Do not cache unless you absolutely have to.
What you're advocating here is side-stepping a serious problem. Database queries should be effortless provided your database is properly configured. Fix that issue and you won't need to go down the caching road.
Saving data from one request to the other is a broken design and if not done perfectly could lead to embarrassing data bleed situations where one user is seeing content intended for another. This is why caching is an option usually pursued after all other avenues have been exhausted.