We are building a web app which will have a lot of images being uploaded. Which is the best Solution to optimize these images and store it in on the website ?
And also is there a way i can also auto enhance the images which are being uploaded ?
Do not store images in DB, store them in file system (as real files). You'll probably need to store information about them in DB though, e.g., filename, time of upload, size, owner etc.
Filenames must be unique. You might use yyyymmddhhiissnnnn, where yyyymmdd is year, month and date, hhiiss - hour, minutes and seconds, nnnn - number of image in that second, i.e., 0001 for first image, 0002 for second image etc. This will give you unique filenames with fine ordering.
Think about making some logic directory structure. Storing millions of images in single folder is not a good idea, so you will need to have something like images/<x>/<y>/<z>/<filename>. This could also be spanned across multiple servers.
Keep original images. You never know what you will want to do with images after year or two. You can convert them to some common format though, i.e., if you allow to upload JPG, PNG and other formats, you might store all of them as JPG.
Create and store all kinds of resized images that are necessary in your website. For example, social networks often have 3 kinds of resized images - one for displaying with user comments in various places (very small), one for displaying in profile page (quite small, but not like icon; might be some 240x320 pixels) and one for "full size" viewing (often smaller than original). Filenames of these related images should be similar to filenames of original images, e.g., suffixes _icon, _profile and _full might be added to filenames of original images. Depending on your resources and the amount of images that are being uploaded at the same time, you can do this either in realtime (in the same HTTP request) or use some background processing (cron job that continuously checks if there are new images to be converted).
As for auto enhancing images - it is possible, but only if you know exactly what must be done with images. I think that analyzing every image and deciding what should be done with it might be too complex and take too much resources.
All good suggestions from binaryLV above. Adding to his suggestion #5, you also probably want to optimize the thumbnails you create. When images are uploaded, they are likely to have metadata that is unnecessary for the thumbnails to have. You can losslessly remove the metadata to make the thumbnail sizes smaller, as suggested here: http://code.google.com/speed/page-speed/docs/payload.html#CompressImages. I personally use jpegtran on the images for my website to automatically optimize my thumbnails whenever they are created. If you ever need the metadata, you can get it from the original image.
Something else to consider if you plan to display these images for users is to host your images on a cookie-free domain or sub-domain as mentioned here: http://developer.yahoo.com/performance/rules.html#cookie_free. If the images are hosted on a domain or sub-domain that has cookies, then every image will send along an unnecessary cookie. It can save a few KB per image requested, which can add up to a decent amount, especially on restricted bandwidth connection such as on a mobile device.
Related
I am working with a project. And there is feature user can upload image. That image will be used in different pages of website, with different sizes(eg: 200*300, 360*520, 700*1000).
I can create thumbnail two ways
while uploading image, create thumbnail with different size and store.
While displaying image src to some server side script, re-size image from there and print image, instead of displaying.
Which is the correct way to do? If I use 1st method, I think disk space will get full very fast, Is there any issue with 2nd method?
The advantage of method 1 is that you won't risk resizing the same image twice simultaneously, and that you can provide quickly a version to display to your user.
But why would your disk get full fast if you size the images beforehand? That would happen if you re-size them to every possible size, so that's not a good idea.
Method 2 is more flexible. You just ask for an image in a given size and your script will produce it on the fly. This is a bit slower, of course, but you could cache the resized image so visitors will get the images fast, unless they are the first one to request an image in a specific size.
My suggestion:
If you know which sizes you use on your website, you could use those sizes to resize the images in an early stage. You could even make a configuration on your website with a bunch of predefined image dimensions, which you can use on your website, and use those same configurations to scale the images when you upload them. This has some advantages:
Having a limited set of sizes will increase the chances of hitting the cache when visitors browse through your website. For instance, if you show a picture of X in the X detail page, and overview page includeing X, search results etcetera, and each of those pages uses a slightly different size, it is a waste of disk space and bandwidth.
And, if disk size is an issue, having a limited number of sizes also limits the disk space that the cached versions of these images consume.
Moreover, if you change or add a dimension, you could pregenerate all images for that size immediately, so visitors would benefit right away from the caches version.
And of course, it also makes it easier to purge the cache for images of a dimension that is no longer in the list, should you remove or change one.
If you choose this method, it makes it very easy to implement method 1 and pre-cache everything, but even if you would choose method 2 (if only as a fallback, should a cached version not exist), then still this would have benefits.
I have a website where users each have their own profile page. Here they can upload a single image that acts as their avatar. This is the only image users can upload to the server across the whole site. There is no archive so it can be overwritten if a user wishes to update their avatar.
I have never had to do anything like this before so I would like to open it up and ask for a suitable, scalable option for this website.
My initial thought is to give each user's image a random name, a string 6-12 characters long. This way you couldn't build a script that just pulls every user's profile pic from a directory (eg 001.png, 002.png etc). My second thought is that there should be only be a certain amount of images per directory to make sure they can be retrieved quickly by the server.
There may well be other things I'm missing here, I'm not sure on exact details hence why I'm asking.
I would recommend storing the images on something like Amazon S3. Depending on how many pictures you're storing, serving images can really take a tow on your web server. S3 is scalable and with multi-zone deployments through CloudFront (Amazon's version of a CDN), you can really speed up this part of your service.
It's good idea to not overload single directory. Very often you can see that images are stored in hierarchy of folders according to theirs first few letters. An example of this is
b5dcv5.jpg -> /b/5/b5dcv5.jpg
bsgb0g.jpg -> /b/s/bsgb0g.jpg
a5dcbt.jpg -> /a/5/a5dcbt.jpg
and so on. I thing you got the principle. Advantage of this is to have access to and image in O(log N) when filenames are uniformly distributed instead of O(N) as it would be in single folder solution.
I've been using base64 to store them within an SQL database. No need to manage files. It works well for relatively low resolution options.
How about not storing them as images at all?
You could leverage an external placeholder for each user, you could cache a random image from lorempixel.com: http://lorempixel.com/100/100. Use an MD5 hash of the user's name or ID. You could also just save the image using the user's ID, for example 442.jpg.
This question already has answers here:
PHP image resize on the fly vs storing resized images
(4 answers)
Closed 9 years ago.
Problem - I wanted to set up an image-uploading feature in my website. But I wanted to show both- the original image and a small thumbnail of the image.
Choices - Which way is better - to create a separate image (thumbnail) in a directory when the image is uploaded or to show a smaller version by reducing its height and width in the fixed ratio every time the image is requested?
How I am doing it currently - The later one sounds better to me because it won't be taking much size on the disk but it has to resize the image again and again. Which one do you think is better?
This is a general question for web application, no language in specific.
Any idea how facebook or google do it?
Question - My question is how to generate thumbnails and show them on a website - by creating a copy of the original image with smaller dimension or by generating the thumbnail dynamically every time it is requested.
Creating the thumbnail on upload is almost always the better option. If storage is a concern you could convert them on the request and then cache the result in a memory store. If its requested again before the cache expires, no conversion will be needed.
Storage is often pretty cheap, so I would probably not go for that extra complexity.
Just create a thumbnail version and save to disk. Hard disk space is very cheap. £70 for a couple of TB.
"Better" depends on the criteria you set.
For most applications, disk space is not an issue - and if storing a thumbnail is a problem, storing the original must be a huge concern - a decent digital camera photo will run to many megabytes, whereas the thumbnail should not exceed 50K.
Bandwidth and performance (as perceived by the client) are usually bigger concerns. If you have lots of people browsing a gallery of image thumbnails, serving 50Kb thumbnails will be significantly faster (and cheaper in bandwidth) than serving multi-megabyte high resolution images.
In addition, by serving thumbnails on a URL like <img src="images/thumbnail/foobar.jpg"> and setting appropriate cache headers, you should get a lot of downstream caching - this is less likely if you serve the image as <img src="thumbnail.php?image=image/foobar.jpg> because caches treat querystrings rather conservatively.
I used to work on a web site that managed hundreds of thousands of product images; we set up ImageMagick to create thumbnails automatically. Depending on your setup, it may make sense to do this when the thumbnail is first requested, rather than when the file is uploaded, because the conversion can be fairly resource hungry, and doing it at upload time would take longer than we wanted to wait. Modern hardware may make that a non-issue.
There's also a question about keeping the thumbnails in sync with the originals - if the user uploads a new image, you have to ensure you get the thumbnail updated; if the original is deleted, you must also delete the thumbnail.
Creating a thumbnail is a better option and it doesn't cost much disk space. Your client will also load smaller size when opening your pages. converting the image upon request will cost event more time to load your page ;)
If you take a look at most CMS with this built in functionality they nearly always create a thumbnail image of the image on upload and store it on the server.
This goes back to the age old saying of "do what google does" but with CMS.
I'm planning to develop an area where the users can upload pictures. I know how to upload a picture on the server using PHP but the problem is what is the best practice to develop a performing system.
The idea is to display in different pages thumbs and I would like to know if it's a better idea to save two different images (thumb + original) on the server or if it's better to save just the original and create all the thumbs on the fly. Thumb + original means more space on the server, whereas the option "thumbs on the fly" means most likely a server overload.
I found couple of good scripts to resize and cropping on the fly but not sure if it's a good idea to use especially if the web site has few thousands visitor per day (or may be more in the future just to be optimistic/pessimistic).
Absolutely generate and save the thumbnails on disk. Storage is cheap.
You can generate some thumbnails and save them on disk but in the long term that's problematic due to different devices needing different sizes, different formats, etc.
If you are already saving the uploaded images on S3, Azure Storage, or Google Cloud I recommend to use some on the fly image processing service like imglab or cloudinary.
With these services you can generate many different types of cropping, and serving them in different (modern) formats like WebP or AVIF so you don't need to generate them before hand. SEO will be improved wit this option too.
Additionaly images will be behind a global CDN so users will get the images in a fast way independent or their location.
I have no idea how the big websites save the pictures on their servers. Could any one tell me how do they save the pictures that are uploaded by the users in their database?
I was thinking, maybe they would just save the file(the picture) in some path and just save that path in the databse is that right?
But I want to do it this way. Is this right? For example, a website named www.photos.com. When a user uploads a picture I would create a folder of the user name and save those pictures in that folder.
I believe we can create a directory using php file concepts. So when a new user uploads his picture or file, I want to create a directory with his name.
Example: if user name is john, I would create a directory like this on photos.com www.photos.com/john/ and then save all his pictures to this directory when he uploads a picture. Is this the right way to do this?
I have no one here that has good knowledge of saving the files to servers so please let me know how to do this? I want to do it the correct and secure way.
All big websites don't save pictures to the database they store them in the disk.
They save a reference to the picture's position in a table. And then link from there.
Why? Performance.
Pulling heavy content from a database is a huge performance bottleneck. And databases don't scale horizontally that well, so it would mean even a bigger problem. All big sites use static content farms to deal with static content such as images. That's servers who won't care less about your identity.
How do they keep the pictures really private you might ask? They don't.
The picture's link is, in itself, the address and the password. Let's take Facebook, for example. If I store a private picture on my account you should not be able to open it. But, as long as you have the correct address you can.
This picture is private. Notice the filename
10400121_87110566301_7482172_n.jpg
(facebook changes the url from time to time so the link may be broken)
It's non sequential. The only way to get the picture is to know it's address.
Based on a previous user photo you can't guess the next one.
It has a huge entropy so even if you start taking random wild guesses you'll have an extensive amount of failures and, if you do get to a picture, you won't be able to, from there, realize the owners identity which, in itself, is protection in anonymity.
Edit (why you should not store images in a "username" folder:
After your edit it became clear that you do intent to put files on disk and not on the database. This edit covers the new scenario.
Even though your logic (create a folder per user) seams more organized it creates problems when you start having many users and many pictures. Imagine that your servers have 1T disk space. And lets also imagine that 1T is more or less accurate with the load the server can handle.
Now you have 11 users, assume they start uploading at the same time and each will upload more than 100GB of files. When they reach 91GB each the server is full and you must start storing images on a different server. If that user/folder structure is followed you would have to select one of the users and migrate all of his data to a different server. Also, it makes a hard-limit on a user who can't upload more than 1T in files.
Should I store all files in the same folder, then?
No, big-sites generally store files in sequential folders (/000001/, /000002/, etc) having an x defined number of files per folder. This is mainly for file-system performance issues.
More on how many files in a directory is too many?
It is usually a bad idea to store images in your database (if your site is popular). Database is, traditionally, one of main bottlenecks in most any application out there. No need to load it more than necessary. If images are in the filesystem, many http servers (nginx, for example) will serve them most efficiently.
The biggest social network in Russia, Vkontakte does exactly this: store images in the filesystem.
Another big social network implemented a sophisticated scalable blob storage. But it's not available to the public, AFAIK.
Summary of this answer: don't store blobs in the database.
is this the right way to do
Yes.
The only thing I'd suggest to use not name but id.
www.photos.com/albums/1234/ would be okay for starter.
Image management may best be achieved by physically uploading images to the server and then recording file location and image details in a database. Subsequently, a Search Form could be configured to permit the user to do a text search, part number search, or other queries. A PHP script could be written to produce a valid HTML image tag based on data found in the table.
uploading images into a MySQL™ BLOB field is such a bad idea such image data is generally problematic if the images are much larger than thumbnails. If the images are large, you can end up having to copy/paste one SQL INSERT statement at a time (into phpMyAdmin). If the images are large and the SQL INSERT statement is broken into two lines by your text editor, you'll never be able to restore the image.