I have a website where users each have their own profile page. Here they can upload a single image that acts as their avatar. This is the only image users can upload to the server across the whole site. There is no archive so it can be overwritten if a user wishes to update their avatar.
I have never had to do anything like this before so I would like to open it up and ask for a suitable, scalable option for this website.
My initial thought is to give each user's image a random name, a string 6-12 characters long. This way you couldn't build a script that just pulls every user's profile pic from a directory (eg 001.png, 002.png etc). My second thought is that there should be only be a certain amount of images per directory to make sure they can be retrieved quickly by the server.
There may well be other things I'm missing here, I'm not sure on exact details hence why I'm asking.
I would recommend storing the images on something like Amazon S3. Depending on how many pictures you're storing, serving images can really take a tow on your web server. S3 is scalable and with multi-zone deployments through CloudFront (Amazon's version of a CDN), you can really speed up this part of your service.
It's good idea to not overload single directory. Very often you can see that images are stored in hierarchy of folders according to theirs first few letters. An example of this is
b5dcv5.jpg -> /b/5/b5dcv5.jpg
bsgb0g.jpg -> /b/s/bsgb0g.jpg
a5dcbt.jpg -> /a/5/a5dcbt.jpg
and so on. I thing you got the principle. Advantage of this is to have access to and image in O(log N) when filenames are uniformly distributed instead of O(N) as it would be in single folder solution.
I've been using base64 to store them within an SQL database. No need to manage files. It works well for relatively low resolution options.
How about not storing them as images at all?
You could leverage an external placeholder for each user, you could cache a random image from lorempixel.com: http://lorempixel.com/100/100. Use an MD5 hash of the user's name or ID. You could also just save the image using the user's ID, for example 442.jpg.
Related
I'm implementing profile pictures in my android application. I came to a decision that I don't know which road to take. The problem is that I have to store the images that users upload and view them also to their contacts / friends, as in telegram I see the images of others and they see mine.
1) First hypothesis:
I know that MySQL can store images in a field with the "blob" (tell me if I'm wrong), so you just have to upload and download images from grabbing this field.
2) Second hypothesis:
The user loads the image, I memorize in a folder (a bit 'like a cloud), unless its path on the server in a field in the database (for example in the field photo I put "http:site/project/image.jpg ") and I download the photos to his friends in this way.
I do not know how to do, which one should I choose? The first method (to store the file directly in mysql) does not make me become the database slower and less accurate?
The application handles messages, post etc. (as in a social network), the pictures can give problems ??
The second method seems faster though less elegant. I trust in your wise advice, thanks for the reply: D
IMHO - store images as files and store only their paths to database.
I'm coding a basic gallery for a website with around 40.000 online people at any given time. Users will be able to create galleries and upload images.
My question is, should I make a seperate folder for each gallery and put the images in them, or make a single folder and put all images in it, but keep the gallery_id for each image in the database? Or, should I make a directory for every user, then another directory inside them for the gallery names?
How would you do this?
Ps. I need it to be as light as it can.
I would store them by id
and i would split them into folders (dependant of filesystem, some don't perform well with lots of files in 1 folder), plus it is easier to find them if you have to manually look at something
Give each file an id, then using the first 3 digits of the file name, split them into folders. (you could start your auto-increment counter at 100000 or zero pad the id, so there is at least 3 levels
/photos/1/0/3/103456.jpg
/photos/9/4/1/941000.jpg
/photos/0/0/0/000001.jpg
You can store the relationship of photo to user / gallery / etc in the database
Or if you want to see how the big boys do it
Needle in a haystack: efficient storage of billions of photos
Typically web servers don't want you to have more than a few thousand images in a single folder (I recently had to deal with 70,000 images causing super slow reads and sorts so trust me on this) so certainly not a single folder if you think you will have thousands of images. I would suggest the best solution would be to host off of amazon's S3 connected to their CDN CloudFront but if that isn't realistic you can still do several things just on your own server.
Make a separate folder for each gallery like you suggest only if you know some bounds on how large a gallery can get and have an idea of how many galleries will be created. (This is what I would suggest for your specific problem right now)
Put the image name through a hash function then use the first 1-3 characters of the hash to name folders to put the images into. The hash ensures that the images are roughly equally split among the folders and you can decide how many folders you need.
At any rate having the information of what gallery and the image id in the actual path will probably be useful to you moving forward both in code and whenever a human has to hunt bugs on the server. I would probably name the folders based on the gallery id and just make sure that no gallery has more than a few thousand images in it.
I store mine like this:
images/userid/photoid
This way I can quickly isolate user images if I need to inspect anything at a later date. It seems more organized than dropping them all in one central directory.
I have no idea how the big websites save the pictures on their servers. Could any one tell me how do they save the pictures that are uploaded by the users in their database?
I was thinking, maybe they would just save the file(the picture) in some path and just save that path in the databse is that right?
But I want to do it this way. Is this right? For example, a website named www.photos.com. When a user uploads a picture I would create a folder of the user name and save those pictures in that folder.
I believe we can create a directory using php file concepts. So when a new user uploads his picture or file, I want to create a directory with his name.
Example: if user name is john, I would create a directory like this on photos.com www.photos.com/john/ and then save all his pictures to this directory when he uploads a picture. Is this the right way to do this?
I have no one here that has good knowledge of saving the files to servers so please let me know how to do this? I want to do it the correct and secure way.
All big websites don't save pictures to the database they store them in the disk.
They save a reference to the picture's position in a table. And then link from there.
Why? Performance.
Pulling heavy content from a database is a huge performance bottleneck. And databases don't scale horizontally that well, so it would mean even a bigger problem. All big sites use static content farms to deal with static content such as images. That's servers who won't care less about your identity.
How do they keep the pictures really private you might ask? They don't.
The picture's link is, in itself, the address and the password. Let's take Facebook, for example. If I store a private picture on my account you should not be able to open it. But, as long as you have the correct address you can.
This picture is private. Notice the filename
10400121_87110566301_7482172_n.jpg
(facebook changes the url from time to time so the link may be broken)
It's non sequential. The only way to get the picture is to know it's address.
Based on a previous user photo you can't guess the next one.
It has a huge entropy so even if you start taking random wild guesses you'll have an extensive amount of failures and, if you do get to a picture, you won't be able to, from there, realize the owners identity which, in itself, is protection in anonymity.
Edit (why you should not store images in a "username" folder:
After your edit it became clear that you do intent to put files on disk and not on the database. This edit covers the new scenario.
Even though your logic (create a folder per user) seams more organized it creates problems when you start having many users and many pictures. Imagine that your servers have 1T disk space. And lets also imagine that 1T is more or less accurate with the load the server can handle.
Now you have 11 users, assume they start uploading at the same time and each will upload more than 100GB of files. When they reach 91GB each the server is full and you must start storing images on a different server. If that user/folder structure is followed you would have to select one of the users and migrate all of his data to a different server. Also, it makes a hard-limit on a user who can't upload more than 1T in files.
Should I store all files in the same folder, then?
No, big-sites generally store files in sequential folders (/000001/, /000002/, etc) having an x defined number of files per folder. This is mainly for file-system performance issues.
More on how many files in a directory is too many?
It is usually a bad idea to store images in your database (if your site is popular). Database is, traditionally, one of main bottlenecks in most any application out there. No need to load it more than necessary. If images are in the filesystem, many http servers (nginx, for example) will serve them most efficiently.
The biggest social network in Russia, Vkontakte does exactly this: store images in the filesystem.
Another big social network implemented a sophisticated scalable blob storage. But it's not available to the public, AFAIK.
Summary of this answer: don't store blobs in the database.
is this the right way to do
Yes.
The only thing I'd suggest to use not name but id.
www.photos.com/albums/1234/ would be okay for starter.
Image management may best be achieved by physically uploading images to the server and then recording file location and image details in a database. Subsequently, a Search Form could be configured to permit the user to do a text search, part number search, or other queries. A PHP script could be written to produce a valid HTML image tag based on data found in the table.
uploading images into a MySQL™ BLOB field is such a bad idea such image data is generally problematic if the images are much larger than thumbnails. If the images are large, you can end up having to copy/paste one SQL INSERT statement at a time (into phpMyAdmin). If the images are large and the SQL INSERT statement is broken into two lines by your text editor, you'll never be able to restore the image.
I am planning to do a photo album website, So each user may upload as many number of images. What is the best way to keep track of images for an individual user. What should be the server configuration to handle this part.
-Lokesh
Depending on the amount of images, you will probably want to store them on a static domain. Then, have a table in whatever database you are using to store the paths to each of the images for each user.
Well like many design topics there are lots of different ways to go about it. Two ways that come to mind right now are as follows.
you could simply have a directory created on the server for each user and then have the images each use uploads saved into that directory. Ofcourse you'd want to make sure they didn't over write any existing images with images of the same name. You could do this by warning them about conflicting names or by adding some sort of noce string (like a time stamp) to the end of of the file name. This is a pretty straight forward solution and means that you can login to your server and see all the images each user has uploaded right there for you to do anything you like with.
Another idea would be to save the images in a database. This can be done by serializing the images to a string and storing it in a database. This is nice becaues it means you don't have to worry about handling directories and duplicate file names. You will have to deserialize each image when you want to display it which will put your DB under load so for a very high traffic volume site this might not really be the way to go.
There are ofcourse combinations of these ideas and many others. It really comes down to working out which solution best fits your exact needs.
which is a better place to upload images to? A database or in the web directory? And why?
You should only store images in your database if you have a specific need to, like security, or like an absolute to-die-for need to keep all custom data in a database.
Other than that, getting large files into databases usually isn't worth the trouble. Storing and retrieving the file get that much more complicated to implement, and database updates/upgrades/conversions have that many more things that can go wrong.
I don't see that there is an advantage storing images in a database. There is certainly no inherent security in this. Files are for the filesystem so store your images in there.
I don't think you can "upload" an image to a database. You can store the image's string value in the database and stream it via "header("Content-Type")" later on. That saves space in your web server, but obviously takes space on your database.
If I were you, I'd upload to a web directory, that way you have the image for a regular URL request later on. If you don't have it in a regular directory, you'll have to connect to the database every time the image is requested, and stream it then.
Well It depends on your requirement.
If you are considering security as a major issue then definitely you should store it in db other wise nothing will leads you to store images in db.
Also retieving images from database is quite complicated as in database images are stored as binary data. So if you have specific need then only store images in database other wise storing images in directory would be fine.
As you can see there are many reasons why to use/why not to use the database for image storage. Personally I prefer not to use the database for storage of files (images, documents etc), except when I'm ordered to store them.
-Sometimes you're tired and screw up a query, something like "SELECT * FROM images", this will kill the server if there are too many images with huge size (2MB and more) in the database.
-The security issue: you can still save the files in the disk and still be secure, how? Well save the files outside the web directory, whenever the file is requested read the file and give it to the user.
-If by any chance you are using MySQL: if your database has got to big (say 2-3 GB), and you are using a shared hosting, well good luck making that backup or trying to restore that image database.
It's just my point of view