I have a social network where users can create posts with embedded images. The images are stored outside the document root. When a page is requested for viewing I want to send a normal HTML response with <img> elements that have an src attribute. The problem is, how to efficiently protect the images so that only authorised users can view them?
The permissions system behind the network is quite complex. The social graph has several different types of nodes and edges, and a user's permission to browse content depends on this graph. I consult the user's subgraph once on each page request, to retrieve a list of permitted content, but I don't want to have to consult it again for each image. It's an expensive exercise and some pages may have 50 or more images.
So, as far as I can see, when I serve the page I will need to send a key (or keys) which, when combined with data by the server, will 'unlock' the image. Here's an example of what I mean - in the example I am putting a key in an url, where the url is the src attribute of an image:
http://example.com/images?id=1234_my_image_id&key=1234123412341234FFF
In this implementation I pass a parameter that uniquely identifies the image. I need to do this since I don't want to get in to translation tables, or user-specific translation tables (because of complexity and performance). I also send an 'image key'.
My idea is that the user's session will include a 'session key'. When the server receives a GET for an image, it will add the session key to the image filename requested and hash them (just like password + salt). If the result of the hash equals the image key, I'll retrieve the image from disk and send it.
The problem with this is that if a hacker gets hold of an image url, they have one input to the hash (ie. the image key), and the required output of the hash (ie. the image file name), so they could (using rainbow tables) work backwards to get a list of potential session keys. Then, through a series of requests and a process of elimination, they could establish the exact session key.
So, how do I get around this? I could change the session key with each request, but that would mean that image urls would change with each request, so there will be no image caching.
I'm sure I'm not the first to have this problem. Is there a 'common' approach to the problem? (I can't find such a thing). I could consider base64 encoding the images in the page response, but I'm guessing that would come at a performance cost.
FYI - I'm using PHP and Apache. I'd use url rewriting to tidy things up a bit.
Well I was thinking the same as you did. So get the image id + an user secret, make a hash et voilâ. You got an easy to use system.
But the problems you got is mostly rainbow tables. So possibilities to tackle that issue;
bcrypt and do it like a realy password would be
make it a time sensitive secret, so the secret is only valid for 30 seconds, which could be as simple as hash(hash(secret + time) + secret), although now you have to send the time it was hashed with it too, so not much of a secret there any more
Other solutions that come to my mind
save with the images a list of allowed users, on changes of rights + upload update this list
you could make a function displayImage(id), what it does it renders a <img src="url" /> but also in the background it sets a flag for x seconds to allow the user to access the image.
Once you store your images outside of the root directory, then the only way to view them is to have the code access them. In PHP, that code will look something like...
$file_location = '../../images/' . $some_id_that_is_authenticated_and_cleansed_for_slashes . '.jpg';
readfile($file_location); // grabs file and shows it to user
Related
I have a website where users each have their own profile page. Here they can upload a single image that acts as their avatar. This is the only image users can upload to the server across the whole site. There is no archive so it can be overwritten if a user wishes to update their avatar.
I have never had to do anything like this before so I would like to open it up and ask for a suitable, scalable option for this website.
My initial thought is to give each user's image a random name, a string 6-12 characters long. This way you couldn't build a script that just pulls every user's profile pic from a directory (eg 001.png, 002.png etc). My second thought is that there should be only be a certain amount of images per directory to make sure they can be retrieved quickly by the server.
There may well be other things I'm missing here, I'm not sure on exact details hence why I'm asking.
I would recommend storing the images on something like Amazon S3. Depending on how many pictures you're storing, serving images can really take a tow on your web server. S3 is scalable and with multi-zone deployments through CloudFront (Amazon's version of a CDN), you can really speed up this part of your service.
It's good idea to not overload single directory. Very often you can see that images are stored in hierarchy of folders according to theirs first few letters. An example of this is
b5dcv5.jpg -> /b/5/b5dcv5.jpg
bsgb0g.jpg -> /b/s/bsgb0g.jpg
a5dcbt.jpg -> /a/5/a5dcbt.jpg
and so on. I thing you got the principle. Advantage of this is to have access to and image in O(log N) when filenames are uniformly distributed instead of O(N) as it would be in single folder solution.
I've been using base64 to store them within an SQL database. No need to manage files. It works well for relatively low resolution options.
How about not storing them as images at all?
You could leverage an external placeholder for each user, you could cache a random image from lorempixel.com: http://lorempixel.com/100/100. Use an MD5 hash of the user's name or ID. You could also just save the image using the user's ID, for example 442.jpg.
I have uploaded some files on server. The link provided to me is pretty simple i-e; no signs/ symbols etc. If I provide the same link to the user for downloading the data, it might result in hacking of my server or loss of data. Now my question is how to encrypt this kind
www.hello.com/myApp/myFile.mp3
of url and provide the encrypted url to the user which the browser can understand.
Regards
Correct me if I misunderstand, but are you trying to prevent someone from downloading the file unless you tell them it's ok to download it?
Then the threat is that someone may find the file linked on a search engine or be able to guess it.
There are a few ways to make that threat less likely.
Make the url very long and unguessable. Simply rename the file to some random value could work. From the command line (linux)
echo http://example.com/file.mpe $(date) | md5sum
d8a5e8d341135379b8ad38f1d06970be
Or even easier, choose a random password from http://tooln.net/pw/ and rename the file to one of the passwords without symbols. Either is difficult to guess.
If you know the person, you could easily share a password and set a password on the directory. You can turn on passwords per directory through apache.
Turn off indexing of the site through robots.txt.
A URL can be encrypted, but if a browser can understand it, decrypting it would be a trivial process for a hacker. I'm assuming what you want to do is to prevent too many people from accessing your URL. To do this, you will have to have either some sort of user login system or an IP based limitation. Both of these would have to be backed by a database.
Instead of linking directly to the file, you would link to something like download.php?fileid=$some_file_id and in your database, you just insert the user ID (or IP address) and file ID every time the file is download. Then to display the file back to the user, you would check how many downloads of that file have been made by the user and if it is less than your threshold, e.g.:
SELECT COUNT(*) FROM downloads WHERE user = :user AND file_id = :id
Then get PHP to echo the contents of the file to the browser.
Add other clauses such as limiting it to X downloads in the past 24 hours, etc. or however you would like to work it.
Other things you could do would be storing the files outside the document root (or protecting direct access with .htaccess or similar), and including a hash of the file name in the link, so someone couldn't just do download.php?fileid=1 and guess the next one is download.php?fileid=2.
I'm developping an App in Android which somehow has avatars like Whatsapp do. As you know, in WhatsApp you can create a group, and set a group picture for it.
I don't have any problems on taking the image, saving, etc. The problem I have is that I'm developing the webservice in Symfony2 (PHP) and I want to receive the image and save it somewhere on the server. However, obviously those images are NOT public and should be only viewed for users with permissions. I've thought about traditional method: saving the image on a folder and giving the link or not, but this is totally easy to hack.
So guys, how would you do this? Maybe saving the binary data into MySql directly? Is there any clean way to achieve this?
Any tips are appreciated.
Thanks.
Another answer is to set the mime type of the PHP call to be an image. A call to a URL like http://xxx/images.php?id=8989031289130 would then return an image instead of an HTML file.
You then have access to the PHP security context and can validate whether the user actually has permissions to view this file.
There are some more details at:
Setting Mime type in PHP
The typical answer here is to use a file naming scheme that precludes guessing. For example, you could take the filename plus a secret salt, hash them together, and append the hash to the filename (before the extension). Thus, what would be /foo/bar/baz.jpg would become /foo/bar/baz_8843d7f92416211de9ebb963ff4ce28125932878.jpg.
So long as your hash salt remains secret, filenames are more or less mathematically protected from random or brute-force discovery. This is, for example, the core of how Facebook protects its' users pictures without having to actually require authentication for each image request (which doesn't scale well at all).
This is theoretical question.
Twitter keeps user profile images as following :
https://twimg0-a.akamaihd.net/profile_images/2044921128/finals_normal.png
It's impossible to imagine that they have a server which contains 2044921128 directories (for example).Maybe this URL is created using mod_rewrite?
So how to store an extremely large number of user images?
How to complete this scheme:
User chooses and PHP script uploads an image that's supposed to be his profile picture.
PHP script renames it, sets the PATH to store this image, moves it and finally adds this path to database for further use.
So how PATH must look like?
Nothing says that Akamai (which stores the pictures for Twitter based on your URL) actually stores the files in a directory structure. It's entirely possible that they are stored in memory (backed by say a directory structure), in a database (SQL / NoSQL) or any other storage mechanism that Akamai finds efficient.
You can route all requests for a URL that start with
https://yourService.com/profile_images/
to a PHP script of your choice that then parses the rest of the URL to determine which image is being requested and store/retrieve from whatever storage mechanism you want (perhaps a database) based on the parsed URL.
Here's a short blog post that shows one method of doing that using mod_rewrite
http://www.phpaddiction.com/tags/axial/url-routing-with-php-part-one/
Most OS-es discourage having more than 1024 directories/files within a single directory as any number above that slowly makes scanning and locating specific resources within it slower, so I think it is safe to think that akamai would not be having 2044921128 directories within profile_images!
Either it is a special unique identifier number generated within profile_images or one of the numerous ways in with url routing can be used to locate a resource. In any case, I do not think it would correspond to the number of directories..
I have no idea how the big websites save the pictures on their servers. Could any one tell me how do they save the pictures that are uploaded by the users in their database?
I was thinking, maybe they would just save the file(the picture) in some path and just save that path in the databse is that right?
But I want to do it this way. Is this right? For example, a website named www.photos.com. When a user uploads a picture I would create a folder of the user name and save those pictures in that folder.
I believe we can create a directory using php file concepts. So when a new user uploads his picture or file, I want to create a directory with his name.
Example: if user name is john, I would create a directory like this on photos.com www.photos.com/john/ and then save all his pictures to this directory when he uploads a picture. Is this the right way to do this?
I have no one here that has good knowledge of saving the files to servers so please let me know how to do this? I want to do it the correct and secure way.
All big websites don't save pictures to the database they store them in the disk.
They save a reference to the picture's position in a table. And then link from there.
Why? Performance.
Pulling heavy content from a database is a huge performance bottleneck. And databases don't scale horizontally that well, so it would mean even a bigger problem. All big sites use static content farms to deal with static content such as images. That's servers who won't care less about your identity.
How do they keep the pictures really private you might ask? They don't.
The picture's link is, in itself, the address and the password. Let's take Facebook, for example. If I store a private picture on my account you should not be able to open it. But, as long as you have the correct address you can.
This picture is private. Notice the filename
10400121_87110566301_7482172_n.jpg
(facebook changes the url from time to time so the link may be broken)
It's non sequential. The only way to get the picture is to know it's address.
Based on a previous user photo you can't guess the next one.
It has a huge entropy so even if you start taking random wild guesses you'll have an extensive amount of failures and, if you do get to a picture, you won't be able to, from there, realize the owners identity which, in itself, is protection in anonymity.
Edit (why you should not store images in a "username" folder:
After your edit it became clear that you do intent to put files on disk and not on the database. This edit covers the new scenario.
Even though your logic (create a folder per user) seams more organized it creates problems when you start having many users and many pictures. Imagine that your servers have 1T disk space. And lets also imagine that 1T is more or less accurate with the load the server can handle.
Now you have 11 users, assume they start uploading at the same time and each will upload more than 100GB of files. When they reach 91GB each the server is full and you must start storing images on a different server. If that user/folder structure is followed you would have to select one of the users and migrate all of his data to a different server. Also, it makes a hard-limit on a user who can't upload more than 1T in files.
Should I store all files in the same folder, then?
No, big-sites generally store files in sequential folders (/000001/, /000002/, etc) having an x defined number of files per folder. This is mainly for file-system performance issues.
More on how many files in a directory is too many?
It is usually a bad idea to store images in your database (if your site is popular). Database is, traditionally, one of main bottlenecks in most any application out there. No need to load it more than necessary. If images are in the filesystem, many http servers (nginx, for example) will serve them most efficiently.
The biggest social network in Russia, Vkontakte does exactly this: store images in the filesystem.
Another big social network implemented a sophisticated scalable blob storage. But it's not available to the public, AFAIK.
Summary of this answer: don't store blobs in the database.
is this the right way to do
Yes.
The only thing I'd suggest to use not name but id.
www.photos.com/albums/1234/ would be okay for starter.
Image management may best be achieved by physically uploading images to the server and then recording file location and image details in a database. Subsequently, a Search Form could be configured to permit the user to do a text search, part number search, or other queries. A PHP script could be written to produce a valid HTML image tag based on data found in the table.
uploading images into a MySQL™ BLOB field is such a bad idea such image data is generally problematic if the images are much larger than thumbnails. If the images are large, you can end up having to copy/paste one SQL INSERT statement at a time (into phpMyAdmin). If the images are large and the SQL INSERT statement is broken into two lines by your text editor, you'll never be able to restore the image.