I am implementing a website which allows users to upload their files. I wonder how the upload folder should be structured so that I can manage the uploaded files easily in the future. Should I provide different users with different sub-folders (for example, upload/user1 for user1, upload/user2 for user2, etc.) or should I just put them all in the same folder? And should I rename the uploaded files? (I know that a lot of websites do this!)
If you won't be using BLOB, and the users may have an arbitrary number of images, then I think you should be using the subfolders. For instance, you could create a subfolder named after the user's id. One of the reasons for this is that when the folder contains A LOT of files, the access times increase as it takes more time to find the file with specified name. You could also rename the files when they're stored just so that there would be no chance of their names overlapping. Just don't forget to store the newly-generated names in the DB.
Renaming them can be useful if you ever want to "hide" them to anyone not logged in, for example. Just store their original name in a DB.
Also, you can always store it all in a SQL table in a BLOB type.. Can be several GB and lets you relate a filename, username, whatever else.
Related
So I have my own webserver now and am hosting a website. I have made a basic screenshot to FTP. It copies the link once uploaded and I'm going to be adding randomized folders e.g /push/eqw8/woeqwe.jpg like puush does.
However, this is not my question. I was wondering how I would be able to grab all folders and images from those folders and create a table like this http://puu.sh/oEyfP/783483492c.png or http://puush.me/account in order from upload date. And then it shows in oderder of date uploaded.
I just wanted to do this because I feel more secure and its a fun project doing it.
How would I get about making this.
can you tell me like what I need to do in words?
I'm not good with web development and only good with languages like c# and such.
Learning php and html atm and its pretty basic but I still need time to learn.
At first it depends on your php skills about how much you can think / do in php.
The main thing you need here to order the files / folders by date or manage them at much professional level , you have to use Mysql database and store Files and Folders information inside the database.
For example , let say you have a Image file extension .jpeg then store it's information about :
Extension File size Parent Folder Uploader id / IP Upload Date Unique random key mime type Random name Original name
And there are many other things you need to consider
Files that are being stored by users should be scanned and non executable from the browser.
Store them in random folders with random names so no body can find the way to the file.
You can use javascript/ ajax for uploading multiple files at the same time.
You can make the files non executable by removing their extensions and saving them inside a folder without their original names and save their original names and extensions inside the database along with random string name and the folder name where it was stored so your system can identify the file and you can show your users the file and then on download you can replace the name with original one along with extension.
this is a one of my projects i have created , if you want you can check it out: Buckty
About any other information you need , you can ask me below in comment.
I have no idea how the big websites save the pictures on their servers. Could any one tell me how do they save the pictures that are uploaded by the users in their database?
I was thinking, maybe they would just save the file(the picture) in some path and just save that path in the databse is that right?
But I want to do it this way. Is this right? For example, a website named www.photos.com. When a user uploads a picture I would create a folder of the user name and save those pictures in that folder.
I believe we can create a directory using php file concepts. So when a new user uploads his picture or file, I want to create a directory with his name.
Example: if user name is john, I would create a directory like this on photos.com www.photos.com/john/ and then save all his pictures to this directory when he uploads a picture. Is this the right way to do this?
I have no one here that has good knowledge of saving the files to servers so please let me know how to do this? I want to do it the correct and secure way.
All big websites don't save pictures to the database they store them in the disk.
They save a reference to the picture's position in a table. And then link from there.
Why? Performance.
Pulling heavy content from a database is a huge performance bottleneck. And databases don't scale horizontally that well, so it would mean even a bigger problem. All big sites use static content farms to deal with static content such as images. That's servers who won't care less about your identity.
How do they keep the pictures really private you might ask? They don't.
The picture's link is, in itself, the address and the password. Let's take Facebook, for example. If I store a private picture on my account you should not be able to open it. But, as long as you have the correct address you can.
This picture is private. Notice the filename
10400121_87110566301_7482172_n.jpg
(facebook changes the url from time to time so the link may be broken)
It's non sequential. The only way to get the picture is to know it's address.
Based on a previous user photo you can't guess the next one.
It has a huge entropy so even if you start taking random wild guesses you'll have an extensive amount of failures and, if you do get to a picture, you won't be able to, from there, realize the owners identity which, in itself, is protection in anonymity.
Edit (why you should not store images in a "username" folder:
After your edit it became clear that you do intent to put files on disk and not on the database. This edit covers the new scenario.
Even though your logic (create a folder per user) seams more organized it creates problems when you start having many users and many pictures. Imagine that your servers have 1T disk space. And lets also imagine that 1T is more or less accurate with the load the server can handle.
Now you have 11 users, assume they start uploading at the same time and each will upload more than 100GB of files. When they reach 91GB each the server is full and you must start storing images on a different server. If that user/folder structure is followed you would have to select one of the users and migrate all of his data to a different server. Also, it makes a hard-limit on a user who can't upload more than 1T in files.
Should I store all files in the same folder, then?
No, big-sites generally store files in sequential folders (/000001/, /000002/, etc) having an x defined number of files per folder. This is mainly for file-system performance issues.
More on how many files in a directory is too many?
It is usually a bad idea to store images in your database (if your site is popular). Database is, traditionally, one of main bottlenecks in most any application out there. No need to load it more than necessary. If images are in the filesystem, many http servers (nginx, for example) will serve them most efficiently.
The biggest social network in Russia, Vkontakte does exactly this: store images in the filesystem.
Another big social network implemented a sophisticated scalable blob storage. But it's not available to the public, AFAIK.
Summary of this answer: don't store blobs in the database.
is this the right way to do
Yes.
The only thing I'd suggest to use not name but id.
www.photos.com/albums/1234/ would be okay for starter.
Image management may best be achieved by physically uploading images to the server and then recording file location and image details in a database. Subsequently, a Search Form could be configured to permit the user to do a text search, part number search, or other queries. A PHP script could be written to produce a valid HTML image tag based on data found in the table.
uploading images into a MySQL™ BLOB field is such a bad idea such image data is generally problematic if the images are much larger than thumbnails. If the images are large, you can end up having to copy/paste one SQL INSERT statement at a time (into phpMyAdmin). If the images are large and the SQL INSERT statement is broken into two lines by your text editor, you'll never be able to restore the image.
What are some ideas out there for storing images on web servers. Im Interacting with PHP and MySQL for the application.
Question 1
Do we change the name of the physical file to a000000001.jpg and store it in a base directory or keep the user's unmanaged file name, i.e 'Justin Beiber Found dead.jpg'? For example
wwroot/imgdir/a0000001.jpg
and all meta data in a database, such as FileName and ReadableName, Size, Location, etc.
I need to make a custom Filemanager and just weighing out some pros and cons of the underlying stucture of how to store the images.
Question 2
How would I secure an Image from being downloaded if my app/database has not set it to be published/public?
In my app I can publish images, or secure them from download, if I stored the image in a db table I could store it as a BLOB and using php prevent the user from downloading it. I want to be able to do the same with the image if it was store in the FileSystem, but im not sure if this is possible with PHP and Files in the system.
Keeping relevant file names can be good for SEO, but you must also make sure you don't duplicate.
In all cases I would rename files to lowercase and replace spaces by underscores (or hyphens)
Justin Beiber Found dead.jpg => justin_beiber_finally_dead.jpg
If the photo's belongs to an article or something specific you can perhaps add the article ID to the image, i.e. 123_justin_beiber_found_dead.jpg. Alternatively you can store the images in an article specific folder, i.e. /images/123/justin_beiber_found_dead.jpg.
Naming the files like a0000001 removes all relevance to the files and adds no value whatsoever.
Store (full) filepaths only in the database.
For part 2;
I'm not sure what the best solution here is, but using the filesystem, I think you will have to configure apache to serve all files in a particular directory by PHP. In PHP you can then check if the file can be published and then spit it out. If not, you can serve a dummy image. This however is not very efficient and will be much heavier on apache.
I am making the small finance management website in php. I have to store quite a few images for my application but i am confused where should i store them. My Scenario is
The user will have
Saving Account
Fixed Deposit
Now for Every Fixed Deposit i need to store the
scanned image of that FD
one or more images for source of money like Bank Cheques
Now where should i store all those images. I mean
Do I need to create folder for each user then each FD and then sources
Or All images in one folder
Don't store all the images in one folder; you'll have issues once you have thousands of files in the same folder. It would make sense and divide up nicely to create a subfolder for each user.
Don't put the folder in a web accessible directory. Instead have an intermediary php script that accepts some parameters, checks whether the requesting user has privileges to view the file, and then use readfile() to return the image. That way you prevent people from viewing other people's images.
Don't use the file name of the uploaded image. Instead you could use the id of database row as the name, or create a UUID; something unique and generated by you. That way you don't have to worry about users uploading files with conflicting names or attempting to hack your system with crazy file names.
I would definitely have a folder (user ID) per user, so you won't have to parse a giant folder of images if you're trying to load one user's images. Beyond that, I think you could just store them in a single folder and use a naming convention or group all images in a sub-folder in a way that makes the retrieval of those images easier.
I have a form where an admin will upload three pictures with different dimensions to three different designated directories. now to make sure that i don't get into the problem of duplicate file names i implemented something like the php will compare the uploaded file name and it will check if that file name exist in the designated directory if yes then it will echo an error and stop the script execution.
Now one of my friend suggested me that it is very bad asking the admin to manually rename the picture file and asking them to take care of the file duplication problem. the solution he suggested was to rename the file automatically and then store it in the database and then direct it to directory.
I am confused about what combination should i give to the renamed file and also make sure it will remain unique file name to be more precise i would like you to understand my directory structure
as i said there will be three pictures files the admin will be uploading namely
a) Title Picture b) Brief Picture c)
Detail Picture
and all the three picture files will be moved to the different respective directory, like title picture goes to title directory and so on.
i am using to script below currently just to move and store the file name with path using varchar in the database.
$ns_pic_title_loc= $_FILES["ns_pic_title"]["tmp_name"];
$ns_pic_title_name = $_FILES["ns_pic_title"]["name"];
move_uploaded_file($ns_pic_title_loc, $ns_title_target.$ns_pic_title_name) or die(mysql_error());
that is just the sample code i havent included the validation function which i am using. i was thinking like i want to rename all the files like
a) In title directory the file should be stored as.
title_1.jpg
title_2.jpg
title_3.jpg
title_4.jpg
and so on
and the same way to rest of the pictures. how do i do that? what function do i use to achieve my target. and if this is not the good way to rename the file i would appreciate any suggestion followed to rename the file.
thanks in advance
Well, here's a possible solution:
Get uploaded filename from $_FILES["ns_pic_title"]["name"] and separate extension OR if we are only talking about image files get the image type with getimagesize($_FILES["ns_pic_title"]["tmp_name"]);
Check your database for the maximum id of the image records and make the the $file_name variable 'title_'.($max_id + 1)
At this point you should have $file_name and $file_extension so do move_uploaded_file($_FILES["ns_pic_title"]["tmp_name"], $ns_title_target.$file_name.'.'.$file_extension)
Hopefully this makes sense and helps.
There are a couple of good options with various pros and cons.
Use php's tempnam when moving the file, and store the path in your mysql database. tempnam generates a unique filename.
Use mysql to store the image content in a blob. This way you will access the image content via an id instead of a pathname.
Instead of having logic to figure out what the latest picture name is and calculate the next number increment, why not just use PHP's tempnam() function? It generates an unique name with a prefix of your choice (i.e., "title", "brief", "detail"). You could also simply prepend a timestamp to the file name -- if you don't have a whole lot of admins uploading pictures at the same time, that should handle most name conflicts.
Since your pictures are going to be sorted into title, brief and detail directories already, it's not really necessary to name each picture title_*, brief_*, and detail_*, right? If it's in the title directory, then it's obviously a title picture.
Also, you're going to be putting the file names in the database. Then elsewhere in the app, when you want to display a picture, I assume you are getting the correct file name from the database. So it isn't really important what the actual file name is as long as the application knows where to find it. If that's correct, it's not necessary to have a very friendly name, thus a tempnam() file name or a timestamp plus the original file name would be acceptable.
Because you are storing references into the DB, I would prefer to just md5 the datetime and use that for the filename and store the disk filename to the DB also. It doesn't matter what name it is written to disk with as long as you can point to it with the unique name into the DB.
I use this methodology, and in none of my testing does the disk name (md5 from the datetime) ever require multiple tries.