I was wondering what would be the good way to organize documents in a filesystem for my php/mysql application. All info of the docs are stored in a db i am curious about filesystem organization. Is this good way to do it? Is there a better way?
Main folder
/documents
One folder per client
/documents/client1
Documents are uploaded here per client
/documents/client1/queue
After users fill the form docs are saved to a database and moved in this folder
/documents/client1/docs
Original and filesystem names of the document are stored in a database and filesystem name is something like md5($time.$filename.$client_id) and the document path looks like this
/documents/client1/docs/6f99caa11e78697612d8f1b4481cd76a.pdf
I need (minimum) first page of pdf for the thumb and auto barcode reading from the first page
/documents/client1/docs/6f99caa11e78697612d8f1b4481cd76a/6f99caa11e78697612d8f1b4481cd76a-0.gif
I also have
/documents/client1/scan
where files from the scanner goes so users can import them in the database and after that they are renamed and moved to:
/documents/client1/docs
I wonder if i should put files for specific date in a date folder like this:
/documents/client1/docs/20110915/6f99caa11e78697612d8f1b4481cd76a.pdf
Or should i use completely different folder structure?
Why not using only one common folder for temporary (uploaded) files ? It could make maintenance routine easier (for example deleting all old files).
I wonder if i should put files for specific date in a date folder like
this:
/documents/client1/docs/20110915/6f99caa11e78697612d8f1b4481cd76a.pdf
If the date is handled by your database it's needless. Nevertheless if for some reason you need to access this file manually (not using your interface) it could be helpful...
Last but not least be aware of your filesystem's limitation. On some filesytem/os you could have a limitation on the number of file per folder (huge but still).
I would prefere a structure that hash the file organisation according to the file name.
/documents/6/f/9/6f99caa11e78697612d8f1b4481cd76a.pdf
This will keep good file access and its easy to make the structure deeper if needed.
The problem with using the date as a directory, is that the file location is dependant on some information that may change (we never know!)
Related
So I have my own webserver now and am hosting a website. I have made a basic screenshot to FTP. It copies the link once uploaded and I'm going to be adding randomized folders e.g /push/eqw8/woeqwe.jpg like puush does.
However, this is not my question. I was wondering how I would be able to grab all folders and images from those folders and create a table like this http://puu.sh/oEyfP/783483492c.png or http://puush.me/account in order from upload date. And then it shows in oderder of date uploaded.
I just wanted to do this because I feel more secure and its a fun project doing it.
How would I get about making this.
can you tell me like what I need to do in words?
I'm not good with web development and only good with languages like c# and such.
Learning php and html atm and its pretty basic but I still need time to learn.
At first it depends on your php skills about how much you can think / do in php.
The main thing you need here to order the files / folders by date or manage them at much professional level , you have to use Mysql database and store Files and Folders information inside the database.
For example , let say you have a Image file extension .jpeg then store it's information about :
Extension File size Parent Folder Uploader id / IP Upload Date Unique random key mime type Random name Original name
And there are many other things you need to consider
Files that are being stored by users should be scanned and non executable from the browser.
Store them in random folders with random names so no body can find the way to the file.
You can use javascript/ ajax for uploading multiple files at the same time.
You can make the files non executable by removing their extensions and saving them inside a folder without their original names and save their original names and extensions inside the database along with random string name and the folder name where it was stored so your system can identify the file and you can show your users the file and then on download you can replace the name with original one along with extension.
this is a one of my projects i have created , if you want you can check it out: Buckty
About any other information you need , you can ask me below in comment.
Since I am just experimenting on this, (only localhost) I may like to ask for some ideas(since nothing is really coming out of my mind) about letting a user, who is going to, for example, register to a mini-social-networking site, with a corresponding username/password, personal details, etc. I would upload the image, and save it to a folder(ON MY HARD DRIVE be it Drive C:\ or D:), for example '/images/username' and the full path of the folder would be the one inserted to a row named img_dir (of course it is a string, instead of putting it as a BLOB, so later i would just use img src="path"). I would not mind where it will be going to be saved. But since I am new to cakephp i haven't really grasped the idea of what I am going to do. I have no problems about registering/login sessions. This was easy in C# but I am too stupid for PHP maybe? :P
While this may not give you a direct solution in CakePHP, you had asked for some ideas.
I've outlines some pros and cons of storing a file on the filesystem (along with some other approaches) in this post.
Hope that helps...
I've written a complete plugin for that kind of task and it's more thought through then just the idea of saving the file path.
A file has some more meta data like it's size and mime type which is useful when the file is served. So an uploaded file should be handled as an entity of it's own. I personally think it is a bad idea to directly save the path to a file within the record it belongs to. What happens if you need two images later? Adding incrementing fields like path1, path2?
It is IMO better to have a separate table for files and associate records with these file records. Expressed in CakePHP associations: User hasOne Avatar or Gallery hasMany Image for example.
Also saving files in path like this uploads\username1\pic.jpg can result in slowing down the app because of file system performance issues if you get a lot directories and files within the same level of the file system.
However, check my plugins readme.md out, there is more about why it does things like it does to solve different kinds of issues you can run into.
What are some ideas out there for storing images on web servers. Im Interacting with PHP and MySQL for the application.
Question 1
Do we change the name of the physical file to a000000001.jpg and store it in a base directory or keep the user's unmanaged file name, i.e 'Justin Beiber Found dead.jpg'? For example
wwroot/imgdir/a0000001.jpg
and all meta data in a database, such as FileName and ReadableName, Size, Location, etc.
I need to make a custom Filemanager and just weighing out some pros and cons of the underlying stucture of how to store the images.
Question 2
How would I secure an Image from being downloaded if my app/database has not set it to be published/public?
In my app I can publish images, or secure them from download, if I stored the image in a db table I could store it as a BLOB and using php prevent the user from downloading it. I want to be able to do the same with the image if it was store in the FileSystem, but im not sure if this is possible with PHP and Files in the system.
Keeping relevant file names can be good for SEO, but you must also make sure you don't duplicate.
In all cases I would rename files to lowercase and replace spaces by underscores (or hyphens)
Justin Beiber Found dead.jpg => justin_beiber_finally_dead.jpg
If the photo's belongs to an article or something specific you can perhaps add the article ID to the image, i.e. 123_justin_beiber_found_dead.jpg. Alternatively you can store the images in an article specific folder, i.e. /images/123/justin_beiber_found_dead.jpg.
Naming the files like a0000001 removes all relevance to the files and adds no value whatsoever.
Store (full) filepaths only in the database.
For part 2;
I'm not sure what the best solution here is, but using the filesystem, I think you will have to configure apache to serve all files in a particular directory by PHP. In PHP you can then check if the file can be published and then spit it out. If not, you can serve a dummy image. This however is not very efficient and will be much heavier on apache.
I'm building a torrent site where users can upload torrents.
What would be a good way to save the .torrent files?
I can think of several options:
Saving the torrent file itself in a folder on the server (not the best option since OS's have limitations saving lots of files in 1 folder)
Saving the torrent file itself in different folders per month
Saving the contents of the torrent file in the database (any limitations / performance issues / any other caveats?)
Any other options?
If you're concerned about having too much files within a directory, you need to distribute the files across multiple directories. Storing them by month, day or week is one way how to do so. It depends a bit how many files you really have I would say.
You can try to more or less equally distribute the files within subdirectories by hashing their filename and use the whole or part of the hash to generate one or multiple subdirectory names:
$hash = md5($fileName);
$srotePath = sprintf('%s/%s', substr($hash,0,2), $fileName);
This would pick the first two character from an md5 hash (00-ff, 256 subdirectories) to generate the subdirectory.
The benefit compared with a date is, that you always can find out in which directory a file is stored when you have it's name.
That does also mean, that you can not have duplicate files with the same name (which might have worked for the date based subfolder).
I'm use this:
Saving the torrent file itself in different folders per month
using database is not at all good. just save them as static files and maybe even gzip them .
just make sure to rename them uniquely with some kind of hashing .
if you don't have any problem in using external provider you can use TorCache
I would say saving the .torrent file in a weekly/monthly folder is the best option.
That way you can use the OS' filesystem cache, even if you store the .torrents outside the document root for limiting user access (in the end you will have to open() the file anyway)
Leaving torrents in the database would eventually lead to slow performance as the DB increases in size.
May be you try Amazon S3? It's cheap, easy and fast.
Uploading them automatically saves .torrent files. http://www.tizag.com/phpT/fileupload.php has a good example. Give it a try.
What is better? To be able to store user docs like word docs, pdfs, scanned images, etc in a database or in a folder. Security is of top issue. I don't want somebody to do something like domain.com/user/social/profile/user_images/23432434.png and be able to see it.
Thanks.
Storing files in the database is often regreted afterwards. Store a filename or id only in the database.
If you want to prevent access, store the files outside of the DOCUMENT_ROOT. Make a wrapper script that handles access, and use either readfile() or X-Sendfile: when invoked. (You would basically have done the same for SQL stored resources.)
If security is of top issue - use .htaccess and close public access to content folders.
You could save the assets behind the public folder (the document root folder where the app lies) and get them like http://myapp.com/assets.php?id=13 where assets.php grabs the file and output the content.
Now in assets.php, you can verify the roles, etc etc.
like for example:
root folder
public_html (where your script lies)
assets (where your uploads lies, they'll be inaccessible to the public eye and you can pull them when someone calls assets.php)
Does not really mather what you do if you store it in DB or on HD. The important thing is how you will be serving the files.
For this purpose you can you any kind of hash function.
add "salt" field to each user in DB
then you can do for example md5(md5({filename}) . salt)
and retrieving those files will be prety hard if you combine it with an captcha.
Store your files above the root of the server.
for example:
/var/www/ - your server root. Publicly available
/var/documents - user files here. above the root and not publicly visible.
Server your documents:
in /var/www/serve.php
#add headers
echo file_read_contents("/var/documents/file.doc");
Saving files in sql may cause some problems like as slower speed.
You can use
# no one gets in here!
deny from all
in your .htaccess and use echo file_get_contents($filename); or readfile($filename);