I'm trying to implement an script that reads the content(files and folders) of a certain directory and writes it in a database. My goal is create an software that allows me to organize those files and folders relating description and tags to them, without affecting the correspondig physical files in the disk.
But for now I'm facing a logical problem: How do I make a direct connection between that physical file and the database register? I want that, even if the physical file, for some reason, is edited or moved to another folder inside the root directory, the software is still able to relate that file with its original register in the database.
My first idea was to use a checksum hash to identify every file but, I'm guessing that if the file is edited, so does the hash, doesn't it? Besides that, I also think that a folder itself can't be checked that way.
Another solution that came up to my mind was applying a unique key in the beginning of every file and folder name in the directory. That may work, but it seems to me like an improvised solution and, therefore, I'mhoping that there may be another way to do it that I haven't considered yet.
Does anyone have an advice on that?
Under Linux, you can track files by inode, which won't change if the file gets moved or renamed. You can even get notified when a file changes.
have you considered storing the file in the database?
You can't.
It looks that there is no way to identify the file: neither by content nor by pathname.
One workaround might be: use path as id (and use them as reference in the DB) and do not use system tools (like mv) to move files but your own script which updates the file system and the database.
Related
This sounds like a dumb question, and the way I am doing this is not secure currently, so I want to be able to improve my security and reduce my chances of malicious use.
I have an "admin panel". I can add/remove users and appoint them admins and such. One of the permissions I might let them have is to edit files. This is a problem though since I store passwords in files and restrict web access to them (MySQL usage for the hosting provider is low) via .htaccess.
Because of this, I provide a file blacklist. However, the people can still see passwords if they edit a PHP file to include() or require() the login handling file and get the session key (which is what my login handling script can do) for the root account then hijack it and have root privileges.
Because of this I want to restrict include/require access.
The first idea I thought of is just simply checking if it is requiring/including the file in the code of the file they edited, but that seems too "risky".
Then, I wondered if I could restrict access of include/require of that certain file/certain files to only a list of specific php files (like the login/control panel files that need that login handler file).
If not, should I do away with a file editing feature altogether, since it seems quite risky even if I do find out how to limit access.
include() doesn't allow limiting the path to a specific folder. Same goes for require(), require_once(), file_get_contents(), highlight_file(), fopen(), etc...
You can change the default include path, but that's a different thing. It would still possible to include using an absolute or a different relative path. Or to simply print the file contents using a different function.
So my recommendation is to really disallow editing files altogether.
If you need users to be able to update text contents, consider using a translation engine (even for just one language). Possibly combined with markdown (stored MD in database, converted to HTML for display) or HTML output from wysiwyg editor if they need to be able to edit the styling as well.
I am new to web development and I'm learning PHP in order to sell a few binary files (shared Linux host). The site is not yet live.
My php scripts (50% borrowed code, 50% self-written, 95% fully understood) login to MySQL to READ the items for sale, and WRITE sale transaction data into another table. Functions.php, located in a subfolder of the webroot, contains the login name and password for MySQL.
Q1. This doesn't seem secure to me. How should the login/password info be stored so the scripts can access it? If functions.php was stored outside the webroot, could the .php files located in webroot #include (PHP "require_once") it? (I did try this once and my scripts broke in a way that seemed permissions-related -- if I knew it should work I'd keep plugging away at it)
Q2. I am unsure where to store the binaries that purchasers can download. Is it correct that savvy users can somehow find / download them (without paying) if I just store them in a subfolder of the webroot? Is it possible to use a .htaccess file to block access to the "binaries" folder within the webroot? Can black-hats get at / modify a .htaccess file?
Q3. Would it be a better idea to store the binaries (max=4Mb) in a MySQL table and copy them from there to a temp file in webroot before each download, then delete?
Q4. Can anyone recommend a set of scripts that manages this sort of thing that I could review / modify rather than reinventing the wheel?
Thanks
Q1 - Your MySQL password and other application specific settings should be stored in a separate file outside of your webroot. You can either put it out of webroot directly or restrict it via .htaccess. You can include the file or read from it as long as you know the path.
Q2 - The binaries should also be stored outside of the webroot. The ideal way to serve them would be to have them downloadable via a PHP file. This way you can do authentication before the file is served and you can make the links temporary so that users can't share it with other people
Q3 - If you use the above method, you don't need to store it as a BLOB in MySQL
Q4 - I haven't really come across anything that does and is a library/autonomous script. Serving them via the correct headers shouldn't be too difficult though.
Not sure if best practice, but this is how I'd approach it:
Q1: I store MySQL login information, along with local paths and other settings, in a config.inc.php file outside of the web root. I can then include that at the start of each script. I also use a database.inc.php which connects to MySQL and selects the database (plus a few database functions). In theory it isn't insecure inside the web root as being called directly will only execute the PHP, not display the contents of it. Storing an XML config or similar is different however!
Q2: If downloadable binaries are stored within the web root then they could be downloaded if the right URL is discovered. Instead they should be stored outside the web root, and a PHP "gateway" script serves the contents of those files if the request meets the right conditions. You may want to store a token with each purchase in your database, and only valid tokens are permitted to download the files. An example of a download script is here.
Q3: I believe it's better to use the file system to store files, rather than a database. It won't improve security over my answer to Q2 if that's what you mean.
Q4: You could try existing shopping cart software. Magento supports downloadable products.
Hope that helps
I would like to write a setup script for my PHP application, which dose a minimum requirements check, gets the DB credentials, DB prefix and saves them, creates the db tables and so on. Now I would like to know what is the best practise to write and save the DB credentials? Write them as an array into a .php file and? Or into an XML file?
I don't think there is a best practise for this, there are so many ways people use configuration files. Some use PHP arrays, some use XML files, some use INI files, some use JSON files and I'm sure some people create proprietary formats.
What you do want to take in account is, where will you store this file. If it is in the document root, people could request it. XML/INI/JSON files are plain-text and by default, will make it easy for people to 'retrieve' the file contents. PHP will be parsed server side so just returns an empty page.
Ideally you'd store the configuration file outside of the document root, but not all webhosts allow you to do so. So I'd say, if you want to release an application people can install themselves easily, a PHP file might be the easiest way to go.
Write them as an array into a .php file. This satisfies speed (no xml parser and file touching is needed per-page), and security (.php files don't get served as text like your xml would).
I also tend to put the private.php that contains my mysql credentials in the directory above the http root, and load it like require_once("../private.php");
You are asking about setting up the environment, correct? If that is the case, then it depends on the script or build system itself. We are using Ant where such configuration is stored in build.properties. For example:
# database credentials
db.host=localhost
db.user=root
db.pass=root
db.name=db_name
This file is working copy specific and as such is not a part of our VC, however, build.properties.dist is. This way A's local settings don't override B's.
If the question is about something else, please, do tell :)
I have a site that allows people to upload files to their account, and they're displayed in a list. Files for all users are stored on different servers, and they move around based on how popular they are (its a file hosting site).
I want to add the ability for users to group files into folders. I could go the conventional route and create physical folders on the hard drive, for each user on the server, and transverse them as expected. The downside to that is the user's files will be bound to a single server. If that server starts running of space (or many files get popular at the same time), it will get very tricky to mitigate it.
What I thought about doing is keeping the stateless nature of files, allowing them to be stored on any of the file servers, and simply storing the folder ID (in addition to the user id who owns the file) with each file in the database. So when a user decides to move a file, it doesn't get physically moved anywhere, you just change the folder ID in the database.
Is that a good idea? I use php and mysql.
Yes, it is.
I don't see any downside, except maybe more queries to the database, but with proper indexing of the parent folder id, this will probably be faster than accessing the filesystem directly.
Forget about folders and let the users tag their files, multiple tags per file. Then let them view files tagged X. This isn't much different to implement than virtual folders but is much more flexible for the users.
I accept file uploads from users. Each file has a pointer in the db which has info on the file location in the filesystem.
Currently, I'm storing the files in the filesystem non categorically, and each file is currently just named a unique value. All categorisation and naming etc is done in the app using the db.
A factor that I'm concerned about is that of file synchronization issues.
If I wanted to set up file system synchronization where, for example, the user's files are automatically updated by bridging with a pc app, would this system still work well?
I have no idea how such a system would work so hopefully I can get some input.
Basically, is representing a file's name and location purely in the database optimal, especially if said file may be synchronized with a pc application?
Yes, the way you are doing this is the best way to do it. You are using a file system to store files and a database to sore structured data.
One suggestion I would make is that you create a directory tree on the file system. You may one day run up against a maximum files per directory limitation of your file system. I have built systems that create a new sub directory for each day or week.
Make sure you have good backups of the database as well as the document repository.
All you need to make such a system work is to make sure the API you use (or, more likely, create) can talk to the database and to the filesystem in a sensible way. Since this is what your site is already doing anyway, it shoudn't be hard to implement.
The mere fact that your files are given identifiers instead of plain-English names is mostly irrelevant with regard to remote synchronization.
Store a file hash in the database rather than a path (i.e. SHA1) and have a separate database connect the hash with the path. Write a small app that will synchronize the hash database so that when you move your files to a different location it'll be easy to build a new database with updated paths.
That way you can also have the system load the file from a different location depending of which hash database you use to locate the file so it offers some transparency if you need people to be able to access the same file from diverse locations (i.e. nfs or webdav).
We use exactly this model for file storage, along with (shameless plug) SabreDAV to make it seem to the end-user it's a normal filesystem.
I think this is a perfectly fine model, as long as looking up the file is documented and easily retrieved there shouldn't be an issue. Just make backups of your DB :)
One other advice I can give, we use an md5() on the file-id to generate a unique filename. We use parts of the files to generate a directory structure, for example.. id 1 will yield: b026324c6904b2a9cb4b88d6d61c81d1, the resulting filename will become:
b02/632/4c6/904b2a9cb4b88d6d61c81d1 The reason for this is that most stable filesystems can become very slow after a high number of files (or directories) in one directory. It's much, much faster too traverse a few sub-directories.
The Boring Answer™:
I think it depends on what you wanna do, as always :)
I mean take your regular web hosting company. Developers are synching files to web servers all the time. Would it make sense for a web server to store hash-generated file names in a db that pointed to physical files? No. Then you couldn't log in with your FTP-client and upload files like that, and you'd have to code a custom module to get Apache to work etc. Instant headache.
Does it make sense for Flickr to use a db? Yes, absolutely! (Then again, you can't log in with an FTP-client and manage your photos—and that's probably a good thing!)
Just remember, a file system is a (very simple) db too. And it's a db that comes with a lot of useful free tools.
my 2¢
/0