Is it better to read and list images directly from file system using simple php, or is it better to store image meta info and filename in the database and access the images by doing a mysql select. What are the pros and cons of both solutions.
Listing files on a file system is probably the easiest way to accomplish what you trying to do but it's going to be very slow if you are trying to cycle through several thousand directories/files on a networked file system (NFS, CIFS, GlusterFS, etc).
Storing files in a database will create a much more overhead since you are now involving an external application to store information. You have to remember that every time you are using a database you are also using network I/O, authentication mechanism, query parser, etc. At the same time all of this overhead might provide for a faster response then using a networked file system.
To conclude - everything depends on amount of files you are working with and underlying infrastructure. Two major things to look out for are going to be disk I/O and network I/O.
I would do the following:
Upload all the images in one directory
Store references to those images that are tied to the uploader's User ID
Then just select the image URLs that are tied to that ID, and output them however necessary.
People find it easier to store their files within folders and parse that folder with php. If you go the database method the database eventually gets larger and larger and larger.
I can see it becoming personal preference, but I personally have gone with parsing folders for images rather than storing it within a database.
Depends on the scale of what you are doing.
This is what I would be doing.
Store the file metadata in the database. You can store quite a bit of information about this image this way.
Store the image file on a distributed storage system like Amazon S3. Store the path in your metadata. Replication is part of the system. And it easily integrates with Cloudfront CDN.
Distribute the the images through Amazon Cloudfront CDN.
Related
I have made a database with tables like projects, employees.
I have some fotos and pdf files (for instance, scan of certificates) related to the entries in my DB, that should be accessible via our internal web site.
Does anyone have any suggestions on how I should manage this?
I was thinking on setting up a subdomain "files.ourdomain.com", and create subdirectories there for each table. And make a directory for each record? Should i create a DB field for "employee.certificates" with the entire path/filename of the certificate foto?`
Or should i actually store the files in the database? (MySql INNODB)
In a comment, Peter said that it's easier and better to store the files in a database for security reasons and there is a case for this - only DBA's and the webservers will have access to the files and they could be encrypted. I don't agree with the "easier" bit though as you would need to encode the image before placing it into the DB and decode again when you want to display it.
You have a choice:
you store the files actually in the database itself
you store the file in a folder (outside of the web tree) and only store the
filename to the file
Each method has pros and cons:
If you store the files actually within the database you could take the backup file and put it onto another server and everything will be present. However, if you're using replication or clustering then each server will have a copy of the file and so storage requirements increase. Obvious, you will also need more space for backups, and backing up and restoration will take proportionally longer.
If you store the file in a central location and only record the location, your DB storage requirements are lower and multiple DB servers can be confident that there is only ever one copy of a file. Again, the files can be backed up separately. The downside is "what happens if your file storage server fails". However, with mirroring and backups, this can be mitigated against.
In both cases, you would need to store a filename for each file so that the webservers can use them.
Have a look at this Stack Overflow question which does a much better job of the pros and cons than I ever could.
Speaking personally, I store a filename which links to an external file.
this question is confusing me so i thought i should listen to an expert voice !.
is it better to upload images to a folder and just save link to mysql, or better upload img itself into a blob mysql field ?
thank you very much
I have often built systems to store images in the database, there are pros and cons to doing this.
Pros:
All your data is kept in one place, if you migrate your website/database the images will just be there
Its easier to sort/delete/etc...
Since you have to serve it via a PHP script, you can perform additional things such as security if required, or image processing (obviously you can do this with flat file too, but you have to make sure the security cant be bypassed by leaving the images in a public directory).
Cons:
Its slower then serving a flat file from the webserver as a PHP script needs to retrieve it, and MySQL needs to return the data.
Your database will become large very fast and not all web hosts take too kindly to this.
The file system is faster for flat file storage and retrieval as thats exactly what a file system is designed for.
Bad. Your webserver does a much better job managing expiry headers and directly loading files from the filesystem. Throughput will be much higher using the filesystem. It's what it's designed for, utilize it.
SQL databases are designed for relational data, not images. You're just loading your database unnecessarily. Store the path/image name instead.
If your application is large i.e you have to display a large number/size of images repeatedly then you should go for first method (storing only image path in database and actual images on file system). This will reduce the processing time to display images moreover consumes less resources. Secondly, if your application requires less number of images then you can store them directly in database . This way it becomes easy to take backups and port application to another OS.
I'm planning a system for serving image files from a server cluster with load-balancing. I'm battling with the architechture and whether to save the actual image files as blobs in the database or in filesystem.
My problem is that, the database connection is required anyways as the users need to be authenticated. Different users have access only to contents of their friends and items uploaded by themselves. Since the connection is required anyways, maybe the images could be retrieved from there aswell?
Images should be stored with no single point of failure. And obviously, the system should be fast.
For database approach:
The database is separate from rest of my application, so my applications main database won't get bloated by all the images. Database would be easy to scale as I just need to add more servers to the cluster. Problem is, that I've heard this might be a slow system from a website with millions, even billions of photos.
For filesystem:
I would be really interested in knowing how could one design a system, where the webservers are load balanced, and none of them is too important for the overall system. All the servers should use a common storage, so they can access the same files in the cluster.
What do you think? Which is the best solution in this case?
What kind of overall architechture and servers would you recommend for a image serving cluster? Note: This cluster only serves images. Applications servers are a whole different story.
I definitely wouldn't store them in the database. If you need to use PHP for authentication, then do that as quickly as possible and use X-SendFile to hand over the actual image serving to your web server.
For the filesystem it sounds like MogileFS would be a good fit.
For the web server I'd suggest nginx. If you can adapt your authentication mechanism to use one of the existing modules, or write your own module for it, you could omit PHP completely (there's already a MogileFS client module).
we are creating a website for hotel booking. we need to store a large number of images. we think it would be a better option to store images in the filesystem and store the path in the database. But do we have to manually save them? We are using web services from another website to get the images. is there a way to save the images dynamically in the filesystem??
You can use PHP's file get contents function or CURL to download all the images you want to the disk or simply refering the foreign image to your clients and you won't need to store them locally on the server.
If you like Python check the Mecanize lib and BeautifulSoup to parse XML if you need.
Storing in disk vs storing in database has it's beneficts. If you need to scale, it's easier and you can always have a lighttpd or a nginx http servers dedicated to images or simply put it out on other server to balance bandwidth.
It depends on the database, and how you are serving up the images. In general it is better to save the images to disk, depending on how you are delivering them to the client.
Getting the images is usually a matter of some process on the server downloading them from websites and saving them. On many systems you could use wget or curl to download the images and save them.
It also depends on how you are getting the data. If it is some inline binary via XML or something, then you will need to extract that using the features of your application language, and save it to disk.
The mechanics of how to do that vary wildly depending on the implementation language and the hosting operating system.
I think that you should store both remote (web service) and local (filesystem) location in the database, with initial file system location blank. If a user requests an image for the first time, download it, update the file field and show it. With this concept you will only have images your clients need.
A while a go I had to developed a music site that allowed audio files to be uploaded to a site and then converted in to various formats using ffmpeg, people would then download the uploaded audio files after purchasing them and a tmp file would be created and placed at the download location and was only valid for each download instance and the tmp file would then get deleted.
Now I am revisiting the project, I have to add pictures and video as upload content also.
I want to find the best method for storing the files,
option 1 : storing the files in a folder and reference them in the database
option 2 : storing the actual file in the database(mysql) as blob.
I am toying around with this idea to consider the security implications of each method, and other issues I might have not calculated for.
See this earlier StackOverflow question Storing images in a database, Yea or nay?.
I know you mentioned images and video, however this question has relevance to all large binary content media files.
The consensus seems to be that storing file paths to the images on the filesystem, rather then the actual images is the way to go.
I would recommend storing as files and storing their locations in the database.
Storage the files in a database requires more resources and makes backing up/restoring databases slower.
Do you really want to have to transfer lots of videos every time you do a database dump?
File systems work very well for dishing out files, and you can back them up/sync them very easily.
I would go for the database option. I've used it on a number of projects, some very larger 100+GB. The storage implementation is key, design it poorly and your performance will be punished. See this example for some good implementation ideas:
Database storage allows more scalability and security.
I would go for storing files directly on the disk, and database holding only their ID/url.
This way accessing those files (that can be large, binary files) doesnt require any php/database operation, and it's done by the webserver directly.
Also it will be easier to move those files to another server if you'd want to.
Actually only one upside I can see atm of storing them in database is easier backup - you wanna backup your DB anyway, this way you'll have all data in one place and you can be sure that each backup is full (i.e. you don't have files on disk that aren't used by database entries; and you don't have image IDs in your database that point to nowhere)
I asked a similar question using Oracle as the backend for a Windows Forms application.
The answer really boils down to your requirements for backing up and restoring the files. If that requirement is important then use the database as it'll be easier (as you're backing up the database anyway, right? :o)