i've thinking about various ways of handling file uploads in a sort of CMS. I write here because i am not satisfied with what i've got right now...
The problem
Uhmm, lets call it the tumblr way ;-) The user shall be able to upload a file or several files directly without a file management view or s.th like that. The bottom side is that if he deletes the file in the WYSIWYG editor the file stays on the server. In my case there is not only a WYSIWYG editor also a media module...
The question
Is there a best practice for handling this? I've never programmed s.th like that. Would you store the filenames in a MySQL table, would you use a cron job to check if the files are really used in the document?
ANY ADVICE WOULD BE REALLY APRECIATED!!!
Muchissimas Gracias y Saludos!!!!
Personally I use a cron job that runs once a day and cleans up any orphan uploaded files (orphans older that x days).
I admit, I'm curious about other ppl approaches.
Why so much hassle for some additional space used? Hard drive space shouldn't be a concern, since it's so cheap. And even if it were not, the images are very lightweight resources.
The only problems I can imagine is that your CMS's users are uploading very large files. In that case, you should process the image before saving them, lowering the quality and the size.
I think a cron job would be more CPU intensive than letting some 'ghosts' files.
However, you could try to catch when an image is deleted, but then again, it could be more trouble.
Related
I am working on a small social media project and one of the things I want todo is allow users to upload an image. I originally was thinking about uploading the actual file to a image directory and saving the files name in the database to call it later.
However there is a limit to how many files I can have in a directory, and file size can be an issue.
So instead of going this route I looked into simply saving the image data using php's file_get_contents() function to the database as a BLOB datatype and then writing another script that renders an image from the image data.
This kinda seems like a no-brainer and a better route since image data would be relatively small in size.
I am worried though because it seems like its too convenient. Is there a reason why I shouldn't do this? I mean a real viable reason? Is something gonna bite me in the ass later because I am going this route?
I hope this was the right place to post this question. Thanks in advance.
Be aware that serving images from a database is usually much, much much slower than serving them from disk. It'll start a PHP process, create the database connection, query the database, transfer it and then pushing out via PHP which would be non-cacheable.
There are some complexities to storing images in a database but it does allow for easier sorting and deleting, and you can perform additional checks if security is of concern, and logging if that is of interest.
See https://stackoverflow.com/a/1638348/5509627 for implementation.
If disk space is only concern you may consider storing the images in AWS S3 or similar.
I've been on a project for the past few days and hit a problem displaying large quantities of images (+20gb total ~1-2gb/directory)in a gallery on one area of the site. The site is built on the bootstrap framework. I've been trying to make massive carousels that ultimately do not function fluidly due to combined /images size. Question A: In this situation do I need i/o from a database and store images there-- is this faster than in /images folder on front end?
And b) in my php script i need to -set directories to variables/ iterate through and display images into < li >, but how do I go about putting controls on the memory usage so as to not overload browser? Any additions, suggestions, or alternatives would be greatly appreciated. Im looking for most direct means to end here.
Though the question is a little generic, here are some thoughts in regards to your two questions:
A) No, performance pulling images from a database would most likely be worse than pulling straight from the file system. In general, it is not a good idea to store images or other binary data in databases unless you absolutely have to, because databases can't do much with this information and you are just adding an extra layer on top of the file system that doesn't need to be there. You would, however, want to store paths to images in your database, potentially along with other characteristics such as image dimensions, thumbnail paths, keywords, etc. Then your application would read the entries for the images to return the correct paths to the images.
B) You will almost certainly want to implement some sort of paging if you are displaying many hundreds or thousands of photos. If the final display must be a carousel, you will want to investigate the Javascript that drives it to determine how you could hook in a function that retrieves more results from your PHP application via an AJAX call when it reaches the end or near end of the current listing of images. If you are having problems with the browser crashing due to too many images, you will also want to remove images from the first part of the list of <li>s when you load new ones so that it keeps the DOM under control.
A) It's a bad idea to store that much binary data into a database, even if the DB allows it, you shouldn't use it, it'll also give you much more memory consumption, all your data will be stored in the database's memory space, then copied into PHP's memory space for you to handle, which eats up twice the memory, plus the overhead of running a database server, and querying, etc.. so no, it's slower to use a database, accessing the filesystem directly is faster, if you also use varnish or other front-end caching system, you'll even be able to serve content much faster too.
What I would do is store files on the filesystem, and the best server to handle static serving like that is either G-WAN or NGINX Source, but do your read up and decide for yourself what suits you best. point is, stay away from apache, and probably host all those static files onto a separate server running a lightweight http server
ProTip: Save multiple copies of the same image with scaled down sizes for example 50% and another version with 25% of the original image size, this way you'll be able to send the thumbnails first for quick browsing, then when a user decides to view an image you serve up the 50% or 100% size, depending on their screen size, this way you save yourself bandwidth and memory. you also save a big 3G bill for mobile users.
B) This is where it makes some sense to use a database, you can index all the directories into a database, and use that to store the location of the image in the FS, and perhaps some tags, and maybe even number of views, etc...
and in the forntend you'll implement a scipt that'll fetch for example 50 thumbnails per page then the user can scroll around using some fancy JQuery, and when you need to fetch more, simply get a new result set with 50 more thumbs, etc..
this way you'll save yourself memory, bandwidth and even the users will thank you for such a lightweight browsing experience !
Another tip:
If you want to be able to handle bigger traffic, you might want to consider using a CDN, there are many CDN services that aren't as expensive as Amazon S3, a simple search will give you tons of resources !
Happy hacking !
I am building a website where I am uploading images to my ftp folder through PHP script. Now I want to display those images on to my HTML pages. I was thinking about using PHP and getting array of all the images from my ftp folder and then display them using image view.
Please tell me if I am doing this the wrong way and if there is any other better alternatives to it. I was reading php manual for ftp_nlist and ftp_rawlist but did not understand.
Well it may depend on how many images you have in there. Probably the most "correct" way to do it would be to store the filenames in a DB. You could scan the entire folder, but for every single request that's potentially a lot of overhead rather than just grabbing them out of a DB.
Are you manually uploading the images? Give us more details on how that works and we can better serve you. If you're using a script to upload images (I've had lots of projects where that's the case), then you can just have the script insert those filepaths into the DB for you. If not, (you're manually uploading them), or if indeed there are not a large number of files, then scanning the folder wouldn't necessarily be a bad thing. I've used that method on smaller projects myself.
Read up on the php readdir function in the docs (which actually works a lot like mysql_fetch_assoc, ironically)- That will provide you with an excellent way to go without setting up a DB. For an approach where an upload script handles it, I recommend a DB. Without more info, it's hard to say.
Good luck!
I would like to build a PHP script to optimize images similarly to how PunyPNG or Kraken.io optimizes images. Essentially, I would need to be able to take .jpeg, .png, and .gif images and reduce their file size as much as possible without losing quality (or with minimal quality loss).
These services offer APIs, but I would like to avoid unnecessary costs, and I do not want to be limited by a specific number of daily uses.
Can this be done with something like ImageMagick? Is it even possible, or is it far too complicated?
talking about re-sizing images, they were never an issue, there are couple of tools that help you do that in bulk. Since you specifically say PHP, I am expecting you would be using it for displaying it on a page. for the very same purpose I wrote this little script not so long ago, which might be of some help to you. Fork it here https://github.com/whizzzkid/phpimageresize
Spatie has a decent package that gets updated regularly, I've been using for a while without problems:
https://github.com/spatie/image-optimizer
I want to create multiple thumbnails using GD library in php, and I already have a script to do this, the question is what is better for me .. is it better to create thumbnail on the fly? or create a physical file on my server each time I want a thumb?? and Why?
Please, consider time consuming and storage capacity and other disadvantages for both
When you create the thumbnail depends on a couple of factors (that I'll get into) but you should never discard the output of something like this (unless you'll never use it again) as it's a really expensive operation.
Anyway your two main choices for "when to generate the thumbnail" are:
When it's first requested. This is common and it means that you don't generate thumbnails that are never used but it does mean if you have a page full of first-time-thumbnails that the server might become overwhelmed with PHP processes generating the thumbnails.
I had a similar issue with Sorl+Django where I was generating 100+ thumbnails per request for the first few requests after uploading and it basically made the entire server hang for 20 minutes. Not good.
Generate all required thumbnails when you upload. Because it takes a long time to upload, you break down the processing quite a lot. You can also pull it out-of-process (ie use another script to process uploads - perhaps not even in PHP).
The obvious downside is you're using up disk space that you otherwise might not need to use up... But unless you're talking about hundreds of thousands of thumbnails, a small percentage of unused ones probably won't break the bank.
Of course, if disk space is an issue, there might be an argument for pushing the thumbnail up to a CDN at the same time as you process it.
One note when you save the thumbnails, it's fairly common that you'll want to resize the thumbnails at some point down the line or perhaps want two small variants. I find it really useful to make the filenames very specific so if the original image is image.jpg, the 200x200 version is image-200x200.jpg.
Neither/both - don't generate the thumbnails till you need them - but keep the files you generate.
That way you'll minimise the amount of work needed and have a self-repairing system
C.
GD is really resource heavy, so you should look at if you can use ImageMagick instead (which also has a clearer syntax).
You definitely will be better off caching the created thumbnail after the first run (regardless of if you run GD or ImageMagick) and serve them from the cache. If you are worried about storage, clear out old files from the cache now and then.
Always cache (= write out to disk) the results of GD operations. They are too expensive both regarding processor time and memory to be done on the fly every time. This becomes increasingly true the more visitors/hits you have.