I have a PHP script where a user can upload an image. This image is stored in the temporary directory and is returned to the user. The user can then use a javascript interface to crop the image. (x1,y2)(x2,y2) is sent to the script which is used to crop the image. It is then returned to the user for another preview and\or crop. Once the user is sufficiently satisfied he will click "save". The temp file is copied over to the original and the temp deleted. These are not per-user images, but rather images of equipment. Any user in the organization can replace any image of equipment. This approach is good but there are a few issues:
1) Let's say the user uploads an image for preview but then closes the browser window. I will be left with a temporary file. This can become an issue. Sure I can have a CRON clean them up but in theory I can have a ton of temporary files (this is ugly). The cron can also delete the user's temp file during an edit.
2) To deal with number 1 I can always have a temporary file per piece of equipment, such as equip1.temp and equip1.jpg. All uploads are stored in equip1.temp, all commits are transfered to equip1.jpg. If two users are trying to upload pictures of the same piece of equipment at the same time this could mess them up (highly unlikely + not an issue, but still ugly)
3) I can always pass the image back and forth (user "uploads" image and it get's echoed back as an <img src="base64....." />. The resulting edits + original base64 string are sent back to PHP for processing). This solution relieves the temp file issue but I noticed it takes several seconds to send high res images back and forth.
How would you deal with this situation?
I had a similar issue like this. If I recall correctly (its been a while), I ended up creating a table in a DB to store file names and session keys/time. Each time the script loaded, if there was a dead session in the database, the corresponding session and image/file was deleted.
I don't know if that's a good solution or not, but it solved the multiple user access problem for me.
I wouldn't recommend #3 due to the reasons you mentioned.
I suggest you do this instead:
User uploads file to a random temporary name. equip1.jpg gets stored as equip1_fc8293ae82f72cf7.jpg. Be sure you script will juggle both file names around. It will allow two users to upload the same equipment, with the last one to upload being the winner, but no conflict along the way.
Everytime your cropper works with the temp image, you should "touch" it to update the modified time.
Let the user finish their edits, move the temp file in place of the final image name.
Have a cron, or a section of your uploader script, that deletes abandoned temp files that have a mtime older than an hour or so. You suggest this is messy, because of the potential of lots of temp files, but do you expect a lot of images to be abandoned? Garbage collection is a very standard method for this problem.
Related
Lets say I'm building a image gallery using PHP, where users would be able to upload their photos.
Every user would have 1 folder on server side with all their images there.
Now lets say I need to provide information in browser. Users would be able to browse images and should see lots of information about them, like image size, image dimensions, even EXIF information etc.
I could do this in 2 ways:
Save all information about image into database when uploading image.
Use PHP functions to browse through folder, and get information from every image.
I have something like file manager class, that can do all manipulations with files on server side, like deleteDir, deleteFile, countItems, getFileSize, getDirSize.
And it would be easy to only write one more class that would inspect images, and then I could just upload images, and get their information right from the folders without a need for relation database.
And now the question you all been waiting for is: ... :)
What would be faster, first or second solution? Lets say that site gets loads of traffic.
What solution would be better if I want it to be fast, and not to stress server to much?
actually, I got this situation like yours, this is my solution:
Save all information about image into database when uploading image.
Why?
I tested 2 ways:
Using php to get the image info for 1000 times.
Getting image info from database for 1000 times.
And the result is :
Getting image info from database is faster and faster.
Last but not least:
What would you do if you want to do a image info analystics?
If you save all info in database ,you can easily get them and analyse them ,but if you using php to get the info? it's hard to image.
So, just save all information about image into database when uploading image.
Good luck.
storing it in the database once
reading the data from the database and store it in cache,
redoing things always costs especially if it happens all the time
Depending on the size of these images, you probably want to show thumbnails instead of the original when people are browsing, which means you need to generate them. I would generate the thumbnail on upload and grab all the file info. Then save the file info in the database and put the original and thumbnail in the file system. If you get a lot of traffic, throw memcache on there too.
Storing data in separate places has a way of creating maintenance headache. I would just serialize the metadata for images in each folder and dump it to a file there. If you use gzip compression on the file, retrieval and storage should be very fast.
I have no idea how the big websites save the pictures on their servers. Could any one tell me how do they save the pictures that are uploaded by the users in their database?
I was thinking, maybe they would just save the file(the picture) in some path and just save that path in the databse is that right?
But I want to do it this way. Is this right? For example, a website named www.photos.com. When a user uploads a picture I would create a folder of the user name and save those pictures in that folder.
I believe we can create a directory using php file concepts. So when a new user uploads his picture or file, I want to create a directory with his name.
Example: if user name is john, I would create a directory like this on photos.com www.photos.com/john/ and then save all his pictures to this directory when he uploads a picture. Is this the right way to do this?
I have no one here that has good knowledge of saving the files to servers so please let me know how to do this? I want to do it the correct and secure way.
All big websites don't save pictures to the database they store them in the disk.
They save a reference to the picture's position in a table. And then link from there.
Why? Performance.
Pulling heavy content from a database is a huge performance bottleneck. And databases don't scale horizontally that well, so it would mean even a bigger problem. All big sites use static content farms to deal with static content such as images. That's servers who won't care less about your identity.
How do they keep the pictures really private you might ask? They don't.
The picture's link is, in itself, the address and the password. Let's take Facebook, for example. If I store a private picture on my account you should not be able to open it. But, as long as you have the correct address you can.
This picture is private. Notice the filename
10400121_87110566301_7482172_n.jpg
(facebook changes the url from time to time so the link may be broken)
It's non sequential. The only way to get the picture is to know it's address.
Based on a previous user photo you can't guess the next one.
It has a huge entropy so even if you start taking random wild guesses you'll have an extensive amount of failures and, if you do get to a picture, you won't be able to, from there, realize the owners identity which, in itself, is protection in anonymity.
Edit (why you should not store images in a "username" folder:
After your edit it became clear that you do intent to put files on disk and not on the database. This edit covers the new scenario.
Even though your logic (create a folder per user) seams more organized it creates problems when you start having many users and many pictures. Imagine that your servers have 1T disk space. And lets also imagine that 1T is more or less accurate with the load the server can handle.
Now you have 11 users, assume they start uploading at the same time and each will upload more than 100GB of files. When they reach 91GB each the server is full and you must start storing images on a different server. If that user/folder structure is followed you would have to select one of the users and migrate all of his data to a different server. Also, it makes a hard-limit on a user who can't upload more than 1T in files.
Should I store all files in the same folder, then?
No, big-sites generally store files in sequential folders (/000001/, /000002/, etc) having an x defined number of files per folder. This is mainly for file-system performance issues.
More on how many files in a directory is too many?
It is usually a bad idea to store images in your database (if your site is popular). Database is, traditionally, one of main bottlenecks in most any application out there. No need to load it more than necessary. If images are in the filesystem, many http servers (nginx, for example) will serve them most efficiently.
The biggest social network in Russia, Vkontakte does exactly this: store images in the filesystem.
Another big social network implemented a sophisticated scalable blob storage. But it's not available to the public, AFAIK.
Summary of this answer: don't store blobs in the database.
is this the right way to do
Yes.
The only thing I'd suggest to use not name but id.
www.photos.com/albums/1234/ would be okay for starter.
Image management may best be achieved by physically uploading images to the server and then recording file location and image details in a database. Subsequently, a Search Form could be configured to permit the user to do a text search, part number search, or other queries. A PHP script could be written to produce a valid HTML image tag based on data found in the table.
uploading images into a MySQL™ BLOB field is such a bad idea such image data is generally problematic if the images are much larger than thumbnails. If the images are large, you can end up having to copy/paste one SQL INSERT statement at a time (into phpMyAdmin). If the images are large and the SQL INSERT statement is broken into two lines by your text editor, you'll never be able to restore the image.
I'm making a real simple "backend" (PHP5) for two flash/air-applications. One of them will upload a photo, the backend will save it to a folder, and the second app will poll the backend for new photo's and show them.
I don't got any access to a database, so the backend has to be pure PHP5 and nothing more. That's why I chose to save the images to a folder (with a timestamp in their names) and use readdir() to get them back.
This all works like a charm. Nevertheless, I would really like to make sure the backend only returns photo's that are completely uploaded, preventing the second app to try to load an unfinished image. Are there any methods/tricks that I can use to validate a file?
You could check the filesize a couple hundred milliseconds apart and see if it changes:
$first = filesize($file);
// wait 100ms
usleep(100000);
$second = filesize($file);
if($first == $second) {
// file is no longer being actively uploaded
}
The usual trick for atomic filesystem operations is to write into a temporary file that is not matched by the reader (e.g. XXX.jpg.tmp) and once it's completely uploaded, rename it to it's target name. Renames on the same volume are atomic, so there is no point where the file is either uncomplete or unavailable.
A really easy and common way to do so would be to create a trigger file based on the files name, so that you get something like
123.jpg
123.rdy
or
123.jpg
123.jpg.rdy
You create that file (just an empty stub) as soon as the upload is complete. The application that grabs files to load only cares about files with a trigger file and then processes those. Alternatively, you could also save the uploaded file as ie. 123.bsy or 123.jpg.bsy while it is still being uploaded and then rename it to the finale name 123.jpg after the upload is done. Since renames in the same directory are usually really cheap operations in term of processing time, the chances of running in a race condition should be pretty low. (This might or might now depend on the OS used, though ...)
If you need to keep the files in that place, you could, of course, use a database where you add a record for each file, as the upload is complete. The other app could then just provide files with a matching database record.
After writing this all down I figgered it out myself. What I did was adding the exact amount of bytes in the filename as well and validate that while outputting the list of images. The .tmp/.bsy-sollution is nice also, but I read it a bit to late :)
Upside to my solution is that no more renaming is required after the upload is done. Thanks everybody for your fast answers!
I'm creating an online game in PHP where users can create playable characters. Each character can have a user-uploaded portrait. A player can simultaneously have multiple characters, and the pictures for them can be changed anytime. Naturally, the pictures have to be resized and re-compressed to avoid huge files. Here's my problem:
When the player changes his data (among it the picture), and then hits "save", server side validation kicks in. It checks for things like non-unique character names, empty mandatory fields, etc. If any errors are found, they are displayed. In this case the form should be pre-populated with the data the player entered, so he only has to change the faulty bit, not re-type everything. But how do you save the picture in such a "temporary" state?
I cannot pre-populate the file upload field, the browsers don't allow that. If I save it in a temporary file, the picture then has to be cleaned up at some point, because the player can simply close his browser and abort the whole process. When should that be? And what file name should I choose for the temporary file? If the player opens the same character to edit in two browser tabs, they should not conflict (each of them should have their own copy).
How would you solve this problem?
I'd save the file to a temporary location and store both a unique identifier as well as the current timestamp in the filename. Then put the filename in the user's session. When a user has successfully created or updated their account, you save the image file to its permanent location and remove the temporary file. You can run a cron process to scan the temporary directory and check the timestamps, deleting any files older than your expiration (an hour perhaps).
If you're unable to run a cron job, you could always just launch the directory clean-up each time you have a successful create/update validation. This might be a bit more inefficient (extra directory reads and possibly file operations for every successful submission) but unless you're dealing with a lot of traffic, you probably won't even notice.
Create a table to hold references to images.
When a file is uploaded, if it's a valid image, do all your resizing, etc, and create a record in the table that points at the file.
Pass the id of the reference record around with the form data. Display the image when you redisplay the form, so the user knows they don't have to re-upload.
When you finally accept the new character object, set avatar_id or whatever.
Run a regular cron-job to cull orphaned image records (deleting the files on disk as well).
You could always populate a disabled text box to hold the name of the picture - it won't populate the browse input field, but is better than nothing. For editing, to avoid conflicts you could create some a "modifing" column for each user's characters, and on a character editing request change the value to true. When the user saves the character, set it back to false. For each edit request, grant it only when the "modifing" is false.
I'd recommend either updating the image immediately, regardless of error in the form, or separating the image updating to a separate form. That way you'll get rid of two problems without complex state machines and cleaning up.
I'm building a site were users can upload images and then "use" them. What I would like is some thoughts and ideas about how to manage temporary uploads.
For example, a user uploads an image but decides not to do anything with it and just leaves the site. I have then either uploaded the file to the server, or loaded it to the server memory, but how do I know when the image can be removed? First, I thought of just having a temporary upload folder which is emptied periodically, but it feels like there must be something better?
BTW I'm using cakePHP and MySQL. Although images are stored on the server, only the location is stored in the dbb.
Save the information about file to MySQL, and save the last time the image was viewed - can be done via some script that would be altered everytime the image is being used.. and check the database for images not used for 30 days, delete them..
You could try to define a "session" in some way and give the user some information about it. For example, in SO, there is a popup when you started an answer but try to leave the site (and your answer would be lost). You could do the same and delete the uploaded image if the user proceeds. Of course, you can still use a timeout or some other rules (maximum image folder size etc.).
I'm not sure what does "temporary upload" mean in your app. The file is either uploaded or not, and under the ownership of a user. If a user doesn't want to do anything at the moment, you have no other choice but to leave the file where it is.
What you can do is put a warning somewhere on your image management page about unused images, but removing them yourself seems like a bad practice (at least from the user perspective).
As a user,When I upload the image to a server(assuming I want to use it later) and leave the site, I don't expect it to be deleted if I am a registered user.
I would prefer it to be there in my acct until I come back.I would suggest thinking in those lines and implementing a solution to save the users' images if possible.
Check the last accessed/modified time of file to see it if has been used.