I have to create a photo online store for my organization with some additional features. But the main idea is to allow users to upload their digital photos (JPEG format only) and sell them in the store. This project is much related to the websites like iStockPhotos, Fotolia etc...
Is there any standard I should follow for the JPEG images like minimum and maximum sizes and quality?
I am able to use libraries like such as Imagine to make watermark and thumbnail images from the original. But my main concern here, is how may I store the original and duplicate files safely in a proper folder structure securely and how do I create download links to the original files when someone purchased it?
I am planning to build this from scratch using PHP Yii2 framework. So please give me any guidelines to make it successful. Thanks in advance.
I'll split my answer in three parts. The upload-, saving- and the download-process. I expect you have experience with PHP and Yii2 so I won't code the whole thing out but give you the gist.
1. Uploading photos
This one is simple. Simply create a form where you can upload your image-files. Make sure you set the correct enctype (multipart/form-data). The basics of file-uploading with Yii2 are explained here.
When saving your file you need to perform two steps:
save the original
create a resized and watermarked copy
For the second task I can recommend you the library yurkinx/yii2-image which I made really good experience with. It has resizing and watermarking built in.
2. Saving the photos
I usually put them in a dedicated folder within #runtime. The runtime folder is within your project-root and therefore not accessible from the web. Your folder could have the following route, which you put in your params.php for later reference.
#runtime/shop_images/
For the actual file-names you could use the id of the model which I'll explain next. Your file-names could be:
1.jpg (the original)
1_thumb.jpg (the resized and watermarked copy)
You should have a model for the photos. Each uploaded photo would be represented by a row in this models db-table. You also need a table to connect the users and their purchased images (m:n). Let me know if you need the exact schema.
Watch out...you need to create the model-instance first and save it. Otherwise you won't have the id to name the actual files! The easiest way is to overwrite the save()-method od your photo-model as beforeSave() is too early (no id yet!) and afterSave() is too late (no more rollback possibility).
3. Downloading the photos
This one is way easier than it sounds. You create an action within your photo-conroller which lets the user download the pictures.
In the action you first check if the user has purchased the picture. You can do this easily as you simply need to check if there is a row in your user_photo-table linking the currently logged in user to the requested photo.
If the connection is there you are ready to output the photo. This involves two steps:
Configure the response for outputting a photo
The actual output
Response configuration
You can do this completely manual or use my extension. This is the manual way:
public function actionPicture($id) {
//find the model (will throw excetion if not)
$model = $this->findModel($id);
//check if user is allowed
if (!UserPhoto::find()->user(Yii::$app->user->id)->photo($model)->exists()) {
//throw not allowed exception here!
}
//get the path to your photo
$photoPath= Yii::getAlias('#runtime/shop_images') . DIRECTORY_SEPARATOR . $model->id . '.jpg');
$photoPath= FileHelper::normalizaPath($imagePath);
//prepare the response
$response = Yii::$app->response;
$response->headers->set('Content-type', 'image/jpg');
$response->headers->set('Content-length', filesize($photoPath));
$response->format = \yii\web\Response::FORMAT_RAW;
//return the contents
return file_get_contents($photoPath);
}
To give you a little insight on this. Whatever you return from an action will be put in the data-attribute of your response. This will then be formatted into the content attribute of the response. How this will happen depends on the response format you set. In our case we chose FORMAT_RAW which leaves the data as is. Now we can simply return the contents of our photo-file and we are done.
You can further specify how your file gets processed by specifying additional headers. This way for example you can force a download with a certain filename:
$headers->set('Content-disposition', 'attachment; filename="' . $myCustomFileName . '"');
The same way you create an action for your thumbs or extend this one with a second parameter which tells you what size of the image to return.
I hope I could help. If you need any more information on one of the steps, don't hesitate to ask!
DB-Schema
Here is a possible migration to create the two tables needed. The user-table is omitted as you probably created this one already.
public function up() {
//the photo table and the relation to the owning user
$this->createTable('{{%photo}}', [
'id'=>$this->primaryKey(),
'owner_user_id'=>$this->integer(),
'original_filename'=>$this->string()->notNull(),
'filesize'=>$this->integer()->notNull(),
'extension'=>$this->string(4)->notNull(),
'meta_data'=>$this->text(),
]);
$this->addForeignKey('FK_photo_user', '{{%photo}}', 'owner_user_id', '{{%user}}', 'id', 'SET NULL', 'CASCADE');
//the m:n-table mapping users to their purchased photos
$this->createTable('{{%user_photo}}', [
'user_id'=>$this->integer()->notNull(),
'photo_id'=>$this->integer()->notNull(),
'PRIMARY KEY (user_id, photo_id)',
]);
$this->addForeignKey('FK_user_photo_user', '{{%user_photo}}', 'user_id', '{{%user}}', 'id', 'CASCADE', 'CASCADE');
$this->addForeignKey('FK_user_photo_photo', '{{%user_photo}}', 'photo_id', '{{%photo}}', 'id', 'CASCADE', 'CASCADE');
return true;
}
If you look at the schema you will get the gist.
Photo-metadata (EXIF)
If you wish to save meta-data for the photos, usually the EXIF-data, you can do this as well of course. I have to warn you though...there really is no real standard existing where the manufacturers of cameras store their values as I currently make the experience in one of our projects. In our example the most important information would have been the "date taken"-field. This alone can be in several places, depending the manufacturer. Making it even more complicated is the type of data within the EXIF. We had big trouble with illegal characters in there and about every charset-encoding you can imagine.
Nonetheless I'll give you a snipped how we save the data in JSON-format in text column. The big advantage is that you have all the in its original structure. I also added the column to the migration above.
$exif = exif_read_data($model->localFilePath, 'FILE,EXIF');
if ($exif !== false) {
$model->meta_Data = Json::encode($exif);
}
Make sure you read the documentation about exif_read_data() as it definitely has its kinks!
Related
What I want to archive:
I want to write a feature test using phpunit in Laravel.
What does the controller I want to test do:
It accepts uploads of test records to store it into a database. Each record consists of two files (xml, yml) with the same file name. Both files have to be read and stored in the database. The uploaded files are never stored on the server, they are directly processes.
What I want to test:
Upload a record and check if the correct data is in the database and available over the API
Check if I get the correct error if one file (xml or yml) is missing
Check if I get the right error if the files are not a valid record
and so on ...
What is my current problem?
I would like to use a template for the xml and yml files and use Faker to generate fake data for this test. The reason: Why not? My understanding of testing is, that you should test as many cases as possible and if static data is enough why do we use Faker and Factory in the Unit tests for the database and so on?
When I look at Laravel: Testing File Uploads, they generate there testing files with UploadedFile::fake(). My understanding of those files is, that they are empty and you can't use a template or something like that, to fill it with useful data. Most solutions I found just kept real files in their project. Like this.
I could use blade for this, as shown here, but I'm not really sure if I should abuse blade like this.
I could fully generate the xml and yml files using Yaml and XMLReader/XMLWriter, but there is a lot of static text in those files and I only need to fill data into some specific points.
Questions:
So what is the best way to create such a fake file? Should I use blade or twig or some other templating engine? A small solution would be appreciated.
Or should I generate the full file by myself and why is this better?
Or is there no point in generating fake data and I should use static data instead?
Here is my current test function:
public function testFullRecordUpload() {
// Generate record files to upload
// TODO Use template files with faker, or something like that
$xml_file = UploadedFile::fake()->create('test_file.xml', $sizeInKilobytes = 566);
$yml_file = UploadedFile::fake()->create('test_file.yml', $sizeInKilobytes = 20);
// Send Files
$response = $this->json('POST', '/upload', [
'xml' => $xml_file,
'yml' => $yml_file
]);
// Check response
$response->assertOk();
// Check if the uploaded data is available over API
// TODO
}
I have a website with images upload/show functionality on it. All images are saved into filesystem on a specific path.
I use Yii2 framework in the project. There isn't straight way to the images and all of them requested by specific URL. ImageController proceses the URL and takes decision about image resizing. ImageModel does the job. The user get image content.
Here the code snippet:
$file = ... // full path to image
...
$ext = pathinfo($file)['extension'];
if (file_exists($file)) {
// return original
return Imagine::getImagine()
->open($file)
->show($ext, []);
}
preg_match("/(.*)_(\d+)x(\d+)\.{$ext}/", $file, $matches);
if (is_array($matches) && count($matches)) {
if (!file_exists("{$matches[1]}.{$ext}")) {
throw new NotFoundHttpException("Image doen't exist!");
}
$options = array(
'resolution-units' => ImageInterface::RESOLUTION_PIXELSPERINCH,
'resolution-x' => $matches[2],
'resolution-y' => $matches[3],
'jpeg_quality' => 100,
);
return Imagine::resize("{$matches[1]}.{$ext}", $matches[2], $matches[3])
->show($ext, $options);
} else {
throw new NotFoundHttpException('Wrong URL params!');
}
We don't discuss data caching in this topic.
So, I wonder about efficient of this approach. Is it ok to return all images by PHP even they aren't changed at all? Will it increase the server load?
Or, maybe, I should save images to another public directory and redirect browser to it? How long does it take to so many redirects on a single page (there are can be plenty images). What about SEO?
I need an advice. What is the best practice to solve such tasks?
You should consider using sendFile() or xSendFile() for sending files - it should be much faster than loading image using Imagine and displaying it by show(). But for that you need to have a final image saved on disk, so we're back to:
We don't discuss data caching in this topic.
Well, this is actually the first thing that you should care about. Sending image by PHP will be significantly less efficient (but still pretty fast, although this may depend on your server configuration) than doing that by webserver. Involving framework into this will be much slower (bootstrapping framework takes time). But this is all irrelevant if you will resize the image on every request - this will be the main bottleneck here.
As long as you're not having some requirement which will make it impossible (like you need to check if the user has rights to see this image before displaying it) I would recommend saving images to public directory and link to them directly (without any redirection). It will save you much pain with handling stuff that webserver already do for static files (handling cache headers, 304 responses etc) and it will be the most efficient solution.
If this is not possible, create a simple PHP file which will only send file to the user without bootstrapping the whole framework.
If you really need the whole framework, use sendFile() or xSendFile() for sending file.
The most important things are:
Do not use Imagine to other things than generating an image thumbnail (which should be generated only once and cached).
Do not link to PHP page which will always only redirect to real image served by webserver. It will not reduce server load comparing to serving image by PHP (you already paid the price of handling request by PHP) and your website will work slower for clients (which may affect SEO) due to additional request required to get actual image.
If you need to serve image by PHP, make sure that you set cache headers and it works well with browser cache - you don't want to download the same images on every website refresh.
I know there is a big amount of questions about this but I cannot get one that involves all I want to be aware of.
What I want to do is to allow the users of my webpage to upload images with a form. And I want to do this process secure, or at least as much secure I can.
I do not know too much about security in terms of deep inside of it, but I am aware of about all the consequences that a insecure webpage can produce. And I cannot be quiet thinking that my webpage is insecure or that anyone is not going to enter into my webpage because it does not have enough visits(I am realist).
At this point, I know that all the checks about security have to be done on server side instead of client side (or in both).
I know that a file can be fooled as an image and run malicious code so I searched about methods to avoid this. This is what I could find to check before store the image on the server:
From $_FILES:
$_FILES['file']['name']: To check that the file that I have uploaded have a name. To know that the file exists.
$_FILES['file']['error']: To check if the image have an error.
$_FILES['file']['size']: To check that the size of the image is bigger than 0.
$_FILES['file']['type']: To check that the type of the file is an image but it is not recommended because PHP does not check it.
General functions:
Check magic numbers to verify the image type.
exif_imagetype(): To check the type of an image.
getimagesize(): To check if it returns a 0 which means that the file is not an image.
imagecreatefromstring(): To create a new image giving a string. If it cannot be created, then is not an image.
imagepng: To create a PNG image to remove all meta-data (using imagecreatetruecolor() and imagecopy()).
But the problem I have is that I do not know if I should use all of these methods or just avoid or add some of them (because some of them seems redundant).
And my questions are:
Should I use all of them?
Have I to add another one method to be more secure?
Could be the order in which I filter the file critic? I mean, is it better to use one filter before another and viceversa? If so, what should be the order and why?
Note: I am not searching about personal opinion. I tried to gather all info I could, but I cannot be sure if it is ok or not talking about security terms. If you can put examples of something that it is forgotten it would be great.
Thanks in advance!
To answer your questions:
You don't need to use all of those methods, and which ones you use are going to be based on personal opinion. Meaning to say, there is more than one perfectly secure way to do it so don't be suprised if you get multiple different answers.
See examples below for additional checks you might have left out
Yes, the order definitely matters.
Depending on your application, the logic for any secure upload should flow something like this:
Is the user logged in? (optional)
// make sure user is logged in
if (!$user->loggedIn()) {
// redirect
}
Does the user have permission? (optional)
// make sure user has permission
if (!$user->isAllowed()) {
// redirect
}
Was the form submitted?
// make sure form was submitted
if ($_SERVER['REQUEST_METHOD'] == 'POST') {
Is the form input valid?
// validate CSRF token
// ...
// make sure there were no form errors
if ($_FILES['file']['error'] == UPLOAD_ERR_OK) {
// make sure the file size is good
if ($_FILES['file']['size'] <= MAX_FILE_UPLOAD) {
// make sure we have a valid image type
$type = exif_imagetype($_FILES['file']['tmp_name']);
if ($type !== false) {
// make sure we check the type against a whitelist
if (in_array(ltrim(image_type_to_extension($type), '.'), array('jpeg', 'jpg', 'png'))) {
Even after validating, never trust user input
// give the file a unique name
$hash = hash_file('sha1', $_FILES['file']['tmp_name']);
$ext = image_type_to_extension($type);
$fname = $hash . $ext;
Save the file (or optionally recreate it with a library to strip out meta-data) but NEVER in a publicly accessible directory
$upload_path = '/path/to/private/folder';
move_uploaded_file($_FILES['file']['tmp_name'], "$upload_path/$fname");
The steps above are perfectly secure and more than reasonable, of course there is always a risk that some other part of your application or server might be vulnerable.
I would do the following with an apparent image upload:
1) Use is_uploaded_file() to ensure you've not been fooled into working on something else entirely
if(!is_uploaded_file($yourfile))
return false;
2) Check the mimetype with exif_imagetype() and block anything you don't want
$allowed_images = array(IMAGETYPE_BMP, IMAGETYPE_GIF, IMAGETYPE_JPEG, IMAGETYPE_PNG);
$uType = exif_imagetype($yourfile);
if(!in_array($uType, $allowed_images))
{
unlink($yourfile);
return false;
}
3) Use Imagick to remake the image and remove all comments and metadata:
$image = new Imagick($yourfile);
$image->resizeImage($image->getImageWidth(), $image->getImageHeight(), Imagick::FILTER_CATROM, 1);
$image->stripImage(); // remove all comments and similar metadata
4) Write the replacement image to the filesystem and erase the original file:
$image->writeImage("/path/to/new/image");
unlink($yourfile);
5) Upload this image to S3.
// your S3 code here
6) Make a note of the image's S3 URL in the database or wherever.
// your database code here
7) Erase the replacement image.
unlink("/path/to/new/image");
If you get enough responses, you might have a good answer! :-)
Operating System
Make sure you have a dedicated volume for the files. Or, at minimum, have quota set on the directory. Make sure you have enough inodes and such, if on Linux/Unix. A bunch of small files can be just as deadly as a few gigantic files. Have a dedicated uploads directory. Set where the temp files should go in your php.ini. Make sure your file permission are safe (chmod), too. Use Linux ACLs, if necessary, to fine tune permissions. Test, test, test.
PHP
Incorporate the knowledge found here into your uploaded file handling algorithm PHP Manual: POST method uploads. Take the MAX_FILE_SIZE bit with a grain of salt.
Make sure you know what your max up load file size is. Set it accordingly. There may be other file related settings. Be sure to lock those in before getting around to the $_FILES superglobal.
Do not work with the uploaded files directly, and do not use the name attribute at all to give the file a real file name. Use, is_uploaded_file() and move_uploaded_file() appropriately.
Use tmp_name appropriately.
Be wary of null bytes on file names! Yes, you still need to filter and validate any string that represents user input (especially if you intend on using it any way).
First things first, check for the presence of a file.
Second, check the size in bytes.
If anything in #5 or #6 fail, the validation process should end. For a robust routine, incorporate the idea that at sometime you may want to upload multiple files at one time (PHP Manual: Uploading Multiple Files). In that case, the $_FILES superglobal may not look like you would expect. See the link above for more details.
GD
You've got the general idea about using these functions to open the submitted file (without using the user submitted name, that is). Just come up with a logical series of progressive steps. I don't have those steps, but if meta-data can be a problem, that would seem high on the list of GD stuff to try early (after basic file presence and size stuff). I could be wrong though.
So im making a website with an image upload functionality and im storing the image name to the database. I took a screenshot of my mac and wanted to upload this photo "Screen shot 2011-02-18 at 6.52.20 PM.png". Well, thats not a nice name to store in mysql! How do people ususally rename photos in such a way that each photo uploaded has a unique name? Also, how would i make sure i keep the file extension in the end when renaming the photo.
I would drop the extension, otherwise Apache (or equivalent) will run a1e99398da6cf1faa3f9a196382f1fadc7bb32fb7.php if requested (which may contain malicious PHP). I would also upload it to above the docroot.
If you need to to make the image accessible above the docroot, you can store a safe copy that is ran through image functions or serve it from some PHP with header('Content-Type: image/jpeg') for example and readfile() (not include because I can embed PHP in a GIF file).
Also, pathinfo($path, PATHINFO_EXTENSION) is the best way to get an extension.
Ensure you have stored a reference to this file with original filename and other meta data in a database.
function getUniqueName($originalFilename) {
return sha1(microtime() . $_SERVER['REMOTE_ADDR'] . $originalFilename);
}
The only way this can generate a duplicate is if one user with the same IP uploads the same filename more than once within a microsecond.
Alternatively, you could just use the basename($_FILES['upload']['tmp_name']) that PHP assigns when you upload an image. I would say it should be unique.
Hash the image name. Could be md5, sha1 or even a unix timestamp.
Here is an (untested) example with a random number (10 to 99)
<?php
function generate_unique_name($file_name)
{
$splitted = split(".", $file_name);
return time() . rand(10,99) . "." . $splitted[count($splitted)-1];
}
?>
You could use an image table like:
id: int
filename: varchar
hash: varchar
format: enum('jpeg', 'png')
The hash can be something like sha1_file($uploaded_file) and used to make sure duplicate images aren't uploaded. (So you could have multiple entries in the image table with the same hash, if you wanted.) The id is useful so you can have integer foreign key links back to the image table.
Next store the images in either:
/image/$id.$format
or
/image/$hash.$format
The second format via the hash would make sure you don't duplicate image data. If you are dealing with lots of images, you may want to do something like:
/image/a/b/c/abcdef12345.jpg
where you use multiple layers of folders to store the images. Many file systems get slowed down with too many files in a single directory.
Now you can link to those files directly, or set up a URL like:
/image/$id/$filename
For example:
/image/12347/foo.jpg
The foo.jpg comes from whatever the user uploaded. It is actually ignored because you look up via the id. However, it makes the image have a nice name if the person chooses to download it. (You may optionally validate that the image filename matches after you look up the id.)
The above path can be translated to image.php via Apache's MultiView or ModRewrite. Then you can readfile() or use X-SendFile (better performance, but not always available) to send the file to the user.
Note that if you don't have X-SendFile and don't want to process things through PHP, you could use a RewriteRule to convert /image/$hash/foo.jpg into /image/a/b/c/$hash.jpg.
HI
I have a forum and I'm trying to think of how to do an "attachment" feature.
You know if you make a thread you can chose to upload a file and attach it in the thread.
Should I make a table called attachment with id of the file id in table files?? Whats the best way. And I want you to be able to upload more than 1 attachment. and if it's a picture show a little miniature of the picture.
How should I check if the file exist etc? How would you do this?
Sorry for my poor english
You question is too broad but I'll give you some pointers:
store the images on the disk, something like /uploads/--thread_id--/1.jpg, /uploads/--thread_id--/2.jpg and so on (this way you don't have to make any changes to your DB)
Regarding the upload process, validation and image resizing you can read more at (I recommend you read them in this order):
http://pt.php.net/manual/en/function.exif-imagetype.php -> image validation
http://php.net/manual/en/function.move-uploaded-file.php -> upload process
http://pt.php.net/manual/en/book.image.php -> image resizing & manipulation
Chacha's plan sounds good to me, but you have to be careful. Make sure the files that you save don't have any execution permissions and that the file isn't on a web-accessible directory on your server. I think you should put the upload directory in a directory higher than your web directory for security purposes.
Another possible way to save the files: save their binary code in blobs in the database. I'm not sure if there are any advantages to this method, but I haven't personally had to deal with file uploads.
Above all else, be careful with uploaded data!
I honestly would create a Column on the table of posts that says 'Attachments', and then do a comma delimited string of attachment file names
file1.png,file2.png,file3.png
then when you get it into PHP, simply explode it
$attachments = explode(',', $string);
and check for each file that you have already put in your upload directory:
foreach($attachments as $file)
{
if(!is_file($upload_directory.$file))
{
$error[] = $file . " is not a valid attachment";
// run cleanup script
}
}
To get the attachments, it is really simple code, but you need to validate and sanitize the incoming file.
foreach($_FILES as $array)
{
// Sanitize Here
die("SANITIZE HERE!");
move_uploaded_file($array['tmp_name'], $upload_dir);
}