I'm writing an app that let's you share files in the cloud.
You make a user-account and can upload your files with and send links to friends.
I'm using Amazon S3 to save the data.
But I'm not sure how I should proceed.
There are buckets, which you can create in S3, and in those buckets you save your files.
I thought about making a bucket for each user, but then I read that you can only have 100 buckets at a time.
Isn't there a better way to managing this then to just save all user files in one "directory".
This will get so messy. I have never used S3 before, I would be very thankful for any advice.
And if this is the only way, what naming convention proved to be the best?
Thanks!
Even though S3 has a flat structure within a bucket, each object has its own path much like the directories you're used to.
So you can structure your paths like so:
/<user-id>/<album-1>/...
One thing to keep in mind is that not all directory related features are available, such as:
Deny access to /<user-123>/*,
Copy from one directory to another.
Related
I am developing Cloud File Storage that can be accessed by specific users.
What is the best way to store folders and files?
What I mean is either the folder created in Database or the folder create in the real folder in server.
These way must be considered to which one is fast and secure.
All the methods has its own advantages and should be used based on your use case.
1) Storing in Database
2) Storing in real folder
or , may be
3) storing in AWS cloud storage like s3
Some of the pros / cons revolve around
Size of the Individual file
How many files are you storing.
Concurrent users trying to download each file.
How do you plan
to do Backup etc.
Do have look at these article
https://habiletechnologies.com/blog/better-saving-files-database-file-system/
https://softwareengineering.stackexchange.com/a/150787
I have a quiz site which creates images from a set of source images, the result images are stored in S3 and i don't care about it. My question is about source images, S3 or EFS is better for storing the source images for this purpose. I am using php to create result images.
Here's a general rule for you: Always use Amazon S3 unless you have a reason to do otherwise.
Why?
It has unlimited storage
The data is replicated for resilience
It is accessible from anywhere (given the right permissions)
It has various cost options
Can be accessed by AWS Lambda functions
The alternative is a local disk (EBS) or a shared file system (EFS). They are more expensive, can only be accessed from EC2 and take some amount of management. However, they have the benefit that they act as a directly-attached storage device, so your code can reference it directly without having to upload/download.
So, if your code needs the files locally, the EFS would be a better choice. But if you code can handle S3 (download from it, use the files, upload the results), then S3 is a better option.
Given your source images will (presumably) be at a higher resolution than those you are creating, and that once processed, they will not need to be accessed regularly after while (again, presumably), I would suggest that the lower cost of S3 and the archiving options available there means it would be best for you. There's a more in depth answer here:
AWS EFS vs EBS vs S3 (differences & when to use?)
Hey just a quick question for anyone who has done this. I want to create a video tube site. I have done file uploads before but was wondering if anyone could give me suggestions on what I am planning to do.
The way I am planning is to have a folder in my web directory and to upload videos into the folder after virus scanning and checking mime. The video will then be converted and compressed using FFMPEG into flv.
I will change the name and store the video reference id in mysql so the file name can be fetched and served.
I will serve the files using HTTP_Download to a flash player
$dl = new HTTP_Download();
$dl->setFile("$path");
$dl->setContentDisposition(HTTP_DOWNLOAD_ATTACHMENT, "$path");
$dl->setContentType('video/flv');
$dl->send();
Anyone have any suggestions? Is it a good idea to put all videos in one directory?
You may want to consider a Java based uploader as PHP can run into timeout problems on large uploads.
Also do you FFMPEG processing as a CRON job not at upload as it takes a long time.
Look in something like Wowza Streaming Server to serve the videos. Allows streaming and everything is above the root. I name each video with a UID and send a parameter to the Flash video player to decide which one to play.
Where and how you store them will largely depend on how secure they need to be (i.e. should people be able to access the files in the directory directly? or should it be stored more securely than that?)
If direct access is fine, then putting them all in one folder is okay. If not, then you may want to obscure folder names, store them in a secure Database, or in a folder that is not accessible outside of the server.
Also, I'm hoping you're aware of the massive amounts of storage space and bandwidth such a service will consume? I hope you have a scaled solution ready to deploy if you're really serious about this..
Is it better to read and list images directly from file system using simple php, or is it better to store image meta info and filename in the database and access the images by doing a mysql select. What are the pros and cons of both solutions.
Listing files on a file system is probably the easiest way to accomplish what you trying to do but it's going to be very slow if you are trying to cycle through several thousand directories/files on a networked file system (NFS, CIFS, GlusterFS, etc).
Storing files in a database will create a much more overhead since you are now involving an external application to store information. You have to remember that every time you are using a database you are also using network I/O, authentication mechanism, query parser, etc. At the same time all of this overhead might provide for a faster response then using a networked file system.
To conclude - everything depends on amount of files you are working with and underlying infrastructure. Two major things to look out for are going to be disk I/O and network I/O.
I would do the following:
Upload all the images in one directory
Store references to those images that are tied to the uploader's User ID
Then just select the image URLs that are tied to that ID, and output them however necessary.
People find it easier to store their files within folders and parse that folder with php. If you go the database method the database eventually gets larger and larger and larger.
I can see it becoming personal preference, but I personally have gone with parsing folders for images rather than storing it within a database.
Depends on the scale of what you are doing.
This is what I would be doing.
Store the file metadata in the database. You can store quite a bit of information about this image this way.
Store the image file on a distributed storage system like Amazon S3. Store the path in your metadata. Replication is part of the system. And it easily integrates with Cloudfront CDN.
Distribute the the images through Amazon Cloudfront CDN.
I will be launching a web application soon which will require users to upload pictures for others to view. I would like to use Amazon S3 to store the images as it scales and is cheap. The user will use a form to upload their file, it will be processed with php and saved to the S3 mount thats attached to the web server.
I am anticipating and hoping tens or hundreds of thousands of images will eventually be uploaded.
My first question is whether an S3 bucket mount is robust and fast enough for such an application, or would I be better off using Amazon EBS. Although I'd like to have my own dedicated box rather than use an EC2 instance.
Also, I am at this point unfamiliar with S3, but when I do upload files, is it appropriate to put them in a single bucket rather than a cascade of directories? It seems it might be ok since each 'bucket' is virtual anyway.
One of the things you can do is to have your users upload to S3 bucket directly, unless you want to do some processing. You can use POST to upload the files to S3 or one of the 3rd party components such as http://flajaxian.com/ This way you can significantly offload your server.
As for your second question it is actually up to you how your design your app. there is no pros and cons.