I have a bunch of images in Amazon S3 that I need to physically rotate. I currently do this by downloading the image to my server, rotating it using GD and overwriting it back to S3.
This process takes ~5 secs per image. I was wondering if there is any AWS API or such that can do this rotation directly in S3, preferably as in a batch mode?
I would appreciate it if anyone who has any experience with that kind of stuff can give me any pointers!
There is no way to rotate an image 'on' S3. Any method you employ is going to have to read the file from S3, do the rotation, and write it back to S3.
If the server you are doing it on now is not an EC2 instance, than its worth a try to do it there - the latency will be reduced quite a bit. Lambda is another option for you in that it will run within the AWS infrastructure, so network overhead will be reduced.
Not quite sure what your constraints might be, but if you're preparing the images for a web page - you could rotate them client-side using CSS. That would prevent the additional calls to S3, and eliminate processing load on your application server.
img {
transform: rotate(90deg);
}
Related
I have a simple image processing server that handles authentication of images, resizing, cropping, etc.
If it recently did a relatively large operation such as resize an image, it will upload the new resized version back to Cloud Storage, which it will use next time that exact size is needed again.
My question is, next time someone requests that image, or the server is serving an unaltered image "directly" from the Google Cloud Storage bucket, what's the best method for my PHP server to "pass on" the data of the image to the user, using as little processing and memory as possible?
Right now my current working code boils down to:
if ($image->exists()) {
header("Content-Type: image/jpeg");
echo $image->downloadAsString();
}
Which, just from looking at it, seems like a bad way to do things. But I don't know so I'm asking.
Edit:
I am not looking for the most efficient way to serve images to the client in general or in any larger scheme, I am looking for the most efficient function(s) I can use already in PHP to accomplish what I'm trying to do with as little latency, CPU usage, and memory as possible.
"What I'm trying to do" being: In a running PHP script, verify that the user can access the image, check if the resized image exists, and if it does, relay it to the client. If it doesn't, resize the image, send it back to cloud storage for later retrieval, then send the resized image to the user. The only part I'm concerned about right now is how to pass the data from the bucket to the user.
Assume all servers including the bucket itself are in the same zone. I am not using a framework. I need to serve through the webserver, and for those that I don't need to, I use a load balancer to serve directly from the cloud storage. I am not using a CDN.
Using downloadAsStream() and stream_copy_to_stream() should give you the best results. Eg:
if ($image->exists()) {
header("Content-Type: image/jpeg");
stream_copy_to_stream($image->downloadAsStream(), STDOUT);
}
In this case PHP should copy file data directly to output as it is retrieved with maximal efficiency, memory or otherwise.
If you scroll up a bit in the source downloadAsString() just calls the stream and returns the whole thing as a string, which will cram all that file data into memory.
Also, for best results, grab the StorageObject metadata and set a Content-Length: header. This generally makes HTTP clients behave a bit better. I believe this will be in what's returned by $image->info().
I've been reading a lot about dynamic image manipulation, storage and content delivery, the company I'm working for already uses AWS for some of their services.
The application I'm working on, store document images to a S3 bucket (not limited to), and i need to display them on demand.
The first version of this application, stored the images locally and performs the image manipulation on-demand on the same server.
Now, the documents storage has increased and a lot of images are being stored, all this via web application, this means that one user may upload say 100+ images and the server needs to process them as fast as it can.
That's why the images are uploaded to an EC2 instance and they are streamed to a S3 bucket internally, that's how we save the original image in the first place, no thumbnails here to speed up the uploading process.
Then a different user may want to preview this images or see them in original size, this is why i need to dynamically re-size them, i will implement Cloudfront for the image caching after they are re-sized, and here comes the issue.
The workflow is like this:
1. User Request CDN image
2.a Cloudfront Serves the cached image
2.b Cloudfront request the image to a custom origin if its not cached
3. The origin server query S3 for the image
4.a If the image size exists on S3
5. Return the image to Cloudfront, Cache and return to user
4.b If the image size does not exists on S3
5. Generate a image size from the original S3 image
6. Save the new size to S3
7. Return the new size to Cloudfront, Cache and return to user
The custom origin is responsible of creating the missing image size and save it to S3, the Cloudfront can use the cached image or request this new image size to S3 as it now exists.
I think this is possible, as i already read a lot of documentation about it, but i still haven't found documentation of someone who has made this before.
Does this looks like a good way of handle the image manipulation, has anyone saw any documentation about how to do this.
I'm a PHP developer but i might be able to implement a non-PHP solution in favor of performance on the image server.
If you are open to non-PHP based solutions https://github.com/bcoe/thumbd is a good option since it already integrates S3, etc. However you will need to know the sizes you need ahead of time. I would recommend such an approach rather than generating sizes on the fly since it means faster response times for your user. Your user will not have to wait while the new size is being generated. Storage on S3 is incredibly cheap and so you will not be wasting any $$ by creating multiple sizes either.
I have another issue with amazon and its related to file uploads.I am using jqueryFileUpload and amazon API's to uplaod files to amazon S3.I have succeeded in uploading it,but it involves a trick.
I had to store the image on my server and then move it to S3 from there using putObjectFile method of S3.Now the plugin comes with great functions to crop/resize images and I have been using them since long.Now when I integrate the plugin with AWS,i am facing performance issues with upload.The time taken for uploads is longer than normal and this raises questions of us using AWS S3 over traditional way.
I had to make changes to my UploadHandler.php file to make it work.These are the changes made.i added a part of AWS upload code to the file from line 735 to 750
$bucket = "elasticbeanstalk-2-66938761981";
$s3 = new S3(awsAccessKey, awsSecretKey);
$response = $s3->putObjectFile($file_path,$bucket,$file->name,S3::ACL_PUBLIC_READ);
$thumbResponse = $s3->putObjectFile('files/thumbnail/'.$file->name,$bucket,'images/'.$file->name,S3::ACL_PUBLIC_READ);
//echo $response;
//echo $thumbResponse;
if ($response==1) {
//echo 'HERER enter!!';
} else {
$file->error = "<strong>Something went wrong while uploading your file... sorry.</strong>";
}
return $file;
Here is a link to s3 class on git.
The normal upload to my current server(not amazon),same image uploads in 15 secs,but on amazon S3 it takes around 23 secs and I am not able to figure out a better solution.I have to store the image on my sever before uploading to S3 as I am not sure if I can process them on the fly and upload directly to S3.Can anyone suggest the right way to approach the problem?Is it possible to resize the images to different sizes in memory and upload directly to S3 avoiding the overhead of saving it to our server?If yes can anyone guide me in the right direction?
Thank you for the attention.
I believe the approximate 8secs is the overhead here for creating versions of image in different sizes.
You may take different approaches to get rid of the resizing overhead at time of upload. The basic idea will be to allow the uploading script to finish execution and return the response, and do the resizing process as a separate script.
I like to suggest following approaches:
Approach 1. Don't resize during the upload! Create resized versions on-the-fly only when it is being requested for the first time and cache the generated images to serve directly for later requests. I saw a few mentions of Amazon CloudFront as a solution in some other threads in Stackoverflow.
Approach 2. Invoke the code for creating resized versions as a separate asynchronous request after the upload of original image. There will be a delay in scaled versions being available. So write necessary code to show some place holder images in the website until the scaled versions become available. You will have to figure out some way to identify whether scaled version is available yet or not(For example check file is existing, or set some flag in database). Some ways for making asynchronous cURL requests are suggested here if you would like to try it out.
I think both approaches will have equal level of complexity.
Some other approaches are suggested as answers for this other question.
I was thinking of using imagemagick to process images uploaded by a user in various ways (creating new images that are scaled, have drop shadows, etc.) but I've been worried about the speed. I don't want the user staring at a loading gif forever.
So I started looking around to see how other sites do it and I found http://www.redbubble.com. Users upload artwork and almost instantly there are tons of variations of the image in the shop processed in various ways. What does it use to process and generate images so fast?
it's relatively hard and inconvenient to maintain client-side image processing (it would be some kind of flash app similar to www.picnik.com with limited functionality)
I see use of ruby, nginx, remote xhr calls, json etc. that means that delayed_jobs/resque might be used to schedule asynchronous image processing using imagemagick, json/xhr to check the status. processed images are requested from ih*.redbubble.net (point to edgecastcdn.net) and seems like they produce them on the fly and let CDN cache them until user changes that image or it expires in the cache.
they have ~800k monthly visitors, you don't want to put load on app/web servers to process images, there is either delayed_jobs or resque behind the scene or ih* (image host?) servers that produce images on the fly (there are 4 of them, but who knows how many behind virtual host/proxy configuration)
all upload requests go to amazon (ec2, that might be a load balanced IP), originals are stored on amazon s3. they can scale by requesting more ec2 instances on demand.
hope you get an idea what's behind... back to your question: no client-side images processing, imagemagick is used, and there is a chance they do it on the fly.
I will be launching an application in the very near future which will, in part, require users to upload files (images) to be viewed by other members. I like the idea of S3 as it is relatively cheap and scales automatically.
My problem is how I will have users upload their images to S3. It seems there are a few options.
1- Use the php REST API. The only problem is that I can't get it to work for uploading variously scaled versions (ie thumbnails) of the same image simultaneously and uploading them directly to s3 (it works for just one image at a time this way). Overall, it just seems less flexible.
http://net.tutsplus.com/tutorials/php/how-to-use-amazon-s3-php-to-dynamically-store-and-manage-files-with-ease/
2- The other option would be to mount an S3 bucket with s3fs. Then just programmatically move my images into the bucket like I would with NFS. From what I've read, it seems some people are dubious of the reliability of mounting S3. Is this true?
http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=fuse+over+amazon
Which method would be better for maximum reliability and speed?
Would EBS be something to consider? I would really like to have a dedicated box rather than use an EC2 instance, though...
For your use case I recommend to use the S3 API directly rather than using s3fs because of performance. Remember that s3fs is just another layer on top of S3's API and it's usage of that API is not always the best one for your application.
To handle the creation of thumbnails, I recommend to decouple that from the main upload process by using Amazon Simple Queue Service. That way your users will receive a response as soon as a file is uploaded without having to wait for it to be processed resulting in shorter response times.
As for using EBS, that is a different scenario. EBS is just a persistent storage for and Amazon EC2 instance and it's reliability doesn't compare with S3.
It's also important to remember that S3 only offers "eventual consistency" as opposed to a physical HDD on your machine or an EBS instance on EC2 so you need to code your app to handle that correctly.