I am currently using an implementation of the php sdk for rackspace to upload files to a container called testcontainer. I am using a library that interacts with the sdk but I think I'll need to write it natively to accomplish what I'm looking for. Before I do this I haven't been having much luck finding out how to clear rackspace's cache.
The problem I run into (in case its not a caching issue for whatever reason) is...
Upload a file called test.jpg
visit the cdn endpoint /test.jpg and see my image
locally change the image to something else but keep the name test.jpg.
Upload the file to the same cdn container replacing the other test.jpg.
visit the cdn endpoint/test.jpg however the image is still the original test.jpg, not my new image.
This looks like a caching issue, I know in your account you can clear the cache and reset it but haven't been able to find any reference in the documentation.
Any ideas? Thanks.
If you have set your CloudFiles containers as CDN what you are seeing is indeed a caching issue. Unfortunately there's no practical way to flush the caches at scale and even if it was just one object, it could take minutes to be propagated globally. For more info how to flush CDN caches for individual objects and limits thereof, please see: here.
And of special attention:
You can use the DELETE operation against a maximum of 25 objects per day using the API
Related
Amazon php sdk (http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-transfer.html) sync operations are performed using the above linked methods. I've successfully coded and configured them, but on each call for the method the last modified date on files under the bucket gets updated to the latest time, given that the files were not modified locally to the previous sync call.
I wonder whether is it a sync operation at all or just overwriting operation of whatever is sent from the local directory?
Why this matters is we are planning to sync gigs of files in between a server and S3 bucket. Using S3 buckets as backup storage, in case of any disruptions we can sync (S3 bucket -> server) the opposite way to make the missing pieces of data available in our server.
Notes:
I've also tried this one from here
Currently I'm using version 3 of the AWS php sdk
Unfortunately I believe the answer is no, I also see complete upload of every file when using the Transfer class.
It used to work, from the v2 API docs:
The uploadDirectory() method of a client will compare the contents of
the local directory to the contents in the Amazon S3 bucket and only
transfer files that have changed.
Perfect, that's what we want!
However, in v3 they have preserved S3Client::uploadDirectory() for API compatibility I guess, but it's just a wrapper for Transfer::promise(), which as we know just uploads without any syncing. Keeping API compatibility but changing behavior like this seems like a really bad idea to me.
I ended up having to add support to my project to use the AWS CLI tool for the actual uploading, which does support sync perfectly. Far from ideal.
If there is a way to use the Transfer class for easy syncing instead of complete uploading, I hope someone can prove me wrong.
I am making a php mysql web app, my idea is to install in the customer home a web server so they can connect with whatever device they want, probably most of the time they will be using an ipad to connect to the app.
Sometimes the client needs to take out the app with them in the ipad, so after discarding other options(like phonegapp because i need to mantain a mysql db for some functions)i realized that Application Cache may be a good solution: They can use the application with the web server(using db functions like randomize the content, generate Statistics)and when they are offline they can access a local copy of the content, with limited function but working.
The problems that i have is that the site have images,video and audio so at least there are 20mb to cache, i read that with application cache you can only store 5mb and the other problem is that my content is dynamic so i cant add all the files that i need to the cache manifest. I want something like make a wget of the site(save an static html file) and to use the dynamic content when online. I dont know if i can make something like that.
Thanks
the cache.manifest for the ipad can store more than 5mb.
the currently ios limit is 50mb.
if you cache more files, automatically the ipad ask if you want increase the storeage to 50mb.
take a look at this
it explains you how to create and implement the cache.manifest. its a great tutorial
hope this help.
I'm trying to create a simple script which takes a URL in a form, download the file and deliver that file to the user. Something like a proxy server, only for downloading files. The only problem is that the server has limited execution time of 10 seconds which will fail for most large files. I can't change the execution time (using set_time_limit) because that's blocked too. Is there ANY way I can get past this?
you can use an cloud storage service to store the file for you
google drive api
dropbox-php
to "deliver" the file to the user you share the link on your cloud storage service.
Ps: Sorry for the previous wrong answer, ftp_get only works if you are trying tpo get it from an ftp server
Get a new web host, a cheap one that I can recommend is Dreamhost which is pretty darn cheap and they have a lot of PHP ini settings you can override (but not all). Or, if you're just playing around and are looking for something temporary, I recommend AWS EC2, the micro instance is as cheap as $0.02/hour depending on the region you select and you get 1 month for free, but most importantly, you get FULL root access.
Edit:
Forgot to mention where to view override PHP settings info: wiki.dreamhost.com/index.php/PHP.ini
(sorry I can't make it a link, I'm a n00b on stackoverflow and am limited to 2 links)
Currently I am looking to move my websites images to a storage service. I have two websites developed in PHP and ASP.NET.
Using Amazon S3 service we can host all our images and videos to serve web pages. But there are some limitations using S3 service when we want to serve images.
If website needs different thumbnail images with different sizes from original image, it is tough. We have again need to subscribe for EC2 also. Though the data transfer from S3 to EC2 is free, it takes time for data transfer before processing image resize operation.
Uploading number of files in zip format and unzipping in S3 is not possible to reduce number of uploads.
Downloading multiple files from S3 is not possible in case if we want to shift to another provider.
Image names are case sensitive in S3. Which will not load images if image name does not match with request.
Among all these first one is very important thing since image resize is general requirement.
Which provider is best suitable to achieve my goal. Can I move to Google AppEngine only for the purpose of image hosting or is there any other vendor who can provide above services?
I've stumbled upon a nice company called Cloudinary that provides CDN image storage service - they also provide a variety of ways that allow on the fly image manipulation (Cropping will mainly concern you as you we're talking about different sized thumbnails).
I'm not sure how they compete with other companies like maxcdn in site speed enhancement but from what I can see - they have more options when it come to image manipulation.
S3 is really slow and also not distributed. Cloudfront in comparison is also one of the slowest and most expensive CDNs you can get. The only advantage is that if you're using other AWS already you'll get one bill.
I blogged about different CDNs and ran some tests:
http://till.klampaeckel.de/blog/archives/100-Shopping-for-a-CDN.html
As for the setup, I'd suggest something that uses origin-pull. So you host the images yourself and the CDN requests a copy of it the first time it's requested.
This would also mean you can use a script to "dynamically" generate the images because they'll be pulled only once or so. Just have to set appropriate cache headers. The images would then be cached until you purge the CDN's cache.
HTH
I've just come across CloudFlare - from what I understand from their site, you shouldn't need to make any changes to your website. Apparently all you need to do is change your DNS settings. Even provides a free option.
If you're using EC2, then S3 is your best option. The "best practice" is to simply pre-render the image in all sizes and upload with different names. I.e.:
/images/image_a123.large.jpg
/images/image_a123.med.jpg
/images/image_a123.thumb.jpg
This practice is in use by Digg, Twitter (once upon a time, maybe not with twimg...), and a host of other companies.
It may not be ideal, but it's the fastest and most simple way to do it. In terms of switching to another provider, you'll likely not do that because of the amount of work to transfer all of the files anyway. If you've got 1,000,000 images or 3,000,000 images, you've still got many megabytes of files.
Fortunately, S3 has an import/export service. You can send them an empty hard drive and they'll format it and download your data to it for a small fee.
In terms of your concern about case sensitivity, you won't find a provider that doesn't have case sensitivity. If your code is written properly, you'll normalize all names to uppercase or lowercase, or use some sort of base 64 ID system that takes care of case for you.
All in all, S3 is going to give you the best "bang for your buck", and it has CloudFront support if you want to speed it up. Not using S3 because of reasons 3 and 4 is nonsense, as they'll likely apply anywhere you go.
My website allows users to upload photographs which I store on Amazon's S3. I store the original upload as well as an optimized image and a thumbnail. I want to allow users to be able to export all of their original versions when their subscription expires. So I am thinking the following problems arise
Could be a large volume of data (possibly around 10GB)
How to manage the download process - eg make sure if it gets interrupted where to start from again, how to verify successful download of files
Should this be done with individual files or try and zip the files and download as one file or a series of smaller zipped files.
Are there any tools out there that I can use for this? I have seen Fzip which is an Actionscript library for handling zip files. I have an EC2 instance running that handles file uploads so could use this for downloads also - eg copy files to EC2 from S3, Zip them then download them to user via Flash downloader, use Fzip to uncompress the zip folder to user's hard drive.
Has anyone come across a similar service / solution?
all input appreciated
thanks
I have not dealt with this problem directly but my initial thoughts are:
Flash or possibly jQuery could be leveraged for a homegrown solution, having the client send back information on what it has received and storing that information in a database log. You might also consider using Bit Torrent as a mediator, your users could download a free torrent client and you could investigate a server-side torrent service (maybe RivetTracker or PHPBTTracker). I'm not sure how detailed these get, but at the very least, since you are assured you are dealing with a single user, if they become a seeder you can wipe the old file and begin on the next.
Break larger than 2GB files into 2GB chunks to accommodate users with FAT32 drives that can't handle > ~4GB files. Break down to 1GB if space on the server is limited, keeping a benchmark on what's been zipped from S3 via a database record
Fzip is cool but I think it's more for client side archiving. PHP has ZIP and RAR libraries (http://php.net/manual/en/book.zip.php) you can use to round up files server-side. I think any solution you find will require you to manage security on your own by keeping records in a database of who's got what and download keys. Not doing so may lead to people leeching your resources as a file delivery system.
Good luck!