I have a basic php script to dynamically resize images on my (virtual shared) server using GD and serve them up as required.
Unfortunately, following the purchase of a better camera with more megapixels, the script falls over due to lack of memory (my hosting allocates ~50m and an uncompressed 20Mpix image is over that limit).
Note that uncompressed means any image file that is loaded with one of:
imageCreateFromGif($filepath);
imageCreateFromJpeg($filepath);
imageCreateFromPng($filepath);
imageCreateFromBmp($filepath);
As these convert the file into the full-resolution raw image data in full 8-bit-per-pixel plus alpha in memory for further processing.
I can detect the image is oversize using getimagesize() and gracefully refuse to load it, but it would be nice to be able to handle these larger images without busting the RAM limit.
Is there anything I can do to solve this?
The server is a cheap shared commercial hosting package so I can't do anything about the restrictions on RAM or change the PHP runtime config or things like that.
Related
Imagine a zip bomb, but in PNG flavour.
When doing any kind of image manipulation in gd it's almost inevitable that at some point the image is held entirely in memory in decompressed form via imagecreatefromjpeg and friends even for simple operations like resizing.
getimagesize only extracts info from the metadata which can be unreliable.
What tools do we have to protect ourselves from such kinds of abuse, for example in the case of scaling down an image to create a thumbnail?
I thought about:
Making the server only accept uncompressed BMP or TIFF images and setting mere upload size limitations, with a client-side javascript that would convert jpg and png files to bmp before sending. Very bad for performance in general.
Actively reducing the available memory for the script in charge of doing the manipulation so that it would fail with an out-of-memory error if the image turns out to be too large (like 16 MB max memory). Bad on a whole new level of bad.
Do imagescale and similar functions need the original image to exist in memory decompressed?
Any other tips?
I've already checked post_max_size upload_max_filesize on my php.ini file, the thing is that my hosting provider doesn't allow more than 25M for each upload.
I need to let the users to upload files 50M - 100M. But before changing my provider I thought maybe there's some workaround to split large files into separate packages, upload them and then join them back together.
Any ideas will be appreciated :)
From Splitting a file before upload?
You can try [Plupload][1]. It can be configured to check whatever
runtime is available on users side, be it - Flash, Silverlight, HTML5,
Gears, etc, and use whichever satisfies required features first. Among
other things it supports image resizing (on users side, preserving
EXIF data(!)), stream and multipart upload, and chunking. Files can
be chunked on users side, and sent to a server-side handler
chunk-by-chunk (requires some additional care on server), so that big
files can be uploaded to a server having max filesize limit set to a
value much lower then their size, for example. And more.
Some runtimes support https I believe, some need testing. Anyway,
developers on there are quiet responsive these days. So you might at
least try ;)
[1]: http://www.plupload.com/index.php
I am debating on storing images in the Mongo GridFS or on an cloud file system. I am leaning towards the cloud because of a few reasons.The language being used is PHP on a Nginx server.
Storing images in the GridFS increases the size of the database. Therefore more of the database has to be in memory and I will spend more time/money managing the servers when it comes to things like sharding.
Retrieving the image from GridFS takes longer than cloud because I have to
a) Query the image using the id
b) read the image into memory
c) use a php header to display the image
The cloud would be better because its a url of the image directly to the cloud.
Does those reasons sound valid or should I be going in a different direction with my thinking?
Its not entirely true.
Storing images in the GridFS increases the size of the database.
Therefore more of the database has to be in memory and I will spend
more time/money managing the servers when it comes to things like
sharding.
Mongodb gridfs splits the huge files into the chunks and only they will be loaded and served (by each chunk) when it is requested . Yes definitely it ll take more memory than the file system. These are all the trade offs when using the in memory data stores.
Retrieving the image from GridFS takes longer than cloud because I
have to a) Query the image using the id b) read the image into memory
c) use a php header to display the image
As i stated in my previous point, it will be getting loaded into the memory at the first time. so you wont be having much of a performance problem, infact it ll be a gain since it served from RAM instead disk. But if you are not satisfied still, i would recommend to cache the images in nginx. so it wont come to mongo after the first one.
I'm enhancing a PHP image hosting script, but I'm facing a problem: the client has requested a function to limit the amount of used bandwidth, for each image, to 100MB / day
This includes thumbnails
Up to now, thumbnails have been served directly without using PHP (with nginx)
I've tried using a php script instead of a direct .jpg, with a VERY heavily cached code (using APC's cache to store the amount of bandwidth used, and serving the image with the X-Sendfile header), but the server just chokes due to the very,very high amount of connections
Is there a more efficient way to measure bandwidth used by thumbnails without choking the CPU and RAM?
How is this problem usually tackled?
Thanks
Parse the server access logs. I believe in nginx they are at /var/log/nginx/access.log . You'll likely need to change permissions to allow access, or you could use cron to regularly copy that file into an accessible folder if you aren't comfortable doing that.
I guess that if you cached results, and only checked once an hour or something then you could do this using grep -c.
I'd be interested to know, if you pick a file you know has been accessed, how long does it take to run:
grep -c filename.jpg /var/log/nginx.access.log
You could then simply multiply that count by the filesize to get the bandwidth for that image.
I would like to create an upload script that doesn't fall under the php upload limit.
There might be an occasion where I need to upload a 2GB, or larger file and I don't want to have to change the whole server execution to above 32MB.
Is there a way to write direct to disk from php?
What method might you propose someone would use to accomplish this? I have read around stack overflow but haven't quite found what I am looking to do.
The simple answer is you can't due to the way that apache handles post data.
If you're adamant to have larger file uploads and still use php for the backend you could write a simple file upload receiver using the php sockets api and run it as a standalone service. Some good details to be found at http://devzone.zend.com/article/1086#Heading8
Though this is an old post, you find it easily via google when looking for a solution to handle big file uploads with PHP.
I'm still not sure if file uploads that increase the memory limit are possible but I think there is a good chance that they are. While looking for a solution to this problem, I found contradicting sources. The PHP manual states
post_max_size: Sets max size of post data allowed.
This setting also affects file upload.
To upload large files, this value must
be larger than upload_max_filesize. If
memory limit is enabled by your
configure script, memory_limit also
affects file uploading. Generally
speaking, memory_limit should be
larger than post_max_size. (http://php.net/manual/en/ini.core.php)
...which implies that your memory limit should be larger than the file you want to upload. However, another user (ragtime at alice-dsl dot com) at php.net states:
I don't believe the myth that 'memory_size' should be the size of
the uploaded file. The files are
definitely not kept in memory...
instead uploaded chunks of 1MB each
are stored under /var/tmp and later on
rebuild under /tmp before moving to
the web/user space.
I'm running a linux-box with only 64MB
RAM, setting the memory_limit to 16MB
and uploading files of sizes about
100MB is no problem at all! (http://php.net/manual/en/features.file-upload.php)
He reports some other related problems with the garbage collector but also states how they can be solved. If that is true, the uploaded file size may well increase the memory limit. (Note, however, that another thing might be to process the uploaded file - then you might have to load it into memory)
I'm writing this before I tried handling large file uploads with PHP myself since I'm evaluating using php or python for this task.
You can do some interesting things based around PHP's sockets. Have you considered writing an applet in Java to upload the file to a listening PHP daemon? This probably won't work on most professional hosting providers, but if you're running your own server, you could make it work. Consider the following sequence:
Applet starts up, sends a request to PHP to open a listening socket
(You'll probably have to write a basic web browser in Java to make this work)
Java Applet reads the file from the file system and uploads it to PHP through the socket that was created in step 1.
Not the cleanest way to do it, but if you disable the PHP script timeout in your php.ini file, then you could make something work.
It isn't possible to upload a file larger than PHP allowed limits with PHP, it's that simple.
Possible workarounds include using a client-side technology - like Java, not sure if Flash and Javascript can do this - to "split" the original file in smaller chunks.