I have an issue with GD not creating a new JPG file, it just fails. No error messages and no indication as to what is happening. This is not new code, it has been in and working for the past six years, but all of a sudden with larger images it has started failing.
As a background this is running on an old server (to be switched off and moved to a new site on PHP8 in a couple of months time) that has PHP5.3.3 with GD version 2.0.34.
The code is creating thumbnails from the high-res image (around 24-30MB) and outputting a series of thumbnails from 150px wide to 1024px wide. It fails on all. I have increased the PHP memory limit on the page to 512MB, and set the GD.JPEG_ignore_warning flag for corrupt JPGs.
But every time with these files, this line:
$src_img = #imagecreatefromjpeg($file_path);
just returns FALSE. But never falls over with an error. The file is definitely there (run the same code with a file of the same name that is <20MB and it works fine) and there is plenty of disc space/memory available for a 60MB file to be processed in memory, so I dont see that that is the issue.
A search of Google, StackOverflow and several other sites has not produced any similar issues.
Can anyone offer any thoughts as to what the issue/solution is? I have been looking at this for two days now, and we need to resolve it - simply using smaller JPG files isn't an option for this.
Thanks to #Lessmore answer above, GD was reporting an invalid JPG file, so a further search revealed this answer on StackOverflow, which solved the problem by reading the JPG from a string, rather than file:
Reading an invalid JPG with GD
Thanks all - as ever!
I am trying to compress an uploaded image using imagefromjpeg but I get this error:
Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 24000 bytes)
The image is only 13215317 bytes big - why do I keep getting this error? I can not ramp up the memory for the server myself - so is there a way to compress without loading the entire image at once?
$image = imagecreatefromjpeg('../../uploads/DSC_0230.jpg');
imagejpeg($image, '../../uploads/DSC_0230.jpg.new', 0.8);
imagedestroy($image);
Loading an image into PHP this way will first uncompress the JPEG into raw pixel data in memory, and then generate the new version. It will therefore take memory equivalent to two raw bitmaps at the image's resolution, plus the other overhead of PHP and your script.
Since all you're doing is recompressing the image with different options, you will probably have better performance with a tool built for just that job, such as jpegtran or jpegstrip, or a general image manipulation tool like ImageMagick.
You could then call those from your PHP script using shell_exec - being very careful that you have validated the filename so that someone can't use it to run arbitrary commands.
Issue:
Your hosting provider offers you service with only a 64Mb memory limit.
Your image upload has a larger than 64Mb memory usage as uncompressed raw pixel data.
Solutions:
1) Increase your memory limit in PHP.ini file, typically with :
memory_limit = 128M
2) Increase your memory limit on your page only, editing the php.ini only for that page execution :
ini_set('memory_limit','128M');
3) Limit the size of the original file upload.
there are a few ways to do this, so please read the PHP manual as well as reaseach some useful posts found via Google Searching.
4) You can also try and resize the image before uploading.
From your statement that you cant edit the PHP.ini with ini_set then it looks like you should use option 4 and 3.
Also, your current code is incorrect.
imagejpeg($image, '../../uploads/DSC_0230.jpg.new', 0.8);
The compression value should be an integer between 0 and 100. to correctly set the saved JPEG to a proper compression level:
imagejpeg($image, '../../uploads/DSC_0230.jpg.new', 80);
This will save the image to a value of 80 compression(the value it looks like you're attempting with 0.8).
I currently use Imagick library on PHP and use the resizing functionality of Image Magick. I've just learned about decompression bombs and how ImageMagick is vulnerable to it.
I have checked how we can ping the image and verify the dimensions of the image without actually loading it into memory/disk. It's also safer to limit the memory and disk limits of ImageMagick so it wouldn't just write a huge file on disk.
I've read and I can do that with setResourceLimit().
http://php.net/manual/en/imagick.setresourcelimit.php
IMagick::setResourceLimit(IMagick::RESOURCETYPE_MEMORY , 100);
IMagick::setResourceLimit(IMagick::RESOURCETYPE_DISK , 100);
$thumb = new Imagick('image.png');
$thumb->resizeImage(320,240,Imagick::FILTER_LANCZOS,1);
However, what happens is that after setting the limit of disk and memory, if an image does hit this limit, all I get is a segmentation fault error, no exceptions are thrown. This makes it impossible for me to handle it properly.
Update:
Here are the package versions I'm using:
dpkg -l | grep magick
ii imagemagick-common 8:6.6.9.7-5ubuntu3.3 image manipulation programs -- infrastructure
ii libmagickcore4 8:6.6.9.7-5ubuntu3.3 low-level image manipulation library
ii libmagickwand4 8:6.6.9.7-5ubuntu3.3 image manipulation library
ii php5-imagick 3.1.0~rc1-1 ImageMagick module for php5
Setting the 'Resource Area' limit only sets the size at which images are not held in memory, and instead are paged to disk. If you want to use that setting to actually restrict the maximum size image that can be openend, you also need to set the 'Resource Disk' limit.
The code below correctly gives a memory allocation error for the image bombs taken from here.
try {
Imagick::setResourceLimit(Imagick::RESOURCETYPE_AREA, 2000 * 2000);
Imagick::setResourceLimit(Imagick::RESOURCETYPE_DISK, 2000 * 2000);
$imagick = new Imagick("./picture-100M-6000x6000.png");
$imagick->modulateImage(100, 50, 120);
$imagick->writeImage("./output.png");
echo "Complete";
}
catch(\Exception $e) {
echo "Exception: ".$e->getMessage()."\n";
}
Output is:
Exception: Memory allocation failed `./picture-100M-6000x6000.png' # error/png.c/MagickPNGErrorHandler/1630
If you want to set the width and height resource, and have a version of ImageMagick >= 6.9.0-1 you should be able to using the values directly of WidthResource = 9, HeightResource = 10
//Set max image width of 2000
Imagick::setResourceLimit(9, 2000);
//Set max image height of 1000
Imagick::setResourceLimit(10, 1000);
These don't have to be set programmatically, you can set them through the policy.xml file installed with ImageMagick. ImageMagick reads that file and uses those settings if none are set in a program - which may be a more convenient way of setting them, as you can change them per machine.
This makes it impossible for me to handle it properly.
It makes it impossible for you to handle it in the same process. You can handle it just fine by running the image processing in a background task.
Personally I think anyway that uses Imagick in a server that is accessed directly by web-browsers is nuts. It is far safer to run it in as a background task (managed by something like http://supervisord.org/) and communicating with that background task via a queue of jobs that need to be processed.
Not only does that solve the 'bad images can bring down my website' problem, it also makes it far easier to monitor resource usage, or shift the image processing to a machine with a faster CPU than a web-front end server needs.
Source - I'm the maintainer of the Imagick extension and I recently added this to the Imagick readme:
Security
The PHP extension Imagick works by calling the ImageMagick library.
Although the ImageMagick developers take good care in avoiding bugs it
is inevitable that some bugs will be present in the code. ImageMagick
also uses a lot of third party libraries to open, read and manipulate
files. The writers of these libraries also take care when writing
their code. However everyone makes mistakes and there will inevitably
be some bugs present.
Because ImageMagick is used to process images it is feasibly possible
for hackers to create images that contain invalid data to attempt to
exploit these bugs. Because of this we recommend the following:
1) Do not run Imagick in a server that is directly accessible from
outside your network. It is better to either use it as a background
task using something like SupervisorD or to run it in a separate
server that is not directly access on the internet.
Doing this will make it difficult for hackers to exploit a bug, even
if one should exist in the libraries that ImageMagick is using.
2) Run it as a very low privileged process. As much as possible the
files and system resources accessible to the PHP script that Imagick
is being called from should be locked down.
3) Check the result of the image processing is a valid image file
before displaying it to the user. In the extremely unlikely event that
a hacker is able to pipe arbitrary files to the output of Imagick,
checking that it is an image file, and not the source code of your
application that is being sent, is a sensible precaution.
Starting with ImageMagick-6.9.0-1, "width" and "height"
resource limits were added. From the commandline, use "-limit width 32000", etc. ImageMagick's PNG decoder will bail out without decompressing the image if the width or height exceeds the specified limit.
The PNG decoder will not attempt to decompress images whose width or height exceeds the limits.
The "area" resource is available in earlier versions of ImageMagick (and Imagick); however the PNG decoder does not reject images based on the "area" limit (see Danack's comment).
In versions of ImageMagick earlier than 6.9.0, the width and height limits come from libpng, and depend upon the libpng version. Current libpng versions (1.0.16 and later, 1.2.6 and later, 1.5.22 and later, and 1.6.17 and later) impose 1,000,000-column and width limits. In versions 1.2.0 through 1.2.5, 1.5.0 through 1.5.23, and 1.6.0 through 1.6.16, the limits were 2.7 billion rows and columns by default.
Look for RESOURCETYPE_AREA in Imagick (I don't see _WIDTH or _HEIGHT in the manual that you referenced, so either Imagick or its manual needs to be updated). So try
IMagick::setResourceLimit(IMagick::RESOURCETYPE_AREA , 100M);
to set a 100MegaPixel limit. Hopefully, some future version of Imagick will support RESOURCETYPE_WIDTH and RESOURCETYPE_HEIGHT, to provide a better solution to the decompression-bomb vulnerability. See Danack's answer about setting these with the current version of IMagick.
all I get is a segmentation fault error, no exceptions are thrown
I'm guessing your segment fault is from resources being set too low for ImageMagick (and related delegates) to operate. The value of the resource are in bytes, not megabytes.
Imagick does throw an exception if a resource is reached. Usually something like...
"cache resource exhausted"
Decompression bombs, or Zip-Bombs, are extremely difficult to identify. What your doing by ping-ing the image, and setting resource limits is the correct course of action. I would roughly outline a solution as...
// Define limits in application settings, or bootstrap (not dynamically!)
define('MY_MAGICK_MEMORY_LIMIT', 5e+8);
// Repeat for AREA, DISK, & etc.
// In application
$image = new Imagick(); // allocate IM structrues
// Set limits on instance
$image->setResourceLimit(Imagick::RESOURCETYPE_MEMORY, MY_MEMORY_LIMIT);
// Repeat for RESOURCETYPE_AREA, RESOURCETYPE_DISK, & etc.
$filename = 'input.png';
if($image->ping($filename)) {
// Validate that this image is what your expecting
// ...
try {
$image->read($filename); // <-- Bomb will explode here
$image->resizeImage(320,240,Imagick::FILTER_LANCZOS,1);
} catch( ImageickException $err ) {
// Handle error
}
}
unset($image)
If you don't trust decompression, you can leverage Imagick::getImageCompression during ping validation to inspect which compression an image requires. The compression type would be a integer that would map to the following enum...
typedef enum
{
UndefinedCompression,
B44ACompression,
B44Compression,
BZipCompression,
DXT1Compression,
DXT3Compression,
DXT5Compression,
FaxCompression,
Group4Compression,
JBIG1Compression,
JBIG2Compression,
JPEG2000Compression,
JPEGCompression,
LosslessJPEGCompression,
LZMACompression,
LZWCompression,
NoCompression,
PizCompression,
Pxr24Compression,
RLECompression,
ZipCompression,
ZipSCompression
} CompressionType;
MagickStudio (written in PERL) offers a good starting point for default resource limits, and how they are checked against uploaded images (search for Ping.)
I'm trying to create a thumbnail / cutout of a larger image and it works for a large percentage of the time but every now and again I get the following:
Warning: imagecreatefromjpeg(): gd-jpeg, libjpeg: recoverable error: Corrupt
JPEG data: 626 extraneous bytes before marker 0xd9 in code.php on line 5
This is line 5 of "code.php":
$srcImg = imagecreatefromjpeg('5f48ecb107a1e297d23392f703992d60.jpg');
The image displays fine in windows but gd just fails to create the resource so I end up with a blank image (where the cutout section was supposed to go).
For frame of reference this is with regard to car titles and the system has 2784 that worked and only 36 that didn't so not a big deal but it has my curiosity piqued.
It sounds much like this problem reported on another web site, and the PHP bug is ticket #29878 (unavailable when I checked).
Although you should also verify that the image file is a valid, non-corrupt, JPEG image file as well. ImageMagick's identify program can identify if the file is corrupt. One potential problem is a JPEG file that uses CYMK rather than RGB colormap. ImageMagick can also likely let you resave the image to a valid JPEG file if corrupted.
In cases such as this error:
Fatal error: Out of memory (allocated
31981568) (tried to allocate 3264
bytes)
Can I use the GD lib to reduce its file size first before getting to this stage?
In short: no.
GD uses memory to reduce the size of an image, if the image is too big the memory limit is exceeded and an error is given.
You can calculate how big an image can be so you can stay under a certain memory limit here: http://www.dotsamazing.com/en/labs/phpmemorylimit
An option, although an unpopular one with shared hosts, is increasing the memory limit, you can do this with ini_set() or in an .htaccess file. Please check if your host allows this. If you have your own host, configure Apache accordingly.
An (also mentioned) option is using Imagemagick, a program that runs on the server that you can call to do the resizing for you. The memory limit for this program can be different than the one for PHP, but there probably will be a limit as well. Contact your host for more info.
You can instead set a higher memory limit.
ini_set('memory_limit', $file_size * 2);
Because if you want to reduce the size using GD, you still need to allocate memory for that file before you can reduce the size.
Also remember to set a file size limit to your image uploads.
You can use the filesize() function to check the file size before reading/opening the file.
Are you reading a file from disk with a PHP script?
This is obviously continuing from your previous post. If I remember rightly it's an uploaded image you're working with? If so what is the size of the image? If it's really large you should consider limiting the size of image uploads.
That would happen when u want the memory to do more than it can .. you might want to look into imagemagick , so instead of resizing via PHP just send request to imagemagick to do the resizing.
Or the easier way would be to increase the memory limit per script of php scripts via ini.