This is my IM command:
/usr/bin/convert
'src.tif'
-limit memory 0
-limit map 0
-limit file 0
-alpha transparent
-clip
-alpha opaque
-resize 800x600
'end.png'
2>&1
So this will remove the white background of my TIFF by clipping the path that is given in the file. It will be resized and saved as transparent PNG.
I got no errors from IM running this.
But if I run this command with PHP to execute it on about 13000 files - I sometimes get these errors:
sh: line 1: 25065 Killed /usr/bin/convert \
'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' \
-limit memory 0 -limit map 0 -limit file 0 -alpha transparent \
-clip -alpha opaque -resize 800x600 \
'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
sh: line 1: 25702 Killed /usr/bin/convert \
'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' \
-limit memory 0 -limit map 0 -limit file 0 -alpha transparent \
-clip -alpha opaque -resize 800x600 \
'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1
But the bigger problem is: Some of the pictures are broken. Below is a "bad" image on the left, a "good" image on the right (ondrag/on a dark background you see the problem better):
On running the command manually the result was ok. Only on running this PHP loop script will provide broken results. ( PHP loop script )
I run the script this way: php55 run.php. A simple loop with find as shell script provides same results.
So I searched, asked in the IM discourse server and run this procedure on 2 machines with different distribution (Debian Wheezy, Ubuntu Server 14.04)
Note/EDIT 1: Running the command in the terminal with the same file provides a perfect result.
EDIT 2: Added example TIFF file here
I'm not sure if this is an answer. For now it is pure speculation. So here goes...
By setting the limits to a 0 value, you are basically telling ImageMagick: "Your resources are not limited at all. You do not need to care for any limits."
What if didn't set any limit? Remove all -limit ... 0 parts from your command. In this case ImageMagick would use its built-in defaults, or the otherise defined settings (which may be contained in the policy.xml file of your IM installation, or through various environment variables). You can query the current limits of your system with the following command:
identify -list resource
On my system, I get these values:
File Area Memory Map Disk Thread Throttle Time
---------------------------------------------------------------------------
192 4.295GB 2GiB 4GiB unlimited 1 0 unlimited
What if you did set these limits to a reasonable value, that matches your system's really available resources? Assuming you have: 8 GByte of RAM, 50 GByte of free disk space and plenty of free inodes on your disk volume. Then try to set it like this: -limit disk 10GB -limit memory 3GB -limit map 6GB.
ImageMagick resource management
For all its processing and intermediate steps, ImageMagick needs access to an intermediate pixel cache memory/storage, before it can deliver the final result.
This need for pixel cache storage can be satisfied by different resources:
heap memory,
anonymous memory map,
disk-based memory map,
direct disk.
ImageMagick makes use of all these resources progressively:
Once heap memory is exhausted, it stores pixels in an anonymous map.
Once the anonymous memory map is exhausted, it creates the pixel cache on disk and attempts to memory-map it.
Once memory-map memory is exhausted, it simply uses standard disk I/O.
Disk storage is cheap but very slow too: it is in the order of 3 magnitudes (a thousand times) slower than memory. Some speed improvements (up to 5 times) can be obtained by using memory mapping to the disk-based cache.
ImageMagick is aware of various ways to control the amount of these resources:
Built-in default values. These limits are: 768 files, 3GB of image area, 1.5GiB memory, 3GiB memory map, and 18.45EB of disk space.
policy.xml config file. Please look up what's in your own policy.xml file. Use convert -list policy to find the location of this file first. Then use cat /some/path/policy.xml to see its contents. (The file uses an XML syntax. Don't forget: anything enclosed in <!-- and --> is a comment!) It also contains comments explaining various details. The policy.xml can define much more things than just the available limit resources. Settings in policy.xml take precedence over the built-in default values if they are defined there.
Environment variables. Here is a list of environment variables which can limit IM resources: MAGICK_AREA_LIMIT (image area limits), MAGICK_DISK_LIMIT (disk space limit), MAGICK_FILE_LIMIT (maximum no. of open files limit), MAGICK_MEMORY_LIMIT (heap memory limit), MAGICK_MAP_LIMIT (memory map limit), MAGICK_THREAD_LIMIT (maximum no. of threads limit) and MAGICK_TIME_LIMIT (maximum elapsed time in seconds). These environment variables, if set, take precedence over the policy.xml config file.
-limit <name> <value> settings on command line. The following <names> are recognized:
width (maximum width of an image). When limit is exceeded, exception is thrown and processing stops.
height (maximum height of an image). When limit is exceeded, exception is thrown and processing stops.
area (maximum number of bytes for any single image to reside in pixel cache memory). When limit is exceeded, automagical caching to disk (possibly memory-mapped) sets in.
memory (maximum memory allocated for the pixel cache from anonymous mapped memory or heap).
map (maximum amount for memory map allocated for pixel cache).
disk (maximum amount of disk space permitted for use by pixel cache). When limit is exceeded, pixel cache is not created and a fatal exception is thrown.
files (maximum number of open pixel cache files). When limit is exceeded, all subsequent pixels cached to disk are closed and reopened on demand.
thread (maximum number of threads which can in parallel).
time (maximum time in seconds a process is permitted to execute). When this limit is exeeded, an exception is thrown and processing stops.
The -limit setting on a command line takes precendence and overrides all other settings.
Related
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4912 bytes) in /var/www/development/example/system/libraries/Image.php on line 130.
The JPEG image in question does not have a particularly large file size (741 KB). We've used this same code to rebuild larger images. However, the image does have unusually large dimensions (4912px x 3264px). Would this have an effect?
What determines memory usage when PHP is rebuilding an image? Is it just the file size? The dimensions? The colour density? The file type?
The line on which it broke was
$f1 = 'imagecreatefrom' . $tag;
$src = $f1($file);
I think that's enough context. It didn't get as far as trying to rebuild the image. Loading it into memory was enough to break it.
As riky said, set the memory limit higher if you can. Also realize that the dimensions are more important than the file size (as the file size is for a compressed image). When you open an image in GD, every pixel gets 3-4 bytes allocated to it, RGB and possibly A. Thus, your 4912px x 3264px image needs to use 48,098,304 to 64,131,072 bytes of memory, plus there is overhead and any other memory your script is using.
Increase your memory buffer size
php_value memory_limit 64M in your .htacess
or ini_set('memory_limit','64M'); in your php file
It depends your implimentation. last time when I was working on csv file with more then 500000 records, I got the same message. Later I introduce classes and try to close the open objects. it reduces it memeory consumption. if you are opening an image and editing it. it means it is loading in a memory. in that case size really matter. if you are operating multiple images. I will record to one per image and then close that image. In my experience when I was working on pdf artwork files to check the crop marks. I was having the same error.
//you can set the memory limits values
// in htaccess
php_value memory_limit 64M
//or in you using following in php
ini_set('memory_limit', '128M');
//or update it in your php.ini file
but if you optimize your code. and use object oriented aproach then you memory consumption will be very less. because in that every object has its own scope and out of that scope it is destroyed.
The size of the used memory depends on the dimension and the color bit depth.
I also ran in to that problem a few years ago, while building a portfolio-website for photographers. The only way to properly solve this is to switch your image library from GD to imagick. Imagick consumes far less memory, and is not tied to the PHP memory limit.
I have to say that the images the photographers uploaded were up to 30MP. And setting the memory limit to over 1024MB makes no sense in my eyes.
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4912 bytes) in /var/www/development/example/system/libraries/Image.php on line 130.
The JPEG image in question does not have a particularly large file size (741 KB). We've used this same code to rebuild larger images. However, the image does have unusually large dimensions (4912px x 3264px). Would this have an effect?
What determines memory usage when PHP is rebuilding an image? Is it just the file size? The dimensions? The colour density? The file type?
The line on which it broke was
$f1 = 'imagecreatefrom' . $tag;
$src = $f1($file);
I think that's enough context. It didn't get as far as trying to rebuild the image. Loading it into memory was enough to break it.
As riky said, set the memory limit higher if you can. Also realize that the dimensions are more important than the file size (as the file size is for a compressed image). When you open an image in GD, every pixel gets 3-4 bytes allocated to it, RGB and possibly A. Thus, your 4912px x 3264px image needs to use 48,098,304 to 64,131,072 bytes of memory, plus there is overhead and any other memory your script is using.
Increase your memory buffer size
php_value memory_limit 64M in your .htacess
or ini_set('memory_limit','64M'); in your php file
It depends your implimentation. last time when I was working on csv file with more then 500000 records, I got the same message. Later I introduce classes and try to close the open objects. it reduces it memeory consumption. if you are opening an image and editing it. it means it is loading in a memory. in that case size really matter. if you are operating multiple images. I will record to one per image and then close that image. In my experience when I was working on pdf artwork files to check the crop marks. I was having the same error.
//you can set the memory limits values
// in htaccess
php_value memory_limit 64M
//or in you using following in php
ini_set('memory_limit', '128M');
//or update it in your php.ini file
but if you optimize your code. and use object oriented aproach then you memory consumption will be very less. because in that every object has its own scope and out of that scope it is destroyed.
The size of the used memory depends on the dimension and the color bit depth.
I also ran in to that problem a few years ago, while building a portfolio-website for photographers. The only way to properly solve this is to switch your image library from GD to imagick. Imagick consumes far less memory, and is not tied to the PHP memory limit.
I have to say that the images the photographers uploaded were up to 30MP. And setting the memory limit to over 1024MB makes no sense in my eyes.
I'm using a php script to create image thumbnails and this error is thrown while creating some thumbs:
Fatal error: Allowed memory size of 31457280 bytes exhausted (tried to
allocate 227 bytes)
this is what top shows:
top - 07:43:49 up 44 days, 22:21, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 171 total, 1 running, 170 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.7%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6097648k total, 3459060k used, 2638588k free, 566924k buffers
Swap: 4194296k total, 0k used, 4194296k free, 1991920k cached
I haven't looked at optimizing phpthumb code. But is there any other way to free the already used memory? May be a cron job can be used to free this memory on regular intervals?
Your image is probably larger than ~10-15MB. PHP has a limit on the amount of memory it can take up per script (memory_limit in php.ini)
What happens is that you load an image in memory (And then resize it, creating a second image)...
Change the memory limit if you're allowed, or don't load such large image ...
AFAIK there is no stream image reader ...
If you can't change the memory limit, a workaround might be calling the commandline ImageMagick or GraphicsMagick tools if they're installed ...
This is a typical php.ini problem, if you are running this script on a VPS or a dedicated server, edit the php.ini file and set memory_limit to 99(or more)MB, also look out for max_run_time as that can stop a script after x number of seconds.
Don't forget to reboot Apache after you have done the changes,
If you are running this on a shared server, you might have some problems trying to solve this as you can't edit the settings file, you can try to set the settings in the actual script, however this usually doesn't .
Hello All i have following ini variable set in phpini file.
max_execution_time 50000
memory_limit 40M
post_max_size 8M
When i try to make thumbnail of image its gives me following error :
Fatal error: Out of memory (allocated 30670848) (tried to allocate 14976 bytes)
image size is 700 kb.
can any one help me ?
Your problem is not that single call to imagecreatefromjpeg() but memory that you have allocated earlier. After all, the allocation fails with a memory request for only ~14kB.
Maybe you created images before in the same script without releasing their memory with imagedestroy() or you have another memory problem. In the latter case you could use a debugging tool (e.g. webgrind) to find the memory hog.
What is the resolution of your 700kb image? It doesn't really matter how big the original .JPG is. If it's a very "simple" image, it could literally be 10,000 x 10,000 pixels. When it's loaded/uncompressed by PHP, those pixels will require 10k x 10k x 3 = 286 megabytes (10x squared times 3 for each red/green/blue component value).
And of course, you say you've set memory limit to 40M, but you've run out of memory at just 29.25 meg. Could be that there's an override somewhere in the setup, perhaps in a site-specific web server .conf file, or a .htaccess, which is setting a lower limit than 40M.
I need to run a PHP CLI script and give it a LOT of memory (it's an automatic documentation generator that needs to cover a large code base). I have a powerful machine and have allocated 5GB to php, both in php.ini and in an inline ini_set('memory_limit','5120M') declaration in the script file.
If I add these lines to top of the script:
phpinfo();
exit();
... it claims that it has a memory limit of 5120M, as I specified.
But the script still errors out, saying
Fatal error: Allowed memory size of 1073741824 bytes exhausted
... which is 1GB, not 5GB as I specified.
Is there any other place that the script might be looking to determine its memory limit? This is running in Fedora Linux.
(Yes, the ultimate solution may be to rewrite the script to be more efficient, but I didn't write it in the first place, so before I resort to that, I want to just throw resources at it and see if that works.)
The heap limit property is a size_t, which is 32 bits on a 32-bit machine. If this is in bytes, this would limit the memory limit to 4 GB. You may try running it on a 64 bit machine, with 64-bit PHP.
Edit: confirmed, heap->limit is a size_t (unsigned int) and is in bytes. A memory_limit of -1 sets the heap->limit to 4GB, and does not disable it as the documentation implies. Setting it to 5GB makes it wrap around to 1GB.