I am using Image Magick in my PHP program on linux to process some images. In one case we are taking layers from a PSD and displaying certain layers and colorizing them. In others, we are overlaying 3 or so PNG images and then returning a final image. We are using PHP but we use the system command to run the commands like at the command prompt and then serve up the image to the browser. It seems very slow. Caching is not an option because of the myriad of possible combinations. The images are not that large and it takes sometimes 5 seconds per image to process and it seems that the server only does them one at a time (I am assuming that the browser is probably asking for more than one image concurrently.). We have put the site on a very beefy server. Is there somewhere that I can allocate more memory or processing power to these Image Magick commands like "convert"?
Related
I'm processing 5MB - 6MB images and reducing them to under 600KB file size using PHP Imagick. Usually on the order of 3000 to 5000 at a time. However, the process is taking 8-12 hours to complete. I tried two different ways of handling this: 1) Retrieving the images remotely using Guzzle Pool and storing them locally, then running the conversion process, and 2) Retrieving the images remotely and storing in an ImageMagick object, processing them, then saving locally. Either method seems to take a huge amount of time to complete. The process of resizing the images and saving them below is the same between the two methods, except for reading the image from file if I already have it saved locally.
$imagick = new Imagick();
$imagick->readImageBlob($source);
$imagick->setImageFormat('jpg');
$imagick->setOption('jpeg:extent', '600kb');
$imagick->stripImage();
$imagick->writeImage($destination);
Wondering if there is something else I can do to speed things up.
I have several suggestions, and you can probably combine them together too if you wish. It would have helped if you had specified your OS and also the image types in your question...
First suggestion - process in parallel
Do them in parallel, with GNU Parallel. So instead of the following sequential code:
for f in *.png; do
./Process "$f"
done
you would do:
parallel ./Process {} ::: *.png
Second suggestion - use "shrink on load"
If you are reducing the pixel dimensions of your images, i.e. from 5,000x3,000 pixels to 800x600 pixels for example, and they are JPEGs, you can use "shrink on load"
magick -define jpeg:size=1600x1200 BigBoy.jpg -define jpeg:extent=600kb BiteSize.jpg
Consider vips
Consider using vips which I suspect will be much faster and lighter on resources
vipsthumbnail BigBoy.jpg -s 800x600 -o BiteSize.jpg
Add --vips-leak to the command above to see how much memory it used.
I'm using a Symfony2 bundle from GregWar to resize up to 12 images at a time from a user upload. I'm using this https://github.com/Gregwar/ImageBundle
I'm resizing them to four different sizes, as these sizes are needed for mobile, desktop, thumbnail, etc. It takes time to do each one of course, but with 12 x 4 resizes it can take a while. Over 30 seconds easily. Which is behind the default PHP timeout and isn't really acceptable for an end user wait time.
I want the resizing to be done at upload so the new sizes are available immediately to the user. Rather than later as a batch process.
This seems like it may be a common problem. So what can I do to improve my situation?
Should I use a different library?
Or reduce my image sizes from four, down to perhaps two to improve processing speed but sacrifice user experience?
Is this normal? Could it be a hardware issue? On my local machine it's even slower.
PHP memory is set to 256MB. I use a ServerGrove VPS, with PHP5.3.
I have moved away from trying to solve this on my server and instead now use a dedicated EC2 instance with Amazon to resize images.
Similar to this approach http://sumitbirla.com/2011/11/how-to-build-a-scalable-caching-resizing-image-server/
If there is GD extension installed on your server, you can use imagecopyresized function.
Today a customer's server went down after a user uploaded a 16MB JPG which then got processed by an automated, cron-controlled script which has been running flawlessly for months, successfully processing at least a thousand images.
After some investigation i figured that indeed that certain image is causing ImageMagick trouble. I downloaded the image to my local system (OSX Mavericks, ImageMagick 6.8.6-6 Q16) and ran the exact same convert command on it:
convert test.jpg icc:profile.icc
convert test.jpg -crop 3567x5340+125+0 +repage -resample 200x200 -resize 3937x5906 -compress lzw -profile ./profile.icc out.tif
Immediately everything stalls, the system gets slow and after a few seconds i get this message:
And in the terminal it says:
convert: unable to extend cache `test.jpg': No space left on device # error/cache.c/OpenPixelCache/3661.
My drive has a clear 73 GB of free space, although i am certain that swap space is not exactly referring to that. Then again i have never before encountered such an issue.
So much for local testing.
On the server (CentOs 6.5 Final, ImageMagick 6.8.6 Q16) the main symptoms are a high CPU load causes by the convert process, causing some kind of stapling of further convert processes which may get called through users navigating the website in yet uncached areas, triggering convert executions for the purpose thumbnail generation. But these never exceed even a percent of CPU usage. The convert process that should work through the image in question however, that one i have seen to go up to almost a 100%.
Sideinfos
I use a normal call to exec() to run convert. Forking as a separate process doesn't seem like much of an option (not even sure if PCNTL is available), nor would it help much, other than that the convert process would then just continue running in circles until infinity==null (so to speak).
PHP's memory limit is set to 128 M on the server (on my test machine i called the process manually from the terminal).
PHP's max_execution_time is set to 300, for we need a certain amount of buffer to assure proper processing of a whole stack of images. Note that this system has worked stable and properly managed to process entire packs of images, not seldom exceeding a count of 25 pieces a time (all of which were larger than the image in question).
Note
Even though i would like to, due to copyright issues i cannot provide the original image. But perhaps someone has ran into the same or a similar issue and can therefore give valuable input that may server in solving the problem or at least finding out what exactly causes it.
I'w writing a PHP script that uses Montage, which is an extension of ImageMagick that creates tiled arrays of images. The Montage syntax is straightforward:
montage image1.png image2.png image3.png -tile x1 -geometry 50x50 out.png
However, I'm generating these images on the fly and I would hate to have to write them to disk just to run this command. Is there some way I can pass there resources in without writing them to file?
(This isn't really a Montage question, but rather a general question that could apply to many different situations.)
Any help would be great!
As I commented, you could find some tmpfs file system (e.g. with df | grep tmpfs) such as /run and put your files inside. They will then live within the virtual memory (and won't usually need any disk I/O). Of course the space they take are consuming virtual memory resources (so are limited).
BTW, on many Linux systems, writing small files don't use a lot of disk I/O because they sit in the file system cache.
Of course, any tmpfs file system loses all its content at every reboot, so you don't want to keep important unrecoverable data inside.
ImageMagick has APIs for both C and PHP. PHP IMagick appears to have montage hooks.
http://us3.php.net/manual/en/imagick.montageimage.php
General info on PHP IMagick usage:
http://us3.php.net/manual/en/imagick.examples-1.php
I have an image GD script which is currently using around 9MB of memory.
I get a lot of traffic and hence sometimes it using up hell lot of RAM on my server.
Is there any way to reduce memory usage for image gd?
Or at-least make script process faster so that it de-allocates the memory which it is using, faster.
I have tried changing image quality, it had no effect.
I also tried changing image pixel size, it reduced the memory usage, but not much.
Thanks.
It's impossible to tell without seeing the code, but unless it contains any major mistakes, then the answer is probably no.
What you might be able to do is use the external imagemagick binary instead - it runs outside PHP's script memory limit - but that is an entirely different technology, and would require you to rewrite your code.
I assume you are already caching GD's results so not every request causes it to run?
Try avoiding using image GD on the fly if you are concerned about memory limits.
It's hard to solve the problem without seeing code, but i can make a suggestion.
Have a different process handle the images, for example, if you want to resize images, don't resize them everytime the user access a page, instead run a cron or a scheduler with window to resize all the images that needs to be resized periodically and save them. so there will be less overhead.
If you provide more code you will get better help