I've noticed that loading an image into imagick ($im = new Imagick($sFilename);) in php is taking 0.6 seconds for an 8MB image. This seems a bit slow to me, so I tried a test and read the file in using file_get_contents instead. About 0.005 seconds. Better. A bit too good tbh, I guess there's some caching going on there?
But I can load the same file a dozen times into imagick and it's always ~0.6 seconds.
Can I tell file_get_contents to bypass the system cache somehow, to give me a better idea of the raw speed with which an 8MB file can be retrieved from my hard drives?
Is there anything that can be done to speed up imagick? Or is 0.6 seconds for this operation completely normal?
The server has two 7200rpm HP sata drives in RAID 1.
Thanks.
Is there anything that can be done to speed up imagick?
Buy a faster CPU
Or is 0.6 seconds for this operation completely normal?
Yes.
This seems a bit slow to me
but it seems a long time for that.
I guess there's some caching going on there?
You're just guessing that something should be faster.....and you'r comparing it to a completely different operation. file_get_contents just reads the bytes in the file off the disk. Creating an image from a JPG means the computer has to read the bytes off the disk, and then decode them from the compressed data to be the actual image data.
If you want to see how much work has to be done during the compression, you can easily see this by writing the image out in an uncompressed format e.g.
$imagick = new Imagick("./testImage.jpg");
$imagick->setImageFormat('BMP');
$imagick->writeImage("./output.bmp");
And yes, this is longer than is reasonable for a HTTP request to take processing. Which is just another reason for why not running Imagick in a webserver is a good idea, but to instead run it as a background task.
Related
I have a PHP file that is 90 KB. It pretty much does all the work for the back end on my site and it gets loaded on every page. However, if I split up the different functions in this file and separated them across multiple files, I could reduce the total size of what is loaded on average on each different page to 45 KB.
Is loading a 90 KB PHP file with "everything on it" for each page going to slow the performance of my site down? Does it make more sense to split up the 90 KB file into smaller files and only load what is necessary for each page? Or is 90 KB small enough that it shouldn't matter.
Turn on Opcache and PHP will store the pre-processed source in memory and then it doesn't matter.
When a file is accessed your OS will usually cache that file in RAM until the memory needs to be reclaimed, and then it doesn't matter.
What is in which file is far better classified as an organization problem for a project and its maintainers, and this incredibly marginal performance consideration [say it with me] doesn't matter.
Thank you Sammitch from the comments
Today a customer's server went down after a user uploaded a 16MB JPG which then got processed by an automated, cron-controlled script which has been running flawlessly for months, successfully processing at least a thousand images.
After some investigation i figured that indeed that certain image is causing ImageMagick trouble. I downloaded the image to my local system (OSX Mavericks, ImageMagick 6.8.6-6 Q16) and ran the exact same convert command on it:
convert test.jpg icc:profile.icc
convert test.jpg -crop 3567x5340+125+0 +repage -resample 200x200 -resize 3937x5906 -compress lzw -profile ./profile.icc out.tif
Immediately everything stalls, the system gets slow and after a few seconds i get this message:
And in the terminal it says:
convert: unable to extend cache `test.jpg': No space left on device # error/cache.c/OpenPixelCache/3661.
My drive has a clear 73 GB of free space, although i am certain that swap space is not exactly referring to that. Then again i have never before encountered such an issue.
So much for local testing.
On the server (CentOs 6.5 Final, ImageMagick 6.8.6 Q16) the main symptoms are a high CPU load causes by the convert process, causing some kind of stapling of further convert processes which may get called through users navigating the website in yet uncached areas, triggering convert executions for the purpose thumbnail generation. But these never exceed even a percent of CPU usage. The convert process that should work through the image in question however, that one i have seen to go up to almost a 100%.
Sideinfos
I use a normal call to exec() to run convert. Forking as a separate process doesn't seem like much of an option (not even sure if PCNTL is available), nor would it help much, other than that the convert process would then just continue running in circles until infinity==null (so to speak).
PHP's memory limit is set to 128 M on the server (on my test machine i called the process manually from the terminal).
PHP's max_execution_time is set to 300, for we need a certain amount of buffer to assure proper processing of a whole stack of images. Note that this system has worked stable and properly managed to process entire packs of images, not seldom exceeding a count of 25 pieces a time (all of which were larger than the image in question).
Note
Even though i would like to, due to copyright issues i cannot provide the original image. But perhaps someone has ran into the same or a similar issue and can therefore give valuable input that may server in solving the problem or at least finding out what exactly causes it.
I am in process of rewriting some scripts to parse machine generated logs from perl to php
The files range from 20mb~400mb
I am running into this problem to decide if I should use file() or fopen()+fgets() combo to go through the file for some faster performance.
Here is the basic run through,
I check for file size before opening it, and if file is larger than 100mb(pretty rare case, but it does happen from time to time) I will go the fopen+fgets route since I only bumped the memory limit for the script to 384mb, any file larger than 100mb will have chance causing fatal error. Otherwise, I use file().
I am only going through the file once from beginning to the end in both method, line by line.
Here is the question, is it worth it to keep the file() part of the code to deal with the small files? I don't know how exactly file() (i use the SKIP_EMPTY_LINE option as well) works in php, does it map the file into the memory directly or does it shove line by line into the memory while going through it? I ran some benchmark on it, performance is pretty close, average difference is about 0.1s on 40mb file, and file() has advantage over fopen+fgets about 80% of the time(out of 200 test on the same fileset).
Dropping the file part could save me some memory from the system for sure, and considering I have 3 instance of the same script running at the same time, it could save me 1G worth of memory on a 12G system that's also hosting the database and other crap. But I don't want to let the performance of the script down also, since there is like 10k of these logs coming in per day, 0.1s difference actually adds up.
Any suggestion would help and TIA!
I would suggest sticking with one mechanism, like foreach(new \SplFileObject('file.log') as $line). Split your input files and process them in parallel, 2-3x per CPU core. Bonus: lower priority than database on same system. In PHP, this would mean spawning off N copies of the script at once, where each copy has its own file list or directory. Since you're talking about a rewrite and IO performance is an issue, consider other platforms with more capabilities here, eg Java 7 NIO, nodejs asynchronous IO, C# TPL.
I have an image GD script which is currently using around 9MB of memory.
I get a lot of traffic and hence sometimes it using up hell lot of RAM on my server.
Is there any way to reduce memory usage for image gd?
Or at-least make script process faster so that it de-allocates the memory which it is using, faster.
I have tried changing image quality, it had no effect.
I also tried changing image pixel size, it reduced the memory usage, but not much.
Thanks.
It's impossible to tell without seeing the code, but unless it contains any major mistakes, then the answer is probably no.
What you might be able to do is use the external imagemagick binary instead - it runs outside PHP's script memory limit - but that is an entirely different technology, and would require you to rewrite your code.
I assume you are already caching GD's results so not every request causes it to run?
Try avoiding using image GD on the fly if you are concerned about memory limits.
It's hard to solve the problem without seeing code, but i can make a suggestion.
Have a different process handle the images, for example, if you want to resize images, don't resize them everytime the user access a page, instead run a cron or a scheduler with window to resize all the images that needs to be resized periodically and save them. so there will be less overhead.
If you provide more code you will get better help
I am using both GD and Curl(one or other) for downloading remote jpeg images to my server. I just want a quick suggestion on which method to use? Which method takes less system resources?
To get load time try to create complex image with both of them (GD and CURL), set microtime at the start of each operation and calculate time-difference at the end. Just do a test at your own.
About memory usage (system resources), I'm not so sure, but I think there's no such big difference.
For me personally, GD and Curl are equal.