I'm trying to convert a png image in PHP the following way:
exec($cmd, $output, $return_code);
Where $cmd contains the following line of code:
/usr/bin/convert 'images/original/Id1741.png' -thumbnail x200 -quality '90' './cache/a3b84c5931d9619d12a9e244a310cb17_h200.png'
Calling this code on the command line works perfectly fine, but executing it on the webserver gives me the following error message:
Tried to execute : convert 'images/original/Id1741.png' -thumbnail x200 -quality '90' './cache/a3b84c5931d9619d12a9e244a310cb17_h200.png', return code: 1, output: Array()
If I remove the thumbnail option the command executes just fine on the webserver, but oviously it does not resize anything. So it's not a problem with permissions or the setup I guess.
PHP Version is 5.2.17.
ImageMagick Version is: 6.6.0-4 2012-04-26
Anyone ever had a similar issue and can help me with this?
Ok, I finally got it fixed. After redirecting stderr to a file I found the following error:
libgomp: Thread creation failed: Resource temporarily unavailable
Seems that my hoster 1&1 recently upgraded the ImageMagick version which apparently uses more memory than the old one (at least that's what the hoster says).
They recommend limiting the number of Threads created by ImageMagick:
putenv('MAGICK_THREAD_LIMIT=1');
I put this code into my init-script and now it works just fine!
You are converting to PNG, but you are setting -quality 90 (seemingly just analogue to the JPEG quality setting).
However, for PNG output, the -quality setting is very unlike JPEG's quality setting (which simply is an integer from 0 to 100).
For PNG it is composed by two single digits:
The first digit (tens) is (largely) the zlib compression level, and it may go from 0 to 9.
(However the setting of 0 has a special meaning: when you use it you'll get Huffman compression, not zlib compression level 0. This is often better... Weird but true.)
The second digit is the PNG data encoding filter type (before it is compressed):
0 is none,
1 is "sub",
2 is "up",
3 is "average",
4 is "Paeth", and
5 is "adaptive".
In practical terms that means:
For illustrations with solid sequences of color a "none" filter (-quality 00) is typically the most appropriate.
For photos of natural landscapes an "adaptive" filtering (-quality 05) is generally the best.
Maybe you want to revisit your -quality 90 setting in the light of this info.
Maybe you were aware of it already. In this case: my apologies for 'preaching to the choir'. :-)
Related
We are using ImageMagick for resizing/thumbnailing JPGs to a specific size. The source file is loaded via HTTP. It's working as expected, but from time to time some images are partially broken.
We already tried different software like GraphicsMagick or VIPS, but the problem is still there. It also only seems to happen if there are parallel processes. So the whole script is locked via sempahores, but it also does not help
We found multiple similar problems, but all without any solution: https://legacy.imagemagick.org/discourse-server/viewtopic.php?t=22506
We also wonder, why it is the same behaviour in all these softwares. We also tried different PHP versions. It seems to happen more often on source images with a huge dimension/filesize.
Any idea what to do here?
Example 1 Example 2 Example 3
I would guess the source image has been truncated for some reason. Perhaps something timed out during the download?
libvips is normally permissive, meaning that it'll try to give you something, even if the input is damaged. You can make it strict with the fail flag (ie. fail on the first warning).
For example:
$ head -c 10000 shark.jpg > truncated.jpg
$ vipsthumbnail truncated.jpg
(vipsthumbnail:9391): VIPS-WARNING **: 11:24:50.439: read gave 2 warnings
(vipsthumbnail:9391): VIPS-WARNING **: 11:24:50.439: VipsJpeg: Premature end of JPEG file
$ echo $?
0
I made a truncated jpg file, then ran thumbnail. It gave a warning, but did not fail. If I run:
$ vipsthumbnail truncated.jpg[fail]
VipsJpeg: Premature end of input file
$ echo $?
1
Or in php:
$thumb = Vips\Image::thumbnail('truncated.jpg[fail]', 128);
Now there's no output, and there's an error code. I'm sure there's an imagemagick equivalent, though I don't know it.
There's a downside: thumbnailing will now fail if there's anything wrong with the image, and it might be something you don't care about, like invalid resolution.
After some additional investigation we discovered that indeed the sourceimage was already damaged. It was downloaded via a vpn connection which was not stable enough. Sometimes the download stopped, so the JPG was only half written.
When I try to upload videos captured from my iPhone in my app, the server performs a conversion from .mov to .mp4 so that it can be played in other platforms. However the problem is that when I shoot the video (in portrait orientation) and it is converted (using ffmpeg) and then played back from the server, it appears to be rotated. Any idea?
FFMPEG changed the default behavior to auto rotate video sources with rotation metadata in 2015. This was released as v2.7.
If your ffmpeg version is v2.7 or newer, but your rotation metadata isn't respected, the problem is likely that you are using custom rotation based on metadata. This will cause the same logic to be applied twice, changing or cancelling out the rotation.
In addition to removing your custom rotation (recommended), there's an option to turn off auto rotation with -noautorotate.
ffmpeg -noautorotate -i input.mp4...
This will also work in some older releases.
For sake of completeness, the reason this is happening is that iPhones only actually capture video in one fixed orientation. The measured orientation is then recorded in Apple-specific metadata.
The effect is that Quicktime Player reads the metadata and rotates the video to the correct orientation during playback, but other software (e.g., VLC) does not and shows it as oriented in the actual codec data.
This is why rotate=90 (or vflip, or transpose, or etc.) will work for some people, but not others. Depending on how the camera is held during recording, the rotation necessary could be 90, 180, or even 270 degrees. Without reading the metadata, you're just guessing at how much rotation is necessary and the change that fixes one video will fail for another.
What you can also do is remove the QuickTime specific metadata when rotate the .mov.
This will make sure that the video is rotated the same way in VLC and QuickTime
ffmpeg -i in.mov -vf "transpose=1" -metadata:s:v:0 rotate=0 out.mov
Here's the documentation on the -metadata option (from http://ffmpeg.org/ffmpeg.html):
-metadata[:metadata_specifier] key=value (output,per-metadata)
Set a metadata key/value pair.
An optional metadata_specifier may be given to set metadata on streams or chapters. See -map_metadata documentation for details.
This option overrides metadata set with -map_metadata. It is also possible to delete metadata by using an empty value.
For example, for setting the title in the output file:
ffmpeg -i in.avi -metadata title="my title" out.flv
To set the language of the first audio stream:
ffmpeg -i INPUT -metadata:s:a:1 language=eng OUTPUT
Depending on which version of ffmpeg you have and how it's compiled, one of the following should work...
ffmpeg -vf "transpose=1" -i input.mov output.mp4
...or...
ffmpeg -vfilters "rotate=90" -i input.mov output.mp4
Use the vflip filter
ffmpeg -i input.mov -vf "vflip" output.mp4
Rotate did not work for me and transpose=1 was rotating 90 degrees
So - I too ran into this issue, and here my $0.02 on it:
1.) some videos DO have Orientation/Rotation metadata, some don't:
MTS (sony AVHCD) or the AVIs I have - DO NOT have an orientation tag.
MOVs and MP4s (ipad/iphone or samsung galaxy note2) DO HAVE it.
you can check the setting via 'exiftool -Rotation file'.
My videos often have 90 or 180 as the rotation.
2.) ffmpeg - regardless of the man-page with the metadata-tag, just doesn't EVER seem to set it in the output file. - the rotation-tag is ALWAYS '0'.
it correctly reports it in the output - but it's never set right to be reported by exiftool. - But hey - at least it's there and always 0.
3.) rotation angles:
if you want rotate +/- 90: transpose=1 for clockwise 90, 2 ccw
now if you need 180 degree - just add this filter TWICE.
remember - it's a filter-chain you specify. :-) - see further down.
4.) rotate then scale:
this is tricky - because you quickly get into MP4 output format violations.
Let's say you have a 1920x1080 MOV.
rotate by 90 gives 1080x1920
then we rescale to -1:720 -> 1080*(720/1920) = 405 horiz
And 405 horizontal is NOT divisable by 2 - ERROR. fix this manually.
FIXING THIS automatically - requires a bit of shell-script work.
5.) scale then rotate:
you could do it this way - but then you end up with 720x1280. yuck.
But the filter-example here would be:
"-vf yadif=1,scale=-1:720,transpose=1"
It's just not what I want - but could work quite OK.
Putting it all together: - NOTE - 'intentionally WRONG Rotation-tag', just to demonstrate - it won't show up AT ALL in the output !
This will take the input - and rotate it by 180 degree, THEN RESCALE IT - resetting the rotation-tag. - typically iphone/ipad2 can create 180deg rotated material.
you just can leave '-metadata Rotation=x' out the line...
/usr/bin/ffmpeg -i input-movie.mov -timestamp 2012-06-23 08:58:10 -map_metadata 0:0 -metadata Rotation=270 -sws_flags lanczos -vcodec libx264 -x264opts me=umh -b 2600k -vf yadif=1,transpose=1,transpose=1,scale=1280:720 -f mp4 -y output-movie.MP4
I have multiple devices - like a settop box, ipad2, note2, and I convert ALL my input-material (regardless whether it's mp4,mov,MTS,AVI) to 720p mp4, and till now ALL the resulting videos play correct (orientation,sound) on every dev.
Hope it helps.
For including into web pages my portrait-format videos from iPhone, I just discovered the following recipe for getting .mp4 files in portrait display.
Step 1: In QuickTime Player, Export your file to 480p (I assume that 720p or 1080p would work as well). You get a .mov file again.
Step 2: Take the new file in QT Player, and export to “iPad, iPhone…”. You get a .m4v file.
Step 3: I’m using Miro Video Converter, but probably any readily-available converter at all will work, to get your .mp4 file.
Works like a (long-winded) charm.
I've filmed the video with Ipad3 and it was oriented upside down, which I suppose is the common situation of all Apple devices at some versions. Besides of it, the 3-minutes long MOV file (1920x1090) took about 500 Mb in size, which made it not available to share easily. I had to convert it to MP4, and analyzing all threads I've found on stackoverflow, here's the final code string for ffmpeg I've used (ffmpeg ver. 2.8.4):
ffmpeg -i IN.MOV -s 960x540 -metadata:s:v rotate="0" -acodec libmp3lame OUT.mp4
I suppose you may just leave '-metadata:s:v rotate="0"' if you don't need the resize and audio codec change. Note that if you resize the video, width and height should fully divide to 4.
Although the topic is old.
Hope this will help some one:
Get ffmpeg latest version : https://www.ffmpeg.org/download.html
The command that worked for me (to flip 180 degrees):
ffmpeg -noautorotate -i input.mp4 -filter:v "rotate=PI" output.mp4
When the degrees are determined by -filter:v "PI/180*degrees"
for example
-filter:v "45*PI/180" for 45 degrees
A nice explanation is here
https://superuser.com/questions/578321/how-to-rotate-a-video-180-with-ffmpeg
Or... to simply change the tag in an existing file:
Read the current rotation
exiftool -Rotation <file>
then, for example:
exiftool -Rotation=180 <file>
to set it to 180
I have an old code base that makes extensive use of GD (no, switching to imagemagick is not an option). It's been running for several years, through several versions. However, when I run it in my current development environment, I'm running into a mysterious gd-png error: cannot allocate image data errors when calling imagecreatefrompng(). The PNG file I'm using is the same as I've been using, so I know it works.
Current setup:
Ansible-provisioned vagrant box
Ubuntu 14.04
PHP 5.5.9
GD 2.1.1
libPNG 1.2.50
PHP's memory limit at the time the script is run is 650M, though it's ultimately the kernel itself that ends up killing the script, and changing PHP's memory limit doesn't seem to have an effect.
The image dimensions are 7200x6600 and are about 500KiB on disk. This hasn't been a problem in my other environments and is only newly-occurring in my development environment. Unfortunately, I don't have access to the other environments anymore to do a comparison, though the setup was similar in the last working one -- Ubuntu 14.04, PHP 5.5, sufficient memory allocations.
What could be happening in this setup that wasn't happening in my previous setups? How can I fix this?
I was browsing a bit through the PHP 5.5.9 source to try and find your specific error string "cannot allocate image data", but the closest I could find is "gd-png error: cannot allocate gdImage struct". The bundled version of GD for PHP 5.5.9 (all the way up to 7.0.0) is version 2.0.35, so it seems you're looking at a custom build(?). Bugging the PHP guys might not help here.
Looking at the GD source (2.1.2, can't seem to find 2.1.1), the only place this error occurs is:
https://github.com/libgd/libgd/blob/master/src/gd_png.c#L435
image_data = (png_bytep) gdMalloc (rowbytes * height);
if (!image_data) {
gd_error("gd-png error: cannot allocate image data\n");
png_destroy_read_struct (&png_ptr, &info_ptr, NULL);
if (im) {
gdImageDestroy(im);
}
if (palette_allocated) {
gdFree (palette);
}
return NULL;
}
Where gdMalloc is:
https://github.com/libgd/libgd/blob/GD-2.1/src/gdhelpers.c#L73
void *
gdMalloc (size_t size)
{
return malloc (size);
}
I'm afraid this is as far as my detective work goes. As far as I can tell the bundled 2.0.35 GD versions in PHP also just use malloc, so at first glance there's no real difference there. I also tried to find some equivalent piece of code in the bundled versions, but so far I haven't found it and it seems to be here:
https://github.com/php/php-src/blob/PHP-5.5.9/ext/gd/libgd/gd_png.c#L323
image_data = (png_bytep) safe_emalloc(rowbytes, height, 0);
I can't seem to find safe_emalloc, but it seems the old PHP bundled versions did use a different memory allocation here than the version your environment uses. Perhaps your best bet is to check with the GD devs?. To avoid a merry goose chase, I think trying another environment is your best bet -after confirming they are indeed using a non-strandard GD version.
After some more PHP source safari, it seems safe_emalloc is used throughout all extensions, so I guess (guess mind you) that this is the preferred way to allocate memory for PHP extensions (looks like it). If your environment is in fact using an unaltered GD 2.1.1, it's likely it is ignoring any PHP memory settings/limits. You might be able to find some other way of specifying the ('stand-alone') GD memory limit?
Edit: Looking at the official libgd faq, near the bottom it states that the PHP extension should indeed respect the memory limit and it specifies that an 8000x8000 pixel image would take about 256MB, so your 650MB limit should really be enough (unless you're working with multiple copies in the same run?) and the fact that it's not working corroborates that something fishy is going on with the memory allocation.
If I was you I would do the following:
Send messages to the php-devel list and see if it is a known issue. You'll also want to search their bug tracker. http://php.net/mailing-lists.php and https://bugs.php.net/
Use GD command line utilities to see if you can process the same image with the same version with cmd line stuff. This is trying to determine if it's a problem with GD and the image or if it is an issue with the PHP-gd lib or some combination.
1.a Create a PHP CLI program to resize the image and see if it works
Figure out exactly what happens when you call the php function in the underlying (probably c) code. This will involve downloading the source to the php-gd module and looking through it.
Code a minimal c application that does the same thing as the underlying PHP-gd library and see if you can reproduce the error
Something in there should tell you exactly what is going on, though it will take a bit of work. You should be able to get all the tools/sources for your environment. The beauty of open source, right?
The other option would be to try different versions of these applications in your environment and see if you find one that works. This is the "shotgun" approach and not the best, IMO.
From my experience:
Try a file with a different row size. I had issues with another image library that did not read images longer than 3000px/row. It was otherwise not restricted in absolute size.
You have quite an image there, if this is in RGBA in memory, you end up with 180M uncompressed image data, 140M on RGB, still 45M as 8Bit. You are sure, your process limits will not be exceeded when this is loaded?
You used the same image with the same libraries, it means libgd and libpng have no restrictions about the memory size (or at least not one for the size you're using), the problem may be php but you are sure the memory limit is 650M (if your php installation is using suhosin, you may want to check suhosin.memory_limit)
You tell the kernel is killing the script (but I can't find why you are telling so, is there a dmesg log about?) but the kernel kill a process only when is out of memory.
One problem could be memory fragmentation / contiguous alloc, nowadays it should be more a kernel space problem than user space, but the size your script is allocating is not an every day size.
Fragmentation could be a problem if your script is running on a server with a long uptime with a lot of process allocating and freeing memory, if is a virtual machine that you just start up and the script fail at the first launch, then the problem isn't fragmentation.
You can do a simple test, command and code below, it use the gcc compiler to create (in the directory where you launch the command) a a.out executable which allocate and set to 0 a memory chunk with size 191 x 1024 x 1024 (about ~190M)
echo '#include <stdlib.h>\n#include <string.h>\n#define SIZ 191*1024*1024\nint main() {void *p=malloc(SIZ);memset(p,0,SIZ);return (p==NULL?1:0);}' \
| gcc -O2 -xc -
run the executable with:
./a.out
If the problem is the system memory the kernel should kill the executable (I'm not sure 100%, the kernel have some policy about which process to kill when there's no memory, to be checked)
You should see the kill message though, with dmesg perhaps.
(I think this shouldn't happen, instead the malloc should just fail)
If the malloc succeed, the executable can allocate enough memory, it should return 0, if it fail should return 1
to verify the return code just use
./a.out
echo $?
(don't execute any other command in the middle)
If the malloc fail you probably just need to add physical or virtual memory to the system.
Keep in mind that ~190M is just the memory for one uncompressed image, depending on how php, libgd and libpng are working the memory for the whole process may be even twice (or more).
I did a simple test and profiled the memory usage, the peak with an image 7600x2200 (1.5M on disk) on my system is about ~342M, here's the test:
<?php
$im = imagecreatefrompng("input.png");
$string = "Sinker sucker socks pants";
$orange = imagecolorallocate($im, 220, 210, 60);
$px = (imagesx($im) - 7.5 * strlen($string)) / 2;
imagestring($im, 3, $px, 9, $string, $orange);
imagepng($im);
imagedestroy($im);
I tried xhprof at first but it returned values too low.
So I tried a simple script memusg
memusg /usr/bin/php -f test.php >output.png
(the test work btw)
I met this problem yesterday and fixed it today.
Yesterday's env:
php-7.0.12
libpng-1.6.26
libgd-2.1.1
This suite will crash when I resize a png image.
After memory check, I thought it might be a bug in the latest php or libpng.
So I changed the env to this:
php-5.6.27
libpng-1.2.56
libgd-2.1.1
I changed php and libpng to mature versions which are used longtime.
Recompile and re-install these - it works well for png and jpeg.
I am trying to complete a project that has to include some OCR. For the job I picked Tesseract OCR but the results are not optimal. I have tried to limit the character set to 1234567890- but the results are not good. Is there an optimal image size I can use or some way to train Tesseract to recognise this kind of string better?
The image is this:
And the result tesseract returns is 05175150152 which is not right, and it should be better since the image is not modified in any way. I use tesseract through PHP with exec with the following command:
"C:\Program Files\Tesseract-OCR\tesseract.exe" C:\wamp\www\a
dwords\phones\center_ctl09_ctl04.png sssd -l eng -psm 7 nobatch letters
Any ideas on what i am doing wrong?
The image resolution of 96 DPI is tough for any OCR engine. Try to rescale it to 300 DPI and you will have better results.
Additionally, JPEG is a lossy image format. Use a different one, like TIFF or PNG, if possible.
A while back I used a PNG optimisation service called (I think) "smush it". You fed it a weblink and it returned a zip of all the PNG images with their filesizes nicely, well, smushed...
I want to implement a similar optimisation feature as part of my website's image upload process; does anyone know of a pre-existing library (PHP or Python preferably) that I can tap into for this? A brief Google has pointed me towards several command line style tools, but I'd rather not go down that route if possible.
Execute with PHP this command line tools
pngcrush -rem gAMA -rem cHRM -rem iCCP -rem sRGB -brute -l 9 -max -reduce -m 0 -q IMAGE
optipng -o7 -q pngout.png
pngout pngout.png -q -y -k0 -s0
advpng -z -4 pngout.png > /dev/null
pngcrush
OptiPNG
pngout
advpng
As long as your PHP is compiled with GD2 support (quite common nowadays):
<?php
$image = imagecreatefromstring(file_get_contents('/path/to/image.original.png'));
imagepng($image, '/path/to/image.smushed.png', 9);
This will read in any image format GD2 understands (not just PNG) and output a PNG gzipped as the maximum compression level without sacrificing quality.
It might be of less use today than years ago though; most image editors already do this, since gzipping doesn't cost as much CPU-wise as it used to.
Have you heard of PNGCrush? You could check out the source, part of PNG and MNG Tools at SourceForge, and transcribe or wrap it in Python.
I would question the wisdom of throwing away other chunks (like gAMA and iCCP), but if that's what you want to do it's fairly easy to use PyPNG to remove chunks:
#!/usr/bin/env python
import png
import sys
input=sys.stdin
out=sys.stdout
def critical_chunks(chunks):
for type,data in chunks:
if type[0].isupper():
yield type,data
chunks = png.Reader(file=input).chunks()
png.write_chunks(out, critical_chunks(chunks))
the critical_chunks function is essentially filtering out all but the critical PNG chunks (the 4 letter type for a critical chunk starts with an uppercase letter).