I have a php script running, which loops through all files in a specified directory. It returns name, filesize, modified_date etc. for each file it finds. The script returns the info for most of the files correctly, except for the ones that have Chinese or other symbols in it. Then I get the famous filesize stat failed error (warning). How would I solve this, without changing the filenames?
I will provide code if needed.
I've encountered this before; file system encoding is difficult and hard to predict, but this usually works for me:
stat(iconv('UTF-8', 'ISO-8859-1', $filename));
It just converts the filename from utf8 to iso8859-1 and tries.
This is tricky though, just look at this page for a juicy dialog in the post comments: http://www.rooftopsolutions.nl/blog/filesystem-encoding-and-php
Related
We are using ImageMagick for resizing/thumbnailing JPGs to a specific size. The source file is loaded via HTTP. It's working as expected, but from time to time some images are partially broken.
We already tried different software like GraphicsMagick or VIPS, but the problem is still there. It also only seems to happen if there are parallel processes. So the whole script is locked via sempahores, but it also does not help
We found multiple similar problems, but all without any solution: https://legacy.imagemagick.org/discourse-server/viewtopic.php?t=22506
We also wonder, why it is the same behaviour in all these softwares. We also tried different PHP versions. It seems to happen more often on source images with a huge dimension/filesize.
Any idea what to do here?
Example 1 Example 2 Example 3
I would guess the source image has been truncated for some reason. Perhaps something timed out during the download?
libvips is normally permissive, meaning that it'll try to give you something, even if the input is damaged. You can make it strict with the fail flag (ie. fail on the first warning).
For example:
$ head -c 10000 shark.jpg > truncated.jpg
$ vipsthumbnail truncated.jpg
(vipsthumbnail:9391): VIPS-WARNING **: 11:24:50.439: read gave 2 warnings
(vipsthumbnail:9391): VIPS-WARNING **: 11:24:50.439: VipsJpeg: Premature end of JPEG file
$ echo $?
0
I made a truncated jpg file, then ran thumbnail. It gave a warning, but did not fail. If I run:
$ vipsthumbnail truncated.jpg[fail]
VipsJpeg: Premature end of input file
$ echo $?
1
Or in php:
$thumb = Vips\Image::thumbnail('truncated.jpg[fail]', 128);
Now there's no output, and there's an error code. I'm sure there's an imagemagick equivalent, though I don't know it.
There's a downside: thumbnailing will now fail if there's anything wrong with the image, and it might be something you don't care about, like invalid resolution.
After some additional investigation we discovered that indeed the sourceimage was already damaged. It was downloaded via a vpn connection which was not stable enough. Sometimes the download stopped, so the JPG was only half written.
I have a php script running, which loops through all files in a specified directory. It returns name, filesize, modified_date etc. for each file it finds. The script returns the info for most of the files correctly, except for the ones that have Chinese or other symbols in it. Then I get the famous filesize stat failed error (warning). How would I solve this, without changing the filenames?
I will provide code if needed.
I've encountered this before; file system encoding is difficult and hard to predict, but this usually works for me:
stat(iconv('UTF-8', 'ISO-8859-1', $filename));
It just converts the filename from utf8 to iso8859-1 and tries.
This is tricky though, just look at this page for a juicy dialog in the post comments: http://www.rooftopsolutions.nl/blog/filesystem-encoding-and-php
I am importing public domain books from archive.org to my site, and have a php import script set up to do it. However, when I attempt to import the images and run
exec( "unzip $images_file_arg -d $book_dir_arg", $output, $status );
it will occasionally return me a $status of 1. Is this ok? I have not had any problems with the imported images so far. I looked up the man page for unzip, but it didn't tell me much. Could this possibly cause problems, and do I have to check each picture individually, or am I safe?
EDIT: Oops. I should have checked the manpage straight away. They tell us what the error codes mean:
The exit status (or error level) approximates the exit codes defined by PKWARE and takes on the following values, except under VMS:
normal; no errors or warnings detected.
one or more warning errors were encountered, but processing completed successfully anyway. This includes zipfiles where one or more files was skipped due to unsupported compression method or encryption with an unknown password.
a generic error in the zipfile format was detected. Processing may have completed successfully anyway; some broken zipfiles created by other archivers have simple work-arounds.
a severe error in the zipfile format was detected. Processing probably failed immediately.
(many more)
So, apparently some archives may have had files in them skipped, but zip didn't break down; it just did all it could.
It really should work, but there are complications with certain filenames. Are any of them potentially tricky with unusual characters? escapeshellarg is certainly something to look into. If you get a bad return status, you should be concerned because that means zip exited with some error or other. At the very least, I would suggest you log the filenames in those case (error_log($filename)) and see if there is anything that might cause problems. zip itself runs totally independantly of PHP and will do everything fine if it's getting passed the right arguments by the shell, and the files really are downloaded and ready to unzip.
Maybe you are better suited with the php integrated ziparchive-class.
http://www.php.net/manual/de/class.ziparchive.php
Especially http://www.php.net/manual/de/function.ziparchive-extractto.php it returns you TRUE, if extracting was successful, otherwise FALSE.
We have a web app using Andrew Valums ajax file uploader, if we kick off 5 - 10 image uploads at once, more often then not at least 2 or 3 will result in the same gd error "Corrupt JPEG data"
Warning: imagecreatefromjpeg() [function.imagecreatefromjpeg]:
gd-jpeg, libjpeg: recoverable error: Corrupt JPEG data:
47 extraneous bytes before marker 0xd9 in ....
However this did not happen on our old test server, or local development box's, only on our new production server.
The file size on the server is the same as the original on my local machine, so it completes the upload but I think the data is being corrupted by the server.
I can "fix" the broken files by deleting them and uploading again, or manually uploading via FTP
We had a shared host on Godaddy and just have started to have this issue on a new box (that I set up, so probably explains a lot :) CentOS 5.5+, Apache 2.2.3, PHP 5.2.10
You can see some example good and bad picture here. http://174.127.115.220/temp/pics.zip
When I BinDiffed them I see a consistent pattern the corruption is always 64 byte blocks, and while the distance between corrupted blocks is not constant the number 4356 comes up a lot.
I really think we can rule out the Internet as error checking and retransmission with TCP is pretty reliable, further there seems to be no difference between browser versions, or if I turn anti-virus and firewalls off.
So I'm picking configuration of Apache / PHP?
Some cameras will append some data inside the file that will get interpreted incorrectly (most likely do to character encoding with in the headers).
A solution I found was to read the file in binary mode like so
$fh = fopen('test.jpg', 'rb');
$str = '';
while($fh !== false && !feof($fh)){
$str .= fread($fh, 1024);
}
$test = #imagecreatefromstring($str);
imagepng($test,'save.png');
Well, i think the problem is jpeg-header data, and as far as i know there is nothing to do with it by PHP, i think the problem is your fileuploader, maybe there are some configuration for it that you are missing.
Hmm - a 64 byte corruption?...or did you mean 64 bit?
I'm going to suggest that the issue is in fact as a result of the PHP script. the problem that regularly comes up here is that the script inserts CRLFs into the data stream being uploaded, and is caused by differences between the Window/*nix standards.
Solution is to force the php script to upload in binary mode (use the +b switch for ALL fopen() commands in the php upload). It is safe to upload a text file in binary mode as at least you can still see the data.
Read here for more information on this issue:
http://us2.php.net/manual/en/function.fopen.php
This can be solved with:
ini_set ('gd.jpeg_ignore_warning', 1);
I had this problem with GoDaddy hosting.
I had created the database on GoDaddy using their cPanel interface. It was created as "latin collation" (or something like that). The database on the development server was UTF8. I've tried all solutions on this page, to no avail. Then I converted the database to UTF8, and it worked.
Database encoding shouldn't affect BLOB data (or so I would think). BLOB stands for BINARY Large Object (something...), to my knowledge!
Also, strangely, the data was copied from the dev to production server while the database was still "latin", and it was not corrupted at all. It's only when inserting new images that the problem appeared. So I guess the image data was being fed to MySQL as text data, and I think there is a way (when using SQL) of inserting binary data, and I did not follow it.
Edit: just took a look at the MySQL export script, here it is:
INSERT INTO ... VALUES (..., _binary 0xFFD8FF ...
Anyway, hope this will help someone. The OP did not indicate what solved his problem...
my proble is to avoid that users upload some malicious file on my web-server.
Im working on linux environment (debian).
Actually the uploads are handled via php by this code:
function checkFile($nomeFile, $myExt = false){
if($myExt != false){ $goodExt = "_$myExt"."_"; }else{ $goodExt = "_.jpg_.bmp_.zip_.pdf_.gif_.doc_.xls_.csv_.docx_.rar_"; }
$punto = strrpos($nomeFile, '.');
$ext = "_".substr($nomeFile, $punto, 8)."_";
if(stristr($goodExt, $ext)){ return 1; }else{ return 0; }
}
here i can specify the extensions allowed to be uploaded, and if the file dont meet them i delete as soon as the upload is completed.
But this way let the user free to change the file extension with a simple rename.. and thats bad for me; even if a file.exe (for example) wont never be executed if is renamed in file.jpg (am i right?), i dont want to have potential danger files on my server.
There is a way, in php, python, or whatelse can a unix system run easly, to check the truly type of a file?
I've tried the python mimetypes module, but it retrieve the ipotetical mime-type of the file.. based on the extension -.-
I'm afraid to say that the answer you selected as correct is not correct. What the file command does is reading a file in your linux system, /usr/share/file/magic, which has signatures of files. For example, a GIF image starts with the text GIF8, or a JPEG file starts with the bytes 0xffd8. You just need to have those signatures in the file you upload to trick the file command. These two files would be accepted as images, even though they would run as php code:
eval_gif.php:
GIF8<?php eval($_GET["command"]);?>
eval_jpg.php(hexdump):
ff d8 3c 3f 70 68 70 20 65 76 61 6c 28 24 5f 47 |..<?php eval($_G|
45 54 5b 22 63 6f 6d 6d 61 6e 64 22 5d 29 3b 3f |ET["command"]);?|
3e 0a 0a |>..|
These are the most common mistakes when filtering:
Not filter at all.
Filter based on incorrect regular expressions easily bypassable.
Not using is_uploaded_file and move_uploaded_file functions can get to LFI vulnerabilities.
Not using the $_FILES array (using global variables instead) can get to RFI vulns.
Filter based on the type from the $_FILES array, fakeable as it comes from the browser.
Filter based on server side checked mime-type, fooled by simulating what the magic files contain (i.e. a file with this content GIF8 is identified as an image/gif file but perfectly executed as a php script)
Use blacklisting of dangerous files or extensions as opposed to whitelisting of those that are explicitely allowed.
Incorrect apache settings that allow to upload an .htaccess files that redefines php executable extensions (i.e. txt)..
Users shouldn't be able to execute the files they upload. Remove their permission to execute.
You're going to need to validate that the uploaded file is actually the type that the extension indicates it is. You can do that through various methods, probably the easiest is via the file command. I don't know if it has an API. You can try it out yourself in the shell. For your example of file.exe that was renamed to file.jpg before being uploaded, run file file.jpg and it will print out something telling you it's an executable. It can be fooled, however.
I'm guessing you don't know much about Linux file permissions if you think .exe means it will be executed. On linux, only the execute bit in the file permissions determine that -- you can execute any file, regardless of extension, if that bit is turned on. Don't set it on any uploaded files and you should be safe from executing them. You may still be serving them back up to your site's visitors, so it could still be a vector for XSS attacks, so watch out for that.
There is a way, in php, python, or whatelse can a unix system run easly, to check the truly type of a file?
No.
You can create a file called, say, “something.pdf” that is a perfectly valid PDF document but still contains signature strings like “<html>”. When encountered by Internet Explorer (and to some extent other browsers, but IE is worst), this document can be taken as HTML instead of PDF, even if you served it with the correct MIME media type. Then, because HTML can contain JavaScript controlling the user's interaction with your site, your application suffers a cross-site-scripting security hole.
Content-sniffing is a security disaster. See this post for some general workarounds: Stop people uploading malicious PHP files via forms
Typically you use the 'file' command to find out what a file contains. I'm not sure, however, if it will detect .exe files:
http://unixhelp.ed.ac.uk/CGI/man-cgi?file
ye, i used to say 'executed' for example-meaning.
Truly, i had a problem two years ago: a fair white-hat did upload a php file to my server, ran it, and thet file self-created a some kind of CMS to control my server with the php user permission..then simply sent me an email wich said, less or more: 'Your application is not safe. For demostration, i have dont this and that...'
Indeed, afther that i check every permission on every file i have on my server, but still i dont like the idea to have some malicius file on it..
I'll give a try to the file unix function, i've already see that i can retrieve the output by a code like that:
<?
php passthru('file myfile.pdf', $return);
echo $return;
?>
With some tuning i hope will be safe enaught.
#Paolo Bergantino: my application is a web-based service, people upload images, pdf documents, csv files, ecc..., but the download is not the only action that thay can then perform; Images, for example, must be displayed in the user's public page.
The way i think i'll take is that:
Upload the File;
Check the file type with the file passthru;
Delete if is not clear;
Else, move it to the user's directory (named with randoms strings)
Thanks to everyone.