I am creating a web page on my CentOS server, where I want to traverse all my photos and videos and then show them on my page. However then it seems that when the files have our Danish national characters included, like æøåÆØÅ, then my exec command cannot access the file - and I need the exec command as I need mediainfo to show-and-tell the format and other details of the file (video, audio or image).
Let's assume I have this data and array that I am traversing (3 files):
$folder = "!My Folder"; // parent folder has a special char
Array
(
[0] => Array
(
[filename] => !file #2.jpg
[descr] => File with special chars, but no national chars
)
[1] => Array
(
[filename] => file with Danish æøå.jpg
[descr] => File with Danish chars, but no special chars
)
[2] => Array
(
[filename] => file with no special.jpg
[descr] => File with nothing special
)
)
I am then reading the mediainfo info from PHP like this:
$param = escapeshellarg("$folder/$filename"); // escape file argument
exec("mediainfo $param", $outputArray); // store line-by-line output in an array
This works fine for file [0] and [2] (I get a populated array), but [1] just returns an empty array from the output:
Array
(
[0] =>
)
As a note then I am able to use mediainfo directly on the server and doing this will work fine and return detailed data:
[usr#srv !My Folder]# mediainfo file\ with\ Danish\ æøå.jpg
So it seems to be the exec that has some problems with this?
I am using PHP 8.1 and I have no problem accessing or storing files on my server, via Samba, with these Danish characters.
An alternative solution would be to rename the files with these characters, but ideally I hope to avoid doing that as it is kind of "destructive" (messing with the original files).
Does anyone have a good idea how to access those files via exec?
### UPDATE 1 - BUT STILL NO SOLUTION ###
Just to make it crystal clear and to "prove" this is something related to exec then I refer to the answer from #CBroe below and adding one additional line to set the locale character encoding in PHP, setlocale + outputting the exec command:
setlocale(LC_CTYPE, "en_US.UTF-8");
$param = escapeshellarg("$folder/$filename");
exec("mediainfo $param", $outputArray);
echo "mediainfo $param";
The echo will output this (and an empty array):
mediainfo '/server/original/!My Folder/file with Danish æøå.jpg'
But if I run this exact same command directly on my server, mediainfo '/server/original/!My Folder/file with Danish æøå.jpg' then it will show me the media info for the file:
I also believe that this is some character encoding issue, but yet I do not know how to solve it ;-)
### UPDATE 2 - BUT STILL NO SOLUTION ###
As suggested below, then I also did try using putenv('LANG=en_US.UTF-8'), but for me that didn't help. I also tried using shell_exec() instead of exec() - same result and it did not help.
putenv("LANG=en_US.UTF-8");
$folder = "!My Folder"; // parent folder has a special char
$filename = "file with Danish æøå.jpg";
$param = escapeshellarg("$folder/$filename");
$output = shell_exec("mediainfo $param");
$outputArray = explode("\n",$output);
print_r($outputArray);
This will result in an array with two empty values:
Array
(
[0] =>
[1] =>
)
Somewhat late, but i stumbled across this today (pretty much the same, filename from glob() function and working further). This is not a problem of exec() but of the environment that mediainfo uses.
If you use UTF-8 filenames, it is essential to set the LANG environment correctly.
putenv('LANG=en_US.UTF-8');
worked for me.
Testing echo escapeshellarg("file with Danish æøå.jpg"); on https://3v4l.org/FEWfI only gives me 'file with Danish .jpg' as result.
Checking the user comments for the function, there is https://www.php.net/manual/en/function.escapeshellarg.php#99213:
When escapeshellarg() was stripping my non-ASCII characters from a UTF-8 string, adding the following fixed the problem:
<?php
setlocale(LC_CTYPE, "en_US.UTF-8");
?>
That indeed appears to fix the problem, https://3v4l.org/1DfpF - result now is 'file with Danish æøå.jpg'
Related
I am using an associative array which I initialized like this:
$img_captions = array();
Then, later in the code I am filling it in a while loop with keys and values coming in from a .txt file (every line in that .txt file contains a pair - a string - separated by '|') looking like this:
f1.jpg|This is a caption for this specific file
f2.jpg|Yea, also this one
f3.jpg|And this too for sure
...
I am filling the associative array with those data like this:
if (file_exists($currentdir ."/captions.txt"))
{
$file_handle = fopen($currentdir ."/captions.txt", "rb");
while (!feof($file_handle) )
{
$line_of_text = fgets($file_handle);
$parts = explode('/n', $line_of_text);
foreach($parts as $img_capts)
{
list($img_filename, $img_caption) = explode('|', $img_capts);
$img_captions[$img_filename] = $img_caption;
}
}
fclose($file_handle);
}
When I test that associative array if it actually contains keys and values like:
print_r(array_keys($img_captions));
print_r(array_values($img_captions));
...I see it contains them as expected, BUT when I try to actually use them with direct calling like, let's say for instance:
echo $img_captions['f1.jpg'];
I get PHP error saying:
Notice: Undefined index: f1.jpg in...
I am clueless what is going on here - can anyone tell, please?
BTW I am using USBWebserver with PHP 5.3.
UPDATE 1: so by better exploring the output of the 'print_r(array_keys($img_captions));' inside Chrome (F12 key) I noticed something strange - THE FIRST LINE OF '[0] => f1.jpg' LOOKS VISUALLY VERY WEIRD tho it looks normal when displayed as print_r() output on the site, I noticed it actually in fact is coded like this in webpage source (F12):
Array
(
[0] => f1.jpg
[1] => f2.jpg
[2] => f3.jpg
[3] => f4.jpg
[4] => f5.jpg
[5] => f6.jpg
[6] => f7.jpg
[7] => f8.jpg
[8] => f9.jpg
[9] => f10.jpg
)
So when I tested anything else than the 1. line it works OK. I tryed to delete completely the file and re-write it once again but still the same occurs...
DISCLAIMER Guys, just to clarify things more properly: THIS IS NOT MY ORIGINAL CODE (that is 'done completely by me'), it is
actually a MiniGal Nano PHP photogalery I had just make to suit my
needs but those specific parts we are talking about are FROM THE
ORIGINAL AUTHOR
I will recommend you to use file() along wth trim().
Your code becomes short, readable and easy to understand.
$parts= file('your text file url', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$img_captions = [];
foreach($parts as $img_capts){
list($img_filename, $img_caption) = explode('|', $img_capts);
$img_captions[trim(preg_replace("/&#?[a-z0-9]+;/i","",$img_filename))] = trim(preg_replace("/&#?[a-z0-9]+;/i","",$img_caption));
}
print_r($img_captions);
So after a while I realize there is something wrong with my .txt file itself as it:-
ALWAYS PUT SOME STRANGE SIGNS IN FRONT OF THE 1st LINE WHATEVER I DO IT WAS ALWAYS THERE EVEN WITH THE NEW FILE CREATED FROM SCRATCH (although those are UNVISIBLE unless seen as a source code on a webpage!!!)
So I decided to test it in another format, this time .log file and all of a sudden everything works just fine.
I do not know if it is just my local problem of some sort (most probably is) or something else I am not aware of.
But my solution to this was changing the file type holding the string pairs (.txt => .log) which solved this 'problem' for me.
Some other possible solution to this as #AbraCadaver said:
(Those strange signs: [0] => f1.jpg) That's the HTML entity for a BYTE ORDER MARK or BOM, save your file
with no BOM in whatever editor you're using.
I have the following code:
$l1 = file($file1['tmp_name']);// get file 1 contents
$l2 = file($file2['tmp_name']);// get file 2 contents
$l3 = array_diff($l1, $l2);// create diff array
Here are the files:
File 1:
6974527983
6974527984
6974527985
File 2:
6974527983
$l3 should be:
6974527984
6974527985
But, instead it is just spitting out the values from File 1:
6974527983
6974527984
6974527985
Am I setting this up right?
UPdate -
Using print_r(), I have verified that the files being loaded are being properly parsed into arrays:
File 1 -
Array ( [0] => 6974527983 [1] => 6974527984 [2] => 6974527985 ) 1
File 2 -
Array ( [0] => 6974527983 ) 1
So I don't believe there are any issues with the newlines in the text files.
If each number is on a new line, you could try splitting each file by line breaks and comparing the arrays that way.
$l1 = explode("\n", file($file1['tmp_name']));
$l2 = explode("\n", file($file2['tmp_name']));
$l3 = array_diff($l1, $l2);
Using the following example you can see that array_diff() works as expected:
$a = array(
6974527983,
6974527984,
6974527985
);
$b = array(
6974527983
);
var_dump(array_diff($a, $b));
Output:
array(2) {
[1] =>
int(6974527984)
[2] =>
int(6974527985)
}
This shows that file($file2['tmp_name']) is the problem in your case. Try:
var_dump(file($file2['tmp_name']));
to check the file's contents.
Okay, I will post an answer as I think this will solve your issue.
Without knowing more about the structure of your files, we can only assume that there is a possible issue with line endings. There are three possible line endings:
Unix: \n
Windows: \r\n
Classic mac: \r
I see two possible scenarios here:
The line endings in each file are different to each other
The line endings in both files are \r (classic mac)
As Mark Baker pointed out, you should use the FILE_IGNORE_NEW_LINES flag as the second argument for each of your file() calls. This, as far as I can make out from quickly experimenting here, should resolve the issue if one file had Unix and the other had Windows line endings.
However, it does not seem to deal well in cases where at least one file has '\r' line endings. In this case, there's an ini setting that might help:
ini_set('auto_detect_line_endings', true);
Consulting the docs for auto_detect_line_endings:
When turned on, PHP will examine the data read by fgets() and file() to see if it is using Unix, MS-Dos or Macintosh line-ending conventions.
This enables PHP to interoperate with Macintosh systems, but defaults to Off, as there is a very small performance penalty when detecting the EOL conventions for the first line, and also because people using carriage-returns as item separators under Unix systems would experience non-backwards-compatible behaviour.
So, TL;DR: debug your line endings to make sure you know what's going on (with file or hexdump or similar), and use a combination of auto_detect_line_endings and FILE_IGNORE_NEW_LINES.
Hope this helps :)
I have a file, "serialized.txt", which contains a serialized array (created by doing serialize($array)).
s:133:"a:7:{i:0;i:640;i:1;i:480;i:2;i:2;i:3;s:24:"width="640" height="480"";s:4:"bits";i:8;s:8:"channels";i:3;s:4:"mime";s:10:"image/jpeg";}";
To fetch the contents I do:
$string = file_get_contents("serialized.txt");
Then I do:
print_r(unserialize($string));
The output that I get:
a:7:{i:0;i:640;i:1;i:480;i:2;i:2;i:3;s:24:"width="640" height="480"";s:4:"bits";i:8;s:8:"channels";i:3;s:4:"mime";s:10:"image/jpeg";}
This is the unserialized version of the string (contents of the file) when it should be printing the unserialized array. If I copy the string and do the following:
print_r(unserialize('a:7:{i:0;i:640;i:1;i:480;i:2;i:2;i:3;s:24:"width="640" height="480"";s:4:"bits";i:8;s:8:"channels";i:3;s:4:"mime";s:10:"image/jpeg";}'));
I get the correct output:
Array
(
[0] => 640
[1] => 480
[2] => 2
[3] => width="640" height="480"
[bits] => 8
[channels] => 3
[mime] => image/jpeg
)
So the problem seems to be isolated to the serialized array when pulling from the file.
According to the unserialize docs the function should be returning false if there is a problem; not the contents of the string.
The serialized data is taken from getimagesize and I have verified that if I serialize another array and place it into the file:
serialize(array("hi"));
I can successfully generate the output:
Array
(
[0] => hi
)
Are there any ideas why this may be happening? A bug with the serialization process relating to a getimagesize array, or potentially a "hidden" character in the file that my copy and paste removes? I have millions of these files already generated so it's not possible for me to change the storage method. I guess the solution may just be to write my own parser to serialize the array? The input is always the same format so that's plausible, but I would like to know of this a bug or my error with something somewhere.
As far as I can see your data is double serialized so the following code should print your array:
$string = file_get_contents("serialized.txt");
print_r(unserialize(unserialize($string)));
Although you should think about how you save to file. You may want to remove one serialization.
Does that solve your problem?
I am having problems extracting the layer contents from a .tmx (Tiled) file.
I would like to get the complete uncompressed data in PHP and make a little image of it.
Getting the header information like width, height and so on is no problem - SimpleXML is doing its job there. But somehow decompressing of the tile layer is not working.
The data itself is stored as a base64 and gzip encoded string (sth like H4sIAAAAAAAAC+3bORKAIBQEUVzuf2YTTSwEA/gL00EnJvJQsAjcSyk7EU3v+Jn3OI) but I am having problems even getting the base64 decoded code (it just gives me wierd characters and when i reopened the map in tiled and saved it as "base64 uncompressed" the result was just an empty string - not using gzip decompressing of course).
I already searched through the web and saw how the data is exactly compressed (Github article). It seems like i have to use the gzinflate() command instead of all the others (e.g. gzuncompress), but this is also not working for me.
The code i have now is the following:
<?php
// Get the raw xml data
$map_xml = new SimpleXML(file_get_contents("map.tmx"));
$data = $map_xml["layer"][0]["data"]["#content"]; // I would make a loop here
$content =gzinflate(base64_decode($map_content)); // gives me and error
var_dump($data); // results in nothing
?>
After some more research I found out that I should use a zlib filter (php.net article).
Now I was really confused I don't know what I should pick - I asked google again and got the following: Compressing with Java Decompressing with PHP. According to the answer I have to crop our the header before using the base64 and gzip methods.
Now my questions: Do I have to crop out the header before? If yes, how do I do that?
If not, how can I get the uncompressed data then?
I really hope that someone can help me in here!
Php's gzinflate and gzuncompress are, as previously noted, incorrectly named. However, we can take advantage of gzinflate which accepts raw compressed data. The gzip header is 10 bytes long which can be stripped off using substr. Using your example above I tried this:
$base64content = "H4sIAAAAAAAAC+3bORKAIBQEUVzuf2YTTSwEA/gL00EnJvJQsAjcSyk7EU3v+Jn3OI";
$compressed = substr( base64_decode($base64content), 10);
$content = gzinflate($compressed);
This gives you a string representing the raw data. Your TMX layer consists mostly of gid 0, 2, and 3 so you'll only see whitespace if you print it out. To get helpful data, you'll need to call ord on the characters:
$chars = str_split($content);
$values = array();
foreach($chars as $char) {
$values[] = ord($char);
}
var_dump( implode(',', $values) ); // This gives you the equivalent of saving your TMX file with tile data stored as csv
Hope that helps.
Wow, these PHP functions are horribly named. Some background first.
There are three formats you are likely to encounter or be able to produce. They are:
Raw deflate, which is data compressed to the deflate format with no header or trailer, defined in RFC 1951.
zlib, which is raw deflate data wrapped in a compact zlib header and trailer which consists of a two-byte header and a four-byte Adler-32 check value as the trailer, defined in RFC 1950.
gzip, which is raw deflate data wrapped in a gzip header and trailer where the header is at least ten bytes, and can be longer containing a file name, comments, and/or an extra field, and an eight-byte trailer with a four-byte CRC-32 and a the uncompressed length module 2^32. This wrapper is defined in RFC 1952. This is the data you will find in a file with the suffix .gz.
The PHP functions gzdeflate() and gzinflate() create and decode the raw deflate format. The PHP functions gzcompress() and gzuncompress() create and decode the zlib format. None of these functions should have "gz" in the name, since none of them handle the gzip format! This will forever be confusing to PHP coders trying to create or decode gzip-formatted data.
There seem to be (but the documentation is not clear if they are always there) PHP functions gzencode() and gzdecode() which, if I am reading the terse documentation correctly, by default create and decode the gzip format. gzencode() also has an option to produce the zlib format, and I suspect that gzdecode() will attempt to automatically detect the gzip or zlib format and decode accordingly. (That is a capability that is part of the actual zlib library that all of these functions use.)
The documentation for zlib_encode() and zlib_decode() is incomplete (where those pages admit: "This function is currently not documented; only its argument list is available"), so it is difficult to tell what they do. There is an undocumented encoding string parameter for zlib_encode() that presumably would allow you to select one of the three formats, if you knew what to put in the string. There is no encoding parameter for zlib_decode(), so perhaps it tries to auto-detect among the three formats.
I know this is old now, but I've literally spent all day playing with this code.
It's been really picky about what I do. However, here's a quick function to turn TMX files into an array of IDs for each tile on each layer.
Credits go to the other answerers who helped me piece together where I was going wrong.
<?php
function getLayer($getLayerName = '')
{
$xml = simplexml_load_file('level.tmx');
$values = array();
foreach($xml->layer as $child)
{
$name = $child->attributes()->name;
if(!empty($getLayerName))
if($name != $getLayerName)
continue;
$data = gzinflate(substr(base64_decode(trim($child->data)), 10));
$chars = str_split($data);
$i = 0;
foreach($chars as $char)
{
$charID = ord($char);
if($i % 4 == 0) // I'm only interested in the tile IDs
{
$values[(String) $name][] = $charID;
}
$i++;
}
}
return $values;
}
print_r(getLayer());
//or you could use getLayer('LayerName') to get a single layer!
?>
On my example 3x3 map, with only one tile image, I get the following:
Array
(
[floor] => Array
(
[0] => 1
[1] => 1
[2] => 1
[3] => 1
[4] => 1
[5] => 1
[6] => 1
[7] => 1
[8] => 1
)
[layer2] => Array
(
[0] => 0
[1] => 0
[2] => 1
[3] => 0
[4] => 1
[5] => 0
[6] => 1
[7] => 1
[8] => 0
)
)
Hopefully this function proves handy for anyone out there who needs it.
I'm trying to help my dad out -- he gave me an export from a scheduling application at his work. We are trying to see if we can import it into a mysql database so he/co-workers can collaborate online with it.
I've tried a number of different methods but none seem to work right -- and this is not my area of specialties.
Export can be seen here: http://roikingon.com/export.txt
Any help / advice on how to go about parsing this would be greatly appreciated!
Thanks !!
I've made an attempt to write a (somewhat dynamic) fixed-with-column parser. Take a look: http://codepad.org/oAiKD0e7 (it's too long for SO, but it's mostly just "data").
What I've noticed
Text-Data is left aligned with padding on the right like "hello___" (_ = space)
Numerical data is right aligned with padding on the left "___42"
If you want to use my code there's yet stuff to do:
The record types 12.x have variable column count (after some static columns), you'd have to implement another "handler" for it
Some of my width's are most probably wrong. I think there is a system (like numbers are 4 characters long and text 8 characters long, with some variations for special cases). Someone with domain knowledge and more than one sample file could figure out the columns.
Getting the raw-data out is only the first step, you have to map the raw-data to some useful model and write that model to the database.
With that file structure you're basically in need of reverse engineering a proprietary format. Yes, it is space delimited but the format does not follow any kind of standard like CSV, YAML etc. It is completely proprietary with what seems to be a header and separate section with headers of their own.
I think your best bet is to try and see if there's some other type of export that can be done such as Excel or XML and working from there. If there isn't then see if there's an html output of some kind that can be screen scraped, and pasted into Excel and seeing what you get.
Due to everything I mentioned above it will be VERY difficult to massage the file in its current form into something that can be sensibly imported into a database. (Note that from the file structure a number of tables would be needed.)
you can use split with a regular expression (zero or more spaces).
I will try and let you know.
There doesnt seem to be a structure with you data.
$data = "12.1 0 1144713 751 17 Y 8 517 526 537 542 550 556 561 567 17 ";
$arr = preg_split("/ +/", $data);
print_r($arr);
Array
(
[0] => 12.1
[1] => 0
[2] => 1144713
[3] => 751
[4] => 17
[5] => Y
[6] => 8
[7] => 517
[8] => 526
[9] => 537
[10] => 542
[11] => 550
[12] => 556
[13] => 561
[14] => 567
[15] => 17
[16] =>
)
Try this preg_split("/ +/", $data); which splits the line by zero or more spaces, then you will have a nice array, that you can process. But looking at your data, there is no structure, so you will have to know which array element corresponds to what data.
Good luck.
Open it with excel and save it as comma-delimited. Treat consecutive delimiters as one, or not. Then resave it with excel as a csv, which will be comma-separated and easier to import to mysql.
EDIT:
The guy who says to use preg_split on "[ +]" is giving you essentially the same answer as I just did above.
The question is what to do after that, then.
Have you determined yet how many "row types" there are? Once you've determined that and defined their characteristics it will be a lot easier to write some code to go through it.
If you save it in csv, you can use the PHP fgetcsv function and related functions. For each row, you would check it's type and perform operations depending on the type.
I noticed that your data rows could possibly be divided on whether or not the first column's data contains a "." so here's an example of how you might loop through the file.
while($row = fgetcsv($file_handle)) {
if(strpos($row[0],'.') === false) {
// do something
} else {
// do something else
}
}
"do something" would be something like "CREATE TABLE table_$row[0]" or "INSERT INTO table" etc.
Ok, and here's some more observation:
Your file is really like multiple files glued together. It contains multiple formats. Notice all the rows starting with "4" next have a 4-letter company abbreviation followed by full company name. One of them is "caco". If you search for "caco", you find it in multiple "tables" within the file.
I also notice "smuwtfa" (days of the week) sprinkled around.
Use clues like that to determine the logic of how to treat each row.