What does "ÿþ" in the content of an URL mean? [duplicate] - php

I'm trying to read ID3 data in bulk. On some of the tracks, ÿþ appears. I can remove the first 2 characters, but that hurts the tracks that don't have it.
This is what I currently have:
$trackartist=str_replace("\0", "", $trackartist1);
Any suggestions would be greatful, thanks!


ÿþ is 0xfffe in UTF-8; this is the byte order mark in UTF-16.
You can convert your string to UTF-8 with iconv or mb_convert_encoding():
$trackartist1 = iconv('UTF-16LE', 'UTF-8', $trackartist1);
# Same as above, but different extension
$trackartist1 = mb_convert_encoding($trackartist1, 'UTF-16LE', 'UTF-8');
# str_replace() should now work
$trackartist1 = str_replace('ÿþ', '', $trackartist1);
This assumes $trackartist1 is always in UTF-16LE; check the documentation of your ID3 tag library on how to get the encoding of the tags, since this may be different for different files. You usually want to convert everything to UTF-8, since this is what PHP uses by default.


I had a similar problem but was not able to force UTF-16LE as the input charset could change. Finally I detect UTF-8 as follows:
if (!preg_match('~~u', $html)) {
For the case that this fails I obtain the correct encoding through the BOM:
function detect_bom_encoding($str) {
if ($str[0] == chr(0xEF) && $str[1] == chr(0xBB) && $str[2] == chr(0xBF)) {
return 'UTF-8';
}
else if ($str[0] == chr(0x00) && $str[1] == chr(0x00) && $str[2] == chr(0xFE) && $str[3] == chr(0xFF)) {
return 'UTF-32BE';
}
else if ($str[0] == chr(0xFF) && $str[1] == chr(0xFE)) {
if ($str[2] == chr(0x00) && $str[3] == chr(0x00)) {
return 'UTF-32LE';
}
return 'UTF-16LE';
}
else if ($str[0] == chr(0xFE) && $str[1] == chr(0xFF)) {
return 'UTF-16BE';
}
}
And now I'm able to use iconv() as you can see in #carpetsmoker answer:
iconv(detect_bom_encoding($html), 'UTF-8', $html);
I did not use mb_convert_encoding() as it did not remove the BOM (and did not convert the linebreaks as iconv() does):


Use regex replacement:
$trackartist1 = preg_replace("/\x00?/", "", $trackartist1);
The regex above seeks the first occurrence of "\x00"(hexadecimal zeros), if possible, and replaces it with nothing.

Related

Charset issue from local to remote server

I have a problem of charset.
On localhost everything works fine, but now on remote server I see strange characters replacing others like à or è. I have read it's a charset issue and I think the problem can be my php.ini (I can't edit it).
To solve it I've tried many things:
I've set
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
on html,
ini_set('default_charset', 'UTF-8');
on php,
AddDefaultCharset utf-8
on my .htaccess file,
if I use utf8_encode on strings letters are replaced by ã or similar, if I leave it without doing anything letters are �
There is another way to solve this problem that I have not found yet?
Sorry, I forgot to say it: strings are retrieved from another site by a file_get_contents (I'm using a Yandex API)
Here's some code:
$yandex = 'https://dictionary.yandex.net/api/v1/dicservice.json/lookup?key=my_api_key&lang=it-it&text=attualità';
// get json from this page
$object = json_decode(file_get_contents($yandex));
$syns_array = array();
$type = '';
// if the word exists
if (!empty($object->def) && $object->def != FALSE && $object->def != NULL)
{
$type = $object->def[0]->tr[0]->pos;
$rows = $object->def[0]->tr;
// if there're synonyms
if (!empty($rows) && $rows != FALSE && $rows != NULL)
{
foreach ($rows as $row)
{
array_push($syns_array, $row->text);
// if there're more rows with syns
if (!empty($row->syn) && $row->syn !== FALSE && $row->syn !== NULL)
{
foreach ($row->syn as $syns_obj)
{
array_push($syns_array, $syns_obj->text);
}
}
}
}
}
// I echo my synonyms from the array
foreach($syns_array as $syn) {
echo $syn;
}

I forgot to say I was using mb_strtolower on those strings. Replacing it with strotolower the problem is solved... Sorry

Exception on php mb_ereg_match

I am using mb_ereg_match to validate that a domain name does not containe illegal characters.
I am using this regex:
'/:\/\/|www[.][a-zA-Zα-ωΑ-ΩάέύήίόώϋϊΐΰΆΈΏΊΎΌΉΫΪÀàÂâÆæÄäÇçÉéÈèÊêËëÎîÏïÔôŒœÖöÙùÛûÜüŸÿ0-9]+[.]|^[-]+|^[.]+|[-]+$|[.]+$|[-]{2,}|[.]{2,}|[^\w-.]|-[.]|[.]-/u'
Which as you can se by your self contain all the basic latin chars, nums, France's letters and the whole Greek alphabet.
My validation code is the following:
$utf8 = (mb_detect_encoding($value) == 'UTF-8') ? TRUE : FALSE;
if ($utf8){
mb_internal_encoding('UTF-8');
mb_regex_encoding('UTF-8');
$matches = mb_ereg_match($pattern, $value);
}else{
preg_match($pattern, $value, $matches);
}
I am trying to validate this:
'geoσσσrge.cσσσσσm.gr'
Here is the error I get:
mb_ereg_match(): mbregex compile err: empty range in char class
The error does not appear all the time. Usually it apears when it stays idle for a long time and after I refresh my page returns to normal.
I don't know how to handle this error or how to approche it in order to find the source of the problem.
Any suggestions?

\w through . is not a range it can understand. Escape the - or move the - to the start; [^\w-.].
$pattern = '/:\/\/|www[.][a-zA-Zα-ωΑ-ΩάέύήίόώϋϊΐΰΆΈΏΊΎΌΉΫΪÀàÂâÆæÄäÇçÉéÈèÊêËëÎîÏïÔôŒœÖöÙùÛûÜüŸÿ0-9]+[.]|^[-]+|^[.]+|[-]+$|[.]+$|[-]{2,}|[.]{2,}|[^\w\-.]|-[.]|[.]-/u';
$value = 'geoσσσrge.cσσσσσm.gr';
$utf8 = (mb_detect_encoding($value) == 'UTF-8') ? TRUE : FALSE;
if ($utf8){
mb_internal_encoding('UTF-8');
mb_regex_encoding('UTF-8');
$matches = mb_ereg_match($pattern, $value);
}else{
preg_match($pattern, $value, $matches);
}
or
$pattern = '/:\/\/|www[.][a-zA-Zα-ωΑ-ΩάέύήίόώϋϊΐΰΆΈΏΊΎΌΉΫΪÀàÂâÆæÄäÇçÉéÈèÊêËëÎîÏïÔôŒœÖöÙùÛûÜüŸÿ0-9]+[.]|^[-]+|^[.]+|[-]+$|[.]+$|[-]{2,}|[.]{2,}|[^-\w.]|-[.]|[.]-/u';
$value = 'geoσσσrge.cσσσσσm.gr';
$utf8 = (mb_detect_encoding($value) == 'UTF-8') ? TRUE : FALSE;
if ($utf8){
mb_internal_encoding('UTF-8');
mb_regex_encoding('UTF-8');
$matches = mb_ereg_match($pattern, $value);
}else{
preg_match($pattern, $value, $matches);
}

Avoid re-conversion of a UTF-8 String PHP

Well, I have a BD with a lot of ISO strings and another with UTF-8 (yes, I ruin everything) and now I'm making a custom function that rewrite all the BD again to have all in UTF-8, the problem, is the conversion with UTF-8 strings... The ? appears:
$field = $fila['Field'];
$acon = mysql_fetch_array(mysql_query("SELECT `$field` as content FROM `$curfila` WHERE id='$i'"));
$content = $acon['content'];
if(!is_numeric($content)) {
if($content != null) {
if(ip2long($content) === false) {
mb_internal_encoding('UTF-8');
if(mb_detect_encoding($content) === "UTF-8") {
$sanitized = utf8_decode($content);
if($sanitized != $content) {
echo 'Fila [ID ('.$i.')] <b>'.$field.'</b> => '.$sanitized.'<br>';
//mysql_query("UPDATE `$curfila` SET `$field`='$sanitized' WHERE id='$i'");
}
}
}
}
}
PD: I check all the columns and rows of all the tables of the BD. (I show all everything before doing anything)
So, how can I detect that?
I tried mb_detect_encoding, but the all the string are in UTF-8... So, which function can I use now?
Thanks in advance.

What is the best PHP method/class for cleaning strings to UTF-8

I've been using a couple of methods on a simple class, which have worked ok for me, but I've noticed that they are really slow due to strtr() and having a ton of translations defined. Also it is really long and thus, more difficult to maintain and understand.
That said, all of the "bad" examples are solutions to real-world issues which have come up, translating strings into UTF8.
Can anyone tell me of there's a well known or more efficient means of doing this? (yes, I've tried the htmlentities() methods and the iconv()methods but neither really replaced all the funky characters correctly.
Here's the class I'm using currently:
https://gist.github.com/2559140

mbstring support is now enabled (but not loaded) by default as of PHP 5.4.0. Loading the extension, this lets you do:
<? //PHP 5.4+
$ensureIsUTF8 = static function($data){
$dataEncoding = \mb_detect_encoding(
$data,
['UTF-8', 'windows-1251', 'iso-8859-1', /*others you encounter*/],
true
);
//UTF-16/32 encoding detection always fails for PHP <= 5.4.1
//Use detection code copied from PHP docs comments:
//http://www.php.net/manual/en/function.mb-detect-encoding.php
if ($dataEncoding === false){
$UTF32_BIG_ENDIAN_BOM = chr(0x00) . chr(0x00) . chr(0xFE) . chr(0xFF);
$UTF32_LITTLE_ENDIAN_BOM = chr(0xFF) . chr(0xFE) . chr(0x00) . chr(0x00);
$UTF16_BIG_ENDIAN_BOM = chr(0xFE) . chr(0xFF);
$UTF16_LITTLE_ENDIAN_BOM = chr(0xFF) . chr(0xFE);
$first2 = \substr($data, 0, 2);
$first4 = \substr($data, 0, 4);
if ($first4 === $UTF32_BIG_ENDIAN_BOM) {
$dataEncoding = 'UTF-32BE';
} elseif ($first4 === $UTF32_LITTLE_ENDIAN_BOM) {
$dataEncoding = 'UTF-32LE';
} elseif ($first2 === $UTF16_BIG_ENDIAN_BOM) {
$dataEncoding = 'UTF-16BE';
} elseif ($first2 === $UTF16_LITTLE_ENDIAN_BOM) {
$dataEncoding = 'UTF-16LE';
} else {
throw new \Exception('Whoa! No idea what that was.');
}
}
if ($dataEncoding === 'UTF-8'){
return $data;
} else {
return \mb_convert_encoding(
$data,
'UTF-8',
$dataEncoding
);
}
};
$utf8Data = $ensureIsUTF8(\file_get_contents('something'));
$utf8Data = $ensureIsUTF8(\file_get_contents('http://somethingElse'));
$utf8Data = $ensureIsUTF8($userProvidedData);
?>

Detect base64 encoding in PHP?

Is there some way to detect if a string has been base64_encoded() in PHP?
We're converting some storage from plain text to base64 and part of it lives in a cookie that needs to be updated. I'd like to reset their cookie if the text has not yet been encoded, otherwise leave it alone.

Apologies for a late response to an already-answered question, but I don't think base64_decode($x,true) is a good enough solution for this problem. In fact, there may not be a very good solution that works against any given input. For example, I can put lots of bad values into $x and not get a false return value.
var_dump(base64_decode('wtf mate',true));
string(5) "���j�"
var_dump(base64_decode('This is definitely not base64 encoded',true));
string(24) "N���^~)��r��[jǺ��ܡם"
I think that in addition to the strict return value check, you'd also need to do post-decode validation. The most reliable way is if you could decode and then check against a known set of possible values.
A more general solution with less than 100% accuracy (closer with longer strings, inaccurate for short strings) is if you check your output to see if many are outside of a normal range of utf-8 (or whatever encoding you use) characters.
See this example:
<?php
$english = array();
foreach (str_split('az019AZ~~~!##$%^*()_+|}?><": Iñtërnâtiônàlizætiøn') as $char) {
echo ord($char) . "\n";
$english[] = ord($char);
}
echo "Max value english = " . max($english) . "\n";
$nonsense = array();
echo "\n\nbase64:\n";
foreach (str_split(base64_decode('Not base64 encoded',true)) as $char) {
echo ord($char) . "\n";
$nonsense[] = ord($char);
}
echo "Max nonsense = " . max($nonsense) . "\n";
?>
Results:
Max value english = 195
Max nonsense = 233
So you may do something like this:
if ( $maxDecodedValue > 200 ) {} //decoded string is Garbage - original string not base64 encoded
else {} //decoded string is useful - it was base64 encoded
You should probably use the mean() of the decoded values instead of the max(), I just used max() in this example because there is sadly no built-in mean() in PHP. What measure you use (mean,max, etc) against what threshold (eg 200) depends on your estimated usage profile.
In conclusion, the only winning move is not to play. I'd try to avoid having to discern base64 in the first place.

function is_base64_encoded($data)
{
if (preg_match('%^[a-zA-Z0-9/+]*={0,2}$%', $data)) {
return TRUE;
} else {
return FALSE;
}
};
is_base64_encoded("iash21iawhdj98UH3"); // true
is_base64_encoded("#iu3498r"); // false
is_base64_encoded("asiudfh9w=8uihf"); // false
is_base64_encoded("a398UIhnj43f/1!+sadfh3w84hduihhjw=="); // false
http://php.net/manual/en/function.base64-decode.php#81425

I had the same problem, I ended up with this solution:
if ( base64_encode(base64_decode($data)) === $data){
echo '$data is valid';
} else {
echo '$data is NOT valid';
}

Better late than never: You could maybe use mb_detect_encoding() to find out whether the encoded string appears to have been some kind of text:
function is_base64_string($s) {
// first check if we're dealing with an actual valid base64 encoded string
if (($b = base64_decode($s, TRUE)) === FALSE) {
return FALSE;
}
// now check whether the decoded data could be actual text
$e = mb_detect_encoding($b);
if (in_array($e, array('UTF-8', 'ASCII'))) { // YMMV
return TRUE;
} else {
return FALSE;
}
}
UPDATE For those who like it short
function is_base64_string_s($str, $enc=array('UTF-8', 'ASCII')) {
return !(($b = base64_decode($str, TRUE)) === FALSE) && in_array(mb_detect_encoding($b), $enc);
}

We can combine three things into one function to check if given string is a valid base 64 encoded or not.
function validBase64($string)
{
$decoded = base64_decode($string, true);
$result = false;
// Check if there is no invalid character in string
if (!preg_match('/^[a-zA-Z0-9\/\r\n+]*={0,2}$/', $string)) {$result = false;}
// Decode the string in strict mode and send the response
if (!$decoded) {$result = false;}
// Encode and compare it to original one
if (base64_encode($decoded) != $string) {$result = false;}
return $result;
}

I was about to build a base64 toggle in php, this is what I did:
function base64Toggle($str) {
if (!preg_match('~[^0-9a-zA-Z+/=]~', $str)) {
$check = str_split(base64_decode($str));
$x = 0;
foreach ($check as $char) if (ord($char) > 126) $x++;
if ($x/count($check)*100 < 30) return base64_decode($str);
}
return base64_encode($str);
}
It works perfectly for me.
Here are my complete thoughts on it: http://www.albertmartin.de/blog/code.php/19/base64-detection
And here you can try it: http://www.albertmartin.de/tools

base64_decode() will not return FALSE if the input is not valid base64 encoded data. Use imap_base64() instead, it returns FALSE if $text contains characters outside the Base64 alphabet
imap_base64() Reference

Here's my solution:
if(empty(htmlspecialchars(base64_decode($string, true)))) {
return false;
}
It will return false if the decoded $string is invalid, for example: "node", "123", " ", etc.

$is_base64 = function(string $string) : bool {
$zero_one = ['MA==', 'MQ=='];
if (in_array($string, $zero_one)) return TRUE;
if (empty(htmlspecialchars(base64_decode($string, TRUE))))
return FALSE;
return TRUE;
};
var_dump('*** These yell false ***');
var_dump($is_base64(''));
var_dump($is_base64('This is definitely not base64 encoded'));
var_dump($is_base64('node'));
var_dump($is_base64('node '));
var_dump($is_base64('123'));
var_dump($is_base64(0));
var_dump($is_base64(1));
var_dump($is_base64(123));
var_dump($is_base64(1.23));
var_dump('*** These yell true ***');
var_dump($is_base64(base64_encode('This is definitely base64 encoded')));
var_dump($is_base64(base64_encode('node')));
var_dump($is_base64(base64_encode('123')));
var_dump($is_base64(base64_encode(0)));
var_dump($is_base64(base64_encode(1)));
var_dump($is_base64(base64_encode(123)));
var_dump($is_base64(base64_encode(1.23)));
var_dump($is_base64(base64_encode(TRUE)));
var_dump('*** Should these yell true? Might be edge cases ***');
var_dump($is_base64(base64_encode('')));
var_dump($is_base64(base64_encode(FALSE)));
var_dump($is_base64(base64_encode(NULL)));

May be it's not exactly what you've asked for. But hope it'll be usefull for somebody.
In my case the solution was to encode all data with json_encode and then base64_encode.
$encoded=base64_encode(json_encode($data));
this value could be stored or used whatever you need.
Then to check if this value isn't just a text string but your data encoded you simply use
function isData($test_string){
if(base64_decode($test_string,true)&&json_decode(base64_decode($test_string))){
return true;
}else{
return false;
}
or alternatively
function isNotData($test_string){
if(base64_decode($test_string,true)&&json_decode(base64_decode($test_string))){
return false;
}else{
return true;
}
Thanks to all previous answers authors in this thread:)

Usually a text in base64 has no spaces.
I used this function which worked fine for me. It tests if the number of spaces in the string is less than 1 in 20.
e.g: at least 1 space for each 20 chars --- ( spaces / strlen ) < 0.05
function normalizaBase64($data){
$spaces = substr_count ( $data ," ");
if (($spaces/strlen($data))<0.05)
{
return base64_decode($data);
}
return $data;
}

Your best option is:
$base64_test = mb_substr(trim($some_base64_data), 0, 76);
return (base64_decode($base64_test, true) === FALSE ? FALSE : TRUE);

Categories