Prepare unique values for utf_unicode_ci in php - php

Question

I need to read values from an ISO-8859-1 encoded file in PHP and use PDO to write them to a database table that's encoded utf8_unicode_ci and has a unique index. Sometimes the data is missing special chars which leads to duplicate key errors. Example: the data contains "Entrainement" and "Entraînement". Is there a PHP string function I can use to avoid this?
Preferably a conversion function so I don't have to iterate over the whole array to check if a value was already inserted.
Here is an example of what I'm trying to do:
$values = array("Entraînement", "Entrainement");
$db = new PDO("mysql:dbname=mydb;host=localhost;charset=utf8", "user", "pw");
$db->exec("SET NAMES 'UTF-8'");
$stmt = $db->prepare("INSERT INTO mytable(myvalue) VALUES(?)");
$already_inserted = array();
foreach($values as $v) {
$v = $v_inserted = iconv('iso-8859-1', 'utf-8', $v);
// Do magic string conversion here
// $v_inserted = collation_convert($v_inserted)
if(isset($already_inserted[$v_inserted])) {
continue;
}
if($stmt->execute(array($v))) {
$already_inserted[$v_inserted] = true;
}
}
This example should only insert "Entraînement" and skip over "Entrainement".
In the original program I'm using Doctrine ORM instead of PDO so I can' do much in SQL. Also, I have special chars in the whole Latin1 range - French, German, Spanish, etc.
I can't change the DB field definition to utf8_binbecause it's part of an ecommerce package - all sorts of things might break.

Answer 1

Well you should definitely convert the values to UTF-8 and use UTF-8 connection encoding. Otherwise your application cannot take advantage of the UTF-8 at all because your application will only be able to send and receive characters that ISO-8859-1 contains. This is very, very little amount compared to Unicode ☹.
That is unrelated to your issue*, in the unicode_ci collation, î is considered same as i.
If you need to consider them as different characters use some other collation:
SELECT 'î' = 'i' COLLATE 'utf8_unicode_ci'
//1
SELECT 'î' = 'i' COLLATE 'utf8_bin'
//0
There is no German** collation so I guess utf8_bin is what you want here.
*There is only an issue when the declared connection encoding does not match the encoding of the physical bytes you send over. I.E. If you send ISO-8859-1 bytes with UTF-8 connection encoding, you will get crap if not an error. And vice versa.
**I looked that up from your profile, if you in fact need some other language there might be a collation for that.