csv "UTF8" character encoding with pgsql - php

Question

Our client has sent us a CSV file of data that I need to import into a specific table in our Postgresql 8.3.9 database. The database uses UTF-8 character encoding, i.e. our CMS allows multiple languages such as French which are inputted into the database via the CMS in French. One particular facility is for the client to upload images to the server and then enter "alt" tags for them in French. However, due to a bulk update required, we have been sent a CSV to feed into a particular table - for the image alt tags, in French.
The CSV has some special characters such as "é" - e.g.
"Bottes Adaptées Amora Cuir Faux-Croco Fauve Photo d'Ensemble"
The images themselves are hosted on two places - one is a CDN, and one is a local database backup and local server (web server) file backup. I am using a PHP script to read the CSV file and do the needful so that the "alt" tags are updated on two places - our web database, and the CDN.
However, when I read the CSV (using PHP), the character does not "come out" as expected.
The data is comming as "Bottes Adapt�es Amora Cuir Faux-Croco Fauve Photo d'Ensemble".
I don't think this has anything to do with the database, but it has something to do with my PHP file reading the CSV data. Even if I print the data that it is reading, the special character above does not print as above, it' prints as if the special character is not recognised. Other characters print fine.
Here is the code I'm using (not some special custom functions are used here to interact with the database but they can be ignored). The CSV file is made up of {column 1} for image name, and {column 2} for the ALT tag.
$handle = fopen($conn->getIncludePath() . "cronjobs/GIB_img_alt_tags_fr.csv", "r");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
//normally I run a query here to check if the data exists - "SELECT imageid, image_fileref FROM table1 WHERE image_fileref = '". $data[0]. "'");
if ($conn->Numrows($result)) { //if rows were found -
$row=$conn->fetchArray($result);
//printing the data from $row here
}
}
fclose($handle);

Answer 1

You've still omitted key information - when asking for help with an UPDATE don't delete the UPDATE statement from the code - and your description of the problem is very confused, but there's some hint of what's going on.
Mismatched encodings
It's highly likely that your PHP connection has a client_encoding set to something other than UTF-8. If you're sending UTF-8 data down the connection without conversion, the connection's client_encoding must be UTF-8.
To confirm, run SHOW client_encoding as a SQL statement from PHP and print the result. Add SET client_encoding = 'UTF-8' to your code before importing the CSV and see if that helps. Assuming, of course, that the CSV file is really UTF-8 encoded. If it isn't, you need to either transcode it to UTF-8 or find out what encoding it is in and SET client_encoding to that.
Read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) and the PostgreSQL manual on character set support.
Better approach
The approach you're taking is unnecessarily slow and inefficient, anyway. You should be:
Opening a transaction
Creating a temporary table in the database with the same structure as the CSV file.
Use pg_copy_from to load the CSV into the temp table, with appropriate options to specify the CSV format.
Merge the contents of the temporary table into the destination table with an INSERT then an UPDATE, eg:
INSERT INTO table1 (image_fileref, ... other fields ...)
SELECT n.image_fileref, ... other fields ...
FROM the_temp_table n
WHERE NOT EXISTS (SELECT 1 from table1 o WHERE o.image_fileref = n.image_fileref);
UPDATE table1 o
SET .... data to update ....
FROM the_temp_table n
WHERE o.image_fileref = n.image_fileref;
Commit the transaction
The INSERT may be more efficiently written as a left outer join with an IS NULL filter to exclude matching rows. It depends on the data. Try it.
I probably could've written a faster CTE-based version, but you didn't say what version of Pg you were using, so I didn't know if your server supported CTEs.
Since you left out the UPDATE I can't be more specific about the UPDATE or INSERT statements. If you'd provided the schema for table1 or even just your INSERT or UPDATE I could've said more. Without sample data I haven't been able to run the statements to check them, and I didn't feel like making up some dummy data, so the above is untested. As it is, completing the code is left as a learning exercise. I will not be updating this answer with fully-written-out statements, you get to work that out.