How to get FGETCSV to read entire Japanese cell - php

Question

I have a large study conducted with about 50 questions and 70,000 entries, so manually editing or using pivot tables just won't really work, I need to upload the data into a database. I can't get the Japanese characters to be read with any accuracy while using fgcsv(). I've tried setting the locale to UTF-8 and SJIS, but neither one seem to want to read all of the Japanese characters. I read somewhere this might be a bug, but I don't know..
The data looks like this:
Q-004 必須回答 あなたは、以下のどちらにお住まいですか？ S/A
1 北海道 Hokkaido
2 青森県 Aomori
3 岩手県 Iwate
4 宮城県 Miyagi
5 秋田県 Akita
Here is my code:
setlocale(LC_ALL, 'ja_JP.SJIS');
$fp = fopen($_POST["filename"],'r') or die("can't open file");
$csv_line = fgetcsv($fp,1024);
$query = "";
$count = 0;
$question = false;
while($csv_line = fgetcsv($fp,1024)) {
if (!$question && strpos($csv_line[0],"Q-")!== false)
{
echo "Found a question: ".$csv_line[2] . "<br>";
$question = true;
}
else if($question && strlen($csv_line[0])==0)
{
echo "<hr>";
$question = false;
}
else if($question && intval($csv_line[0])>0)
{
echo $csv_line[0]. " has value ". $csv_line[2]." - ".$csv_line[3]. "<br>";
}
$count++;
}
echo "$count records read successfully";
fclose($fp) or die("can't close file");
Here is the result:
Found a question: A以下のどちらにお住まいですか？
1 has value k海道 - Hokkaido
2 has value X県 - Aomori
3 has value - Iwate
4 has value {城県 - Miyagi
5 has value H田県 - Akita

Answer 1

When it comes to reading a CSV in PHP, I would say... don't do it, and use an SQL database instead, wherein you can set a collation such as ujis_japanese_ci in MySQL.
You should be able to easily import your CSV into a MySQL database using phpMyAdmin, if that is what you have, and then render the data from the MySQL database instead of reading a CSV file.
It is a work-around, granted, but my general experience is that CSV + foreign/special characters == problems.
I believe it is at least worth the try. Good luck