This question already has answers here:
UTF-8 all the way through
(13 answers)
Closed 7 years ago.
So I was having an issue with json_encode returning null that I found the solution for here, but I don't understand why it was an issue in the first place. The MySQL tables from which I was drawing the data are defined like
CREATE TABLE `super_table` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) DEFAULT NULL,
`values` text,
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;
so shouldn't values and name be utf8 encoded already when I pull them out in PHP? A simplified version of what I'm doing:
$sql = "SELECT * FROM super_table";
$query = $mysqli->query($sql);
$data = (object) $query->fetch_all(MYSQLI_ASSOC);
foreach ($data as $key => $val) {
$data->$key = utf8_encode($val);
}
$result = array('success'=>$success, 'data'=>$data);
echo json_encode($result);
Why does not doing the extra utf8_encode step sometimes yield a null result when I try to json_encode it?
Indeed, MySQL's utf8 is not universally understood UTF-8. Even though this is not your issue here... MySQL's utf8 is a subset of actual UTF-8 only covering the BMP and not supporting 4-byte characters. But this just means that high characters will get discarded; otherwise it's still UTF-8 compatible.
Your actual issue is that MySQL is just storing the data in utf8, but that says nothing about how you will receive the data in your database client. MySQL converts text on the fly from the stored encoding to the connection encoding (and vice versa). When connecting to the database in your PHP code, you can choose which encoding you prefer to receive your data in. Use $mysqli->set_charset('utf8') to retrieve your data in UTF-8.
Updated Question :
I had one old script with mysql 5.5.46 . i was storing my text in one columnt with blob type. my text was persian like this : سلام خوبی جه خبر .
after some month's i changed blob column to longblob column with phpmyadmin with gui(without any converting query).
everything work's correctly but i get backup from my mysql and restored this db after 2 years. now it's not show my persian characters correctly.english text is ok but persian text's is going someting like the hex i mentioned in my question.
I need text's stored in page_about and page_contact columns in this table:
Creating Table :
mysql_query("CREATE TABLE `".$prefix."ProFolio_info` (
`id` int(5) NOT NULL auto_increment,
`page_about` blob NOT NULL,
`page_contact` blob NOT NULL,
PRIMARY KEY (`id`)
) TYPE=MyISAM AUTO_INCREMENT=1") or die(mysql_error());
Updating Data :
$info_query = mysql_query("SELECT * FROM ".$prefix."ProFolio_info ORDER BY id DESC LIMIT 0,10");
while($info_row = mysql_fetch_array($info_query)){
$about_page = html_entity_decode($info_row['page_about']);
$contact_page = html_entity_decode($info_row['page_contact']);
if(isset($_POST['change_settings']) && $LOGGEDIN == 'yes'){
$new_aboutpage = clean_page($_POST['about_page']);
$new_contactpage = clean_page($_POST['contact_page']);
if($about_page != $new_aboutpage){
mysql_query("UPDATE ".$prefix."ProFolio_info SET page_about = '$new_aboutpage' WHERE id = '$info_id'");
}
if($contact_page != $new_contactpage){
mysql_query("UPDATE ".$prefix."ProFolio_info SET page_contact = '$new_contactpage' WHERE id = '$info_id'");
}
get texts from DB :
<textarea name="contact_page"><? echo str_replace('<br />', '', $contact_page); ?></textarea>
</div>
i tested some query like cast and convert and convert column to longtest but result is the same and i have wrong chracters.
I think it stored with latin1 collation but I select column with utf-8
HEX
BC28620264F736C6173683B26756D6C3B264F736C6173683B26736563743B265567726176653BC284264F736C6173683B26736563743B20265561637574653B266D6163723B265567726176653BC281264F736C6173683B266F7264663B265567726176653BC28520264F736C6173683B26736563743B2655636972633BC28C265567726176653BC286264F736C6173683B266E6F743B264F736C6173683B26736563743B20265567726176653BC286265567726176653BC2852655636972633BC28C265561637574653B266D6163723B265567726176653BC286264F736C6173683B266E6F743B265567726176653BC28720264F736C6173683B26736563743B265567726176653BC285264F736C6173683B26736563743B20265567726176653BC285264F736C6173683B26737570333B265567726176653BC286265561637574653B26636F70793B265567726176653BC28720265561637574653B266D6163723B265567726176653B
i uploaded my script in github
also uploaded script in my server and sending text from script but data going to db correctly ! i think it's just because of my backup of database.
you can check my ready script from here and view my text's from kave note's menu;
There are 4 places to "say" utf8:
The data in the client must be utf8-encoded. (It probably was.)
SET NAMES utf8 or equivalent. (You took care of that in new PDO.)
CHARACTER SET utf8 on the table or column declaration. Please provide `SHOW CREATE TABLE, but do not try to fix it if it says latin1.
On the html page, <meta ... charset=UTF-8 ...>
Please provide SELECT col, HEX(col) FROM tbl WHERE ... so we can see whether the data was messed up in the table. There two possible fixes for the data; we need to see the hex to know which fix to apply.
I have too got the same problem the main reason is windows uses different kind of character encoding for different characterset. I have found the package convert character set. It has solved my problem and it may solve your too.
I have csv files what i parse to array and pass to mysql table.
Some CSV files does not contains some columns what are in database and when i convert it to array than i have smaller number of columns that are in table and i get "syntax error".
From controller i call:
function sendHistoric(){
$this->load->model('Historic_model');
$this->load->library('csvreader');
foreach($this->divisions as $div){
$result = $this->csvreader->parse_file("assets/csv/1516{$div}.csv");//path to csv file
$this->Historic_model->loadCSVtoDB($result);
//var_dump($result);
}
}
in model i have:
function loadCSVtoDB($data){
$sql = "call ins_historic(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
foreach($data as $row){
var_dump($row);
$this->db->query($sql,$row);
}
//echo $this->db->conn_id->error_message();
}
Can i somehow send NULL instead where he find that i have smaller number of data or i need to check if each element of array exists, if not that set it to null?
One option to consider is not writing slower PHP looping but to harness the flexible code possibilties inside a LOAD DATA INFILE block, such as the processing seen in this answer link here. The functions, conversions possible are endless.
Maybe 6 of one, half dozen of another. But nice for simpler routines too, and I would suggest a fast data ingest.
Edit: (based on OP comment below)
create table t62
( c1 int not null,
c2 varchar(10) not null,
c3 int not null,
c4 char(5) not null,
someOther int not null,
-- an addtion 57 columns here
primary key(c1,c2,c3,c4)
);
-- Mimic the LOAD DATA INFILE or PHP loop:
insert t62(c1,c2,c3,c4,someOther) values (1,'t',1,'01742',777);
insert t62(c1,c2,c3,c4,someOther) values (1,'t',1,'01742',777); -- error 1062: Dupe PK
insert t62(c1,c2,c3,c4,someOther) values (1,'t',2,'01742',777); -- happy
So I don't know, how would you suggest code would pick that composite PK for you? If so, how would that made-up-data play with its relationships with other data?
I have data with accents in my database. like image below
when I want to display the data with my controller it gives me this
Here is the code
header('Content-Type: text/html; charset=utf-8');
$icozim=$this->Icozim->find('all');
debug($icozim,0,0);
when I run my function I have this
How can I solve this problem?
sql for my table is
CREATE TABLE `icozims` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`synonymes` MEDIUMTEXT NOT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB;
In your APP/Config/database.php look for the line:
// 'encoding' => 'utf8',
and uncomment it.
There are several things that you have to cover. All of these Encode has to be the same (assigned properly). then you can retrieve symbols or special characters properly.
Encode that you are using for Database
At the same time Encode you are using for Table
Encode of your HTML header
Encode of php code when ever you are retrieving/ printing / saving your data.
Also there are some functions that you can play around which does that for you.
e.g. mb_convert_encoding($value, 'UTF-8', 'HTML-ENTITIES')
If you are using any kind of frameworks you might need to set the Encode in core class
If your data has been saved with this specific symbol then you will need to edit all of them or write a function to convert them to a symbol or character you want to show. I remember I had the same problem with one of my project with wordpress which we removed all manually but after that I find an article that there was a plugin which sort out that problem automatically for you.
I'm using SQLite 3 with PHP.
The SQLite table consists of 2 INTEGER and one FLOAT column. When saving data into this table using PHP floats are not saved correctly (default value is stored instead). The two integer columns are saved. Any ideas what could be wrong? Thank you.
Simplified code that actually works correctly:
$conn = new SQLite3('dbFileName');
$conn->query("CREATE TABLE data (
id INTEGER NOT NULL DEFAULT 0 ,
ts INTEGER NOT NULL DEFAULT 0 ,
value FLOAT NOT NULL DEFAULT 0
);"
);
$conn->query("REPLACE INTO data(id,ts,value) VALUES ('1','1234567890','12.1')");
-> 1|1234567890|0
This is just a suggestion, seeing as I have never used SQLite, but are you sure the numbers should be quoted? That seems somewhat odd to me.
Try:
$conn->query("REPLACE INTO data(id,ts,value) VALUES (1, 1234567890, 12.1)");
Sqlite data types are typeless, so your queries have to work:
all the following queries works for me
REPLACE INTO data(id,ts,value) VALUES ('1','1234567890','12.1');
REPLACE INTO data(id,ts,value) VALUES ('1','1234567890','12,123')
REPLACE INTO data(id,ts,value) VALUES ('1','1234567890','abc');
Check your PHP variable, you might have a bug and you pass a null variable.