I have a MySQL database with all the table fields collation as
latin1_swedish_ci
It has almost 1000 of the records already stored and now I want to convert all these data into
utf8_general_ci
So that I can display any language content. I have already altered the field collations into utf8_general_ci but this does not CONVERT all the old records into utf8_general_ci
one funny thing.
CONVERT TO CHARSET and CONVERT()/CAST() suggested by Anshu will work fine if charset in the table is in right encoding.
If for some reason latin1 column containts utf8 text, CONVERT() and CAST() will not be able to help. I had "messed" my database with that setup so spend bit more time on solving this.
to fix this in addition to character set conversion, there are several exercises required.
"Hard one" is to recreate the database from dump that will be converted via console
"Simple one" is to convert row by row or table by table:
INSERT INTO UTF8_TABLE (UTF8_FIELD)
SELECT convert(cast(convert(LATIN1_FIELD using latin1) as binary) using utf8)
FROM LATIN1_TABLE;
basically, both cases will process string to original symbols and then to right encoding, that won't happen with simple convert(field using encoding) from table; command.
Export your table.
Drop the table.
Open the export file in the editor.
Edit it manually where the table structure is created.
old query:
CREATE TABLE `message` (
`message_id` int(11) NOT NULL,
`message_thread_id` int(11) NOT NULL,
`message_from` int(11) NOT NULL,
`message_to` int(11) NOT NULL,
`message_text` longtext NOT NULL,
`message_time` varchar(50) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
new query: ( suppose you want to change message_text field. )
CREATE TABLE `message` (
`message_id` int(11) NOT NULL,
`message_thread_id` int(11) NOT NULL,
`message_from` int(11) NOT NULL,
`message_to` int(11) NOT NULL,
`message_text` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`message_time` varchar(50) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
save the file and import back to the database.
Related
I have noticed when dealing with some names that are not of normal spelling ie standard alphabet UK/US are getting lost from my inserting of a record to what actually shows up in the database. I have done quiet a bit of reading regarding the Collation type, which is what I thought was causing the issue, but not sure if this is the case or I'm still doing it wrong as my problem is still persisting.
Below is an example of a record I am creating as well as my database structure, and as you can also see the last_name field has "ö", when I lookup the record I actually see the last_name "Körner"
CREATE TABLE `data` (
`id` bigint(20) NOT NULL,
`profile_id` int(11) NOT NULL,
`first_name` varchar(100) NOT NULL,
`last_name` varchar(100) NOT NULL,
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
ALTER TABLE `data`
ADD PRIMARY KEY (`id`),
ADD UNIQUE KEY `profile_id` (`profile_id`);
ALTER TABLE `data`
MODIFY `id` bigint(20) NOT NULL AUTO_INCREMENT;
INSERT IGNORE INTO data (profile_id, first_name, last_name) VALUES (1, 'Brent', 'Körner');
The field collation on the last_name is set to 'utf8_general_ci' which that I understand or should I say thought would sort this issue out.
This seems to be something I am doing wrong / missing with PHP, as when I execute the INSERT query within PhpMyAdmin it saves fine.
it seems the issue was down to PHP in the end, and i wasn't setting the charset.
For mysql
mysql_set_charset('utf8');
For mysqli
mysqli_set_charset('utf8');
ref https://akrabat.com/utf8-php-and-mysql/
I am trying to json_encode data from the database but the file does not display more than 255 characters.
I have a table column with 812 characters and I want to return a json.
The column in which that data is saved is of type TEXT and it allows to save the 812 characters. But when I run json_encode from the table, it doesnot show anything and when I reduce the content, it returns the json and yet I need all the content.
This is what i get after i run: SHOW CREATE TABLE
CREATE TABLE news` (
id int(10) unsigned NOT NULL AUTO_INCREMENT,
headline varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
details text COLLATE utf8mb4_unicode_ci NOT NULL,
file varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
created_at timestamp NULL DEFAULT NULL,
updated_at timestamp NULL DEFAULT NULL, PRIMARY KEY (id)
) ENGINE=MyISAM AUTO_INCREMENT=8 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci`
I have a started to learn php & mysql and currently i`m thinking about possibility how to store and retrieve tags for some products.
I would like to assigne tags for specified products and after that display using while method or similiar
At the first step i was thinking to store it in one field, but i have not idea how to retrive it every words as unical item.
CREATE TABLE `products` (
`id` int(11) NOT NULL,
`brand_id` int(11) NOT NULL,
`category_id` int(11) NOT NULL,
`product_name` text COLLATE utf8_unicode_ci NOT NULL,
`product_description` text COLLATE utf8_unicode_ci NOT NULL,
`product_keywords` text COLLATE utf8_unicode_ci NOT NULL,
`product_image` text COLLATE utf8_unicode_ci NOT NULL,
`product_specification` text COLLATE utf8_unicode_ci NOT NULL,
`product_spec_link` text COLLATE utf8_unicode_ci NOT NULL,
`product_webstie` text COLLATE utf8_unicode_ci NOT NULL,
`status` text COLLATE utf8_unicode_ci NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Any ideas ?
I read here that the best way is to create separate table for this purpose and assign them to the specific product ID, but currently i cant imagine it, how it will work when i want to add new tag for it.
After many of research time i think i found the best way to realize my idea.
First we have to do separate table for tags and create relationships with other tables using many to many or one to many option, depends of way which i want to use.
In case of multiple products which will have multiple tags which can be repeated then i have to use relationships many to many.
Please correct me if i made mistake.
I have one old database which I must use. The problem is that the old data(mostly text) is stored in 1252(latin1_general_ci) and is showed like ?????? on the page. Then I've converted whole database and the table to UTF-8 collation like this:
ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
But the problem whit old records remains. I know that the queries above are just change the fields collation. My question is there an way to show those ????? records properly on the web page now?
1) Create dump
mysqldump --default-character-set=latin1 --skip-set-charset mydatabase mytable > ./mytable.sql
2) In mytable.sql replace latin1 in utf8
CREATE TABLE `test` (
`id` int(10) unsigned NOT NULL,
`name` char(255) NOT NULL default '',
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
3) Import DB
mysql --user=login -p --database=mydatabase < ./mytable.sql
mysqldump — A Database Backup Program
I've recently started using laravel for a project I'm working on, and I'm currently having problems displaying data from my database in the correct character encoding.
My current system consists of a separate script responsible for populating the database with data, while the laravel project is reponsible for displaying the data. The view that is used, is set to display all text as utf-8, which works as I've successfully printed special characters in the view. Text from the database is not printed as utf8, and will not print special characters the right way. I've tried using both eloquent models and DB::select(), but they both show the same poor result.
charset in database.php is set to utf8 while collation is set to utf8_unicode_ci.
The database table:
CREATE TABLE `RssFeedItem` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`feedId` smallint(5) unsigned NOT NULL,
`title` varchar(250) COLLATE utf8_unicode_ci NOT NULL,
`url` varchar(250) COLLATE utf8_unicode_ci NOT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`text` mediumtext COLLATE utf8_unicode_ci,
`textSha1` varchar(250) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `url` (`url`),
KEY `feedId` (`feedId`),
CONSTRAINT `RssFeedItem_ibfk_1` FOREIGN KEY (`feedId`) REFERENCES `RssFeed` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6370 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
I've also set up a test page in order to see if the problem could be my database setup, but the test page prints everything just fine. The test page uses PDO to select all data, and prints it on a simple html page.
Does anyone know what the problem might be? I've tried searching around with no luck besides this link, but I haven't found anything that might help me.
I did eventually end up solving this myself. The problem was caused by the separate script responsible for populating my database with data. This was solved by running a query with SET NAMES utf8 before inserting data to the database. The original data was pulled out, and then sent back in after running said query.
The reason for it working outside laravel, was simply because the said query wasn't executed on my test page. If i ran the query before retrieving the data, it came out with the wrong encoding because the query stated that the data was encoded as utf8, when it really wasn't.