i have a very strange phenomena trying to migrate an older php project from iso-8859-1 to utf-8
when typing a german umlaut like "äöüß" into a textfield an submitting it i receive broken ones like "äöüÃ�"
header is set to
<?php header("Content-Type:text/html;charset=UTF-8"); ?>
<meta charset="UTF-8" />
the form is set to
<form accept-charset="utf-8">
and so on - i tried everything i could think of - anybody got an ide or experience with an alike problem?
best regards,
Lupo
i found a solution thats working for my case:
i had to change the accept-charset from UTF-8 back to ISO-8859-15 then the field values look ok again - to be able to process the received data i have to utf8_encode them - seems a bit weird but it works fine now.
Thank you all for your time and Jukka for the help.
best regards,
Lupo
I'm glad you managed to find a solution.
And if you don't want to worry about so many different charset codings or if htmlentities doesn't work for others, here the alternative:
I used mysqli DB connection (and PHPV5) Form post for writing/inserting to MySQl DB.
$Notes = $_POST['Notes']; //can be text input or textarea.
$Notes = str_replace('"', '"', $Notes); //double quotes for mailto: emails.
$von = array("ä","ö","ü","ß","Ä","Ö","Ü"," ","é"); //to correct double whitepaces as well
$zu = array("ä","ö","ü","ß","Ä","Ö","Ü"," ","é");
$Notes = str_replace($von, $zu, $Notes);
echo " Notes:".$Notes."<br>" ;
$Notes = mysqli_real_escape_string($link, $Notes); //for recommended mysqli DB connection.
//$Notes = mysql_real_escape_string($Notes); //for old deprecated mysql DB connection.
// mysqli_real_escape_string Escapes special characters in a string for use in an SQL statement
echo " Notes:".$Notes ; //ready for inserting
//Optinal:
$charset = mysqli_character_set_name($link); //only works for mysqli DB connection
printf ("To check your character set but not necessary %s\n",$charset);
Related
I am having issues inserting double quotes into my mysql database.
I had to change headers to UTF8 since I will be displaying Spanish (ñ, accents).
I run a page (PHP) to insert a message in my database:
header('Content-Type: text/html; charset=utf-8');
$newMessage = $_POST['newChatMessage'];
$currentUser = $_SESSION['mundialUser'];
$currentUserId = $_SESSION['idPlayer'];
$newMessage = $_POST['newChatMessage'];
date_default_timezone_set('America/New_York');
$messageDate = date( "Y-m-d H:i:s", time());
$chatMessageQuery = "INSERT INTO chat (message, user, user_id, date) values (:message, :user, :user_id, :date)";
$con->setAttribute(PDO::MYSQL_ATTR_INIT_COMMAND, "SET NAMES 'utf8'");
$chatMessage = $con->prepare($chatMessageQuery);
$chatMessage->execute(array(':message'=>$newMessage, ':user'=>$currentUser, ':user_id'=>$currentUserId, ':date'=>$messageDate));
When I check in the database, double quotes come up like \" instead of simply ". The Spanish ñ is inserted like this: ñ.
Then to display, my main page has this in the Head:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
And I print the message retrieved from the database using utf8_decode.
The outcome is: the Spanish ñ shows correctly but the double quotes are displayed like \" and not "
What am I missing when inserting into my db or displaying?
double quotes come up like \" instead of simply "
Some of your code is adding these slashes. Or ancient feature of magic quotes is turned off. Either should be disabled.
The Spanish ñ is inserted like this: ñ.
It makes very little sense sending mysql init command when the actual init has been done already.
SET NAMES is not a magic chant that require some special dance to perform, but rather plain and simple SQL query. So, you can run it via query() method. Or, better, set charset in DSN.
You should look the charset of you tables/database. At least in Portuguese it can be a headache
I´m creating a simple register and login system for a school project where you sign up through a form.
This is my code for the sign up process (without the bunch of tests´ that determine if fields were left empty etc.), this is just the query part.
Sorry about the variables containing Danish signs (ÆØÅ), and being in danish, but i have made a couple of tests where it didn´t matter if i used æøå or some other kind of letters.
I simply can´t understand why this little piece of code won´t work:
//I retrieve alot of variables from a form
$Fornavn = mysql_real_escape_string($_POST["Fornavn"]);
$Efternavn = mysql_real_escape_string($_POST["Efternavn"]);
$Koen = mysql_real_escape_string($_POST["Koen"]);
$Etnicitet = mysql_real_escape_string($_POST["Etnicitet"]);
$Brugernavn = mysql_real_escape_string($_POST["Brugernavn"]);
$Password = mysql_real_escape_string($_POST["Password"]);
$Mail = mysql_real_escape_string($_POST["Mail"]);
$Haarfarve = mysql_real_escape_string($_POST["Haarfarve"]);
$Oejenfarve = mysql_real_escape_string($_POST["Oejenfarve"]);
$Vaegt = mysql_real_escape_string($_POST["Vaegt"]);
$Hoejde = mysql_real_escape_string($_POST["Hoejde"]);
//The query
mysqli_query($con, "INSERT INTO bruger (Fornavn, Efternavn, Køn, Etnicitet, Brugernavn, Password, Mail, Hårfarve, Øjenfarve, Vaegt, Højde)
VALUES ('$Fornavn', '$Efternavn', '$Koen', '$Etnicitet', '$Brugernavn', '$Password', '$Mail', '$Haarfarve', '$Oejenfarve', '$Vaegt', '$Hoejde')");
echo $Fornavn;
Edit:
I know it doesn´t work since nothing appears in my mysql database after i run the code.
It just returns zero rows (i use phpmyadmin btw.) And i don´t really get any error messages or anything.
As suggested i have tried to use Mysqli_* for my variables instead:
//I retrieve alot of variables from a form
$Fornavn = mysqli_real_escape_string($con,$_POST["Fornavn"]);
$Efternavn = mysqli_real_escape_string($con,$_POST["Efternavn"]);
$Koen = mysqli_real_escape_string($con,$_POST["Koen"]);
$Etnicitet = mysqli_real_escape_string($con,$_POST["Etnicitet"]);
$Brugernavn = mysqli_real_escape_string($con,$_POST["Brugernavn"]);
$Password = mysqli_real_escape_string($con,$_POST["Password"]);
$Mail = mysqli_real_escape_string($con,$_POST["Mail"]);
$Haarfarve = mysqli_real_escape_string($con,$_POST["Haarfarve"]);
$Oejenfarve = mysqli_real_escape_string($con,$_POST["Oejenfarve"]);
$Vaegt = mysqli_real_escape_string($con,$_POST["Vaegt"]);
$Hoejde = mysqli_real_escape_string($con,$_POST["Hoejde"]);
//The query
mysqli_query($con, "INSERT INTO bruger (Fornavn, Efternavn, Køn, Etnicitet, Brugernavn, Password, Mail, Hårfarve, Øjenfarve, Vaegt, Højde)
VALUES ('$Fornavn', '$Efternavn', '$Koen', '$Etnicitet', '$Brugernavn', '$Password', '$Mail', '$Haarfarve', '$Oejenfarve', '$Vaegt', '$Hoejde')");
echo $Fornavn;
But it doesn´t work.
Everything is stored as "Varchar" in the database with collation "utf8_danish_ci".
Answer to comments:
#John #Jayaram I do not retrieve any error messages, or perhaps i don´t know where they go? Im really new to Mysql and PHP.
#Adunahay Vaegt and Højde are stored as Varchar in the table.
#Gordon The code for the connection is as follows:
//We check the connection
$con = mysqli_connect("localhost","Mads","","meat-market");
//If there is a fail.
if (mysqli_connect_errno())
{
echo "<h1> Det er vores fejl:</h1><br /><h2>Kunne ikke forbinde til Databasen:</h2><br /> " . mysqli_connect_error();
exit;
}
Altough im sure the table exist and the database does since i can see it in Phpmyadmin. The user Mads also has all of the necessary permissions.
#Raphaël I´m not aware if you can use special chars, but i have made a couple of tests before where i tried with special chars and it seemed to work fine.
#dboals I don´t know what you mean (i´m new to php), further explanation would be welcomed.
It's because you're using mysql_* for your POST variables and mysqli_* for your DB connection and in your query using $con as the first variable, which is mysqli_* syntax.
Change:
$Fornavn = mysql_real_escape_string($_POST["Fornavn"]);
to
$Fornavn = mysqli_real_escape_string($con,$_POST["Fornavn"]);
and do the same for the others.
Sidenote: Use all mysqli_* functions exclusively for your entire code --- mysqli_* and mysql_* functions do not mix together.. (which I recommend you use and with prepared statements, or PDO)
An example of preparing and binding for mysqli_ can be found HERE.
I found out what was wrong.
First off i needed to change mysql_* to mysqli_* in my variables.
But after that it still didn´t work.
Then i tried to change the letters "æ ø å" to something that wasn´t special letters in the database.
It didn´t seem to matter in the variable names tho.
So instead:
Æ = AE
Ø = OE
Å = AA
So apparently special characters are no go, eventough it seems to be a bit inconsistent since i have done it with special characters before... I don´t know if it is dependent of the collaction of the database...
But thanks to everyone!
EDIT!:
I´m using the UTF8_Danish_ci collation in my table and all the rows.
But after this i still had the problem that if i entered a special character in the form, it would show up as A| or something like that in the database.
I have fixed this by doing the following for each variable that comes from the form:
$Fornavn =utf8_decode($_POST["Fornavn"]);
$Fornavn = mysqli_real_escape_string($con,$Fornavn);
And add the following parameter to my FORM tag:
accept-charset="UTF-8"
Still haven´t been able to execute mysqli queries where the row names contains Æ Ø or Å.
My Project builds on PHP and connect to MS SQL Server. I am using sqlsrv library. The fields type in MS SQL is nvarchar. When I define the parameters for connection I also put "utf8". It is:
global $cnf;
$cnf = array();
$cnf["mssql_user"] = "xxx";
$cnf["mssql_host"] = "xxx";
$cnf["mssql_pw"] = "xxx";
$cnf["mssql_db"] = "xxx";
$cnf["CharacterSet"] = "**UTF-8**";
When Insert records to database, for Vietnamese content and Chinese content I use:
$city = iconv('UTF-8', 'utf-16le', $post['city']);
$params = array(array($city, null, SQLSRV_PHPTYPE_STRING(SQLSRV_ENC_BINARY)));
$sql= "INSERT INTO tblCityGarden (city) VALUES(?)
$stmt = sqlsrv_query( $this->dbhandle, $sql, $params);
It inserts data OK for Vietnamese and Chinese language (the data stored in database for Vietnamese and Chinese is correct).
However when I load the records back into web, It appears the strange character (?, �).
I try some php as iconv, mb_detect_encoding, mb_convert_encoding and search many results on internet, but It cannot work. How can I display correct data
Please someone who has experiences about this issues
I had this same problem (�), but with single quotes, double qoutes, and "Rights Reserved" characters... Here is what I've found:
The CharacterSet specified seems to "rule all", so what you set this to will determine the encoding for the connection (as it should). I did not have ANY CharacterSet configured on my connection(s). Simply setting this resolved my issue, along with making sure that the values that were inserted into my DB were not double encoded via htmlspecialchars().
header()'s must be set before ANY output (this is really important)
headers cannot be set to something different later in the document
Sometimes trailing spaces before and/or after the closing ?> in your PHP file can cause issues (I don't use this closing tag, but I saw this mentioned a lot while searching)
I am not familiar with iconv(), and I am most certainly not experienced with encoding in general, but I solved my issue just by taking the time to check my headers and ensure they meet the above standards...
Your query parameters also look strange:
$params = array(array($city, null, SQLSRV_PHPTYPE_STRING(SQLSRV_ENC_BINARY)));
I have not seen a multidimensional array passed into that argument... (just a note)
I had a registration form which will insert into a member table, the collation is in utf8_general_ci, and I use SET NAMES utf8 in php script, OK here is the problem, I can't insert a string other than pure alphabet, I try input 'Hélène' in a form field, after the sql query run, I check with my db table, the field is inserted as 'H', the rest of the alphabet cannot be insert whenever the string come with special alphabet such as é, ř on behind.
Can someone please help? thanks.
SOLUTION:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
mysql_query("SET NAMES utf8");
above two line is crucial part to accept input into database and output on a webpage.
Thanks everyone for the advise.
try to insert data after encoding. try mysql_real_escape_string() to encode. then execute insert query.
EDIT:-
This answer was posted one year ago. Now mysql_real_escape_string() for php 5 will work in mysqli::real_escape_string this format. please check here
And if you don't want to worry about so many different charset codings or if htmlentities doesn't work for you, here the alternative:
I used mysqli DB connection (and PHPV5) Form post for writing/inserting to MySQl DB.
$Notes = $_POST['Notes']; //can be text input or textarea.
$charset = mysqli_character_set_name($link); //only works for mysqli DB connection
printf ("To check your character set but not necessary %s\n",$charset);
$Notes = str_replace('"', '"', $Notes); //double quotes for mailto: emails.
$von = array("ä","ö","ü","ß","Ä","Ö","Ü"," ","é"); //to correct double whitepaces as well
$zu = array("ä","ö","ü","ß","Ä","Ö","Ü"," ","é");
$Notes = str_replace($von, $zu, $Notes);
echo " Notes:".$Notes."<br>" ;
$Notes = mysqli_real_escape_string($link, $Notes); //for recommended mysqli DB connection.
//$Notes = mysql_real_escape_string($Notes); //for old deprecated mysql DB connection.
// mysqli_real_escape_string Escapes special characters in a string for use in an SQL statement
echo " Notes:".$Notes ; //ready for inserting
The reason you are not able to do so is because PHP prevents a set of special characters to be sent to database to avoid things like sql injection. You can use the escape string function if you wish to escape but the best way to do so is by using the prepared statements.
$connection = <"Your database connection">;
$userinput = "Special characters Hélène";
$stmt = $con->prepare("insert into `table` (`fieldname`)values(?)");
$stmt->bind_param("s",$input);
$stmt->execute();
I'm generating a XML file with PHP using DomDocument and I need to handle asian characters. I'm pulling data from the MSSQL2008 server using the pdo_mssql driver and I apply utf8_encode() on the XML attribute values. Everything works fine as long as there's no special characters.
The server is MS SQL Server 2008 SP3
The database, table and column collation are all SQL_Latin1_General_CP1_CI_AS
I'm using PHP 5.2.17
Here's my PDO object:
$pdo = new PDO("mssql:host=MyServer,1433;dbname=MyDatabase", user123, password123);
My query is a basic SELECT.
I know storing special characters into SQL_Latin1_General_CP1_CI_AS columns isn't great, but ideally it would be nice to make it work without changing it, because other non-PHP programs already use that column and it works fine. In SQL Server Management Studio I can see the asian characters correctly.
Considering all the details above, how should I process the data?
I found how to solve it, so hopefully this will be helpful to someone.
First, SQL_Latin1_General_CP1_CI_AS is a strange mix of CP-1252 and UTF-8.
The basic characters are CP-1252, so this is why all I had to do was UTF-8 and everything worked. The asian and other UTF-8 characters are encoded on 2 bytes and the php pdo_mssql driver seems to hate varying length characters so it seems to do a CAST to varchar (instead of nvarchar) and then all the 2 byte characters become question marks ('?').
I fixed it by casting it to binary and then I rebuild the text with php:
SELECT CAST(MY_COLUMN AS VARBINARY(MAX)) FROM MY_TABLE;
In php:
//Binary to hexadecimal
$hex = bin2hex($bin);
//And then from hex to string
$str = "";
for ($i=0;$i<strlen($hex) -1;$i+=2)
{
$str .= chr(hexdec($hex[$i].$hex[$i+1]));
}
//And then from UCS-2LE/SQL_Latin1_General_CP1_CI_AS (that's the column format in the DB) to UTF-8
$str = iconv('UCS-2LE', 'UTF-8', $str);
I know this post is old, but the only thing that work for me was
iconv("CP850", "UTF-8//TRANSLIT", $var);
I had the same issues with SQL_Latin1_General_CP1_CI_AI, maybe it work for SQL_Latin1_General_CP1_CI_AS too.
You can try so:
header("Content-Type: text/html; charset=utf-8");
$dbhost = "hostname";
$db = "database";
$query = "SELECT *
FROM Estado
ORDER BY Nome";
$conn = new PDO( "sqlsrv:server=$dbhost ; Database = $db", "", "" );
$stmt = $conn->prepare( $query, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL, PDO::SQLSRV_ATTR_CURSOR_SCROLL_TYPE => PDO::SQLSRV_CURSOR_BUFFERED, PDO::SQLSRV_ENCODING_SYSTEM) );
$stmt->execute();
while ( $row = $stmt->fetch( PDO::FETCH_ASSOC ) )
{
// CP1252 == code page Latin1
print iconv("CP1252", "ISO-8859-1", "$row[Nome] <br>");
}
For me, none of the above was the direct solution--though I did use parts of above solutions. This worked for me with the Vietnamese alphabet. If you come across this post and none of the above work for you, try:
$req = "SELECT CAST(MY_COLUMN as VARBINARY(MAX)) as MY_COLUMN FROM MY_TABLE";
$stmt = $conn->prepare($req);
$stmt->execute();
while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
$str = pack("H*",$row['MY_COLUMN']);
$str = mb_convert_encoding($z, 'HTML-ENTITIES','UCS-2LE');
print_r($str);
}
And a little bonus--I had to json_encode this data and was (duh) getting html code instead of the special characters. to fix just use html_entity_decode() on the strings before sending with json_encode.
No need for crazy stuff. Collation SQL_Latin1_General_CP1_CI_AS character encoding is: Windows-1252
This works perfect for me: $str = mb_convert_encoding($str, 'UTF-8', 'Windows-1252');
By default, PDO uses PDO::SQLSRV_ENCODING_UTF8 for sending/receiving data.
If your current collate is LATIN1, have you tried specifiying PDO::SQLSRV_ENCODING_SYSTEM to let PDO know that you want to use the current system encoding instead of UTF-8 ?
You could even use PDO::SQLSRV_ENCODING_BINARY which returns data in a binary form (no encoding or translation is done when transfering data). This way, you could handle character encoding on your side.
More documentation here: http://ca3.php.net/manual/en/ref.pdo-sqlsrv.php
Thanks #SGr for answer.
I found out a better way for doing that :
SELECT CAST(CAST(MY_COLUMN AS VARBINARY(MAX)) AS VARCHAR(MAX)) as MY_COLUMN FROM MY_TABLE;
and also try with:
SELECT CAST(MY_COLUMN AS VARBINARY(MAX)) as MY_COLUMN FROM MY_TABLE;
And in PHP you should just convert it to UTF-8 :
$string = iconv('UCS-2LE', 'UTF-8', $row['MY_COLUMN']);