I am using below mysql query to check which records vary from the trimmed value
SELECT id, BINARY(username) as binary_username, TRIM(username) as trim_username FROM table.
Above query returns binary value and trimmed value as shown below.
Result of mysql query:
Highlighted values in above image show that binary value vary from trimmed value.
I tried below 2 things:
calculating length of both binary and trimmed column but it is same LENGTH(binary_username) != LENGTH(trim_username).
equating them directly binary_username != trim_username.
but both of them are returning empty records.
How can I fetch these highlighted entries using mysql?
Edit 1: I have added HEX value in the query result
SELECT id, BINARY(username) as binary_username, TRIM(username) as trim_username, HEX(username) as hex_username FROM table
Thanks in advance...
To avoid storing, trimming, etc, the trailing zeros, use VARBINARY instead of BINARY. Why, pray tell, are you using BINARY for text strings??
Please do SELECT HEX(username) FROM ... so we can further diagnose the problem. That screenshot is suspect -- we don't know what the client did to "fix" the output.
Well, none of those are encoded in UTF-8, nor anything else that I recognize. The 'bad' characters (02, 04, 0c 17), are all "control codes" in virtually all encodings. ("Unicode" is not an encoding method, so it is not relevant.)
Would you like a REGEXP that tests for control codes?
In PHP, json_encode has an option for JSON_UNESCAPED_UNICODE. See https://www.php.net/manual/en/function.json-encode.php
But that generates \u1234 type text.
When storing binary data into MySQL, use the binding or escaping mechanism in PDO or mysqli.
Related
I have copied this table from Wikipedia into a PostgreSQL database. The column Cultivated land (km2) became a column of type real. Then I use the PHP command
echo rtrim(rtrim(sprintf('%.10F',$v),'0'),'.');
to display the numbers ($v) in a table (both integers and float), but some values lose precision. For instance, the value from United States, 1669302, becomes 1669300, what is strange, since I expected 10 decimal digits of precision. I thought I have lost the precision when saving into a real column, but converting the column to double precision makes the difference (02) appear again, so it was there somewhere.
I don't think I need double precision, so how can I display the real value correctly? Keep in mind that some columns have decimal places, while some others are bigint, and they also should be displayed correctly.
The problem seems to originate from the way PHP returns results. The values are not returned as the corresponding data type, but rather formatted as a string using PostgreSQL default formatting. This formatting, is different for real and double precision types hence you are seeing different results when you convert the column types of your table. The reason you are seeing this specific result is that PostgreSQL guarantees 6 decimal places for real types and 15 decimal places for double precision.
Setting extra_float_digits
The manual states
Note: The extra_float_digits setting controls the number of extra significant digits included when a floating point value is converted to text for output. With the default value of 0, the output is the same on every platform supported by PostgreSQL. Increasing it will produce output that more accurately represents the stored value, but may be unportable.
Therefore, a simple solution to your problem is to increase extra_float_digits before issuing your SELECT-query:
pg_query($connection, "set extra_float_digits = 3");
Alternatively, you can also specify this change when connecting to your database by adding options to your connection string as follows:
$connection = pg_connect("host=localhost port=5432 dbname=test user=php password=pass connect_timeout=5 options='-c extra_float_digits=3'");
Another option would be to set this flag in the postgresql.conf configuration file of the PostgreSQL server if you have access to the server and want to change the option globally.
Casting the values
A different solution would be to have PostgreSQL return a different string to the PHP backend. This can be achieved by casting your columns to types with different default formatting which avoids cutting off some of the digits. In your case you could either cast to integer or double precision, i.e. instead of using
select cultivated_land from table
you could use
select cultivated_land::integer from table
or
select cultivated_land::double precision from table
Changing data types
Looking at the data you specified, I noticed that all numerical values except those columns specifying percentages contain integers, hence the usage of the integer data type is more suitable in this case. It can fit all the integer values of this table (the maximum being 149,000,000, therefore bigint is not required), requires the same storage size as real (4 bytes) and implies the default formatting of integers that you are looking for.
Update: Background on PostgreSQL-PHP interface and floating point representation
As mentioned above the way the PostgreSQL-PHP interface works is that all values sent from PostgreSQL to PHP are formatted as a string in some type-dependent way. Neither any of the pg_fetch_* functions nor pg_copy_to will provide raw values and all of these functions convert the values to strings in the same manner. As far as I am aware the current PHP interface will not provide you with anything different from a string (which, in my opinion, is not the best interface design).
The reason 18.22 is returned as 18.2199993 can be found in how PostgreSQL converts float4 to strings. You can check the code of how PostgreSQL is internally using float4out and find this relevant line that does the string-conversion:
snprintf(ascii, MAXFLOATWIDTH + 1, "%.*g", ndig, num);
num is the float4-number to be printed as a string. Note however that C will promote the float-variable to a double-variable when calling snprintf. This conversion to double precision results in the value 18.219999313354492 which is why you end up seeing 18.2199993 (you can check this here and will also find some details on floating point number representation on this site).
The takeaway message is that all your float4 values will be converted using this function and the only parameter you can influence is ndig by varying extra_float_digits, however no single value for this variable will suffice all your needs in representing the values as you want them. So as long as you keep using float4 as your data type and use the current PHP-interface to obtain the data you will run into these problems.
I therefore still recommend choosing different data types for your columns. If you think you have a requirement for decimal numbers you might want to investigate decimal data types where you can specify precision and scale as required for your application. If you would like to stick with floating point numbers I suggest rounding the values in PHP before displaying them to the user.
if I want to change a column data type from REAL to something else, like NUMERIC, what data type and precision I should use? i.e. NUMERIC(?,?)
please help~~ thank you.
this is what data I have in current column with data type REAL.
0.5
0.0086
2E-07
NULL
4
0.5
0.00375
1E-07
I need get exactly same value as usual when doing query,
but since php driver have some problem with the REAL data type,
I cannot use REAL anymore.
I tried with a new column in float,
I use update sql clause to copy value to the new column
and the new column shown exactly as I queried in PHP... :-(
and it's annoying, because if I insert 16.7 into the REAL column,
I got 16.7 in SMS, 16.700000762939 in PHP query.
that's why I trying to find a alternative data type.
Thank you all in advanced.
If you want the exact values then stay away from float datatype as it is an Imaginary or Approximate representation of the number. Therefore using Float datatype you can get slightly different results then expected.
Use one of the NUMERIC or DECIMAL datatype. Which are only datatype allows you to store fractional values with exact numeric representation (Debateable but somewhat exact). :)
Read here MSDN Documentation to learn more about different Datatypes in Sql Server.
I am in need of storing a score in a mysql field but this score could be a time,integer or float. My first way of doing this was to identify the score type and then enter it into one of three fields but if the need arises to add a new score type I dont want to have to continually add a field to the database. I remember somewhere down the line someone told me that if you store somethign as a varchar then is wont be able to be read as an integer or float or even date. My question is, can I store all three of those as one specific type but read it any way I need when taking it from the database and using it in my php code?
In my opinion you could model the field as FLOAT except if you absolutely need to know about the type of variable stored. Time can be converted to an integer value by converting to timestamp. Integers are a subset of the real (floating point) numbers set actually so I guess that way you have everything covered. Floating point arithmetic can cause some issues with precision and equality testing though so be careful!
You can use CAST and CONVERT functions to convert the string datatype into another MySQL datatype such as INT, FLOAT, DECIMAL, DATE, DATETIME etc.
There are a few issues. How do you know what datatype is stored in a row? MySQL does have RegExp support but I do not recommend using it in your WHERE clauses. Store the datatype in another column.
Also, using cast functions in the where clause of your query will make them run slow. If you need to search upon/sort by the data you should use proper datatypes. Perhaps you should add one column for each datatype; and for each row, populate only the corresponding column.
mysql will happily convert text to numbers of the appropriate type if you perform a mathematical operation on it. However, it will also convert non-numeric text to zero and perform the same operation on it, so you need to check that you're only pulling fields of the appropriate type beforehand.
This forum post shows how to add a regular expression condition to your query to ensure that you're only pulling fields with numeric data in them. However, I think it's probably wiser to use a separate column to indicate what type of score each record is, and use that to retrieve the appropriate ones.
I don't know how to convert text to a date (putting it through date() doesn't work). However, note that the mysql date format (2012-05-08 11:20:23) has the date elements in order of descending significance. If you just want to find the highest / lowest date, or sort by date, treating them as strings will work fine.
Can we declare a variable with fixed length in PHP?
I'm not asking about trimming or by putting condition do substring.
Can we declare variable just like database char(10).
The reason I'm asking am doing an export process, PHP export data to DB.
In DB I have a field with size 100, and I'm passing a field with length 25, using PHP.
When I look in DB, it's showing some extra space for that field.
Maybe it's your database that is the problem.
The CHAR datatype will always fill up the remaining unused characters when storing data. If you have CHAR(3) and pass 'hi', it will store it as 'hi '. This is true for a lot of relational database engines (MySQL, Postgres, SQLite, etc.).
This is why some database engines also have the VARCHAR datatype (which is variable, like the name says). This one doesn't pad the content with spaces if the data stored in isn't long enough.
In most cases, you are looking for the VARCHAR datatype. CHAR is mostly useful when you store codes, etc. that always have the same length (e.g.: a CHAR(3) field for storing codes like ADD, DEL, CHG, FIX, etc.).
No, a string in PHP is always variable length. You could trim the string to see if extra space is still passed to your DB.
Nope. PHP has no provision to limit string size.
You could simulate something in an object using setter and getter variables, though, throwing an error (or cutting off the data) if the incoming value is larger than allowed.
No, but I really don't think you're having a problem with php. I think you should check your DB2 configuration, perhaps it automatically completes strings with spaces... How much spaces are added? Are they added before? After?
As others have said: No.
I don't understand how it would help anyway. I'm not familiar with DB2 but it sounds like if you have extra spaces, they are either coming in the variable (and thus it should be trimmed) or DB2 does space padding to make the value have 100 characters. If your input is only 25 characters long then if it is doing space padding, it seems it would do it anyway.
If you want to store variable length strings in DB2 then go with VARCHAR, if you always want the same length for each string in the column, define the exact length using CHAR (for postal codes, for instance).
Details on character strings is available here: http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/topic/com.ibm.db2.luw.sql.ref.doc/doc/r0008470.html with a good summary:
Fixed-length character string (CHAR)
All values in a fixed-length string column have the same length, which is determined by the length attribute of the column. The length attribute must be between 1 and 254, inclusive.
Varying-length character strings
There are two types of varying-length character strings:
A VARCHAR value can be up to 32,672 bytes long.
A CLOB (character large object) value can be up to 2 gigabytes minus 1 byte (2,147,483,647 bytes) long.
Of course it then gets more detailed, depending on what sort of encoding you're using, etc... ( like UTF-16 or UTF-32 )
I try to store the PHP floating point value 63.59072952118762 into a double precision column in postgres. Postgres stores the value as 63.59073. Does anyone know why? 8 byte should be more than enough for that value. I've tried with the data type numeric, which works when specifying the precision, but that shouldn't really be necessary.
Update: The same problem is present when trying to store 63.5907295, so the suggestion that something happens with the double before it's getting stored seems realistic.
Update II (partly solved): The line where I assign the double parameter looks like this:
$stmt->bindParam(4, $this->latitude);
The thing I didn't know is that PDO defaults its param type to string. I changed it to PDO::PARAM INT in lack of a better alternative (PARAM DOUBLE was not an option), and got 10 digits precision in the double stored in postgres (some progress, at least). I've checked that the numeric type works well, so it seems that numeric is the way to go when using PDO and doubles that has to have a precision of more than 10 decimals.
Anyways, as someone has mentioned, I don't know if it's a must for me to have this kind of precision, but I think the problem in itself deserved to be investigated.
How do you determine what PostgreSQL is storing?
How do you send the data to PostgreSQL?
How do you get the data back again?
How do you display it?
What type is the column in the database?
There are many, many places on the path between PHP and PostgreSQL where there could be confusion about how to represent the data.
It is important to explain how data is inserted into the DBMS. Using a literal value in the INSERT statement leads to a different set of problems from using bound parameters. If you wrote the value out in the SQL:
INSERT INTO SomeTable(SomeColumn) VALUES(63.xxxxxxxxx);
and the data was truncated, you'd have a problem down in PostgreSQL. If you bind the variable, you have to be sure to understand what PHP and the PDO PostgresSQL modules do with the value - is it sent as a double, or as a string, and which code deals with the conversion, and so on.
You run into analogous issues with Perl + DBI + DBD::YourDBMS (DBD::Pg in your case).
Consider using the DECIMAL/NUMERIC type if you need that much precision
PostgreSQL accepts float(1) to float(24) as selecting the real type, while float(25) to float(53) select double precision.
On most platforms PG, the real type has a range of at least 1E-37 to 1E+37 with a precision of at least 6 decimal digits. The double precision type typically has a range of around 1E-307 to 1E+308 with a precision of at least 15 digits (REF)
Which one do you use?