I have a problem with MySql
I have a table with parsed informations from websites. A strange string interpretation appear:
the query
select id, address from pagesjaunes_test where address = substr(address,1,length(address)-1)
return a set of values instead of none
at beginning I executed functions as:
address = replace(address, '\n', '')
address = replace(address, '\t', '')
address = replace(address, '\r', '')
address = replace(address, '\r\n', '')
address = trim(address)
but the problem still persist.
Values of field 'address' have some french chars , but the query returned also values that contains only alfanumeric english chars.
Another test: I tried to check the length of strings and ... the strlen() from PHP and LENGTH() from MYSQL display different results! Somewhere difference is by 2 chars, somewhere by 1 character without a specific "rule".
Visual I can't see any space or tabs or something else.
After I modified an address manualy(I deleted all string and I wrote it again), the problem is solved, but I have ~ 6000 values, so this is not a solution :)
What can be the problem?
I suppose that strings can have something as an "empty char", but how to detect and remove it?
Thanks
P.S.
the problem is not just length. I need to join this table with other one and using a condition that check if values from fields 'address' are equals. Even if the fields have the same collation and tables have the same collation, query returns that no addresses match
E.g.
For query:
SELECT p.address,char_length(p.address) , r.address, char_length(r.address)
FROM `pagesjaunes_test` p
LEFT JOIN restaurants r on p.name=r.name
WHERE
p.postal_code=r.postal_code
and p.address!=r.address
and p.phone=''
and p.cuisines=''
LIMIT 10
So: p.address!=r.address
The result is:
+-------------------------------------+------------------------+--------------------------+------------------------+
| address | char_length(p.address) | address | char_length(r.address) |
+-------------------------------------+------------------------+--------------------------+------------------------+
| Dupin Marc13 quai Grands Augustins | 34 | 13 quai Grands Augustins | 24 |
| 39 r Montpensier | 16 | 39 r Montpensier | 16 |
| 8 r Lord Byron | 14 | 3 r Balzac | 10 |
| 162 r Vaugirard | 15 | 162 r Vaugirard | 15 |
| 32 r Goutte d'Or | 16 | 32 r Goutte d'Or | 16 |
| 2 r Casimir Périer | 18 | 2 r Casimir Périer | 18 |
| 20 r Saussier Leroy | 19 | 20 r Saussier Leroy | 19 |
| Senes Douglas22 r Greneta | 25 | 22 r Greneta | 12 |
| Ngov Ly Mey44 r Tolbiac | 23 | 44 r Tolbiac | 12 |
| 33 r N-D de Nazareth | 20 | 33 r N-D de Nazareth | 20 |
+-------------------------------------+------------------------+--------------------------+------------------------+
As you see, "162 r Vaugirard", "20 r Saussier Leroy" contains only ASCII chars, have the same length but aren't equals!
Maybe have a look at the encoding of the mysql text fields - UTF8 encodes most of its characters with 2 bytes - only a small subset of UTF8 (ASCII characters for example) get encoded with one byte.
MySQL knows UTF8 and counts right.
PHP text functions aren't UTF8 aware and count the bytes itself.
So if PHP counts more than MYSQL, this is probably the cause and you could have a look at utf8decode.
br from Salzburg!
The official documentation says:
Returns the length of the string str, measured in bytes. A multi-byte character counts as multiple bytes. This means that for a string containing five two-byte characters, LENGTH() returns 10, whereas CHAR_LENGTH() returns 5.
So, use CHAR_LENGTH instead :)
select id, address from pagesjaunes_test
where address = substr(address, 1, char_length(address) - 1)
Finally, I found the problem. After changed collation to ascii_general_ci all non-ascii chars was transformed to "?". Some spaces also was replaced with "?". After check initial values, function ORD() from MySQL returned 160 (instead of 32) for these spaces. So,
UPDATE pagesjaunes_test SET address = TRIM(REPLACE(REPLACE(address, CHAR(160), ' '), ' ',' ')
resolved my question.
Related
I want to update a field on a really huge (1m rows) table. I want to update it from:
+-----------------------------------------------------------+
| ref |
+-----------------------------------------------------------+
| 0001___000000000003616655___IVANTI UK___TEMPLATE MATERIAL |
+-----------------------------------------------------------+
to:
+-------------------------------+
| ref |
+-------------------------------+
| IVANTI UK___TEMPLATE MATERIAL |
+-------------------------------+
So basically its just changing the ref (which is not fixed length) from sid___sku___mfr___pnum to mfr___pnum format.
In PHP I'd do it like so (pseduo code):
list($p['sid'], $p['sku'], $p['mfr'], $p['pnum']) = explode('___', $row['ref']);
$row['ref'] = $p['mfr'] . '___' . $p['pnum'];
Wondering if its possible to do it directly with MySQL with a performant query?
select SUBSTRING_INDEX(ref,'___',-2) from test
0001___000000000003616655___IVANTI UK___TEMPLATE MATERIAL
=>
IVANTI UK___TEMPLATE MATERIAL
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring-index
SUBSTRING_INDEX(str,delim,count)
Returns the substring from string str before count occurrences of the
delimiter delim. If count is positive, everything to the left of the
final delimiter (counting from the left) is returned. If count is
negative, everything to the right of the final delimiter (counting
from the right) is returned. SUBSTRING_INDEX() performs a
case-sensitive match when searching for delim.
Ok, I have such script
#!/bin/bash
keyOrPass=$1
intercom=$2
flat=$3
number=$4
mysql -ulogin -ppass db_name -e "select cli.codeGuestEmail, cli.codePrivateEmail, cliKey.rf_id, cliKey.emailNotification from mbus_clients as cli join mbusClientKeys as cliKey on cliKey.id_client=cli.id WHERE cli.flat=${flat} and cli.domophone=${intercom};";
php -q sendNotifications.php
It works fine, but I should pass some result fields from select into php arguments. Any ideas how to do it?
OUTPUT:
+----------------+------------------+------------+-------------------+
| codeGuestEmail | codePrivateEmail | rf_id | emailNotification |
+----------------+------------------+------------+-------------------+
| 1 | 0 | 2774490192 | 0 |
| 1 | 0 | 2774490193 | 0 |
| 1 | 0 | 2774490194 | 0 |
| 1 | 0 | 2774490195 | 1 |
+----------------+------------------+------------+-------------------+
mysql is capable of generating output formatted differently. With the -B or --batch option, mysql produces the output with TAB as the column separator. The special characters in the fields are escaped (e.g. TAB is output as "\t") so you can use cut to extract fields.
Many times in cases like this it's helpful to use the -N or --skip-column-names option as well, in order to remove column names from the output.
For example, this is my table, which is called example:
--------------------------
| id | en_word | zh_word |
--------------------------
| 1 | Internet| 互联网 |
--------------------------
| 2 | Hello | 你好 |
--------------------------
and so on...
And I tried using this SQL Query:
SELECT * FROM `example` WHERE LENGTH(`zh_word`) = 3
For some reason, it wouldn't give me three, but would give me a lot of single letter characters.
Why is this? Can this be fixed? I tried this out in PhpMyAdmin.
But when I did it with JavaScript:
"互联网".length == 3; // true
And it seems to work fine. So how come it doesn't work?
you should use CHAR_LENGTH instead of LENGTH
LENGTH() returns the length of the string measured in bytes.
CHAR_LENGTH() returns the length of the string measured in characters.
LENGTH returns length in bytes (and chinese is multibyte)
Use CHAR_LENGTH to get length in characters
http://dev.mysql.com/doc/refman/5.0/en/string-functions.html#function_char-length
http://dev.mysql.com/doc/refman/5.0/en/string-functions.html#function_length
I am trying to run a script through command prompt in PHP and trying to show the result in tabular form. But due to different character length of words I am not able to show the result properly align.
I want result like this
Book ISBN Department
Operating System 101 CS
C 102 CS
java 103 CS
Can anyone please help me to get this output like this in php on console.
Thanks in advance
If you don't want (or not allowed for some reason) to use libraries, you can use standard php printf / sprintf functions.
The problem with them that if you have values with variable and non-limited width, then you will have to decide if long values will be truncated or break table's layout.
First case:
// fixed width
$mask = "|%5.5s |%-30.30s | x |\n";
printf($mask, 'Num', 'Title');
printf($mask, '1', 'A value that fits the cell');
printf($mask, '2', 'A too long value the end of which will be cut off');
The output is
| Num |Title | x |
| 1 |A value that fits the cell | x |
| 2 |A too long value the end of wh | x |
Second case:
// only min-width of cells is set
$mask = "|%5s |%-30s | x |\n";
printf($mask, 'Num', 'Title');
printf($mask, '1', 'A value that fits the cell');
printf($mask, '2', 'A too long value that will brake the table');
And here we get
| Num |Title | x |
| 1 |A value that fits the cell | x |
| 2 |A too long value that will brake the table | x |
If neither of that satisfies your needs and you really need a table with flowing width columns, than you have to calculate maximum width of values in each column. But that is how PEAR::Console_Table exactly works.
You can use PEAR::Console_Table:
Console_Table helps you to display tabular data on a
terminal/shell/console.
Example:
require_once 'Console/Table.php';
$tbl = new Console_Table();
$tbl->setHeaders(array('Language', 'Year'));
$tbl->addRow(array('PHP', 1994));
$tbl->addRow(array('C', 1970));
$tbl->addRow(array('C++', 1983));
echo $tbl->getTable();
Output:
+----------+------+
| Language | Year |
+----------+------+
| PHP | 1994 |
| C | 1970 |
| C++ | 1983 |
+----------+------+
Your best option is to use the Pear Package Console_Table ( http://pear.php.net/package/Console_Table/ ).
To use - on a console you need to install the pear package, running:
pear install Console_Table
this should download the package and install. You can then use a sample script such as:
require_once 'Console/Table.php';
$tbl = new Console_Table();
$tbl->setHeaders(
array('Language', 'Year')
);
$tbl->addRow(array('PHP', 1994));
$tbl->addRow(array('C', 1970));
$tbl->addRow(array('C++', 1983));
echo $tbl->getTable();
You could try the recent simple PHP library ConsoleTable if you don't want to use the standard PHP functions printf/sprintf or the pear package PEAR::Console_Table.
Example:
require_once 'ConsoleTable.php';
$table = new LucidFrame\Console\ConsoleTable();
$table
->addHeader('Language')
->addHeader('Year')
->addRow()
->addColumn('PHP')
->addColumn(1994)
->addRow()
->addColumn('C++')
->addColumn(1983)
->addRow()
->addColumn('C')
->addColumn(1970)
->display()
;
Output:
+----------+------+
| Language | Year |
+----------+------+
| PHP | 1994 |
| C++ | 1983 |
| C | 1970 |
+----------+------+
See more example usages at its github page.
Too old, but i went trough the same now and used str_pad, just set the lenght as the size of your column and thats it
regards.
The CLIFramework table generator helps you get the job done very easily and it supports text alignment, text color, background color, text wrapping, text overflow handling.. etc
Here is the tutorial: https://github.com/c9s/CLIFramework/wiki/Using-Table-Component
Sample code: https://github.com/c9s/CLIFramework/blob/master/example/table.php
use CLIFramework\Component\Table\Table;
$table = new Table;
$table->setHeaders([ 'Published Date', 'Title', 'Description' ]);
$table->addRow(array(
"September 16, 2014",
"Title",
"Description",
29.5
));
$table->addRow(array(
"November 4, 2014",
"Hooked: How to Build Habit-Forming Products",
["Why do some products capture widespread attention whil..."],
99,
));
echo $table->render();
Just in case someone wants to do that in PHP I posted a gist on Github
https://gist.github.com/redestructa/2a7691e7f3ae69ec5161220c99e2d1b3
simply call:
$output = $tablePrinter->printLinesIntoArray($items, ['title', 'chilProp2']);
you may need to adapt the code if you are using a php version older than 7.2
after that call echo or writeLine depending on your environment.
Answers provided have all been great, I mentioned in the comments of Alnitak's answer that I would need to go take a look at my CSV Generation script because for whatever reason it wasn't outputting UTF-8.
As was correctly pointed out, it WAS outputting UTF-8 - the problem existed with Ye Olde Microsoft Excel which wasn't picking up the encoding the way I would have liked.
My existing CSV generation looked something like:
// Create file and exit;
$filename = $file."_".date("Y-m-d_H-i",time());
header("Content-type: application/vnd.ms-excel");
header("Content-disposition: csv" . date("Y-m-d") . ".csv");
header( "Content-disposition: filename=".$filename.".csv");
echo $csv_output;
It now looks like:
// Create file and exit;
$filename = $file."_".date("Y-m-d_H-i",time());
header("Content-type: text/csv; charset=ISO-8859-1");
header("Content-disposition: csv" . date("Y-m-d") . ".csv");
header("Content-disposition: filename=".$filename.".csv");
echo iconv('UTF-8', 'ISO-8859-1', $csv_output);
-------------------------------------------------------
ORIGINAL QUESTION
Hi,
I've got a form which collects data, form works ok but I've just noticed that if someone types or uses a '£' symbol, the MySQL DB ends up with '£'.
Not really sure where or how to stop this from happening, code and DB information to follow:
MySQL details
mysql> SHOW COLUMNS FROM fraud_report;
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | mediumint(9) | | PRI | NULL | auto_increment |
| crm_number | varchar(32) | YES | | NULL | |
| datacash_ref | varchar(32) | YES | | NULL | |
| amount | varchar(32) | YES | | NULL | |
| sales_date | varchar(32) | YES | | NULL | |
| domain | varchar(32) | YES | | NULL | |
| date_added | datetime | YES | | NULL | |
| agent_added | varchar(32) | YES | | NULL | |
+--------------+--------------+------+-----+---------+----------------+
8 rows in set (0.03 sec)
PHP Function
function processFraudForm($crm_number, $datacash_ref, $amount, $sales_date, $domain, $agent_added) {
// Insert Data to DB
$sql = "INSERT INTO fraud_report (id, crm_number, datacash_ref, amount, sales_date, domain, date_added, agent_added) VALUES (NULL, '$crm_number', '$datacash_ref', '$amount', '$sales_date', '$domain', NOW(), '$agent_added')";
$result = mysql_query($sql) or die (mysql_error());
if ($result) {
$outcome = "<div id=\"success\">Emails sent and database updated.</div>";
} else {
$outcome = "<div id=\"error\">Something went wrong!</div>";
}
return $outcome;
}
Example DB Entry
+----+------------+--------------+---------+------------+--------------------+---------------------+------------------+
| id | crm_number | datacash_ref | amount | sales_date | domain | date_added | agent_added |
+----+------------+--------------+---------+------------+--------------------+---------------------+------------------+
| 13 | 100xxxxxxx | 10000000 | £10.93 | 18/12/08 | blargh.com | 2008-12-22 10:53:53 | agent.name |
What you're seeing is UTF-8 encoding - it's a way of storing Unicode characters in a relatively compact format.
The pound symbol has value 0x00a3 in Unicode, but when it's written in UTF-8 that becomes 0xc2 0xa3 and that's what's stored in the database. It seems that your database table is already set to use UTF-8 encoding. This is a good thing!
If you pull the value back out from the database and display it on a UTF-8 compatible terminal (or on a web page that's declared as being UTF-8 encoded) it will look like a normal pound sign again.
£ is 0xC2 0xA3 which is the UTF-8 encoding for £ symbol - so you're storing it as UTF-8, but presumably viewing it as Latin-1 or something other than UTF-8
It's useful to know how to spot and decode UTF-8 by hand - check the wikipedia page for info on how the encoding works:
0xC2A3 = 110 00010 10 100011
The bold parts are the actual
"payload", which gives 10100011,
which is 0xA3, the pound symbol.
In PHP, another small scale solution is to do a string conversion on the returned utf8 string:
print iconv('UTF-8', 'ASCII//TRANSLIT', "Mystring â"); //"Mystring "
Or in other platforms fire a system call to the inconv command (linux / osx)
http://php.net/manual/en/function.iconv.php#83238
You need to serve your HTML in utf-8 encoding (actually everyone needs to do this I think!)
Header like:
Content-Type: text/html; charset=UTF-8
Or the equivalent. Double check the details though. Should always be declaring the charset as a browser can default to anything it likes.
To remove a  use:
$column = str_replace("\xc2\xa0", '', $column);
Credits among others: How to remove all occurrences of c2a0 in a string with PHP?
Thanks a lot. I had been suspecting mysql for being currupting the pound symbol. Now all i need to do is wherever the csv record is generated, just use wrap them incov funciton. Though this is a good job, I am happy, at least someone showed exactly what to do. I sincerly appreciate dislaying the previous and the new 'header' values. It was a great help to me.
-mark
If you save line "The £50,000 Development Challenge" in two different data type column i.e. "varchar" & "text" field.
Before i save i have replaced the symbol with html equi value using following function.
str_replace("£", "£", $title);
You will find that value stored in text fields is £ where as in varchar its "£".