Automatically clip strings to the proper length on insert - php

I have a table with various VARCHAR fields in MySQL. I would like to insert some user data from a form via PHP. Obviously if I know the field lengths in PHP, I can limit the data length there with substr(). But that sort of violates DRY (field length stored in MySQL and as a constant in my PHP script). Is there a way for me to configure an INSERT so it automatically chops off excessively-long strings, rather than fails?
edit: it's failing (or at least causing an exception) in PHP/PDO, when I have excessively long strings. Not sure what I have to do in PHP/PDO so it Does The Right Thing.
edit 2: Ugh. This is the wrong approach; even if I get it to work ok on INSERT, if I want to check for a duplicate string, it won't match properly.

Actually, MySQL truncates strings to the column width by default. It generates a warning, but allows the insert.
mysql> create table foo (str varchar(10));
mysql> insert into foo values ('abcdefghijklmnopqrstuvwxyz');
Query OK, 1 row affected, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+------------------------------------------+
| Level | Code | Message |
+---------+------+------------------------------------------+
| Warning | 1265 | Data truncated for column 'str' at row 1 |
+---------+------+------------------------------------------+
mysql> select * from foo;
+------------+
| str |
+------------+
| abcdefghij |
+------------+
If you set the strict SQL mode, it turns the warning into an error and rejects the insert.
Re your comment: SQL mode is a MySQL Server configuration. It probably isn't PDO that's causing it, but on the other hand it's possible, because any client can set SQL mode for its session.
You can retrieve the current global or session sql_mode value with the following statements:
SELECT ##GLOBAL.sql_mode;
SELECT ##SESSION.sql_mode;
The default should be an empty string (no modes set). You can set SQL mode in your my.cnf file, with the --sql-mode option for mysqld, or using a SET statement.
Update: MySQL 5.7 and later sets strict mode by default. See https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html

You can use column metadata to check string length. (PHP Manual on PDOStatement->getColumnMeta)
Get metadata for whole table this way
$query = $conn->query("SELECT * FROM places");
$numcols = $query->columnCount();
$tablemeta = array();
for ($i = 0; $i < $numcols; $i++) {
$colmeta = $query->getColumnMeta($i);
$tablemeta[$colmeta['name']] = $colmeta;
}

you could query information_schema to get the lenght of the column and use that data to truncate before insert, but that's more overhead than necessary. just turn off strict SQL mode for this statement:
SET #prev_mode = ##sql_mode;
SET sql_mode = '';
INSERT blah....;
SET sql_mode = #prev_mode;

Related

Why does this not error out

I'm working with mysql and php and I'm attempting to test the error handling of a call, but I can't figure out why this doesn't give an error. I'm executing the following line:
if (! mysql_query("UPDATE Accounts SET disabled='0' WHERE id='15'")) { ... }
Here's the scenario... There is a table called 'Accounts', but there isn't a record with an id of 15 (which is the primary key). I have tried this from the command line and via a web browser, but this line executes without problems. I checked the php manual for this and here's a quote from their pages:
For other type of SQL statements, INSERT, UPDATE, DELETE, DROP, etc, mysql_query() returns TRUE on success or FALSE on error.
Why is this not generating an error? Any help would greatly be appreciated!
The query is not failing.
Just because the ID doesn't exist, doesn't mean that the query fails. Mysql successfully looked for the record, found none, and didn't apply any action. This is much different that what the !mysql_query statement suggests. That implies that mysql was unable to run your command.
Here your command ran successfully, just didn't affect your table due to the nonexistent row.
Your query will update no record.
This is not an error, it happens every time the conditions in the WHERE clause are not met.
There are many ways to cause your query to fail. One of them would be to use a non-existing field:
UPDATE Accounts SET blablabla='0' WHERE id='15'
There's a difference between an empty result set, and an error. A query which results in no changes is NOT an error, it's simply a valid result that happens to be empty, e.g.
this can never return anything:
mysql> select now() from dual where 1=0;
Empty set (0.01 sec)
but is still not an error. It's just an empty set. By comparison, this will always return one row:
mysql> select now() from dual where 1=1;
+---------------------+
| now() |
+---------------------+
| 2013-05-03 09:51:19 |
+---------------------+
1 row in set (0.00 sec)
and then there's errors. This will not return an empty set, because the query itself failed at the parser level:
mysql> select now() from dual where abc=def;
ERROR 1054 (42S22): Unknown column 'abc' in 'where clause'

lots of queries - slow?

To begin with, I apologize if this has been asked already, I could not find anything at least.
Anyway, I'm going to run a cron task each 5 minutes. The script loads 79 external pages, whereas each page contain ~200 values I need to check in database (in total, say 15000 values). 100% of the values will be checked if they exist in database, and if they do (say 10% does) I will use an UPDATE query.
Both queries are very basic, no INNER etc.. It's the first time I use cron and I'm already assuming I will get the response "don't use cron for that" but my host doesn't allow daemons.
The query goes as:
SELECT `id`, `date` FROM `users` WHERE `name` = xxx
And if there was a match, it will use an UPDATE query (sometimes with additional values).
The question is, will this overload my mysql server? If yes, what are the suggested methods? I'm using PHP if that matters.
If you are just checking the same query over and over, there are a few options. Off the top of my head, you can use WHERE name IN ('xxx','yyy','zzz','aaa','bbb'...etc). Other than that, you could possibly do a file import into another table and probably run one query to do an insert/update.
Update:
//This is what I'm assuming your data looks like after loading/parsing all the pages.
//if not, it should be similar.
$data = array(
'server 1'=>array('aaa','bbb','ccc'),
'server 2'=>array('xxx','yyy','zzz'),
'server 3'=>array('111','222', '333'));
//where the key is the name of the server and the value is an array of names.
//I suggest using a transaction for this.
mysql_query("SET AUTOCOMMIT=0");
mysql_query("START TRANSACTION");
//update online to 0 for all. This is why you need transactions. You will set online=1
//for all online below.
mysql_query("UPDATE `table` SET `online`=0");
foreach($data as $serverName=>$names){
$sql = "UPDATE `table` SET `online`=1,`server`='{$serverName}' WHERE `name` IN ('".implode("','", $names)."')";
$result = mysql_query($sql);
//if the query failed, rollback all changes
if(!$result){
mysql_query("ROLLBACK");
die("Mysql error with query: $sql");
}
}
mysql_query("COMMIT");
About MySQL and lot of queries
If you have enough rights on this server - you may try to increase Query Cache.
You can do it in SQL or in mysql config file.
http://dev.mysql.com/doc/refman/5.1/en/query-cache-configuration.html
mysql> SET GLOBAL query_cache_size = 1000000;
Query OK, 0 rows affected (0.04 sec)
mysql> SHOW VARIABLES LIKE 'query_cache_size';
+------------------+--------+
| Variable_name | Value |
+------------------+--------+
| query_cache_size | 999424 |
+------------------+--------+
1 row in set (0.00 sec)
Task sheduler in MySQL
If your updates may work only on data stored in database (there are no PHP variables) - consider using EVENT in MySQL instead of running SQL scripts from PHP.

MySQL data does not appear to be in the character encoding it should be in

First off, thanks to whomever is reading this.
I have a very strange problem with character encoding in a MySQL database that I am using PHP's PDO to interface with. The tables are all encoded using UTF8, the webapp uses utf-8, but it seems that the data stored in the database is not actually utf-8 but latin-1.
Things had been working fine for quite some time, but this is causing issues with when importing utf-8 encoded data files or conducting fulltext searches that contain special characters such as "é" or "ë".
EDIT:
some replies have suggested that this is an issue with my terminal. It is not:
foreach($dbh->query("SELECT c FROM t") as $row){
echo $row['c'] ."\n";
echo urlencode($row['c'])."\n";
}
$dbh->exec("SET NAMES 'latin1'");
foreach($dbh->query("SELECT c FROM t") as $row){
echo $row['c'] ."\n";
echo urlencode($row['c'])."\n";
}
$dbh->exec("SET NAMES 'utf8'");
foreach($dbh->query("SELECT c FROM t") as $row){
echo $row['c'] ."\n";
echo urlencode($row['c'])."\n";
}
Outputs the following:
é
%C3%A9f
é
%C3%A9f
é
%C3%83%C2%A9f
Thanks to everyone so far.
END EDIT
So, first I check that the tables are working as they should:
USE information_schema;
mysql> SELECT table_collation FROM tables WHERE table_schema="mydb" and table_name="mytable";
+-----------------+
| table_collation |
+-----------------+
| utf8_general_ci |
+-----------------+
1 row in set (0.00 sec)
mysql> SELECT character_set_name,collation_name FROM information_schema.columns WHERE table_schema="mydb" and table_name="t" and column_name="c";
+--------------------+-----------------+
| character_set_name | collation_name |
+--------------------+-----------------+
| utf8 | utf8_general_ci |
+--------------------+-----------------+
1 row in set (0.00 sec)
However, the data does not appear to be utf-8, but latin-1:
mysql> use mydb;
Database changed
mysql> SET NAMES 'latin1';
Query OK, 0 rows affected (0.00 sec)
mysql> select c from t;
+---+
| c |
+---+
| é |
+---+
1 row in set (0.00 sec)
mysql> SET NAMES 'utf8';
Query OK, 0 rows affected (0.00 sec)
mysql> select c from t;
+----+
| c |
+----+
| é |
+----+
1 row in set (0.00 sec)
So I have two questions:
1) Most importantly, what can I do about the data already in the DB?
2) Is there a way to set up the DB so that it actually uses utf-8 when I connect or do I have to execute the SET NAMES query every time?
Many thanks for your time and help,
Matt
it's your terminal appeared to be in latin1, not data :)
Is there a way to set up the DB so that it actually uses utf-8
you have set it up already.
only thing you need is to set up client encoding, which is done by SET NAMES 'utf8'
in fact, by using SET NAMES, you can make your data appear in whatever encoding you set. that's the only purpose of the SET NAMES magic word.
in case you have some encoding issues not covered in this question,
what can I do about the data already in the DB?
whatever you wish, as long as your db returns no ? marks.
to recover your data you have to set names to the data encoding set for the table. this will prevent mysql from recoding the data. so, you can fetch or dump it and then load it again with proper settings.
EDIT
after some consideration, i am going to say that your data is in utf8 while table encoding is set to latin1 somehow.
%C3%A9 is a perfectly valid utf-8 representation of the é character. (dunno where did you get trailing f though)
while %C3%83%C2%A9 is utf-f encoded version of %C3%A9. So, it seems your database thinks that your data is in latin1 and encode it in utf8.
so, when you set names to latin1 it don't mind and don't recode.
Conclusion:
doublecheck your table (and fields) encoding. it should be latin1
yes, to save your data you have to do something like
***God I HATE this autoformatting issue which prevents me from posting the code right after list item
mysqldump --default_charset=latin1 mydb > mydb.sql
then check this dump and change every appearance of latin1 to utf8.
then load it back.
do not forget to backup your data first!
In case you received data from a HTML page and saved this to your DB, don't forget to set the correct text encoding in the HTML's head section:
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
If you don't set the text encoding, browsers may return text differently encoded.
Regarding 1): Have a look at PHP's
string utf8_decode ( string $data )
function described here. Retrieve records from your DB, transcode to the desired encoding and write it back.
use
SET character_set_client = "UTF-8";
when connecting with your client. in PHP you will achieve this by using function:
set_charset($encoding)
before you do any actual data insert / update
Connect to your Database
Set connection to UTF-8
SET NAMES 'utf8';
In your HTML files:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

Transaction rollback doesn't work

I have made a database wrapper with extra functionality around the PDO system (yes, i know a wrapper around a wrapper, but it is just PDO with some extra functionality). But i have noticed a problem.
The folowing doesn't work like it should be:
<?php
var_dump($db->beginTransaction());
$db->query('
INSERT INTO test
(data) VALUES (?)
;',
array(
'Foo'
)
);
print_r($db->query('
SELECT *
FROM test
;'
)->fetchAll());
var_dump($db->rollBack());
print_r($db->query('
SELECT *
FROM test
;'
)->fetchAll());
?>
The var_dump's shows that the beginTransaction and rollBack functions return true, so no errors.
I expected that the first print_r call show a array of N items and the second call show N-1 items. But that issn't true, they both show same number of items.
My $db->query(< sql >, < values >) call nothing else then $pdo->prepare(< sql >)->execute(< values >) (with extra error handling ofcourse).
So i think or the transaction system of MySQL doesn't work, or PDO's implenmentaties doesn't work or i see something wrong.
Does anybody know what the problem is?
Check if your type of database equals innoDB. In one word you must check if your database supports transactions.
Two possible problems:
The table is MyISAM which doesn't support transaction. Use InnoDB.
Check to make sure auto-commit is OFF.
http://www.php.net/manual/en/pdo.transactions.php
I'm entering this as an answer, as a comment is to small to contain the following:
PDO is just a wrapper around the various lower level database interface libraries. If the low-level library doesn't complain, either will PDO. Since MySQL supports transactions, no transaction operations will return a syntax error or whatever. You can use MyISAM tables within transactions, but any operations done on them will be done as if auto-commit was still active:
mysql> create table myisamtable (x int) engine=myisam;
Query OK, 0 rows affected (0.00 sec)
mysql> create table innodbtable (x int) engine=innodb;
Query OK, 0 rows affected (0.00 sec)
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into myisamtable (x) values (1);
Query OK, 1 row affected (0.00 sec)
mysql> insert into innodbtable (x) values (2);
Query OK, 1 row affected (0.00 sec)
mysql> rollback;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> select * from myisamtable;
+------+
| x |
+------+
| 1 |
+------+
1 row in set (0.00 sec)
mysql> select * from innodbtable;
Empty set (0.00 sec)
mysql>
As you can see, even though a transaction was active, and some actions were performed on the MyISAM table, no errors were thrown.
MySQL doesn't support transactions on the MyISAM table type, which is unfortunately the default table type.
If you need transactions, you should switch to the InnoDB table type.
Another reason this may happen is certain types of SQL statements cause an immediate auto-commit. I had a large script that ran in a transaction that was getting committed immediately and ignored the transaction. I eventually found out it was because any ALTER TABLE statement immediately causes a commit to happen.
Types of statements that cause auto commits are:
Anything that modifies a table or the database, such as ALTER TABLE, CREATE TABLE, etc.
Anything that modifies table permissions, such as ALTER USER or SET PASSWORD
Anything that locks that tables or starts a new transaction
Data loading statements
Administrative statements, such as ANALYZE TABLE, FLUSH, or CACHE INDEX
Replication control statements, such as anything to do with a slave or master
More info and a complete list can be found here: https://dev.mysql.com/doc/refman/8.0/en/implicit-commit.html
If you're having this problem only with a specific script and you're sure you're using InnoDB, you might want to look to see if any SQL statements in your script match these.

MySQLi prepared statements and REPLACE INTO

I have to following code:
http://www.nomorepasting.com/getpaste.php?pasteid=22987
If PHPSESSID is not already in the table the REPLACE INTO query works just fine, however if PHPSESSID exists the call to execute succeeds but sqlstate is set to 'HY000' which isn't very helpful and $_mysqli_session_write->errno and
$_mysqli_session_write->error are both empty and the data column doesn't update.
I am fairly certain that the problem is in my script somewhere, as manually executing the REPLACE INTO from mysql works fine regardless of whether of not the PHPSESSID is in the table.
Why are you trying to doing your prepare in the session open function? I don't believe the write function is called more then once during a session, so preparing it in the open doesn't do much for you, you might as well do that in your session write.
Anyway I believe you need some whitespace after the table name, and before the column list. Without the whitespace I believe mysql would act as if you where trying to call the non-existent function named session().
REPLACE INTO session (phpsessid, data) VALUES(?, ?)
MySQL sees no difference between
'COUNT ()' and 'COUNT()'
Interesting, when I run the below in the mysql CLI I seem to get a different result.
mysql> select count (*);
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '*)' at line 1
mysql> select count(*);
+----------+
| count(*) |
+----------+
| 1 |
+----------+
1 row in set (0.00 sec)
REPLACE INTO executes 2 queries: first a DELETE then an INSERT INTO.
(So a new auto_increment is "By Design")
I'm also using the REPLACE INTO for my database sessions, but I'm using the MySQLi->query() in combination with MySQLI->real_escape_string() in stead of a MySQLi->prepare()
So as it turns out there are other issues with using REPLACE that I was not aware of:
Bug #10795: REPLACE reallocates new AUTO_INCREMENT (Which according to the comments is not actually a bug but the 'expected' behaviour)
As a result my id field keeps getting incremented so the better solution is to use something along the lines of:
INSERT INTO session(phpsessid, data) VALUES('{$id}', '{$data}')
ON DUPLICATE KEY UPDATE data='{$data}'
This also prevents any foreign key constraints from breaking and potential causing data integrity problems.

Categories