I have a PHP script where users create questionnaires, and the script tables in the DB to store the incoming data. The site's been live for a while, and there are about 100 tables in the database.
My script was awfully flawed! It calls for "tinytext" fields in places where I really need "text". Is there a way to bulk update all of the tinytext columns to text?
Thanks!
Solution without stored procedures (using only phpMyAdmin or any other DBA tool).
Run the following query
SELECT
CONCAT('ALTER TABLE ',
TABLE_NAME,
' CHANGE COLUMN ',
COLUMN_NAME,
' ',
column_name,
' TARGET_TYPE ',
CASE
WHEN IS_NULLABLE = 'NO' THEN ' NOT '
ELSE ''
END,
' NULL;') AS que
FROM
information_schema.columns
WHERE
table_schema = 'MY DB'
AND data_type = 'SOURCE_TYPE';
This query will return you all the statements to fire. You can run them or save into a SQL Upgrade script
Example (from tinyint to bit):
ALTER TABLE mytable CHANGE COLUMN redacted redacted BIT NULL;
ALTER TABLE mytable CHANGE COLUMN redacted2 redacted2 BIT NOT NULL;
One way of doing this, is to find all the tinytext columns in the given database and then create ALTER TABLE statement for the each column. Working solution using stored procedure:
DELIMITER $$
DROP PROCEDURE IF EXISTS `BULK_RETYPE` $$
CREATE PROCEDURE `BULK_RETYPE`(IN SCHEMA_NAME VARCHAR(255), IN FROM_TYPE VARCHAR(255), IN TO_TYPE VARCHAR(255))
BEGIN
DECLARE `done` INT DEFAULT FALSE;
DECLARE tn VARCHAR(255);
DECLARE fn VARCHAR(255);
DECLARE `cur1` CURSOR FOR
SELECT
`TABLE_NAME`,
`COLUMN_NAME`
FROM
`information_schema`.`COLUMNS`
WHERE
`DATA_TYPE` = FROM_TYPE AND `TABLE_SCHEMA` = SCHEMA_NAME;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO `tn`, `fn`;
IF done THEN
LEAVE read_loop;
END IF;
SET #ALTER_SQL = CONCAT('ALTER TABLE ', '`', tn,'`' , ' MODIFY ', '`', fn,'`' , ' ', TO_TYPE);
PREPARE stmt1 FROM #ALTER_SQL;
EXECUTE stmt1;
END LOOP;
CLOSE cur1;
END $$
DELIMITER ;
CALL BULK_RETYPE('test', 'tinytext', 'text');
Not really, though if you use PHPMyAdmin, it's a rather quick task... alternatively, you could export a list of all the tables and fields you need to change and put together the necessary ALTER TABLE statements and then execute them.
You can just select from schema and prepare SQL for mass alter
SELECT table_scheme, table_name, columun_name FROM information_schema.`COLUMNS`
WHERE DATA_TYPE='tinyint'
Related
I have a table with a column 'search_text' type text.
In that field I have values:
1. 'MyBook MyBook PDF PDF',
2. 'Example 1 Example 2 Example 3'
3. 'John Snow John Snow'
I would like to distinct clean these fields.
Expected result:
1. 'MyBook PDF',
2. 'Example 1 2 3'
3. 'John Snow'
The approach I came up with goes as follows:
read the field for each record, split it by space (' '), put each text in array, do array_unique in PHP, then put the array back to string with join in PHP.
The thing is, this is a PHP based solution, I would like to have an MySQL solution for this. I have over 180.000 records I need to clean, I don't know what impact it would have to run this on PHP.
I have found a solution for MS SQL: Remove duplicate values in a cell SQL Server
Help greatly appreciated.
SQL of my test data:
CREATE TABLE IF NOT EXISTS `test` (
`id` int(10) unsigned NOT NULL,
`search_text` text COLLATE utf8_unicode_ci NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
INSERT INTO `test` (`id`, `search_text`) VALUES
(1, 'MyBook MyBook PDF PDF'),
(2, 'Example 1 Example 2 Example 3'),
(3, 'John Snow John Snow'),
(4, 'test test test test formula test test test formula test test test formula test test test formula test test test formula test test test formula '),
(5, '');
ALTER TABLE `test`
ADD PRIMARY KEY (`id`);
ALTER TABLE `test`
MODIFY `id` int(10) unsigned NOT NULL AUTO_INCREMENT,AUTO_INCREMENT=6;
Try this to sort by count :)
SELECT DISTINCT SUBSTRING_INDEX(SUBSTRING_INDEX(test.search_text, ' ', numbers.n), ' ', - 1) col_name
FROM (
SELECT 1 n
UNION ALL
SELECT 2
UNION ALL
SELECT 3
UNION ALL
SELECT 4
) numbers
INNER JOIN test ON CHAR_LENGTH(test.search_text) - CHAR_LENGTH(REPLACE(test.search_text, ' ', '')) >= numbers.n - 1
ORDER BY col_name;
You will need to write a MySQL function to do this for you. I would think that a PHP page will be just fine. 180,000 records isn't that many and it should (unless you are using a low spec server) run without putting much strain on anything else.
I wrote 2 for you that you might be able to make use of:
DROP PROCEDURE IF EXISTS explode;
DELIMITER //
CREATE PROCEDURE explode(str_string TEXT)
NOT DETERMINISTIC
BEGIN
DROP TABLE IF EXISTS explosion;
CREATE TABLE explosion (id INT AUTO_INCREMENT PRIMARY KEY NOT NULL, word VARCHAR(100));
SET #sql := CONCAT('INSERT INTO explosion (word) VALUES (', REPLACE(QUOTE(str_string), " ", '\'), (\''), ')');
PREPARE myStmt FROM #sql;
EXECUTE myStmt;
END //
DELIMITER ;
This procedure creates an "explode" function for use in MySQL. It uses a temporary table and explodes the words, separated by spaces into it
Then this function will read that table in, and put them into another temporary table with the duplicates removed:
DROP PROCEDURE IF EXISTS removeDuplicates;
DELIMITER //
CREATE PROCEDURE removeDuplicates(str TEXT)
BEGIN
DECLARE temp_word TEXT;
DECLARE last_word TEXT DEFAULT "";
DECLARE result TEXT;
DECLARE finished INT DEFAULT false;
DECLARE words_cursor CURSOR FOR
SELECT word FROM explosion;
DECLARE CONTINUE handler FOR NOT found
SET finished = true;
CALL explode(str);
DROP TABLE IF EXISTS temp_words;
CREATE TABLE temp_words (id INT AUTO_INCREMENT PRIMARY KEY NOT NULL, t VARCHAR(100));
OPEN words_cursor;
loop_words: LOOP
FETCH words_cursor INTO temp_word;
IF finished THEN
LEAVE loop_words;
END IF;
IF last_word = "" THEN
INSERT INTO temp_words (t) VALUES (temp_word);
SET last_word = temp_word;
ITERATE loop_words;
END IF;
IF last_word = temp_word THEN
SET last_word = temp_word;
ITERATE loop_words;
END IF;
INSERT INTO temp_words (t) VALUES (temp_word);
END LOOP loop_words;
CLOSE words_cursor;
END //
DELIMITER ;
So all you need to do is work out how to get the records in temp_words into your current database table.
I went for the PHP solution here:
$s = 'John Snow John Snow';
//remove duplicate values in string
$tmpArray = explode(" ", $s);
$tmpArray = array_unique($tmpArray);
$s = join(" ", $tmpArray);
Which is run before INSERT, and it does what I wanted.
I'm trying to alt WP tables with the function. Mysql code works fine if I call it from mysql client (adds columns), but when I called it via $wpdb->query() it gave me error pointing to DELIMITER line. Upon some googling, I found out that I should use mysqli for this purpose. Now the error log is empty but columns are still not created. Anyone got an idea of what I'm doing wrong? Function code is below
global $wpdb;
$tables=array(
'wp_users',
'wp_usermeta',
'wp_term_taxonomy',
'wp_term_relationships',
'wp_terms',
'wp_taxonomymeta',
'wp_posts',
'wp_postmeta',
'wp_p2p',
'wp_p2pmeta',
'wp_options',
'wp_fb_user_timezone',
'wp_fb_coaching_call_report_team_user',
'wp_fb_coaching_call_report_team',
'wp_fb_3d_notes',
'wp_comments',
'wp_commentmeta',
'wp_coaching_teams_student',
'wp_coaching_teams_coach',
'wp_coaching_teams',
'wp_coaching_call_count'
);
if (is_array($tables)&&count($tables)>0)
{
foreach ($tables as $table)
{
$aaa=mysqli_multi_query($wpdb->dbh,'DELIMITER $$
DROP PROCEDURE IF EXISTS add_createdat_and_updatedat_fields $$
CREATE PROCEDURE add_createdat_and_updatedat_fields()
BEGIN
IF EXISTS(SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema=DATABASE() AND table_name=\''.$table.'\') THEN
IF NOT EXISTS ((SELECT * FROM information_schema.columns WHERE table_schema=DATABASE() AND table_name=\''.$table.'\' AND column_name=\'created_at\')) THEN
ALTER TABLE `'.$table.'` ADD `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP;
END IF;
IF NOT EXISTS ((SELECT * FROM information_schema.columns WHERE table_schema=DATABASE() AND table_name=\''.$table.'\' AND column_name=\'updated_at\')) THEN
ALTER TABLE `'.$table.'` ADD `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP;
END IF;
END IF;
END $$
CALL add_createdat_and_updatedat_fields() $$
DROP PROCEDURE IF EXISTS add_createdat_and_updatedat_fields $$
DELIMITER;');
}
}
Do not use multiquery, run each command separately with mysqli_query() instead, that way you don't need manipulating separators at all.
I've written this code for mysql pivot table:
SET #SQL = NULL;
SET ##group_concat_max_len = 6000;
SELECT GROUP_CONCAT( DISTINCT CONCAT( 'MAX(IF(questiondetails = \'', questiondetails, '\', answer, null)) AS \'', questiondetails, '\' ' )) INTO #SQL FROM wtfeedback;
SET #SQL = CONCAT( 'SELECT trialid, productsku, userkey, category, ', #SQL, ' FROM wtfeedback GROUP BY trialid' );
PREPARE stmt FROM #SQL;
EXECUTE stmt;
This works fine in Sequel Pro (mysql gui editor)
But when I paste into my php page to run this code it is showing a syntax error:
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SET ##group_concat_max_len = 6000;
SELECT GROUP_CONCAT( DISTINCT CONCAT( 'MAX(I' at line 3
I'm struggling to see what the error might be.
Any ideas ? Thanks in advance.
MySQL doesn't use '' to escape single quotes. If you want to embed single quotes in your in-sql strings, then use \':
CONCAT('MAX(IF(questiondetails = \'', questiondetails, '\', answer, null)) AS "', questiondetails, '" ')
^^---------------------^^
From PHP, issue only one statement at a time. I deduce that this is the problem since the error is pointing at the beginning of the second SET.
Here is a stored proc to generate a pivot SELECT for you.
I was struggling with the same issue as Guy Murray, but fought my way out.
Basically I made a stored procedure that lets you run a pivot table on selectable rows and columns, with optional filtering. It does so by first storing the result of a "group by" select query in a temp table, and then fiddling that to a pivot table with the "group_concat" function. Same trick as Guy does.
The advantage is that it goes through the main table only once, which may save time if there are zillions of records in it.
Here is a sample table:
CREATE TABLE `Data` (
`Period` INT(2) NOT NULL,
`Product` VARCHAR(20) NOT NULL DEFAULT '',
`Amount` DOUBLE NOT NULL
) ENGINE=INNODB DEFAULT CHARSET=latin1;
INSERT INTO `Data` (`Period`, `Product`, `Amount`)
VALUES
(1,'PrdA',15484),
(1,'PrdA',45454),
(1,'PrdB',478),
(2,'PrdB',985),
(2,'PrdB',741),
(2,'PrdB',985),
(3,'PrdA',7515),
(3,'PrdA',454),
(3,'PrdB',4584),
(2,'PrdB',445),
(1,'PrdB',669);
And this is the stored procedure. Additional comment in the code.
DELIMITER ;;
CREATE DEFINER=`root`#`localhost` PROCEDURE `pivot`(
source VARCHAR(1000),
val VARCHAR(40),
rws VARCHAR(40),
cls VARCHAR(40),
filter VARCHAR(1000))
BEGIN
/*
Creates a pivot table from any table, view or SQL statement.
Mandatory: source, value, rows, and columns to be pivoted.
Optional filtering.
Sample call strings:
CALL pivot('data', 'amount', 'period', 'product', '');
CALL pivot('(select * from data)', 'amount', 'product', 'period', 'WHERE amount>1000');
*/
/*just to be sure*/
DROP TEMPORARY TABLE IF EXISTS temp1;
/*increase the value of group concat, otherwise the number of columns is very limited*/
SET SESSION group_concat_max_len = 100000;
/*perform a "select...group by" on the source and store it in a temp table1*/
SET #a=CONCAT(
'CREATE TEMPORARY TABLE temp1 (
SELECT ',
rws,' AS rows, ',
cls,' AS cols,
SUM(',val,') AS val
FROM ',source,' S ',
filter, '
GROUP BY '
,rws,', ',
cls,');'
);
PREPARE stmt FROM #a;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
/*use "distinct columns" from temp1 to make a text string #coltext, that contains the column statements, to be used in the final step
Produced text string looks like this: sum(CASE WHEN cols='PrdA' THEN val END) AS 'PrdA', sum(CASE WHEN cols='PrdB' THEN val END) AS 'PrdB' */
SELECT GROUP_CONCAT(
' SUM(CASE WHEN cols=\'',cols,'\' THEN val END) AS \'',cols,'\'')
INTO #coltext
FROM (SELECT DISTINCT(cols) AS cols FROM temp1) A;
/*build the final statement in #b*/
SET #b=CONCAT(
'SELECT
IFNULL(rows, \'Total\') AS ',rws,', '
,#coltext,',
SUM(val) AS Total
FROM temp1
GROUP BY
rows
WITH ROLLUP;');
/*and launch it*/
PREPARE stmt FROM #b;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
/*clean up*/
DROP TEMPORARY TABLE IF EXISTS temp1;
SET #a=NULL;
SET #b=NULL;
SET #coltext=NULL;
END;;
DELIMITER ;
The result looks like this:
period PrdA PrdB total
1 60938 1147 62085
2 NULL 3156 3156
3 7969 4584 12553
total 68907 8887 77794
Hope this shows up correctly on stack overflow. It's my first post here.
edit 2015-10-19: when reading others solutions here, I realised that the code could be cleaned up and improved: it's now free of any hardcoded references. Just plug it in any database and it will work.
I have a stored procedure to fetch an id. It concat's the last name with first name as "Stone, Cold" and compares with the _fullname passed which when i var_dump gives
String(13) Stone, Cold
It should give an id
IF NOT ISNULL(_fullname) THEN
SET _fullname = TRIM(_fullname);
SET clause = CONCAT( clause , ' AND CONCAT(c.lname, ', ', c.fname) LIKE CONCAT('%',_fullname,'%')');
END IF;
When i try the same query in MySQL it works perfectly fine but doesn't work in procedure. I'm sure that the problem is syntax in stored procedure.
Try:
DELIMITER //
DROP PROCEDURE IF EXISTS `sp_test`//
CREATE PROCEDURE `sp_test`(IN `_fullname` VARCHAR(20))
BEGIN
DECLARE `clause` VARCHAR(500) DEFAULT '';
IF NOT ISNULL(`_fullname`) THEN
SET `_fullname` := TRIM(`_fullname`);
SET `clause` := CONCAT(`clause`, ' AND CONCAT(`c`.`lname`, '', '', `c`.`fname`) LIKE ''', '%', `_fullname`, '%''');
END IF;
SELECT `clause`;
END//
DELIMITER ;
SQL Fiddle demo
I am trying to run a procedure in which i will send the table name to fetch all records from it..so that i don't have to create different procedure...but i am facing some problem in that.
ALTER PROCEDURE [dbo].[getTableData]
-- Add the parameters for the stored procedure here
#tableName Varchar(100),
#whrFldName NVarchar(100)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- Insert statements for procedure here
DECLARE #ActualTableName AS NVarchar(255)
SELECT #ActualTableName = QUOTENAME( TABLE_NAME )
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = #tableName
DECLARE #sql AS NVARCHAR(MAX)
SELECT #sql = 'SELECT * FROM ' + #ActualTableName + ' WHERE ' +
#whrFldName + ' = ''y'' ;'
--PRINT #sql
EXEC(#SQL)
END
The PHP code is this..
$sql ="EXEC [dbo].[getTableData] 'tbl_services','serviceStatus'";
$rst = odbc_exec($connection, $sql);
$i = 0;
while($result = odbc_fetch_array($rst))
{
$returnPageData[$i] = $result;
$i++;
}
It executes just fine in server but when I call it from my PHP code, it returns null.
Here if I remove * and place fields it works fine..I have tested my code well,it specially creates the problem for a Text type field..
If i change the procedure to this,it works fine..
ALTER PROCEDURE [dbo].[getTableData]
-- Add the parameters for the stored procedure here
#rowsPerPage as bigint,
#pageNum as bigint
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
WITH SQLPaging AS (
SELECT TOP(#rowsPerPage * #pageNum) ROW_NUMBER() OVER (ORDER BY creationDate DESC)
AS resultNum, *
FROM [DB_SB].[dbo].[tbl_blog] )
SELECT
blogTitle,
blogSlug,
blogImage,
substring( blogContent, 1, 210 ) AS blogContent,
creationDate,
blogAddedBy
FROM SQLPaging WITH (nolock) WHERE resultNum > ((#pageNum - 1) * #rowsPerPage)
END
But this is no logical..if i send fields everytime..this is not what i want exactly..
Any solution??please help...
You need to use SQL Server Profiler to see what’s actually getting to database engine.
If your stored procedure is executed correctly when run from SSMS then there is a problem somewhere in PHP part.
Are you using the same database using when testing from SSMS? There might be some issue in that…