I have gotten the gist of most of mysql query and how to set-up my table however I am a little lost on how I can do this.
I currently have a table that has this format:
tag_target_id | tag_id | sub_tag_id | tag_target_name | tag_type_id
int(11) | int(11) | int(11) | varchar(255) | int(11
<PK> | <FK> | <FK> | | <FK>
A single tag_target_id corresponds to one piece of content ie. a video, article etc. now in the table there should be multiple entries with the same tag_target_id and tag_target_name and different tag_id , sub_tag_id , tag_type_id simple enough
So my problem is when I am inserting data into the table I have a PDO query that looked like this:
"INSERT INTO phpro_tag_targets (tag_id, sub_tag_id, tag_target_name, tag_type_id)
VALUES (:tag_id, :sub_tag_id,:tag_target_name,:tag_type_id)"
... I used to have tag_target_id as auto increment however each new query obviously incremented the tag_target_id so a single piece of content would have tag-target_id = 5, 6, 7, 8, 9 for eg. where I would like to have each piece of new content have one single tag_target_id
Now I would like to somehow have each new piece of content have a new tag_target_id but not each entry into the table for a single piece of content have it's own unique idea
so for eg. for one piece of content in the table all data for that content could look like this:
tag_target_id | tag_id | sub_tag_id | tag_target_name | tag_type_id
int(11) | int(11) | int(11) | varchar(255) | int(11
54 | 22 | 64 | url_to_content | 16
54 | 66 | 82 | url_to_content | 24
54 | 22 | 77 | url_to_content | 18
54 | 87 | 55 | url_to_content | 16
54 | 66 | 92 | url_to_content | 20
So how can I change the structure of the table or write a query set to achieve this desired output?
Assuming that You have separate table with info about content itself You may create a composite primary key on fields tag_target_id, tag_id and sub_tag_id.
And make tag_target_id as FK for this content table.
For existing table the query will be
ALTER TABLE phpro_tag_targets ADD PRIMARY KEY uniq_target_tag_sub (tag_target_id,tag_id,sub_tag_id)
Well you still need a unique id to distinguish the records. I would create a new column called something like "tag_content_id" and use that to store the grouping id (in your example 54). This way your table will have a unique id to use for the key (tag_target_id) and also a way to distinguish between different content entries and group items together (tag_content_id). When all else fails, add a column!
Related
I'm pretty new to programming and trying to design a web application which provides a front-end to update data for an online course database (tables incl. users, assignments, questions, answers, etc). All data is coming into this database upon submission via course management system. Currently working with dummy data for development purposes.
The idea is to allow a user to update from the front end, rather than have updates occur automatically from the back end (as in using triggers). This is because we have a relatively small data set, and will just need the updated tables for users not familiar with mysql to export into data analysis programs.
Using multiple joins, I've created a list of assignments already taken by users, which looks like this:
+----+---------------+---------------------------------------+
| id | assignment_id | quiz_name |
+----+---------------+---------------------------------------+
| 1 | 2 | Guidance Counselors (Post-Assessment) |
| 2 | 3 | Guidance Counselors (Pre-Assessment) |
| 3 | 4 | Guidance Counselors (Pre-Assessment) |
+----+---------------+---------------------------------------+
In PHP, I've coded a basic front-end for displaying and updating these assignments in a multi-selection dropdown, which looks like this:
Assignment list table
Whenever a user wants to update the list, php runs this if statement (the same one used to generate the above table):
//update assignment list when button is clicked
if(isset($_POST['updatelist'])){
//query to update the assignment list
$sql_update = "DELETE FROM assignment_list;";
$sql_update .= "TRUNCATE assignment_list;";
$sql_update .= "INSERT INTO assignment_list....
#values added from joined assignment, quizzes, and user_quizzes tables
SELECT...
INNER JOIN ...
INNER JOIN ...
...
GROUP BY assignment_id";
$update_result = mysqli_multi_query($conn,$sql_update);
//success and error messages here
}
ETA: Which is then reflected in the dropdown menu. This menu then should allow users to select one or more assignments for which to update data, let's say if there are new user submissions and scores available for that assignment (i.e. the data contained within each assignment -- structurally they are all the same).
Updating the list seems to be working, but I am struggling to figure out the best way to update what we're calling "consolidated data" tables (appended with "_(assignment_id)" for each individual assignment. So when a user selects "update data", the data in the table for the selected assignment(s) above should update. As an example, consolidated_data_4 looks like this (some fields omitted for better readability):
+------------------+---------+---------------+---------+-----------------------------+----------------------------+--------------------------------------+-----------+
| consolidation_id | user_id | assignment_id | quiz_id | assignment_pass_score_point | assignment_pass_score_perc | quiz_name | answer_id |
+------------------+---------+---------------+---------+-----------------------------+----------------------------+--------------------------------------+-----------+
| 1 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175973 |
| 2 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175981 |
| 3 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175985 |
| 4 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175991 |
| 5 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175995 |
| 6 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 175999 |
| 7 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 176002 |
| 8 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 176009 |
| 9 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 176015 |
| 10 | 34 | 4 | 50 | 5.00 | 50.00 | Guidance Counselors (Pre-Assessment) | 176021 |
+------------------+---------+---------------+---------+-----------------------------+----------------------------+--------------------------------------+-----------+
Each table is currently identified by its assignment id, e.g. "consolidated_data_4" is a table in which assignment_id = 4 for every record in that table.
I tried looping through each table (using a foreach loop) and performing a similar query as the one above for the assignments list, but I receive an error unless I separate the queries out like so (also abridged):
//for each assignment user selects, start with an empty table and join data from corresponding course database tables
foreach($_POST['selectassign'] as $assignment){
$table = "quiz_data_update_application.consolidated_data_".$assignment;
//query updates consolidated data table(s) for selected assignments
$sql_del = "DELETE FROM $table";
$sql_trunc = "TRUNCATE $table";
//records added from joining tables in course db
$sql_ins = //sql omitted here for brevity
....
$del_sel_res = mysqli_query($conn,$sql_del);
$trunc_sel_res = mysqli_query($conn,$sql_trunc);
$ins_sel_res = mysqli_query($conn,$sql_ins);
}
(The only difference between this and the code that doesn't work is that the queries are combined in the same fashion as the first PHP snippet above; this seems redundant to post here).
Using single queries seems inefficient and will slow down the application. I am wondering if there is a better way to approach this than using part of a table name as a variable in PHP loops. (Don't see too many people asking about this or iterating through actual tables as opposed to fields) It seems there's either some silly syntax mistake I'm missing, or mysqli_multiquery() can't be used in loops/is overall a poor approach?
Some of the higher-ups where I am working have suggested either:
(1) creating separate loops - one to temporarily create a new table based on a users' selection of assignments, and another to split that up into separate tables by assignment (this part they are saying is required for optimal data analysis. IMO this makes everything more challenging than having a large "master" table of data for ALL assignments -- which I did in fact have success doing -- but alas this is how they're requesting it be done). Those tables would then be deleted after an update completes;
(2) Using stored procedures. This I am not as familiar with, and not sure how that'd work, but if it's more feasible than (1), I could look into it more.
Any other alternative, more feasible suggestions would be appreciated as well. I've made a lot of progress with this, but have been stuck here the past few weeks and not finding much in the way of online resources.
Apologies for the length of the post. I thought it necessary to provide more context rather than less.
Single queries is the way to go. The perceived inefficiency is not there -- the operations are more costly than the mysqli_query.
If you have thousands of tables that to work with, then I question the wisdom of having multiple similar tables. Instead, have an extra column to indicate which thing is involved.
DELETE (all of a table) and TRUNCATE have virtually the same effect; there is no need to do both. TRUNCATE is faster.
And deleting one indexed row from a table is a lot faster than truncate + insert.
Furthermore, batching the deletes into a single SQL statement is even faster. So is batch inserting multiple rows at once. This is easily 10 times as fast.
#Rick James here is SHOW CREATE TABLE for one of the tables, although the structure is the same for all of them.
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| consolidated_data_2 | CREATE TABLE `consolidated_data_2` (
`consolidation_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`assignment_id` int(11) NOT NULL,
`quiz_id` int(11) NOT NULL,
`assignment_pass_score_point` decimal(10,2) DEFAULT NULL,
`assignment_pass_score_perc` decimal(10,2) DEFAULT NULL,
`quiz_name` varchar(3800) NOT NULL,
`question_id` int(11) NOT NULL,
`question_text` varchar(3800) CHARACTER SET utf8 NOT NULL,
`question_type_id` int(11) NOT NULL,
`answer_id` int(11) DEFAULT NULL,
`answer_textA` varchar(800) CHARACTER SET utf8 DEFAULT NULL,
`answer_added_ts` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`group_id` int(11) DEFAULT NULL,
`user_answer_text` varchar(3800) CHARACTER SET utf8 DEFAULT NULL,
`answer_image` varchar(800) DEFAULT NULL,
`is_correct` int(11) DEFAULT NULL,
`correct_answer_text` varchar(3800) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`consolidation_id`)
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=latin1 |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
The reason for having multiple tables is so that there's one table unique to each assignment id, so each assignment table can be updated separately/selectively for analysis purposes.
It isn't necessarily just updating one row, since each assignment is typically 10 questions. So each new user submission would add 10 more rows in most cases.
I am attaching an ER diagram which shows all the tables that have source data for the consolidated data tables.
ER Diagram for consolidated_data tables
(Blue = fields used for joins; yellow = all other fields included in the table; data type info on some for reference)
I'm using Mysql 5.5 and by example I have a table like this
+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| idgroups | int(11) | NO | PRI | NULL | auto_increment |
| group_id | int(11) | YES | | NULL | |
| group_name | varchar(45) | YES | | NULL |
Where some people are allowed to do inserts,update and delete but I want to know which is the last modified row or row's id in a given time
Any ideas?
Thanks in advance
My suggestion would be to create a second table. something like edit_history for recording modifications. You can put triggers on your groups table above that says "Any time a record is inserted, deleted, or updated, create a record in my edit_history table".
A trigger can be created as follows:
CREATE TRIGGER trigger_name
AFTER INSERT
ON table_name FOR EACH ROW
BEGIN
-- For each row inserted
-- do something...
END;
Since your field is auto_increment, you can just select the maximum value of idgroups to get the most recently inserted value:
select max(idgroups) from tbl
to get last modified in general will require additional structure to your table. In particular, if you are deleting, you will need to store what you have most recently deleted somewhere.
I am having a mysql table
content_votes_tmp
+------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+------------------+------+-----+---------+----------------+
| up | int(11) | NO | MUL | 0 | |
| down | int(11) | NO | | 0 | |
| ip | int(10) unsigned | NO | | NULL | |
| content | int(11) | NO | | NULL | |
| datetime | datetime | NO | | NULL | |
| is_updated | tinyint(2) | NO | | 0 | |
| record_num | int(11) | NO | PRI | NULL | auto_increment |
+------------+------------------+------+-----+---------+----------------+
surfers can vote up or vote down on posts i.e. content, a record gets inserted everytime a vote is given same as rating , in the table along with other data like ip , content id
Now i am trying to create cronjob script in php which will SUM(up) and SUM(down) of votes
like this,
mysqli_query($con, "SELECT SUM(up) as up_count, SUM(down) as down_count, content FROM `content_votes_tmp` WHERE is_updated = 0 GROUP by content")
and then by using while loop in php i can update the main table for the specific content id,
but i would like to set the records which are part of SUM to be marked as updated i.e. SET is_updated = 1, so the same values wont get summed again and again.
How can i achieve this ? using mysql query ? and work on same data set as , every second/milisecond the records are getting inserted in the table ,.
i can think of another way of achieving this is by getting all the non-updated records and doing sum in the php and then updating every record.
The simplest way would probably be a temporary table. Create one with the record_num values you want to select from;
CREATE TEMPORARY TABLE temp_table AS
SELECT record_num FROM `content_votes_tmp` WHERE is_updated = 0;
Then do your calculation using the temp table;
SELECT SUM(up) as up_count, SUM(down) as down_count, content
FROM `content_votes_tmp`
WHERE record_num IN (SELECT record_num FROM temp_table)
GROUP by content
Once you've received your result, you can set is_updated on the values you just calculated over;
UPDATE `content_votes_tmp`
SET is_updated = 1
WHERE record_num IN (SELECT record_num FROM temp_table)
If you want to reuse the connection to do the same thing again, you'll need to drop the temporary table before creating it again, but if you just want to do it a single time in a page, it will disappear automatically when the database is disconnected at the end of the page.
We are implementing a system that analyses books. The system is written in PHP, and for each book loops through the words and analyses each of them, setting certain flags (that translate to database fields) from various regular expressions and other tests.
This results in a matches table, similar to the example below:
+------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| id | bigint(20) | NO | PRI | NULL | auto_increment |
| regex | varchar(250) | YES | | NULL | |
| description | varchar(250) | NO | | NULL | |
| phonic_description | varchar(255) | NO | | NULL | |
| is_high_frequency | tinyint(1) | NO | | NULL | |
| is_readable | tinyint(1) | NO | | NULL | |
| book_id | bigint(20) | YES | | NULL | |
| matched_regex | varchar(255) | YES | | NULL | |
| [...] | | | | | |
+------------------------+--------------+------+-----+---------+----------------+
Most of the omitted fields are tinyint, either 0 or 1. There are currently 25 fields in the matches table.
There are ~2,000,000 rows in the matches table, the output of analyzing ~500 books.
Currently, there is a "reports" area of the site which queries the matches table like this:
SELECT COUNT(*)
FROM matches
WHERE is_readable = 1
AND other_flag = 0
AND another_flag = 1
However, at present it takes over a minute to fetch the main index report as each query takes about 0.7 seconds. I am caching this at a query level, but it still takes too long for the initial page load.
As I am not very experienced in how to manage datasets such as this, can anyone advise me of a better way to store or query this data? Are there any optimisations I can use with MySQL to improve the performance of these COUNTs, or am I better off using another database or data structure?
We are currently using MySQL with MyISAM tables and a VPS for this, so switching to a new database system altogether isn't out of the question.
You need to use indexes, create them on the columns you do a WHERE on most frequently.
ALTER TABLE `matches` ADD INDEX ( `is_readable` )
etc..
You can also create indexes based on multiple columns, if your doing the same type of query over and over its useful. phpMyAdmin has the index option on the structure page of the table at the bottom.
Add multi index to this table as you are selecting by more than one field. Below index should help a lot. Those type of indexes are very good for boolean / int columns. For indexes with varchar values read more here: http://dev.mysql.com/doc/refman/5.0/en/create-index.html
ALTER TABLE `matches` ADD INDEX ( `is_readable`, `other_flag`, `another_flag` )
One more thing is to check your queries by using EXPLAIN {YOUR WHOLE SQL STATEMENT} to check which index is used by DB. So in this example you should run query:
EXPLAIN ALTER TABLE `matches` ADD INDEX ( `is_readable`, `other_flag`, `another_flag` )
More info on EXPLAIN: http://dev.mysql.com/doc/refman/5.0/en/explain.html
I have a mySQL database with a tad under 2 million rows. The database is non-interactive, so efficiency isn't key.
The (simplified) structure I have is:
`id` int(11) NOT NULL auto_increment
`category` varchar(64) NOT NULL
`productListing` varchar(256) NOT NULL
Now the problem I would like to solve is, I want to find duplicates on productListing field, merge the data on the category field into a single result - deleting the duplicates.
So given the following data:
+----+-----------+---------------------------+
| id | category | productListing |
+----+-----------+---------------------------+
| 1 | Category1 | productGroup1 |
| 2 | Category2 | productGroup1 |
| 3 | Category3 | anotherGroup9 |
+----+-----------+---------------------------+
What I want to end up is with:
+----+----------------------+---------------------------+
| id | category | productListing |
+----+----------------------+---------------------------+
| 1 | Category1,Category2 | productGroup1 |
| 3 | Category3 | anotherGroup9 |
+----+----------------------+---------------------------+
What's the most efficient way to do this either in pure mySQL query or php?
I think you're looking for GROUP_CONCAT:
SELECT GROUP_CONCAT(category), productListing
FROM YourTable
GROUP BY productListing
I would create a new table, inserting the updated values, delete the old one and rename the new table to the old one's name:
CREATE TABLE new_YourTable SELECT GROUP_CONCAT(...;
DROP TABLE YourTable;
RENAME TABLE new_YourTable TO YourTable;
-- don't forget to add triggers, indexes, foreign keys, etc. to new table
SELECT MIN(id), GROUP_CONCAT(category SEPARATOR ',' ORDER BY id), productListing
FROM mytable
GROUP BY
productListing