Ok so i have a database table called requests with this structure
mysql> desc requests;
+------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| artist | varchar(255) | YES | | NULL | |
| song | varchar(255) | YES | | NULL | |
| showdate | date | YES | | NULL | |
| amount | float | YES | | NULL | |
+------------+--------------+------+-----+---------+----------------+
Here is some example data
+----+-----------+-------------------------+------------+--------+
| id | artist | song | showdate | amount |
+----+-----------+-------------------------+------------+--------+
| 6 | Metallica | Hello Cruel World | 2010-09-15 | 10.00 |
| 7 | someone | Some some | 2010-09-18 | 15.00 |
| 8 | another | Some other song | 2010-11-10 | 45.09 |
+----+-----------+-------------------------+------------+--------+
I need a way to be able to give user a way to upload a csv with the same structure and it updates or inserts based on whats in the csv. I have found many scripts online but most have a hard coded csv which is not what i need. I need the user to be able to upload the csv...Is that easy with php....
Here is an example csv
id artist song showdate amount
11 NewBee great stuff 2010-09-15 12.99
10 anotherNewbee even better 2010-09-16 34.00
6 NewArtist New song 2010-09-25 78.99
As you can see i have id 6 which is already in the database and needs to be updated..The other two will get inserted
I am not asking for someone to write the whole script but if i can get some direction on the upload and then where to go from there....thanks
Create store procedure as below and test it. It is works
CREATE proc csv
(
#id int,
#artist varchar(50),
#songs varchar(100),
#showdate datetime,
#amount float
)
as
set nocount on
if exists (select id from dummy1 where id=#id) -- Note that dummy1 as my table.
begin
update dummy1 set artist= #artist where id=#id
update dummy1 set songs=#songs where id=#id
update dummy1 set showdate=#showdate where id=#id
update dummy1 set amount=#amount where id=#id
end
else
insert into dummy1 (artist,songs,showdate,amount)values(#artist,#songs,#showdate,#amount)
Go
upload the file to a directory using move_uploaded_file
use fgetcsv to read the uploaded csv and process each row as you like.
delete the csv file
Related
I have a table called ratings with the following fields:
+-----------+------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------+------+-----+---------+----------------+
| rating_id | bigint(20) | NO | PRI | NULL | auto_increment |
| user_id | int(11) | NO | MUL | NULL | |
| movie_id | int(11) | NO | | NULL | |
| rating | float | NO | | NULL | |
+-----------+------------+------+-----+---------+----------------+
Indexes on this table:
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| ratings | 0 | PRIMARY | 1 | rating_id | A | 100076 | NULL | NULL | | BTREE | | |
| ratings | 0 | user_id | 1 | user_id | A | 564 | NULL | NULL | | BTREE | | |
| ratings | 0 | user_id | 2 | movie_id | A | 100092 | NULL | NULL | | BTREE | | |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
I have another table called movie_average_ratings which has the following fields:
+----------------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------+------+-----+---------+-------+
| movie_id | int(11) | NO | PRI | NULL | |
| average_rating | float | NO | | NULL | |
+----------------+---------+------+-----+---------+-------+
As it is obvious by this point I want to calculate the average rating of movies from ratings table and update the movie_average_ratingstable. I tried the following SQL query.
UPDATE movie_average_ratings
SET average_rating = (SELECT AVG(rating)
FROM ratings
WHERE ratings.movie_id = movie_average_ratings.movie_id);
Currently, there are around 10,000 movie records and 100,000 rating records and I get Lock wait timeout exceeded; try restarting transaction error. The number of records can grow significantly so I don't think increase timeout is a good solution.
So, how can I write 'scalable' query to acheive this? Is iterating the movie_average_ratings table records and calculate averages individually the most efficient solution to this?
Without an explain, it's hard to be clear on what's holding you up. It's also not clear that you will get a performance improvement by storing this aggregated data as a denormalized table - if the query to calculate the ratings executes in 0.04 seconds, it's unlikely querying your denormalized table will be much faster.
In general, I recommend only denormalizing if you know you have a performance problem.
But that's not the question.
I would do the following:
delete from movie_average_ratings;
insert into movie_average_ratings
Select movie_ID, avg(rating)
from ratings
group by movie_id;
I just found something in another post:
What is happening is, some other thread is holding a record lock on
some record (you're updating every record in the table!) for too long,
and your thread is being timed out.
This means that some of your records are locked you can force unlock them in the console:
1) Enter MySQL mysql -u your_user -p
2) Let's see the list of locked tables mysql> show open tables where in_use>0;
3) Let's see the list of the current processes, one of them is locking
your table(s) mysql> show processlist;
4) Kill one of these processes mysql> kill put_process_id_here;
You could redesign the movie_average_ratings table to
movie_id (int)
sum_of_ratings (int)
num_of_ratings (int)
Then, if a new rating is added you can add it to movie_average_ratings and calculate the average if needed
I'm trying to make a small logging table on my database.
Users
+----+------+
| id | name |
+----+------+
| 1 | FOO |
| 2 | BAR |
| 3 | LOS |
+----+------+
Log_Users
+-------------+-------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| old_id | int(11) | YES | | NULL | |
| old_name | varchar(100) | YES | | NULL | |
| new_id | int(11) | YES | | NULL | |
| new_name | varchar(100) | YES | | NULL | |
| action_type | enum('C','U','D') | YES | | NULL | |
| time | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| doers | int(11) | YES | | NULL | |
+-------------+-------------------+------+-----+-------------------+-----------------------------+
I have a small application created using PHP to save user's id into session. How do i send this user's id value (on PHP's session) to a trigger of one of the tables to log their activities, like deleting another users or updating them? I've tried to use a trigger on log table to do all of the things, something like this.
CREATE TRIGGER userTrigger BEFORE INSERT ON Log_Users FOR EACH ROW
BEGIN
IF(new.action_type = 'C') THEN
INSERT INTO Users(id, name) VALUE(new.new_id, new.new_name);
ELSEIF(new.action_type = 'U') THEN
UPDATE Users SET id = new.new_id, name = new.new_name WHERE id = new.old_id;
ELSEIF(new.action_type = 'D') THEN
SET new.old_name = (SELECT name FROM Users WHERE id = new.old_id);
DELETE FROM Users WHERE id = new.old_id;
END IF;
END~
But, I'm struggling on the problem when users updating multiple records on the same column. At the end, what is and how to make an optimal activities logging using PHP and MySQL and how to do it? I have no solution for this problem for now. Thank you.
I've never done this using triggers so I can't help you with that sadly. How I usually do this:
Your users should NEVER have direct access to mysql or phpmyadmin, they should create, edit, delete and anything else using a PHP script you provide. This way you have total control over what your users can and can't do, and you narrow a lot the posible actions performed, so creating logs of them is much easier. For example:
You have a php scrip that users use to do some stuff and insert a new row, right after that you do a insert on the log table recording this last action.
I'm creating a portfolio website that has galleries that contain images. I want the user of this portfolio to be able to order the images within a gallery. The problem itself is fairly simple I'm just struggling with deciding on a solution to implement.
There are 2 solutions I've thought of so far:
Simply adding an order column (or priority?) and then querying with an ORDER BY clause on that column. The disadvantage of this being that to change the order of a single image I'd have to update every single image in the gallery.
The second method would be to add 2 nullable columns next and previous that simply store the ID of the next and previous image. This would then mean there would be less data to update when the order was changed; however, it would be much more complex to set up and I'm not entirely sure how I'd actually implement it.
Extra options would be great.
Are those options viable?
Are there better options?
How could / should they be implemented?
The current structure of the two tables in question is the following:
mysql> desc Gallery;
+--------------+------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+------------------+------+-----+-------------------+-----------------------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| title | varchar(255) | NO | | NULL | |
| subtitle | varchar(255) | NO | | NULL | |
| description | varchar(5000) | NO | | NULL | |
| date | datetime | NO | | NULL | |
| isActive | tinyint(1) | NO | | NULL | |
| lastModified | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+--------------+------------------+------+-----+-------------------+-----------------------------+
mysql> desc Image;
+--------------+------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+------------------+------+-----+-------------------+-----------------------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| galleryId | int(10) unsigned | NO | MUL | NULL | |
| description | varchar(250) | YES | | NULL | |
| path | varchar(250) | NO | | NULL | |
| lastModified | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+--------------+------------------+------+-----+-------------------+-----------------------------+
Currently there is no implementation of ordering in any form.
while 1 is a bit ugly you can do:
UPDATE table set order=order+1 where order>='orderValueOfItemYouCareAbout';
this will update all the rest of the images and you wont have to do a ton of leg work.
As bart2puck has said and I stated in the question, option 1 is a little bit ugly; it is however the option I have chosen to go with to simplify the solution all round.
I have added a column (displayOrder int UNSIGNED) to the Image table after path. When I want to re-order a row in the table I simply swap rows around. So, if I have 3 rows:
mysql> SELECT id, galleryId, description, displayOrder FROM Image ORDER BY displayOrder;
+-----+-----------+----------------------------------+--------------+
| id | galleryId | description | displayOrder |
+-----+-----------+----------------------------------+--------------+
| 271 | 20 | NULL | 1 |
| 270 | 20 | Tracks leading into the ocean... | 2 |
| 278 | 20 | NULL | 3 |
+-----+-----------+----------------------------------+--------------+
3 rows in set (0.00 sec)
If I want to re-order row 278 to appear second rather than third, I'll simply swap it with the second by doing the following:
UPDATE Image SET displayOrder =
CASE displayOrder
WHEN 2 THEN 3
WHEN 3 THEN 2
END
WHERE galleryId = 20
AND displayOrder BETWEEN 2 AND 3;
Resulting in:
mysql> SELECT id, galleryId, description, displayOrder FROM Image ORDER BY displayOrder;
+-----+-----------+----------------------------------+--------------+
| id | galleryId | description | displayOrder |
+-----+-----------+----------------------------------+--------------+
| 271 | 20 | NULL | 1 |
| 278 | 20 | NULL | 2 |
| 270 | 20 | Tracks leading into the ocean... | 3 |
+-----+-----------+----------------------------------+--------------+
3 rows in set (0.00 sec)
One possible issue that some people may find is that you can only alter the position by one place with this method, i.e. to move image 278 to appear first I'd have to make it second, then first, otherwise the current first image would appear third.
This question is for my pastebin app written in PHP.
I did a bit of a research, although I wasn't able to find a solution that matches my needs. I have a table with this structure:
+-----------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+----------------+
| id | int(12) unsigned | NO | PRI | NULL | auto_increment |
| author | varchar(50) | YES | | | |
| authorid | int(12) unsigned | YES | | NULL | |
| project | varchar(50) | YES | | | |
| timestamp | int(11) unsigned | NO | | NULL | |
| expire | int(11) unsigned | NO | | NULL | |
| title | varchar(25) | YES | | | |
| data | longtext | NO | | NULL | |
| language | varchar(50) | NO | | php | |
| password | varchar(60) | NO | | NULL | |
| salt | varchar(5) | NO | | NULL | |
| private | tinyint(1) | NO | | 0 | |
| hash | varchar(12) | NO | | NULL | |
| ip | varchar(50) | NO | | NULL | |
| urlkey | varchar(8) | YES | MUL | | |
| hits | int(11) | NO | | 0 | |
+-----------+------------------+------+-----+---------+----------------+
This is for a pastebin application. I basically want paste revisions so that if you open paste #1234, it shows all past revisions of that paste.
I thought of three ways:
Method 1
Have a revisions table with id and old_id or something and for each ID, I would insert all old revisions, so if my structure looks like this:
rev3: 1234
rev2: 1233
rev1: 1232
The table will contain this data:
+-------+----------+
| id | old_id |
+-------+----------+
| 1234 | 1233 |
| 1234 | 1232 |
| 1233 | 1232 |
+-------+----------+
The problem which I have with this is that it introduces a lot of duplicate data. And the more the revisions get, it has not only more data but I need to do N inserts for each new paste to the revisions table which is not great for a large N.
Method 2
I can add a child_id to the paste table at the top and just update that. And then, when fetching the paste, I will keep querying the db for each child_id and their child_id and so on... But the problem is, that will introduce too many DB reads each time a paste with many revisions is opened.
Method 3
Also involves a separate revisions table, but for the same scenario as method 1, it will store the data like this:
+-------+-----------------+
| id | old_id |
+-------+-----------------+
| 1234 | 1233,1232 |
| 1233 | 1232 |
+-------+-----------------+
And when someone opens paste 1234, I'll use an IN clause to fetch all child paste data there.
Which is the best approach? Or is there a better approach? I am using Laravel 4 framework that has Eloquent ORM.
EDIT: Can I do method 1 with a oneToMany relationship? I understand that I can use Eager Loading to fetch all the revisions, but how can I insert them without having to do a dirty hack?
EDIT: I figured out how to handle the above. I'll add an answer to close this question.
If you are on Laravel 4, give Revisionable a try. This might suite your needs
So here is what I am doing:
Say this is the revision flow:
1232 -> 1233 -> 1234
1232 -> 1235
So here is what my revision table will look like:
+----+--------+--------+
| id | new_id | old_id |
+----+--------+--------+
| 1 | 1233 | 1232 |
| 2 | 1234 | 1233 |
| 3 | 1234 | 1232 |
| 4 | 1235 | 1232 |
+----+--------+--------+
IDs 2 and 3 show that when I open 1234, it should show both 1233 and 1232 as revisions on the list.
Now the implementation bit: I will have the Paste model have a one to many relationship with the Revision model.
When I create a new revision for an existing paste, I will run a batch insert to add not only the current new_id and old_id pair, but pair the current new_id with all revisions that were associated with old_id.
When I open a paste - which I will do by querying new_id, I will essentially get all associated rows in the revisions table (using a function in the Paste model that defines hasMany('Revision', 'new_id')) and will display to the user.
I am also thinking about displaying the author of each revision in the "Revision history" section on the "view paste" page, so I think I'll also add an author column to the revision table so that I don't need to go back and query the main paste table to get the author.
So that's about it!
There are some great packages to help you keeping model revisions:
If you only want to keep the models revisions you can use:
Revisionable
If you also want to log any other actions, whenever you want, with custom data, you can use:
Laravel Activity Logger
Honorable mentions:
Activity Log. It also has a lot of options.
I have a mysql table called "Data",
+---------+------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+-------------------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| data | text | YES | | NULL | |
| created | timestamp | NO | MUL | CURRENT_TIMESTAMP | |
+---------+------------------+------+-----+-------------------+----------------+
The field "data" has values like this:
606 | {"first_name":"JOHN","last_name":"SLIFKO","address":"123 main AVE","city":"LAKEWOOD","state":"OH","zip":"20190","home_phone":2165216359,"email":"john#gmail.com",} | 2012-12-04 16:37:23 |
So, it is saving the records in a JSON Format from a PHP Script that I have.
THIS IS THE THING:
How can I structure this table to make faster searchs or consults by every single field like doing searches or queries like:
SELECT * FROM Data WHERE first_name = john;
how can I do this???
Help please......
Yikes. Not a good design. About the best you could do is use the like keyword
Select * from Data Where data like '%"first_name":"JOHN"%'