Background: A group of students including me are creating a website for a merit system for our school. We basically had no experience in html, css, php or sql databases at the start. I will say we've managed decently.
However, we've run into a problem with updating the database. Every week, we need to merge the table for students from a csv, which is updated with classes and all from a department of education system. So there are thousands of students and each of them have certain details that when updated can change but others that need to stay the same. For example, we have the number of merits which needs to stay the same when we update but details such as their year group or classes can change. The thing is, is that this update is done of a csv file which may also have new students that need to be updated to the database and the csv is not in any order. So we are trying to find a solution which allows one field to be the same but allow others to change.
So we've tried LOAD DATA INFILE with a unique key but the problem we found was that while it preserved the merits, it ignored the other details and just added new students.
EDIT: So the students have a class field which contains the classes of the student. When we import, we need to be able to change that but still maintain the merit field which will not be in the csv file. We understand that we could just manually import it but suspect that will take a long time to run.
Our code will be in PHP by the way.
tl;dr
Need to find out how to merge/import databases from csv, changing only specific fields of existing students and also adding new students
Thanks in advance for your help and sorry for the long post. This is my first post so do forgive me if I make any mistakes.
You will need a php script which reads the contents of the file, line by line preferably since it is a CSV. You will then need to use the primary key unique identifier from each CSV line to check whether a record with that primary key value already exists in the database. If a record with that primary key exists in the database, use the UPDATE query to update the merits only. If a record cannot be found with that primary key, use the INSERT statement to add that new record to the database.
NOTE: Bolded bits are steps required, in chronological order.
Related
Here goes a very basic question.
Here is my stroll in trying to create a form for work:
I created the HTML form. Added some JavaScript to make it do some things I needed. Stylized it with CSS, wrote PHP code and created a Database (I had no idea of how to do it at first) for the entered data to be saved.
I didn't know how to do any of that, but in the past two weeks I've managed to make it exactly the way I needed, and I'm very pleased with myself. After a lot of work, the form perfectly sends the data to the database and displays it in the page after you hit submit, and also looks really good too.
The thing is... this that I am creating is an Activities Bank for us to use here at work (I teach English) and the page (base) I have created is only ONE of MANY that are needed in this data bank. Let me explain... Let's say the page I've created is the post and display of, say, Book3 Chapter1 Activities, and I need to have many other pages (which will be exact copies of this one).
My question is... will I have to create (actually, it's copy and paste) new databases/tables manually (which will be more than one hundred of them) or there is a way to automatize this process?
I mean, all the pages will share the same variables and the same form... the only different thing will be the title and the entered data, of course.
Will I have to create a database for each page? Or a new table for each page in the same database?
If you still don't understand what I need, here is how this is supposed to be:
Book1 has 40 chapters, so, 40 copies of the same form (which already works fine);
PLUS
Book2 that has 40 more chapters, etc.
Thanks in advance for any clarification.
Sorry if this is such a basic question, but if it isn't, if otherwise, what I wanna do is very complicated, I don't mind that I don't know much about all this and I will take the challenge, like I have, when I was making this form from scratch, without ever hearing about "databases". Any words of help are appreciated.
That isn't how databases or tables work. You should be creating a new row in one or more tables for each form submission. You should almost never be dynamically creating tables, and even less often databases.
It sounds like you want a books table, and a chapters table. Each row in the books table will have many rows in the chapters table "pointing" to it via foreign keys.
I think in your case your case two tables.in total.
As you have already added one, you need another one, where the first one will contain the common data with a primary key column and the other table will keep the primary key of the first table and the data which occurs in multiple. Then later you needs to join (Sql join)rotatable to get your data.
I'm looking to create a PHP script that creates a new table within a database that would be tied to a label and then within the table there would be rows of data relating to the status of the label. However, I'm not sure how I can get the PHP script (or MySQL) to increment the name of the table. All I can find is a lot of detail on auto incrementing columns for rows.
Thoughts?
You're doing it wrong. If you have scripts that, during the project live phase, create and delete regular tables, more often than not it is an indicator of bad design.
If you're keen on OOP, you may consider a table like a Class definition, and each row as an object (or an entity, if you wish) - i know it is a stretch, but it has some similarities.
Take some time to read about database normalization and database design, this project and everyone after this will benefit much more than spending time to research a working solution for the current problem you are facing.
I have 2 sql servers on 2 diferent locations.
One is a web server and the other a crm system.
People update and register on web, when they do changes i need to insert or update the changes to my crm server.
I have a view on web server where i can select from but i need to
insert into on duplicate update only fields that changed and then in a description
show
wich fields were updated?
I have no clue how to start.
You can not determine the differences on fields after changing them.
You can however select and store the contents prior to the update and then compare it with the new contents.
The question then becomes: Do you need the differences per column?
If yes: Pre-select and do the difference yourself (in the
application).
If no: Use the method described by #Ogelami (and accept his answer :)
On a side note: The Pre-Select thing won't work as well, when you start using several mysql servers, since you might run into issues with drifting data (ie one server is behind in inserted data). When this occurs, the method will get a bit more complex.
Perhaps something like this?
INSERT INTO table ON DUPLICATE UPDATE table SET field = value WHERE field != 'value'
and you might want to look into this to see if there are Affected rows.
it is possible to create column at run time in mysql table using add column in php.
but how to insert data at run time into the newly created field(s) (i.e.column(s))?
thanks in advance
EDIT:
some said that why i need to create a new column at run time as i know before hand that the column will be necessary. No, columns quite unknown at the beginning may be needed to be created Just take the problem as a programming issue.
PLZ GIVE THE EXPLANATION OF -VE RATING
i donno why my post was rated -ve ly. uff where is the solution to the prob that someone needs to create columns in a mysql table beforehand and the thing is quite unpredictable. Suppose a site admin needs to add column to a membership club table from time to time. Can't this be any real situation? Why is there no explanation of -ve rating , plz?
Oh that,
You know you need to create the address column already. Then what you need to do is do a 2 step registration.
You need to check whether this column is empty when the student logs in. If it is empty then you ask the student to put in their address. When they have typed in the address the just update the column and change it from blank to the address provided by the student. You dont need the dynamic columns as that would be slow and complex operation.
Check database for the address of the student or check if the user is loggin in the first time
If the users address is empty or the user is logging in the first time
Then
Ask the user for their address
If the address is given then
Update the address column with the provided address
Endif
Endif
Done
Yes, you can execute a MySQL command in PHP code to add a column, create a table, alter a table and more.
Use a function that alters your table and which returns true, if the altering was successfull. If you get this, you can use normal UPDATE functions.
But you should never, never, never design your application this way. Think about all the ibformations, that you will get now and in the future. Then design your tables referring to this informations.
In your special case you already know, that you will aks for the address informations, so implement this NOW in your code. user658911 showed you how to do this.
In the future it might be neccessary to add more columns or tables to your system, because the requirements will change. If so, you can realize those changes on your development system, test them with your current database content and after all deploy it to your production system.
Altering a table during runtime should never be an option.
I want to create a simple auditing system for my small CodeIgniter application. Such that it would take a snapshot of a table entry before the entry has been edited. One way I could think of would be to create a news_audit table, which would replicate all the columns in the news table. It would also create a new record for each change with the added column of date added. What are your views, and opinions of building such functionality into a PHP web application?
There are a few things to take into account before you decide which solution to use:
If your table is large (or could become large) your audit trail needs to be in a seperate table as you describe or performance will suffer.
If you need an audit that can't (potentially) be modified except to add new entries it needs to have INSERT permissions only for the application (and to be cast iron needs to be on a dedicated logging server...)
I would avoid creating audit records in the same table as it might be confusing to another developer (who might no realize they need to filter out the old ones without dates) and will clutter the table with audit rows, which will force the db to cache more disk blocks than it needs to (== performance cost). Also to index this properly might be a problem if your db does not index NULLS. Querying for the most recent version will involve a sub-query if you choose to time stamp them all.
The cleanest way to solve this, if your database supports it, is to create an UPDATE TRIGGER on your news table that copies the old values to a seperate audit table which needs only INSERT permissions). This way the logic is built into the database, and so your applications need not be concerned with it, they just UPDATE the data and the db takes care of keeping the change log. The body of the trigger will just be an INSERT statement, so if you haven't written one before it should not take long to do.
If I knew which db you are using I might be able to post an example...
What we do (and you would want to set up archiving beforehand depending on size and use), but we created an audit table that stores user information, time, and then the changes in XML with the table name.
If you are in SQL2005+ you can then easily search the XML for changes if needed.
We then added triggers to our table to catch what we wanted to audit (inserts, deletes, updates...)
Then with simple serialization we are able to restore and replicate changes.
What scale are we looking at here? On average, are entries going to be edited often or infrequently?
Depending on how many edits you expect for the average item, it might make more sense to store diff's of large blocks of data as opposed to a full copy of the data.
One way I like is to put it into the table itself. You would simply add a 'valid_until' column. When you "edit" a row, you simply make a copy of it and stamp the 'valid_until' field on the old row. The valid rows are the ones without 'valid_until' set. In short, you make it copy-on-write. Don't forget to make your primary keys a combination of the original primary key and the valid_until field. Also set up constraints or triggers to make sure that for each ID there can be only one row that does not have it's valid_until set.
This has upsides and downsides. The upside is less tables. The downside is far more rows in your tables. I would recommend this structure if you often need to access old data. By simply adding a simple WHERE to your queries you can query the state of a table at a previous date/time.
If you only need to access your old data occasionally then I would not recommend this though.
You can take this all the way to the extreme by building a Temportal database.
In small to medium size project I use the following set of rules:
All code is stored under Revision Control System (i.e. Subversion)
There is a directory for SQL patches in source code (i.e. patches/)
All files in this directory start with serial number followed by short description (i.e. 086_added_login_unique_constraint.sql)
All changes to DB schema must be recorded as separate files. No file can be changed after it's checked in to version control system. All bugs must be fixed by issuing another patch. It is important to stick closely to this rule.
Small script remembers serial number of last executed patch in local environment and runs subsequent patches when needed.
This way you can guarantee, that you can recreate your DB schema easily without the need of importing whole data dump. Creating such patches is no brainer. Just run command in console/UI/web frontend and copy-paste it into patch if successful. Then just add it to repo and commit changes.
This approach scales reasonably well. Worked for PHP/PostgreSQL project consisting of 1300+ classes and 200+ tables/views.