I currently have a script that runs every 5 minutes and selects the data from a table on server 1 and an identical table on server2. This is a workaround for replication, essentially, since we don't have that option currently.
The script is successful but I've realized that it misses records sometimes, for whatever reason. The current script selects all records from the destination table, stores the max primary key, selects all data from the source table and then inserts anything with a greater Primary key into the dest. table.
I'd like to modify the script slightly and instead of using max id, just say "if a row has an primary key that doesn't exist in the destination table, insert that row there."
Again these are cloned tables so the structure is the same and they both use AI Primary Keys.
Here's the current working script:
$latest_result = $conn2->query("SELECT MAX(`SESSIONID`) FROM
`ambition`.`session`");
$latest_row = $latest_result->fetch_row();
$latest_session_id = $latest_row[0];
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "SELECT * FROM
`cdrdb`.`session` WHERE `SESSIONID` > $latest_session_id");
// Loop on the results
while($source = $source_data->fetch_assoc()) {
// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM
ambition.session WHERE SESSIONID = '".$source['SESSIONID']."' ") or
die(mysqli_error($conn2));
//if query returns false, rows don't exist with that new ID.
if ($row_exists->num_rows == 0){
//Insert new rows into ambition.session
$stmt = $conn2->prepare("INSERT INTO ambition.session (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE //etc. There are a lot of columns so I
ommitted the others
Is there a way I can slightly modify this to just insert what doesn't exist rather than relying on the MAX ID?
Or is there something here that would be a culprit as to why it's missing records?
You could use INSERT INTO SELECT and check if value is already in target:
INSERT INTO trg_table (cols)
SELECT cols
FROM src_table s
WHERE NOT EXISTS (SELECT 1 FROM trg_table t WHERE t.id = s.id);
Related
I used INSERT INTO SELECT to copy values (multiple rows) from one table to another. Now, my problem is how do I insert rows with its corresponding IDs from different tables (since it's normalized) into a gerund table because it only outputs one row in my gerund table. What should I do to insert multiple rows and their corresponding IDs in the gerund table.
My code for the gerund table goes like this.
$insert = "INSERT INTO table1 SELECT * FROM sourcetable"; // where id1 is pk of table1.
$result =mysqli_query($conn,$insert)
$id1=mysqli_insert_id($conn);
Now table 1 has inserted multiple rows same as the other 2 tables.
Assuming id.. are the foreign keys
INSERT INTO gerundtable (pk, id1,id2,id3) VALUES ($id1,$id2,$id3);
My problem is it doesn't yield multiple rows.
According to MySql documentation:
For a multiple-row insert, LAST_INSERT_ID() and mysql_insert_id() actually return the AUTO_INCREMENT key from the first of the inserted rows. This enables multiple-row inserts to be reproduced correctly on other servers in a replication setup.
So, grab the number of records being copied, and the LAST_INSERT_ID() and you should be able to map exact IDs with each copied row.
In the lines of:
$mysqli->query("Insert Into dest_table Select * from source_table");
$n = $mysqli->affected_rows; // number of copied rows
$id1 = $mysqli->insert_id; // new ID of the first copied row
$id2 = $mysqli->insert_id + 1; // new ID of the second copied row
$id3 = $mysqli->insert_id + 2; // new ID of the third copied row
...
$mysqli->query("INSERT INTO gerundtable (pk, id1,id2,id3) VALUES ($id1,$id2,$id3)");
Thank you for trying to understand and also answering my question. I resolved my own code. I used while loop to get the ids of every row and didn't use INSERT INTO SELECT.
Here is the run down. SInce I'm just using my phone bare with my way posting.
$sqlselect = SELECT * FROM table1;
While($row=mysqli_fetch_array(table1){
$insertquery...
$id1=mysqli_insert_id($conn)
$insertgerundtable = INSERT INTO gerundtable VALUES ( $id1, $id2);
}
I am trying to create a table that logs steps depending on date and the user id. But when I run my code, it happens that I get duplicate rows if a user logs their steps several times a day. I can't have a date with a unique key because that would cause all other users unable to log steps if a any other user has logged steps the same day. So my point is that I want to remove the option of having duplicate rows where user id and date is identical. I have two tables
Table a and table b, and I will refer to them as something.a and something.b
I have a problem with returning a valid row when using $entry = "SELECT * FROM table.a WHERE userid.a = '$user_id.b' AND date=NOW()"
I want to use it as a conditional to decide to either UPDATE or INSERT INTO table.a. I have user_id.b from an previous query which works as it is, so I will leave that as it is for now.
Here is how I query the database:
$entry_result = mysqli_query($conn, $entry);
Which is used here:
if (mysqli_num_rows($entry_result) > 0){
$conn->query("UPDATE steplogger SET steps='$steps' WHERE userid='$user_id' AND date=NOW()");
} else {
$conn->query("UPDATE users SET totalsteps = totalsteps + ('$steps') WHERE username = '$user'");
$conn->query("INSERT INTO steplogger (steps, userid, date) VALUES ('$steps', '$user_id', NOW())");
}
Any thoughts on what I am doing wrong?
PS. When I echo $entry_result I get a mysqli object.
As you said :
I want to remove the option of having duplicate rows where user id and date
The best way is to create an UNIQUE index on user_id and date, this way you won't be able to insert two rows with same user_id and date.
With an UNIQUE index, you can use INSERT...ON DUPLICATE KEY UPDATE that will do what you want : you will insert a new row (new user_id + date) and if a row already exists with the same user_id and date, you will update the row.
Here is the documentation : https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html
You can try like this
if (mysqli_num_rows($entry_result) > 0){
$conn->query("UPDATE steplogger SET steps='$steps' WHERE userid='$user_id' AND date=".NOW().")";
} else {
$conn->query("UPDATE users SET totalsteps = totalsteps + ('$steps') WHERE username = '$user'");
$conn->query("INSERT INTO steplogger (steps, userid, date) VALUES ('$steps', '$user_id', ".NOW()."))";
}
To get current date in NOW() function, you can use this function.
And also format of the two conditions should be same.
Following is my database in mysql:
Id Username Password
1 admin admin
2 jay jay1
3 suman xyza
4 chintan abcde
This is my code in php:
$fetchid = mysql_query(" SELECT MAX(Id) As max From user;");
$row = mysql_fetch_array($fetchid);
$largest = $row['max'];
$largest++;
$user= $_POST['username'];
$pass= $_POST['password'];
$result = mysql_query(" INSERT INTO `proshell`.`user` (
`Id` ,
`Username` ,
`Password`
)"."
VALUES (
'".$largest."', '".$user."', '".$pass."'
);");
Problem:
Now if I delete row with Id=1 and then re-enter the data then it should use ID=1 then Again I reinsert the data it use ID=5
It works like this:
if I delete row with Id=1 and then re-enter the data the Id it gets is 5 but then 1 is free so,
What should I write to perform that task.
First, if you set your Id column to AUTO_INCREMENT you don't need the following part in your code at all:
$fetchid = mysql_query(" SELECT MAX(Id) As max From user;");
$row = mysql_fetch_array($fetchid);
$largest = $row['max'];
$largest++;
Because AUTO_INCREMENT will automatic add new value to your ID colume.
But if you don't set it to AUTO_INCREMENT, the above code will grab the MAXIMUM ID value (in this case, 4).
When you re-enter your data again after you delete the row 1, the MAXIMUM ID still 4, so your new ID value will be 5 (from $largest++;).
.....
If you really need to use consecutive ids as you PK, you need to re-write you code but I suggest you to use UUID for you ID column instead.
You can easily generate UUID by using uuid().
How about the UUID performance? Refer to Dancrumb's answer about this:
A UUID is a Universally Unique ID. It's the universally part that you should be considering here.
Do you really need the IDs to be universally unique? If so, then UUIDs
may be your only choice.
I would strongly suggest that if you do use UUIDs, you store them as a
number and not as a string. If you have 50M+ records, then the saving
in storage space will improve your performance (although I couldn't
say by how much).
If your IDs do not need to be universally unique, then I don't think
that you can do much better then just using auto_increment, which
guarantees that IDs will be unique within a table (since the value
will increment each time)
see. UUID performance in MySQL?
EDIT: I don't suggest you run query on the whole table just to find the MAX ID value before inserting new value everytime, because it will give you a performance penalty (Imagine that if you have million rows and must query on them everytime just to insert a new row, how much workload causes to your server).
It is better to do the INSERT just as INSERT, no more than that.
EDIT2:
If you really want to use consecutive ids, then how about this solution?
Create new TABLE just for store the ids for insert (new ids and the ids that you deleted).
For example:
CREATE TABLE cons_ids (
ids INT PRIMARY KEY,
is_marker TINYINT DEFAULT 0
);
then initial ids with values from 1-100 and set marker to be '1' on some position, e.g. 80th of whole table. This 'marker' uses to fill your ids when it's nearly to empty.
When you need to INSERT new Id to your first table, use:
$result = mysql_query("SELECT ids, marker FROM cons_ids ORDER BY ids ASC LIMIT 1;");
$row = mysql_fetch_row($result);
and use $row[0] for the following code:
INSERT INTO yourtable (Id, Username, Password)
VALUES ($row[0], $username, $password);
DELETE FROM cons_ids
WHERE ids = $row[0];
This code will automatically insert the lowest number in cons_ids as your Id and remove it from the cons_ids table. (so next time you do insert, it will be the next lowest number)
Then following with this code:
if ($row[1] == 1) {
//add new 100 ids start from the highest ids number in cons_ids table
//and set new marker to 80th position again
}
Now each time you delete a row from your first table, you just add the Id from the row that you deleted to cons_ids, and when you do INSERT again, it will use the Id number that you just deleted.
For example: your current ids in cons_ids is 46-150 and you delete row with Id = 14 from first table, this 14 will add to your cons_ids and the value will become 14, and 46-150. So next time you do INSERT to your first table, your Id will be 14!!.
Hope my little trick will help you solve your problem :)
P.S. This is just an example, you can modify it to improve its performance.
First of all, as I understand, you are selecting highest column ID which should be always the last one (since you set auto-increment on ID column).
But what are you trying to do is actually filling up holes after delete query, right?
If you are really looking for such approach, try to bypass delete operation by making new boolean column where you flag record if it is active or not (true/false).
SQL table change:
Id Username Password Active
1 admin admin false
2 jay jay1 true
3 suman xyza false
4 chintan abcde true
PHP request:
$fetchid = mysql_query(" SELECT MIN(Id) As min FROM user WHERE active = false;");
$result = mysql_query(" INSERT INTO `proshell`.`user` (
`Id` ,
`Username` ,
`Password`
`Active`
)"."
VALUES (
'".$largest."', '".$user."', '".$pass."', 'true'
);");
I have a php script that uploads csv files into a mysql database.
The database has several columns. Among these columns is an 'email' field. I wrote some mysql that would remove rows that contained duplicate values in the email column. Below is the mysql:
$sql = "CREATE TABLE new_table as SELECT * FROM auto WHERE 1 GROUP BY email";
mysql_query($sql, $conn);
$query = mysql_query("SELECT COUNT(*) FROM new_table");
list($number) = mysql_fetch_row($query);
$query = mysql_query("SELECT COUNT(*) FROM auto");
list($number2) = mysql_fetch_row($query);
$result = $number2 - $number;
mysql_query("DROP TABLE auto");
mysql_query("RENAME TABLE new_table TO auto");
The code works, it removes duplicate values.
Problem:
It removes rows that contain no values. So it assumes that two or more emails values that are empty are duplicates and removes they're rows.
Question:
How do I tell mysql to ignore empty values.
Thanks for the help.
Edit
The where is my database table. One table.
The when is when I execute the code. I plan on putting in a php file to be executed on demand.
The result I expect is a mysql table without duplicate emails.
Something like this would work for a one-time alteration, by allowing NULL in email, and adding a UNIQUE constraint:
-- set empties to NULL
UPDATE tablename SET email = NULL WHERE LENGTH(email)=0;
-- drop all rows violating the UNIQUE constraint on email:
ALTER IGNORE TABLE tablename ADD UNIQUE (email);
I'm building a database for making hotel reservations. One table called "reservations" holds the general details of the reservation, while another called "rooms" holds details about specific rooms (each reservation has many rooms, each room belongs to only one reservation).
I would like to be able to easily generate duplicate reservations records (except for the primary key, of course). My problem is in generating the rooms data as an array which is then inserted into the rooms table while being associated to its reservation.
I've come as far as the following trivial code (stripped down to the bare essentials for discussion purposes).
if (isset($_POST['action']) and $_POST['action'] == 'Duplicate')
{
include $_SERVER['DOCUMENT_ROOT'] . '/includes/connect.inc.php';
$id = mysqli_real_escape_string($link, $_POST['id']);
// retrieve reservation
$sql = "SELECT type_of_reservation FROM reservations WHERE id='$id'";
$result = mysqli_query($link, $sql);
$row = mysqli_fetch_array($result);
$type_of_reservation = $row['type_of_reservation'];
// create new reservation record
$sql = "INSERT INTO reservations SET type_of_reservation ='$type_of_reservation'";
$id = mysqli_insert_id($link);
// retrieve rooms
$sql = "SELECT reservation_id, in_date FROM rooms WHERE reservation_id='$id'";
$result = mysqli_query($link, $sql);
while ($row = mysqli_fetch_array($result))
{
$rooms[] = array('reservation_id' => $row['reservation_id'], 'in_date' => $row['in_date']);
}
The big question is, now what? Everything I've tried either generates an error or no new entries, and I can't seem to find any discussion that addresses this specific need. Thanks for your help.
PeterC, there is no code listed that shows you inserting the ROOM record information. In the //retrieve room section of your code, you are pulling the data and putting it into an array. If you really want to create a duplicate records, I would use in insert inside the database, then you don't have to pull the records out just to put them back in.
The bit of code you want will be something like this. It will be in place of the //retrieve rooms code you have listed: (psuedo code) [note: $id represents the newly selected id from your sql insert for the duplicated reservation]
INSERT INTO rooms(res_id, other, data)
SELECT $id, other, data FROM rooms WHERE id = $_POST['id'];
This will allow you to duplicate the room data, adding the new reservation_id right inside the database. No need to pull out the records, create inserts, and then put them back in. You can read more about INSERT INTO ... SELECT statements here: http://dev.mysql.com/doc/refman/5.0/en/ansi-diff-select-into-table.html
// create new reservation record
$sql = "INSERT INTO reservations SET type_of_reservation ='$type_of_reservation'";
//ADD HERE CODE BELOW
$id = mysqli_insert_id($link);
with mysql_insert_id you get the inseted id, but you should insert it into db.. so add
mysqli_query($link, $sql);
before retrieving data
If you simply need to duplicate records, you can do it this way:
INSERT INTO
reservations
(
SELECT
null, # assume first column is auto incrementing primary key, so leave null
`all`,
`other`,
`column`,
`names`
FROM
reservations
WHERE
reservation_id = $oldReservationId # id of reservation to duplicate
)
Then for the rooms use the last inserted id (for instance retrieved with mysql_insert_id), like this:
INSERT INTO
rooms
(
SELECT
null, # assume first column is auto incrementing primary key, so leave null
$newReservationId, # this is the new reservation id
`all`,
`other`,
`column`,
`names`
FROM
rooms
WHERE
reservation_id = $oldReservationId # id of reservation to duplicate
)