Refactor PHP script to do mysql update between 2 database servers - php

Question

I'm trying to finish a script that connects to two databases, each on a different server, and preforms an update. Basically, the tables being selected from and inserted to are identical: I did a dump/import the other day. The script needs to keep my local table up to date from the remote once since there will be daily records inserted into the remote one and I need to keep it up to date locally.
The key here is that I'm determining the new rows on the remote server by looking at the Auto-incremented Primary key that the tables share, SESSIONID . I'm trying to get my loop below to say, if the id exists in remote server and not local, then insert those records in local server.
I run the below script in powershell by typing php 'filename', and I get both of my successful connection messages, but then it hangs. After about 10 minutes, it had a memory error so I added ini_set('memory_limit', '256M');. After this it would still hang for about 10 minutes, and then say that MySQL server has gone away and result header couldn't be found, both errors occurring on the line where I check to see if the $rowCount failed.
Note: Replication and large dump/import/table recreations are not an option for us in this situation. We have several similar scripts to this running and we want to keep the same process here. I'm merely looking to resolve these errors or have someone give me a more efficient way of coding this script, perhaps using a max id or something along those lines.
Here's the script:
ini_set('memory_limit', '256M');
// Create connection
$conn = new mysqli($servername, $username, $password);
$conn2 = new mysqli($servername2, $username2, $password2);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
echo "Connected successfully";
// Check connection2
if ($conn2->connect_error) {
die("Connection failed: " . $conn2->connect_error);
}
echo "Connected successfully";
//Start queries
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "select * from cdrdb.session");
// Loop on the results
while($source_item = $source_data->fetch_assoc()) {
// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM ambition.session WHERE SESSIONID = '".$source_item['SESSIONID']."' ");
//if query returns false, rows don't exist with that new ID.
if (!$row_exists){
//Insert new rows into ambition.session
$conn2->query("INSERT INTO ambition.session (SESSIONID,SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,RESPONSIBLEUSEREXTENSIONID,ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)
VALUES ('".$source['SESSIONID']."' ,
'".$source['SESSIONTYPE']."' ,
'".$source['CALLINGPARTYNO']."' ,
'".$source['FINALLYCALLEDPARTYNO']."',
'".$source['DIALPLANNAME']."',
'".$source['TERMINATIONREASONCODE']."',
'".$source['ISCLEARINGLEGORIGINATING']."',
'".$source['CREATIONTIMESTAMP']."',
'".$source['ALERTINGTIMESTAMP']."',
'".$source['CONNECTTIMESTAMP']."',
'".$source['DISCONNECTTIMESTAMP']."',
'".$source['HOLDTIMESECS']."',
'".$source['LEGTYPE1']."',
'".$source['LEGTYPE2']."',
'".$source['INTERNALPARTYTYPE1']."',
'".$source['INTERNALPARTYTYPE2']."',
'".$source['SERVICETYPEID1']."',
'".$source['SERVICETYPEID2']."',
'".$source['EXTENSIONID1']."',
'".$source['EXTENSIONID2']."',
'".$source['LOCATION1']."',
'".$source['LOCATION2']."',
'".$source['TRUNKGROUPNAME1']."',
'".$source['TRUNKGROUPNAME2']."',
'".$source['SESSIONIDTRANSFEREDFROM']."',
'".$source['SESSIONIDTRANSFEREDTO']."',
'".$source['ISTRANSFERINITIATEDBYLEG1']."',
'".$source['SERVICEEXTENSION1']."',
'".$source['SERVICEEXTENSION2']."',
'".$source['SERVICENAME1']."',
'".$source['SERVICENAME2']."',
'".$source['MISSEDUSERID2']."',
'".$source['ISEMERGENCYCALL']."',
'".$source['NOTABLECALLID']."',
'".$source['RESPONSIBLEUSEREXTENSIONID']."',
'".$source['ORIGINALLYCALLEDPARTYNO']."',
'".$source['ACCOUNTCODE']."',
'".$source['ACCOUNTCLIENT']."',
'".$source['ORIGINATINGLEGID']."',
'".$source['SYSTEMRESTARTNO']."',
'".$source['PATTERN']."',
'".$source['HOLDCOUNT']."',
'".$source['AUXSESSIONTYPE']."',
'".$source['DEVICEID1']."',
'".$source['DEVICEID2']."',
'".$source['ISLEG1ORIGINATING']."',
'".$source['ISLEG2ORIGINATING']."',
'".$source['GLOBALCALLID']."',
'".$source['CADTEMPLATEID']."',
'".$source['CADTEMPLATEID2']."',
'".$source['ts']."',
'".$source['INITIATOR']."',
'".$source['ACCOUNTNAME']."',
'".$source['APPNAME']."',
'".$source['CALLID']."',
'".$source['CHRTYPE']."',
'".$source['CALLERNAME']."',
'".$source['serviceid1']."',
'".$source['serviceid2']."')");
}
}

Answer 1

// Check if row exists in destination phone database
$row_exists = $conn2->query("SELECT SESSIONID FROM ambition.session WHERE SESSIONID = '".$source_item['SESSIONID']."' ");
//if query returns false, rows don't exist with that new ID.
if (!$row_exists){
This is incorrect: if query returns false, the query failed to execute. You need to check if ($row_exists->num_rows == 0) instead. The way your code is now, it will always insert every record, again and again. Since you're not checking for errors on the INSERT query you're not noticing the failures you're getting on the duplicate entries for your SESSIONID column (I assume that's the primary key column in your local database as well).
Additionally, it would probably be a lot faster if you only SELECT the sessions you don't already have. Since you're working with an auto increment column, you can pretty much assume that anything in the remote database with a SESSIONID that came after the latest SESSIONID from your local database is new:
//Start queries
$latest_result = $conn2->query("SELECT MAX(`SESSIONID`) FROM `ambition`.`session`");
$latest_row = $latest->fetch_row();
$latest_session_id = $latest_row[0];
//Select All rows from the source phone database
$source_data = mysqli_query($conn, "SELECT * FROM `cdrdb`.`session` WHERE `SESSIONID` > $latest_session_id");
Tables in MySQL are usually sorted by the primary key column by default, but if you're concerned about the order in which you're inserting data in your local database in case your script gets interrupted for some reason, you could add an explicit ORDER BY `SESSIONID` ASC to the query.