Speeding up an insert/ignore between database server connections - php

Question

I've successfully refactored the below script to select records from a table on server 1, and then connect to server 2 and insert/ignore the missing records into the cloned table there.
This works but takes about 1.5 minutes to run. I'm hoping someone can help with a slightly faster and more efficient way of doing this, since it's successful but expensive.
I don't have the option to do federated storage or replication, so this has to be done with a script. I previously did this by using the max ID of the source table but after the insert I was missing up to 15 records a day.
Here's the script:
$source_data = mysqli_query($conn,
"select * from `cdrdb`.`session` where ts >= now() - INTERVAL 1 DAY");
while($source = $source_data->fetch_assoc()) {
//Insert new rows into ambition.session
$stmt = $conn2->prepare(
"INSERT IGNORE INTO ambition.session (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)
VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
or die(mysqli_error($conn2)) ;
mysqli_stmt_bind_param($stmt,
"iisssiissssiiiiiiiiissssiiissssiiiisssiisiiiiiiiiisisssisii"
,$source['SESSIONID']
,$source['SESSIONTYPE']
,$source['CALLINGPARTYNO']
//omitting other columns for sake of space
);
$stmt->execute() or die(mysqli_error($conn2));
}

Answer 1

A simple improvement would be to move the call to prepare() to before the loop. Since the prepared statement is the same each time through the loop, there's no need to contact the DB server each time.
You can also move the call to bind_param() outside the loop, since the variables are the same each time. bind_param binds to a reference, so updating the variable will change what gets inserted when you call execute().
However, these are likely to make only a small difference. One of the most effective ways to improve speed of INSERT queries is to insert multiple rows at a time. This is much easier to do with PDO than mysqli, because you can provide an array of parameters in the call to $stmt->execute(). The code would look like:
$params = array();
$count = 0;
$batch_size = 100;
$placeholders = implode(", ", array_fill(0, $batch_size, "(SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)"));
$stmt = $pdo->prepare("INSERT INTO ambition.sentence (<columns>) VALUES $placeholders");
while ($row = $source_data->fetch(PDO::FETCH_NUM)) {
$params += $row; // Append this row to $params
$count++;
if ($count != $batch_size) {
continue;
}
$stmt->execute($params);
// Reset variables for next batch
$params = array();
$count = 0;
}
if ($count) { // Handle the last batch that isn't the full size
$placeholders = implode(", ", array_fill(0, $count, "( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"));
$stmt = $pdo->prepare("INSERT INTO ambition.sentence (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2) VALUES $placeholders");
$stmt->execute($params);
}
For this to work as I've written it, you need to ensure that the columns returned by the SELECT query are in the same order as the list you're inserting. Avoid using SELECT * when doing this, so you don't get any surprises if there's a change in the schema of the source table.

Answer 2

You probably know by now that inserting one row at a time, one transaction per row, is the slowest way to load data.
I have tested different ways of loading data, and I gave a talk at Percona Live 2017:
Load Data Fast!
TL;DR: Use LOAD DATA INFILE, even if you have to dump your source data into a CSV file first.
Here's an example, though I have not tested it:
$all_columns = "
`SESSIONID`, `SESSIONTYPE`, `CALLINGPARTYNO`, `FINALLYCALLEDPARTYNO`,
`DIALPLANNAME`, `TERMINATIONREASONCODE`, `ISCLEARINGLEGORIGINATING`,
`CREATIONTIMESTAMP`, `ALERTINGTIMESTAMP`, `CONNECTTIMESTAMP`,
`DISCONNECTTIMESTAMP`, `HOLDTIMESECS`, `LEGTYPE1`, `LEGTYPE2`,
`INTERNALPARTYTYPE1`, `INTERNALPARTYTYPE2`, `SERVICETYPEID1`,
`SERVICETYPEID2`, `EXTENSIONID1`, `EXTENSIONID2`, `LOCATION1`, `LOCATION2`,
`TRUNKGROUPNAME1`, `TRUNKGROUPNAME2`, `SESSIONIDTRANSFEREDFROM`,
`SESSIONIDTRANSFEREDTO`, `ISTRANSFERINITIATEDBYLEG1`, `SERVICEEXTENSION1`,
`SERVICEEXTENSION2`, `SERVICENAME1`, `SERVICENAME2`, `MISSEDUSERID2`,
`ISEMERGENCYCALL`, `NOTABLECALLID`, `RESPONSIBLEUSEREXTENSIONID`,
`ORIGINALLYCALLEDPARTYNO`, `ACCOUNTCODE`, `ACCOUNTCLIENT`, `ORIGINATINGLEGID`,
`SYSTEMRESTARTNO`, `PATTERN`, `HOLDCOUNT`, `AUXSESSIONTYPE`, `DEVICEID1`,
`DEVICEID2`, `ISLEG1ORIGINATING`, `ISLEG2ORIGINATING`, `GLOBALCALLID`,
`CADTEMPLATEID`, `CADTEMPLATEID2`, `ts`, `INITIATOR`, `ACCOUNTNAME`, `APPNAME`,
`CALLID`, `CHRTYPE`, `CALLERNAME`, `serviceid1`, `serviceid2`"
$select_sql = "
SELECT $all_columns FROM `cdrdb`.`session`
WHERE ts >= NOW() - INTERVAL 1 DAY";
$source_data = mysqli_query($conn, $select_sql);
$tmpfilename = tempnam('/tmp', 'data');
fp = fopen($tmpfilename, 'w'); // do proper error handling and use
while ($source = $source_data->fetch_assoc()) {
fputcsv($fp, array_values($source));
}
fclose($fp);
$tmpfilename = mysqli_real_escape_string($conn2, $tmpfilename);
$load_sql = "
LOAD DATA LOCAL INFILE '$tmpfilename'
IGNORE INTO TABLE `ambition`.`session`
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
( $all_columns )";
$conn2->query($load_sql);