Related
I've successfully refactored the below script to select records from a table on server 1, and then connect to server 2 and insert/ignore the missing records into the cloned table there.
This works but takes about 1.5 minutes to run. I'm hoping someone can help with a slightly faster and more efficient way of doing this, since it's successful but expensive.
I don't have the option to do federated storage or replication, so this has to be done with a script. I previously did this by using the max ID of the source table but after the insert I was missing up to 15 records a day.
Here's the script:
$source_data = mysqli_query($conn,
"select * from `cdrdb`.`session` where ts >= now() - INTERVAL 1 DAY");
while($source = $source_data->fetch_assoc()) {
//Insert new rows into ambition.session
$stmt = $conn2->prepare(
"INSERT IGNORE INTO ambition.session (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)
VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
or die(mysqli_error($conn2)) ;
mysqli_stmt_bind_param($stmt,
"iisssiissssiiiiiiiiissssiiissssiiiisssiisiiiiiiiiisisssisii"
,$source['SESSIONID']
,$source['SESSIONTYPE']
,$source['CALLINGPARTYNO']
//omitting other columns for sake of space
);
$stmt->execute() or die(mysqli_error($conn2));
}
A simple improvement would be to move the call to prepare() to before the loop. Since the prepared statement is the same each time through the loop, there's no need to contact the DB server each time.
You can also move the call to bind_param() outside the loop, since the variables are the same each time. bind_param binds to a reference, so updating the variable will change what gets inserted when you call execute().
However, these are likely to make only a small difference. One of the most effective ways to improve speed of INSERT queries is to insert multiple rows at a time. This is much easier to do with PDO than mysqli, because you can provide an array of parameters in the call to $stmt->execute(). The code would look like:
$params = array();
$count = 0;
$batch_size = 100;
$placeholders = implode(", ", array_fill(0, $batch_size, "(SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)"));
$stmt = $pdo->prepare("INSERT INTO ambition.sentence (<columns>) VALUES $placeholders");
while ($row = $source_data->fetch(PDO::FETCH_NUM)) {
$params += $row; // Append this row to $params
$count++;
if ($count != $batch_size) {
continue;
}
$stmt->execute($params);
// Reset variables for next batch
$params = array();
$count = 0;
}
if ($count) { // Handle the last batch that isn't the full size
$placeholders = implode(", ", array_fill(0, $count, "( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"));
$stmt = $pdo->prepare("INSERT INTO ambition.sentence (SESSIONID,
SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
RESPONSIBLEUSEREXTENSIONID,
ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2) VALUES $placeholders");
$stmt->execute($params);
}
For this to work as I've written it, you need to ensure that the columns returned by the SELECT query are in the same order as the list you're inserting. Avoid using SELECT * when doing this, so you don't get any surprises if there's a change in the schema of the source table.
You probably know by now that inserting one row at a time, one transaction per row, is the slowest way to load data.
I have tested different ways of loading data, and I gave a talk at Percona Live 2017:
Load Data Fast!
TL;DR: Use LOAD DATA INFILE, even if you have to dump your source data into a CSV file first.
Here's an example, though I have not tested it:
$all_columns = "
`SESSIONID`, `SESSIONTYPE`, `CALLINGPARTYNO`, `FINALLYCALLEDPARTYNO`,
`DIALPLANNAME`, `TERMINATIONREASONCODE`, `ISCLEARINGLEGORIGINATING`,
`CREATIONTIMESTAMP`, `ALERTINGTIMESTAMP`, `CONNECTTIMESTAMP`,
`DISCONNECTTIMESTAMP`, `HOLDTIMESECS`, `LEGTYPE1`, `LEGTYPE2`,
`INTERNALPARTYTYPE1`, `INTERNALPARTYTYPE2`, `SERVICETYPEID1`,
`SERVICETYPEID2`, `EXTENSIONID1`, `EXTENSIONID2`, `LOCATION1`, `LOCATION2`,
`TRUNKGROUPNAME1`, `TRUNKGROUPNAME2`, `SESSIONIDTRANSFEREDFROM`,
`SESSIONIDTRANSFEREDTO`, `ISTRANSFERINITIATEDBYLEG1`, `SERVICEEXTENSION1`,
`SERVICEEXTENSION2`, `SERVICENAME1`, `SERVICENAME2`, `MISSEDUSERID2`,
`ISEMERGENCYCALL`, `NOTABLECALLID`, `RESPONSIBLEUSEREXTENSIONID`,
`ORIGINALLYCALLEDPARTYNO`, `ACCOUNTCODE`, `ACCOUNTCLIENT`, `ORIGINATINGLEGID`,
`SYSTEMRESTARTNO`, `PATTERN`, `HOLDCOUNT`, `AUXSESSIONTYPE`, `DEVICEID1`,
`DEVICEID2`, `ISLEG1ORIGINATING`, `ISLEG2ORIGINATING`, `GLOBALCALLID`,
`CADTEMPLATEID`, `CADTEMPLATEID2`, `ts`, `INITIATOR`, `ACCOUNTNAME`, `APPNAME`,
`CALLID`, `CHRTYPE`, `CALLERNAME`, `serviceid1`, `serviceid2`"
$select_sql = "
SELECT $all_columns FROM `cdrdb`.`session`
WHERE ts >= NOW() - INTERVAL 1 DAY";
$source_data = mysqli_query($conn, $select_sql);
$tmpfilename = tempnam('/tmp', 'data');
fp = fopen($tmpfilename, 'w'); // do proper error handling and use
while ($source = $source_data->fetch_assoc()) {
fputcsv($fp, array_values($source));
}
fclose($fp);
$tmpfilename = mysqli_real_escape_string($conn2, $tmpfilename);
$load_sql = "
LOAD DATA LOCAL INFILE '$tmpfilename'
IGNORE INTO TABLE `ambition`.`session`
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
( $all_columns )";
$conn2->query($load_sql);
I am trying to insert two rows at once using mysqli:bind::param on that few data has to repeat in two rows on mysql and few data has to change according to the array of the name[]. But the problem which i am facing only the array with index 0 is inserting in the mysql though in print_r($sql) is displaying two times true here is my function of inserting the data:
function insertSellingItem($customer_details,$po_no,$new_customer,$date,$address,$offer_no,$gstin,$dispatched,$invoice_no,$state,$invoice_date,$code,$challan_no,$remarks,$challan_date,&$serial_no,&$mat_des,&$pl_serial_no,&$hsn_code,&$unit,&$purchase_rate,&$tax_amount,&$total_cost,&$rate,&$quantity,&$item_value,&$discount,&$sgst,&$cgst,&$igst,&$amount,$pay_cost){
global $mysqli;
$stmt = $mysqli->prepare("INSERT INTO sellItems(
customer_details,
po_no,
new_customer,
date,
address,
offer_no,
gstin,
dispatched,
invoice_no,
state,
invoice_date,
code,
challan_no,
remarks,
challan_date,
serial_no,
mat_des,
pl_serial_no,
hsn_code,
unit,
purchase_rate,
tax_amount,
total_cost,
rate,
quantity,
item_value,
discount,
sgst,
cgst,
igst,
amount,
pay_cost
)
VALUES(
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?,
?
)");
$stmt->bind_param("ssssssssssssssssssssssssssssssss",$customer_details,$po_no,$new_customer,$date,$address,$offer_no,$gstin,$dispatched,$invoice_no,$state,$invoice_date,$code,$challan_no,$remarks,$challan_date,$serial_no,$mat_des,$pl_serial_no,$hsn_code,$unit,$purchase_rate,$tax_amount,$total_cost,$rate,$quantity,$item_value,$discount,$sgst,$cgst,$igst,$amount,$pay_cost);
$stmt->execute();
$inserted_id = $mysqli->insert_id;
$stmt->close();
if($inserted_id>0)
{
return true;
}
else
{
return false;
}}
And the loop how i am sending the data in the in this function :
if(!empty($mat_des)){
for($i = 0; $i<count($mat_des); $i++) {
$sql = insertSellingItem($customer_details,$po_no,$new_customer,$date,$address,$offer_no,$gstin,$dispatched,$invoice_no,$state,$invoice_date,$code,$challan_no,$remarks,$challan_date,$serial_no[$i],$mat_des[$i],$pl_serial_no[$i],$hsn_code[$i],$unit[$i],$purchase_rate[$i],$tax_amount[$i],$total_cost[$i],$rate[$i],$quantity[$i],$item_value[$i],$discount[$i],$sgst[$i],$cgst[$i],$igst[$i],$amount[$i],$pay_cost);
print_r($sql);
}}
Where print_r(($mat_des)); is displaying count no 2 as i am passing the two rows in the name = mat_des[];
Any advice will be highly appreciated!
I recieve the echo before the bind_param statment but not after it
$stmt = $this->conn->prepare("INSERT INTO restaurants(unique_id, name, type, longitude, latitude, value_for_money, cleanliness, view, atmosphere, staff created_at) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NOW())");
echo "ezzat wasal";
$stmt->bind_param("sssddiiiii", $uuid, $name, $type, (double)$longitude, (double)$latitude, (int)$value_for_money, (int)$cleanliness, (int)$view, (int)$atmosphere, (int)$staff);
echo "ana zeh2et";
You are missing a comma between staff and created_at. Also I would suggest quoting all column names in the query (because some of them are reserved words in mySQL: name, type, view):
$stmt = $this->conn->prepare("INSERT INTO `restaurants`
(`unique_id`, `name`, `type`, `longitude`, `latitude`, `value_for_money`,
`cleanliness`, `view`, `atmosphere`, `staff`, `created_at`)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NOW())");
I was always using normal querys for inserting data into the database but now I want to make it with prepared statements. I'm already using statements to select data in all my files but insert never worked... And now I ran out of ideas again. Maybe someone can see what I did wrong.
$animeId = $_POST['animeId'];
$username = $_POST['username'];
$rating = $_POST['rating'];
$story = $_POST['story'];
$genre = $_POST['genre'];
$animation = $_POST['animation'];
$characters = $_POST['characters'];
$music = $_POST['music'];
//Datum auslesen
$date = date("Y-m-d H:i:s");
if($insertRating = $con->prepare("INSERT INTO anime_rating (animeId, rating, story, genre, animation, characters, music, user, date) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?"))
{
$insertRating->bind_param("iiiiiiiss", $animeId, $rating, $story, $genre, $animation, $characters, $music, $username, $date);
$insertRating->execute();
$insertRating->close();
}
You have an errant comma in your query:
music, user,) VALUES (?, ?, ?, ?, ?, ?, ?
^^^
HERE
It should be
music, user) VALUES (?, ?, ?, ?, ?, ?, ?
In the statement:
INSERT INTO anime_rating (
animeId,
rating,
story,
genre,
animation,
characters,
music,
user /* 8 columns */)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?") /* 10 parameters */
There are 8 columns listed to insert values into and 10 parameters specified in the values section. Also as pointed out there is the extra comma in the list of values.
The number of columns must match the number of parameters and the number of parameters binding in the following statement:
`$insertRating->bind_param("iiiiiiiss", $animeId, $rating, $story, $genre, $animation, $characters, $music, $username, $date);`
Two errors in the statement:
INSERT INTO anime_rating (animeId, rating, story, genre, animation, characters, music, user,) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?"
^ here and ^ ^
remove the comma
add a closing parentheses before the end of the string.
remove one ,?
Furthermore you should chop one is from the binding:
$insertRating->bind_param("iiiiiiss", $animeId, $rating, $story, $genre, $animation, $characters, $music, $username, $date);
if($insertRating = $con->prepare("INSERT INTO anime_rating (animeId, rating, story, genre, animation, characters, music, user, date) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?"))
The last (") should be placed after the first ) at the end
New code:
if($insertRating = $con->prepare("INSERT INTO anime_rating (animeId, rating, story, genre, animation, characters, music, user, date) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)")
I am receiving error 'Number of variables doesn't match number of parameters in prepared statement'
$stmt = $this->conn->prepare( "INSERT INTO user
(
st1, u1, e1, sa1,
h1, roles_id, name_titles_id, first_name,
last_name, phone, mobile, address_road,
address_area, address_region, post_code, city,
country_id, creation_date, activated_at, modified_date_time,
created_by, referred_by, gender, ad1, status
)
VALUES
(
?, ?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?,
?, ?, ?, ?, ?
)"
)
$stmt->bind_param('i',$st1);
$stmt->bind_param('s',$u1);
$stmt->bind_param('s',$e1);
$stmt->bind_param('s',$sa1);
$stmt->bind_param('s',$h1);
$stmt->bind_param('i',$roles_id);
$stmt->bind_param('i',$name_titles_id);
$stmt->bind_param('s',$first_name);
$stmt->bind_param('s',$last_name);
$stmt->bind_param('s',$phone);
$stmt->bind_param('s',$mobile);
$stmt->bind_param('s',$address_road);
$stmt->bind_param('s',$address_area);
$stmt->bind_param('s',$address_region);
$stmt->bind_param('s',$post_code);
$stmt->bind_param('s',$city);
$stmt->bind_param('i',$country_id);
$stmt->bind_param('s',$creation_date);
$stmt->bind_param('s',$activated_at);
$stmt->bind_param('s',$modified_date_time);
$stmt->bind_param('i',$created_by);
$stmt->bind_param('i',$referred_by);
$stmt->bind_param('s',$gender);
$stmt->bind_param('s',$ad1);
$stmt->bind_param('i',$status);
Edit:
Just make a small test and it confirms, we can't use multiple bind_param with mysqli.
Not work:
$stmt->bind_param('s',$a);
$stmt->bind_param('s',$b);
Work:
$stmt->bind_param('ss',$a, $b);
Hopefully it'll be useful for future searches.
Your problem is simple. You are trying to do the thing manually, while the number of data asks for the automated process. You have to make a program to create a query for you.
Suppose You have an array with data already. All you need is to define the list of fields to insert
$fields = "st1,u1,e1,sa1,h1,roles_id,name_titles_id,first_name,last_name,phone,";
$fields .= "mobile,address_road,address_area,address_region,post_code,city,";
$fields .= "country_id,creation_date,activated_at,modified_date_time,";
$fields .= "created_by,referred_by,gender,ad1,status" ;
$fields = explode(",",$fields);
and then use some programming. Luckily, it's already done:
include 'safemysql.class.php';
$db = new safeMysql();
$insert = $db->filterArray($_POST,$fields);
$db->query("INSERT INTO user SET ?u", $insert);
And yeah, you are using bind_param wrong way. Correct usage can be seen in the manual page.