Select table from Sql server and insert data to Mysql table - php

Question

I have a running ms sql server and want some data copied to a mysql database.
i already can connect to them both so i made something like:
$pdo = new PDO('SQLSERVER', $user, $password);
$sql = "SELECT id, name FROM users";
$stmt = $pdo->prepare($sql);
$stmt->execute();
$json_users = array();
while ($row = $stmt->fetchObject()){
$json_users[] = $row;
}
$pdo = new PDO('MYSQLDB', $user, $password);
foreach ($json_users as $key => $value){
$sql = "INSERT INTO users (id, name) VALUES (:id, :name)"
$stmt = $pdo->prepare($sql);
$stmt->bindParam('id', $value->id, PDO::PARAM_INT);
$stmt->bindParam('name', $value->name, PDO::PARAM_STR);
$stmt->execute();
}
this does work but takes a lot of time cause its a big table.
so my question is can i insert the complete results from sqlserver to mysql at once with only one insert query? without the foreach?
Update: the table contains 173398 rows and 10 columns

Answer 1

With prepared statements (especially for multi-insert) you want to have your prepared statement outside your loop. You only need to set the query up once, then supply your data in each subsequent call
$sql = "INSERT INTO users (id, name) VALUES (:id, :name)";
$stmt = $pdo->prepare($sql);
foreach($json_users as $key => $value){
$stmt->bindParam('id', $value->id, PDO::PARAM_INT);
$stmt->bindParam('name', $value->name, PDO::PARAM_STR);
$stmt->execute();
}

Answer 2

You can export this into CSV file first from MSSQL then import that file into MySQL.
$pdo = new PDO('SQLSERVER', $user, $password);
$sql = "SELECT id, name FROM users";
$stmt = $pdo->prepare($sql);
$stmt->execute();
$fp = fopen('/tmp/mssql.export.csv', 'w');
while ($row = $stmt->fetch(PDO::FETCH_NUM)){
fputcsv($fp, array_values($row));
}
fclose($fp);
$pdo = new PDO('MYSQLDB', $user, $password, array(PDO::MYSQL_ATTR_LOCAL_INFILE => 1));
$sql = <<<eof
LOAD DATA LOCAL INFILE '/tmp/mssql.export.csv'
INTO TABLE user_copy
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
(id,name)
eof;
$pdo->exec($sql);
one drawback with the above, you need to have this configuration enabled in my.cnf ( MySQL configuration )
[server]
local-infile=1
Since MySQL cannot read files that are owned by others unless it it opened with --local-infile=1

Answer 3

I would suggest not bind values, but generate the query string:
$sql = "";
foreach ($json_users as $key => $value){
if ($sql=="") {
$sql = "INSERT INTO users (id, name) VALUES ";
$sql =." (".$value->id.',"'.$value->name.'")';
} else {
$sql .= ", (".$value->id.',"'.$value->name.'")';
}
}
$stmt = $pdo->prepare($sql);
$stmt->execute();
It is not best practice, but since you trust data source it could help.

Answer 4

Consider doing bulk inserts instead of one row at a time.
$sourcedb = new \PDO('SQLSERVER', $sourceUser, $sourcePassword);
$targetdb = new \PDO('MYSQLDB', $targetUser, $targetPassword);
$sourceCountSql = "SELECT count(*) count FROM users;";
/**
* for mssql server 2005+
*/
$sourceSelectSql = "
SELECT
id,
name
FROM
(
SELECT
ROW_NUMBER() OVER (ORDER BY id) RowNum,
id,
name
FROM
users
) users
WHERE
RowNum >= %d
AND RowNum < %d
ORDER BY
RowNum
";
/**
* for mssql server 2012+
*/
$sourceSelectSql = "
SELECT
FROM TableName ORDER BY id OFFSET 10 ROWS FETCH NEXT 10 ROWS ONLY;
SELECT
id,
name
FROM
users
ORDER BY
id
OFFSET %d ROWS
FETCH NEXT %d ROWS ONLY
";
$sourceCount = $sourcedb->query($sourceCountSql, \PDO::FETCH_COLUMN, 0);
$rowCount = 1000;
$count = 0;
$count2 = 0;
for($x = 0; $x < $sourceCount; $x += $rowCount) {
$sourceRecords = $sourcedb->query(sprintf($sourceSelectSql, $x, $rowCount), \PDO::FETCH_ASSOC);
$inserts = [];
foreach($sourceRecords as $row) {
$inserts[] = sprintf("(:id_%1$d, :name_%1$d)", $count++);
}
$stmt = $targetdb->prepare(sprintf("INSERT INTO users (id, name) VALUES %s;", implode(',', $inserts));
foreach($sourceRecords as $row) {
$stmt->bindParam(sprintf('id_%d', $count2), $row['id'], \PDO::PARAM_INT);
$stmt->bindParam(sprintf('name_%d', $count2), $row['name'], \PDO::PARAM_STR);
$count2++;
}
$targetdb->execute();
unset($inserts);
}