php inserting data in mysql from json runs too slowly - php

Question

I have the following code to read a JSON and store the results in a DDBB.
The code works, but it takes more than a minute to insert just 400 records.
If I open the json, it loads pretty fast.
What I'm doing wrong?
$db = new PDO('', '', '');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
if(tableExists($db, 'locations') == 1)
{
$sql=$db->prepare("DROP TABLE locations");
$sql->execute();
}
$sql ="CREATE TABLE `locations` (id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, evid INT(6) NOT NULL, place VARCHAR(150), country VARCHAR(150), reg_date TIMESTAMP)" ;
$db->exec($sql);
$json = file_get_contents('thejson.php');
$data = array();
$data = json_decode($json);
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
$stmt->bindValue(':evid', $id, PDO::PARAM_INT);
$stmt->bindValue(':places', $place, PDO::PARAM_STR);
$stmt->bindValue(':country', $country, PDO::PARAM_STR);
$stmt->execute();
}
}

Answer 1

So first 2 things i would try would be moving the prepare outside the loop, and wrapping it in a transaction:
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt->bindValue(':evid', $id, PDO::PARAM_INT);
$stmt->bindValue(':places', $place, PDO::PARAM_STR);
$stmt->bindValue(':country', $country, PDO::PARAM_STR);
$stmt->execute();
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
Another thing you could do is try using bindParam to bind the vars by reference - this way you only need to call bindParam once on each variable name at the start, then just overwrite the values of those variables on each iteration and call execute.
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
$id = 0;
$place = '';
$country = '';
$stmt->bindParam(':evid', $id, PDO::PARAM_INT);
$stmt->bindParam(':places', $place, PDO::PARAM_STR);
$stmt->bindParam(':country', $country, PDO::PARAM_STR);
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt->execute();
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
Similar to this instead of calling bind* you could just pass the values in via execute:
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
foreach ($data as $key => $object)
{
if(is_object($object))
{
$params = array(
':id' => $object->id,
':places' => $object->name,
':country' => substr(strrchr($object->name, "-"), 2)
);
$stmt->execute($params);
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
I suspect using a transaction will get you a performance gain, but I dont know that there will be a ton of difference between switching up the binding methods.
Your best bet is probably to insert all the records in a single query as #PavanJiwnani suggests:
// first we need to compile a structure of only items
// we will insert with the values properly transformed
$insertData = array_map(function ($object) {
if (is_object($object)) {
return array(
$object->id,
$object->name,
substr(strrchr($object->name, "-"), 2)
);
} else {
return false;
}
}, $data);
// filter out the FALSE values
$insertData = array_filter($insertData);
// get the number of records we have to insert
$nbRecords = count($insertData);
// $records is an array containing a (?,?,?)
// for each item we want to insert
$records = array_fill(0, $nbRecords, '(?,?,?)');
// now now use sprintf and implode to generate the SQL like:
// INSERT INTO `locations` (evid, place, country) VALUES (?,?,?),(?,?,?),(?,?,?),(?,?,?)
$sql = sprintf(
'INSERT INTO `locations` (evid, place, country) VALUES %s',
implode(',', $records)
);
$stmt = $db->prepare($sql);
// Now we need to flatten our array of insert values as that is what
// will be expected by execute()
$params = array();
foreach ($insertData as $datum) {
$params = array_merge($params, $datum);
}
// and finally we attempt to execute
$stmt->execute($params);

Answer 2

Try to echo timestamps with milliseconds to see what is running slow. Probably executing 400 insert queries (including opening/closing connections).

Answer 3

There are many factors that affects the performance of the database, please provide detailed information about the database system, PHP version and related hardware.
The bottleneck could be at:
file_get_contents('thejson.php')
if the JSON contents are fetched from a remote host, i.e. DB is running normally, network is slow.
You may also want to consider move:
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
out of the foreach loop.