Drupal 7 database API very slow compared to PHP mysqli_connect() - php

Question

I am trying to loop over some data coming to me from a SOAP request and insert the records into a custom table in my Drupal install.
At first I created a custom module and used standard mysqli_connect() syntax to connect to the database and loop through the records and insert them. This was working great and fetched and inserted my remote data in about 2 seconds without a hitch.
I then remembered that Drupal has a database API (I am fairly new to Drupal) so I decided to do it right and use the API instead. I converted my code to how I think I should be doing it per the API docs, but now the process takes more like 5 or 6 seconds and sometimes even randomly hangs and doesn't complete at all and I get weird Session errors. The records end up inserting fine, but it just takes forever.
I'm wondering if I am doing it wrong. I would also like to wrap the inserts into a transaction, because I will first be deleting ALL of the records in the destination table first and then inserting the new data and since I am deleting first, I want to be able to roll back if the inserts fail for whatever reason.
I did not add transaction code to my original PHP only code, but did try to attempt it with the Drupal API, although completely removing the transaction/try/catch code doesn't seem to affect the speed or issues at all.
Anyway here is my original code:
$data = simplexml_load_string($jobsXml);
$connection = mysqli_connect("localhost","user","pass","database");
if (mysqli_connect_errno($connection))
{
echo "Failed to connect to MySQL: " . mysqli_connect_error();
exit();
}
// delete * current jobs
mysqli_query($connection,'TRUNCATE TABLE jobs;');
$recordsInserted = 0;
foreach ($data->NewDataSet->Table as $item) {
//escape and cleanup some fields
$image = str_replace('http://www.example.com/public/images/job_headers/', '', $item->job_image_file);
$specialty_description = mysqli_real_escape_string($connection, $item->specialty_description);
$job_board_title = mysqli_real_escape_string($connection, $item->job_board_title);
$job_board_subtitle = mysqli_real_escape_string($connection, $item->job_board_subtitle);
$job_state_code = ($item->job_country_code == 'NZ') ? 'NZ' : $item->job_state_code;
$sql = "
INSERT INTO jobs (
job_number,
specialty,
specialty_description,
division_code,
job_type,
job_type_description,
job_state_code,
job_country_code,
job_location_display,
job_board_type,
job_image_file,
job_board_title,
job_board_subtitle
) VALUES (
$item->job_number,
'$item->specialty',
'$specialty_description',
'$item->division_code',
'$item->job_type',
'$item->job_type_description',
'$job_state_code',
'$item->job_country_code',
'$item->job_location_display',
'$item->job_board_type',
'$image',
'$job_board_title',
'$job_board_subtitle'
)
";
if (!mysqli_query($connection,$sql))
{
die('Error: ' . mysqli_error($connection) . $sql);
}
$recordsInserted++;
}
mysqli_close($connection);
echo $recordsInserted . ' records inserted';
and this is my Drupal code. Can anyone tell me if maybe I am doing this wrong or not the most efficient way?
$data = simplexml_load_string($jobsXml);
// The transaction opens here.
$txn = db_transaction();
// delete all current jobs
$records_deleted = db_delete('jobs')
->execute();
$records_inserted = 0;
try {
$records = array();
foreach ($data->NewDataSet->Table as $item) {
$records[] = array(
'job_number' => $item->job_number,
'specialty' => $item->specialty,
'specialty_description' => $item->specialty_description,
'division_code' => $item->division_code,
'job_type' => $item->job_type,
'job_type_description' => $item->job_type_description,
'job_state_code' => ($item->job_country_code == 'NZ') ? 'NZ' : $item->job_state_code,
'job_country_code' => $item->job_country_code,
'job_location_display' => $item->job_location_display,
'job_board_type' => $item->job_board_type,
'job_image_file' => str_replace('http://www.example.com/public/images/job_headers/', '', $item->job_image_file),
'job_board_title' => $item->$job_board_title,
'job_board_subtitle' => $item->job_board_subtitle,
);
$records_inserted++;
}
$fields = array(
'job_number',
'specialty',
'specialty_description',
'division_code',
'job_type',
'job_type_description',
'job_state_code',
'job_country_code',
'job_location_display',
'job_board_type',
'job_image_file',
'job_board_title',
'job_board_subtitle'
);
$query = db_insert('jobs')
->fields($fields);
foreach ($records as $record) {
$query->values($record);
}
$query->execute();
} catch (Exception $e) {
// Something went wrong somewhere, so roll back now.
$txn->rollback();
// Log the exception to watchdog.
watchdog_exception('Job Import', $e);
echo $e;
}
echo $records_deleted . ' records deleted<br>';
echo $records_inserted . ' records inserted';

Answer 1

How big is the dataset you are trying to insert? If the dataset is very large then perhaps you might right into query size issues. Try looping over records and inserting each record one by one like you did with PHP.