Why XMLReader repeat mysql insert function inside while and can't stop? - php

Question

I am using XMLReader to import huge xml file by elements to MySQL database. Xml contains 1 547 772 tags (element) named 'RECORD'.
XML example
<?xml version="1.0" encoding="utf-8"?>
<RECORD><NAME>ДОШКІЛЬНИЙ НАВЧАЛЬНИЙ ЗАКЛАД №1 ЗАГАЛЬНОГО РОЗВИТКУ УЖГОРОДСЬКОЇ МІСЬКОЇ РАДИ ЗАКАРПАТСЬКОЇ ОБЛАСТІ</NAME><SHORT_NAME>ДНЗ №1</SHORT_NAME><EDRPOU>34888585</EDRPOU><ADDRESS>88000, Закарпатська обл., місто Ужгород, ВУЛИЦЯ М.ВОВЧКА, будинок 47, "А"</ADDRESS><BOSS>НАКОНЕЧНА ОЛЕНА АНАТОЛІЇВНА</BOSS><KVED>85.10 Дошкільна освіта</KVED><STAN>зареєстровано</STAN><FOUNDERS><FOUNDER>УПРАВЛІННЯ ОСВІТИ УЖГОРОДСЬКОЇ МІСЬКОЇ РАДИ, розмір внеску до статутного фонду - 0.00 грн.</FOUNDER>...</FOUNDERS></RECORD>...
For MySQL connection use
function connectBase(){
include __DIR__ . '/../../settings/sql.set.php';
$mysql = mysqli_connect($_sqlhost, $_sqluser, $_sqlpass, $_sqldb);
mysqli_query($mysql, "Set charset utf8");
mysqli_query($mysql, "Set character_set_client = utf8");
mysqli_query($mysql, "Set character_set_connection = utf8");
mysqli_query($mysql, "Set character_set_results = utf8");
mysqli_query($mysql, "Set collation_connection = utf8_general_ci");
return $mysql;
}
Main function for parsing
function XMLReaderToDB($setting = false)
{
$mysql = connectBase();
$dir = __DIR__ . '/../../tmp/';
$xmlURL = $dir . $setting['file'];
$xml = new XMLReader();
$xml->open($xmlURL);
$start_time = time();
$start = $setting['start'];
$limit = $setting['limit'];
$stop = $start + $limit;
$i = 0;
$count = 0;
$result = 1;
while($xml->read())
{
if ($xml->nodeType == XMLReader::ELEMENT && $xml->name == $setting['tag']) {
$item[] = "('items', '" . mysqli_real_escape_string($mysql, $xml->readOuterXML()) . "')";
}
if ($xml->nodeType == XMLReader::END_ELEMENT && $xml->name == $setting['tag']) {
$i++;
$count++;
if ($count >= 500) {
insertXMLtoDB($mysql, $item);
$item = array();
$count = 0;
}
}
if($i == $stop){
break;
}
}
$xml->close();
insertXMLtoDB($mysql, $item);
$mysql->close();
$end_time = time();
$time_elapsed_secs = $end_time - $start_time;
echo '<br/>Items: ' . $i . '<br/>';
echo 'Start: ' . date('H:i:s', $start_time) . '<br/>';
echo 'End: ' . date('H:i:s', $end_time) . '<br/>';
echo $time_elapsed_secs . ' sec. (' . ($time_elapsed_secs /60) . ' min.)';
die;
}
And for MySQL insert
function insertXMLtoDB($mysql, $data = false){
mysqli_query($mysql,"INSERT INTO _parse_tmp (parse_key, parse_value) VALUES " . implode(", ", $data));
$data = array();
// echo 'Success';
}
But, MySQL requests don't stop after 1 547 772 inserts and "while" continues to run. I notice if uncommenting echo 'Success'; in insertXMLtoDB function, "while" is stopping at 1 547 772 inserts and finishing correctly.
What is wrong in my functions?

Answer 1

I've never personally used XMLReader however, I'd assume you could try to add another condition to your statement such as while($xml->read() && $i != $stop) opposed to having it within your loop directly. Ah also just noticed after your usage of the function insertXMLtoDB you re-declare $data although you'll notice you set it to NULL every time your call the function, therefore setting it to an array is pointless.

Answer 2

I can't understand why the script doesn't work fine. That's why I decided to change it.
function XMLReaderToDB($setting = false)
{
$mysql = connectBase();
$dir = __DIR__ . '/../../tmp/';
$xmlURL = $dir . $setting['file'];
$xml = new XMLReader();
$xml->open($xmlURL);
$start_time = time();
$start = $setting['start'];
$limit = $setting['limit'];
$stop = $start + $limit;
$i = 0;
$count = 0;
while (($valid = $xml->read()) && $xml->name !== $setting['tag']) ;
while ($valid) {
$i++;
$count++;
$item[] = "('" . $setting['type'] . "', '" . mysqli_real_escape_string($mysql, $xml->readOuterXML()) . "')";
if ($count == 500) {
insertXMLtoDB($mysql, $item);
$item = array();
$count = 0;
}
$valid = $xml->next($setting['tag']);
}
$xml->close();
insertXMLtoDB($mysql, $item);
$mysql->close();
$end_time = time();
$time_elapsed_secs = $end_time - $start_time;
echo '<br/>Items: ' . $i . '<br/>';
echo 'Start: ' . date('H:i:s', $start_time) . '<br/>';
echo 'End: ' . date('H:i:s', $end_time) . '<br/>';
echo $time_elapsed_secs . ' sec. (' . ($time_elapsed_secs /60) . ' min.)';
return;
}