creating json or xml with mysql in php too slow - php

Question

I am working on a project for a lecture at the university and I am searching for a solution for more than 2 weeks now, and I just can't get it right.
We have a project where we need to generate specific JSON or XML files to visualize them later with for example D3 or Sigma.
We have a mysql database and all the code is in Javascript (as you can see with the libraries) and we use pho to get the data from the database and to get it in the right format. Here is an example xml file I tried to create with php (it's a gexf-file for the visualization with Sigma, but it's just the same as xml):
<?xml version="1.0" encoding="UTF-8"?>
<gexf xmlns="http://www.gexf.net/1.2draft" version="1.2">
<meta lastmodifieddate="2009-03-20">
<creator>Gexf.net</creator>
<description>A hello world! file</description>
</meta>
<graph mode="static" defaultedgetype="directed">
<nodes>
<node id="0" label="Hello" />
<node id="1" label="Word" />
</nodes>
<edges>
<edge id="0" source="0" target="1" />
</edges>
</graph>
</gexf>
And Here is my php code where I tried to create the xml:
<?php
set_time_limit(500000000);
ini_set('memory_limit', '-1');
class XmlWriter2 {
var $xml;
var $indent;
var $stack = array();
function XmlWriter($indent = ' ') {
$this->indent = $indent;
$this->xml = '<?xml version="1.0" encoding="utf-8"?>'."\n";
}
function _indent() {
for ($i = 0, $j = count($this->stack); $i < $j; $i++) {
$this->xml .= $this->indent;
}
}
function push($element, $attributes = array()) {
$this->_indent();
$this->xml .= '<'.$element;
foreach ($attributes as $key => $value) {
$this->xml .= ' '.$key.'="'.htmlentities($value).'"';
}
$this->xml .= ">\n";
$this->stack[] = $element;
}
function element($element, $content, $attributes = array()) {
$this->_indent();
$this->xml .= '<'.$element;
foreach ($attributes as $key => $value) {
$this->xml .= ' '.$key.'="'.htmlentities($value).'"';
}
$this->xml .= '>'.htmlentities($content).'</'.$element.'>'."\n";
}
function emptyelement($element, $attributes = array()) {
$this->_indent();
$this->xml .= '<'.$element;
foreach ($attributes as $key => $value) {
$this->xml .= ' '.$key.'="'.htmlentities($value).'"';
}
$this->xml .= " />\n";
}
function pop() {
$element = array_pop($this->stack);
$this->_indent();
$this->xml .= "</$element>\n";
}
function getXml() {
return $this->xml;
}
}
/*
$xml = new XmlWriter2();
$array = array(
array('monkey', 'banana', 'Jim'),
array('hamster', 'apples', 'Kola'),
array('turtle', 'beans', 'Berty'),
);
$xml->push('zoo');
foreach ($array as $animal) {
$xml->push('animal', array('species' => $animal[0]));
$xml->element('name', $animal[2]);
$xml->element('food', $animal[1]);
$xml->pop();
}
$xml->pop();
print $xml->getXml();
<?xml version="1.0" encoding="utf-8"?>
<zoo>
<animal species="monkey">
<name>Jim</name>
<food>banana</food>
</animal>
<animal species="hamster">
<name>Kola</name>
<food>apples</food>
</animal>
<animal species="turtle">
<name>Berty</name>
<food>beans</food>
</animal>
</zoo>
*/
mysql_connect("127.0.0.1", "root", "manager") or die(mysql_error());
mysql_select_db("enrondata") or die(mysql_error());
$data1 = mysql_query("SELECT DISTINCT sId FROM mailToId WHERE date
BETWEEN '01.06.2002' AND '30.06.2002' UNION SELECT DISTINCT rId FROM
mailToId WHERE date BETWEEN '01.06.2002' AND '30.06.2002'") or die
(mysql_error());
$data2 = mysql_query("SELECT sender, recipient, count(*) AS numMails
FROM mailTo WHERE date BETWEEN '01.06.2002' AND '30.06.2002' GROUP
BY sender, recipient") or die (mysql_error());
$users = array();
$id = 0;
while($tmpUsers = mysql_fetch_array($data1)){
$tmpArray['id'] = $tmpUsers['sId'];
$user = mysql_query("SELECT email FROM users WHERE id=".$tmpUsers['sId']);
while($tmpUser = mysql_fetch_array($user)){
$tmpArray['email'] = $tmpUser['email'];
}
array_push($users, $tmpArray);
}
$xml = new XmlWriter2();
$xml->push('gexf', array('xmlns' => 'http://www.gexf.net/1.2draft" version="1.2'));
$xml->push('meta', array('lastmodifieddate' => '2009-03-20'));
$xml->element('creator', 'Gexf.net');
$xml->element('description', 'A hello world! file');
$xml->pop();
$xml->push('graph', array('mode' => 'static', 'defaultedgetype' => 'directed'));
$xml->push('nodes');
for($i = 0; $i < count($users); $i++){
$xml->push('node', array('id' => $users['id'],
'label' => $users['email']));$xml->pop();
}
$xml->pop();
$xml->push('edges');
while($tmp = mysql_fetch_array($data2)){
$xml->push('edge', array('id' => id,
'source' => $tmp['sender'], 'target' => $tmp['recipient'], 'weight'
=> $tmp['numMails']));$xml->pop();
$id++;
}
$xml->pop();
$xml->pop();
$xml->pop();
print $xml->getXml();
?>
And it works, the code is correct, but it takes hours. Really, even after 30min it is not finished doing all that. And I have no idea how to improve it and get it very fast. Or is there another possiblity to get the data from the mysql database in the right format without using php?
Please help me. My deadline is really close and I have no ideas and didn't find anything on the web that fits my problem.

Answer 1

So you doing over 500 000 querys, that a big overhead,
try just join in the email in the first query:
/* puggan added left join email */
$data1 = mysql_query("SELECT DISTINCT mailToId.sId, users.email FROM mailToId LEFT JOIN users ON (users.id = mailToId.sId) WHERE date BETWEEN '01.06.2002' AND '30.06.2002' UNION SELECT DISTINCT mailToId.rId, users.email FROM mailToId LEFT JOIN users ON (users.id = mailToId.rId) WHERE date BETWEEN '01.06.2002' AND '30.06.2002'") or die(mysql_error());
$data2 = mysql_query("SELECT sender, recipient, count(*) AS numMails FROM mailTo WHERE date BETWEEN '01.06.2002' AND '30.06.2002' GROUP BY sender, recipient") or die (mysql_error());
$users = array();
$id = 0;
while($tmpUsers = mysql_fetch_array($data1))
{
$tmpArray['id'] = $tmpUsers['sId'];
/* Puggan added: */
$tmpArray['email'] = $tmpUsers['email'];
// $user = mysql_query("SELECT email FROM users WHERE id=".$tmpUsers['sId']);
// while($tmpUser = mysql_fetch_array($user))
// {
// $tmpArray['email'] = $tmpUser['email'];
// }
array_push($users, $tmpArray);
}