Getting faster results from 2 databases to form 1 resultset - php

Question

So here is my scenario...
The bug_tracker table is in one server and task_traker is in another.
I want to show a combined result but can't since there are in two separate databases remotely.
So I am calling the task tracker first and then getting the bug details per iteration.
$task = oci_parse($task_conn, "select * from task_table where ....");
oci_execute($task);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
$bug = oci_parse($bug_conn, "select * from bug_table where id = " . $task_row['BUGID'] );
oci_execute($bug);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
... //output
}
... //output
}
But this entire process is making it very slow... since there are large number of records and columns.
Is there any way to make it even slightly faster? Note: I don't have access so can't setup oracle db links.

Answer 1

You could improve it using the IN statement:
<?php
$task = oci_parse($task_conn, "select * from task_table where ....");
oci_execute($task);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
$bugs[] = $task_row['BUGID'];
$users[] = $task_row['USER'];
$status[] = $task_row['TASK_STATUS'];
}
$bug = oci_parse($bug_conn, "select * from bug_table where id IN (" . implode(',', $bugs) . ");" );
oci_execute($bug);
while ($task_row = oci_fetch_array($task, OCI_ASSOC+OCI_RETURN_NULLS)) {
// ...
}
?>
On a sidenote, why are you not using PDO? I believe using it will already give you a performance boost.

Answer 2

PHP is not meant for this kind of operation, neither should you try to write your own join function.
One proper way of solving this issue is to dump the data from both databases into a local database, and there do the join.
You do not need anything fancy for the local database, an SQLite3 is probably enough.
Just dump the data from each database into a CSV files using a bash script that you put into cron. After the dump, (re)create each table in your SQLite3, and load the CSVs into these tables. After this you can do a join once and push the result into a new table which you then are free to query.
This is what in the datawarehouse world is often referred to as an ETL process, just in this case, very very simplified.