MySQL > sum up into a single query (incl. google maps) - php

Question

This Q is posted as a follow up to a comment on one of my Qs by #Vincent Savard.
Currently I'm trying (with some success - hitting timeout after 60 sec. for google-maps XML requests) to split data from a (pretty) large table into lots of smaller tables - plus converting/modifying/etc them on the fly. For this task I'm using php close to the following example:
// The following happens inside some functions.
// The main table has some "groups" of content.
// fn_a creates the new small tables
"
CREATE TABLE {$small_table_a}
col_group_a_id int UNSIGNED NOT NULL AUTO_INCREMENT,
col_group_a_fname tinytext,
col_group_a_lname tinytext,
col_group_a_valA tinytext,
col_group_a_valB tinytext,
col_group_a_address tinytext,
col_group_a_lat decimal(9,3),
col_group_a_lng decimal(9,3),
PRIMARY KEY (id)
"
"
CREATE TABLE {$small_table_b}
col_group_b_id int UNSIGNED NOT NULL AUTO_INCREMENT,
col_group_b_fname tinytext,
col_group_b_lname tinytext,
col_group_b_valA tinytext,
col_group_b_valB tinytext,
col_group_b_address tinytext,
col_group_b_lat decimal(9,3),
col_group_b_lng decimal(9,3),
PRIMARY KEY (id)
"
// fn_b loads the content from the big table, modifies it and saves it row per row into the small tables
$sql = "
SELECT *
FROM {$big_table}
"
foreach ( $sql as $data )
{
$id = $data->id;
$group_a_fname = $data->group_a_fname;
$group_a_lname = $data->group_a_lname;
$group_a_lname = "{$group_a_fname}, {$group_a_lname}";
$group_a_valA = $data->group_a_valA ? $data->group_a_valA : '-';
$group_a_valA = $data->group_a_valB ? $data->group_a_valB : 'none';
$group_a_valA = $data->group_a_address;
$group_b_fname = $data->group_b_fname;
$group_b_lname = $data->group_b_lname;
$group_b_name = "{$group_b_fname}, {$group_b_lname}";
$group_b_valA = $data->group_b_valA ? $data->group_b_valA : '/';
$group_b_valA = $data->group_b_valB ? "€ {$data->group_b_valB}" : null;
"
INSERT INTO {$small_table_a} ... VALUES ...
"
}
// fn_c pulls in data from the small tables, asks the google map API for lat & lng and _should_ update the small table
$sql = "
SELECT *
FROM {$small_table_a}
"
foreach ( $sql as $data )
{
$output['id'] = $data->id;
$address = urlencode( $data->address );
$url = "http://maps.google.com/maps/api/geocode/xml?address={$address}&sensor=false";
$content = file_get_contents( $url );
$file_data = new SimpleXMLElement( $content );
$file_data = $file_data->result ? $file_data->result : null;
if ( ! $file_data )
continue;
$location = $file_data->geometry->location;
$output['lat'] = (string) $location->lat;
$output['lng'] = (string) $location->lng;
}
foreach ( $output as $data )
{
"
UPDATE {$table}
SET lat=SET lat={$data['lat']}, lng={$data['lng']}
WHERE id=$data['id']
}
Question: How could I do this in one query? Or how could I reduce DB-queries? And how would I add the lat/lng to the tables without interrupting the query building when my geocoding limit was exceeded for today - I don't want to drop everything just because I've gone over my limit.
Thanks!
Note: The example was written by hand straight out of my mind. Failures may be in there.

Answer 1

We need to know what the INSERT INTO query is in you foreach loop, because this is the one that can be summed into one query. Basically, here is the idea:
INSERT INTO {$small_table} -- you can specify which columns to fill,
-- i.e. INSERT INTO table (col_a, col_b)
SELECT group_a_fname, group_a_lname,
group_a_valA, group_a_valB,
group_a_address, group_b_fname,
group_b_lname, group_b_valA, group_b_valB -- etc
FROM {$big_table};
Obviously, you'll have to adapt the query to fill your needs. You just need to grasp the idea behind it : you can insert rows with a SELECT query.
The UPDATE query is different because you have to rely on external data (a website). I don't think there is an easy way to do it in one query, but I may be wrong.