reading off multiple CSV files - php

Question

need to pull a ton of info, i.e.
file1:
10948|Book|Type1
file2:
SHA512||0||10948
file3:
0|10948|SHA512|c3884fbd7fc122b5273262b7a0398e63
I'd like to get it into soething like
c3884fbd7fc122b5273262b7a0398e63|SHA512|Type1|Book
I do not have access to an actual database, is there any way to do this? Basically looking for a $id = $file1[0]; if($file3[1] == $id) or something unles sthere's more efficient.
Each CSV file is anywhere from 100k-300k lines. I don't care if it takes a while, I can just let it run on EC2 for a while.

Answer 1

$data = array();
$fh = fopen('file1') or die("Unable to open file1");
while(list($id, $val1, $val2) = fgetcsv($fh, 0, '|')) {
$data[$id]['val1'] = $val1;
$data[$id]['val2'] = $val2;
}
fclose($fh);
$fh = fopen('file2') or die ("Unable to open file2");
while(list($method, null, null, null, $id) = fgetcsv($fh, 0, '|')) {
$data[$id]['method'] = $method;
}
fclose($fh);
$fh = fopen('file3') or die("Unable to open file3");
while(list(null, $id, null, $hash) = fgetcsv($fh, 0, '|')) {
$data[$id]['hash'] = $hash;
}
fclose($fh);
Tedious, but should you get an array with the data you want. Outputting it it as another csv is left as an exercise to the reader (hint: see fputcsv()).

Answer 2

All three files appear to have a common field (i.e. in your example, "10948" was common to all three lines). If you're not worried about using a lot of memory, you could load all three files in different array, setting the common field as the array key and using a foreach loop to reassemble all three.
For example:
$result = array();
// File 1
$fh = fopen('file1');
while ( ($data = fgetcsv($fh, 0, '|')) !== FALSE )
$result[$data[0]] = $data;
fclose($fh);
// File 2
$fh = fopen('file2')
while ( ($data = fgetcsv($fh, 0, '|')) !== FALSE )
$result[$data[5]] = array_merge($result[$data[3]], $data);
fclose($fh);
// File 3
$fh = fopen('file3')
while ( ($data = fgetcsv($fh, 0, '|')) !== FALSE )
$result[$data[1]] = array_merge($result[$data[1]], $data);
fclose($fh);

Answer 3

I would suggest to perform a merged-sort using basic unix tools:
a) sort your .CSV files by the columns common between each file, sort -d" " -K? -K? -K?
b) Using the unix 'join' command to output records common between pairs of .CSV files.
The 'join' command only works with 2 files at a time, so you'll have to 'chain' the results for multiple data sources:
# where 'x' is field number from file A, and 'y' is field number from file B
sort -kx "fileA"
sort -ky "fileB"
join -1x -2y "fileA" "fileB" > file1
sort -kx "fileC"
join -1x -2y "file1" "fileC" > file2
sort -kx "fileD"
join -1x -2y "file2" "fileD" > file3
etc...
This is very fast, and allows you to filter your .CSV files as if an impromptu database join occurred.
If you have to write your own merge-sort in php: (Read Here: Merge Sort )
The easiest implementing to merge-sort of .CSV files is 2-stage: a) unix sort your files, then B) 'merge' all the sources in parallel, reading in a record from each, looking for the case where your value in your common fields match all the other sources (JOIN in database terminology):
rule 1) Skip the record that is less than (<) ALL the other sources.
rule 2) When a record's common value is equal to (==) ALL other sources do you have a match.
rule 3) When a record's common value is equal to (==) is SOME of the other source, you can use 'LEFT-JOIN' logic if desired, otherwise skip that record from all sources.
Pseudo code for a join of multiple files
read 1st record from every data source;
while "record exists from all data sources"; do
for A in each Data-Source ; do
set cntMissMatch=0
for B in each Data-Source; do
if A.field < B.field then
cntMissMatch+=1
end if
end for
if cntMissMatch == count(Data-Sources) then
# found record with lowest values, skip it
read next record in current Data-source;
break; # start over again looking for lowest
else
if cntMissMatch == 0 then
we have a match, process this record;
read in next record from ALL data-sources ;
break; # start over again looking for lowest
else
# we have a partial match, you can choose to have
# 'LEFT-JOIN' logic at this point if you choose,
# where records are spit out even if they do NOT
# match to ALL data-sources.
end if
end if
end for
done
Hope that helps.