How to optimize loops for large CSV files data extraction - php

Question

I have a question about code optimization.
I haven't coded anything besides simple loops in over ten years.
I created the code below, which works fine but is super slow for my needs.
In essence, I have 2 CSV files:
a source CSV file that has about 500 000 records, let's say: att1, att2, source_id, att3, att4 (in reality there are about 40 columns)
a main CSV file that has about 120 million records, let's say: att1, att2, att3, main_id, att4 (in reality there are about 120 columns)
For each source_id in the source file, my code parses the main file for all the lines where main_ id == source_id and it writes each of those lines in a new file.
Do you have any suggestion on how I could optimize the code, to go much much faster?
<?php
$mf = "main.csv";
$mf_max_line_length = "512";
$mf_id = "main_id";
$sf = "source.csv";
$sf_max_line_length = "884167";
$sf_id = "source_id";
if (($mf_handle = fopen($mf, "r")) !== FALSE)
{
// Read the first line of the main CSV file
// and look for the position of main_id
$mf_data = fgetcsv($mf_handle, $mf_max_line_length, ",");
$mf_id_pos = array_search ($mf_id, $mf_data);
// Create a new main CSV file
if (($nmf_handle = fopen("new_main.csv", "x")) !== FALSE)
{
fputcsv($nmf_handle,$mf_data);
} else {
echo "Cannot create file: new_main.csv" . $sf;
break;
}
}
// Open the source CSV file
if (($sf_handle = fopen($sf, "r")) !== FALSE)
{
// Read the first line of the source CSV file
// and look for the position of source_id
$sf_data = fgetcsv($sf_handle, $sf_max_line_length, ",");
$sf_id_pos = array_search ($sf_id, $sf_data);
// Go trhough the whole source CSV file
while (($sf_data = fgetcsv($sf_handle, $sf_max_line_length, ",")) !== FALSE)
{
// Open the main CSV file
if (($mf_handle = fopen($mf, "r")) !== FALSE)
{
// Go trhough the whole main CSV file
while (($mf_data = fgetcsv($mf_handle, $mf_max_line_length, ",")) !== FALSE)
{
// If the source_id matches the main_id
// then we write it into the new_main CSV file
if ($mf_data[$mf_id_pos] == $sf_data[$sf_id_pos])
{
fputcsv($nmf_handle,$mf_data);
}
}
fclose($mf_handle);
}
}
fclose($sf_handle);
fclose($nmf_handle);
}
?>

Answer 1

Sounds like a job for mysql.
First, you'll need to create tables based on all your fields. See here
Then, you'll load your data. See here
Finally, you'll create a query like:
SELECT * INTO OUTFILE '/tmp/something.csv'
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM source_table INNER JOIN main_table ON
source_table.source_id=main_table.main_id;