Processing huge files with php into a database - php

Question

I have a text file (which essentially is a csv without the extension) that has 150,000 lines in it. I need to remove duplicates by key then insert them into the database. I'm attempting fgetcvs to read it line by line, but I don't want to do 150,000 queries. So this is what I came up with so far: (keep in mind i'm using laravel)
$count = 0;
$insert = [];
if (($handle = fopen("myHUGEfile.txt", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$count++;
//See if this is the top row, which in this case are column headers
if ($count == 1) continue;
//Get the parts needed for the new part
$quantity = $data[0];
$part_number = $data[1];
$manufacturer = $data[2];
$new_part = [
'manufacturer' => $manufacturer,
'part_number' => $part_number,
'stock' => $quantity,
'price' => '[]',
'approved' => 0,
];
$insert[] = $new_part;
}
fclose($handle);
} else {
throw new Exception('Could not open file for reading.');
}
//Remove duplicates
$newRows = [];
$parsedCount = 0;
foreach ($insert as $row) {
$x = 0;
foreach ($newRows as $n) {
if (strtoupper($row['part_number']) === strtoupper($n['part_number'])) {
$x++;
}
}
if ($x == 0) {
$parsedCount++;
$newRows[] = $row;
}
}
$parsed_rows = array_chunk($newRows, 1000, true);
$x = 0;
foreach ($parsed_rows as $chunk) {
//Insert
if (count($chunk) > 0)
if (DB::table('search_parts')->insert($chunk))
$x++;
}
echo $x . " chunks inserted.<br/>" . $count . " parts started with<br/>" . $parsedCount . " rows after duplicates removed.";
But it's very clunky, I have only tested it with a little over 1000 rows and it works using localhost. But i'm afraid if I push it up to production it won't be able to handle all 150,000 rows. The file is about 4mb.
Can someone show me a better more efficient way to do this?

Answer 1

Right now, you're keeping the first duplicate record. If you're ok keeping the last dupe, you can just change
$insert[] = $new_part;
to
$insert[strtoupper($part_number)] = $new_part
That way, your $insert array will only have one value for each $part_number. Your inserts will be a little slower, but you can drop all of the code which checks for duplicates which looks very, very slow.

Answer 2

4Mb is not remotely a "huge" file. I'd just read the whole thing into an assoc array keyed by part number, which will inherently de-dupe, giving you the last row whenever a duplicate is encountered. Something like this maybe:
$parts = [];
foreach (explode("\n", file_get_contents('file')) as $line) {
$part = str_getcsv($line);
$parts[$part[1]] = [
'manufacturer' => $part[2],
'part_number' => $part[1],
'stock' => $part[0],
'price' => '[]',
'approved' => 0,
];
}
// $parts now contains unique part list
foreach ($parts as $part) {
$db->insert($part);
}

Answer 3

If you don't want duplicates on a certain or multiple keys, you can make it easy on yourself and just add a UNIQUE INDEX on the key you don't want duplicates for on the table.
This way, all you have to worry about is processing the file. When it reaches a duplicate key, it will not be able to insert it and will continue.
It would also make it easier in the future because you wouldn't have to modify your code if you need to do checks on additional columns. Just modify the index.