Checking for partial duplications in a CSV in PHP - php

Question

I'm having an issue with a memory leak in this code. What I'm attempting to do is to temporarily upload a rather large CSV file (at least 12k records), and check each record for a partial duplication against other records in the CSV file. The reason why I say "partial duplication" is because basically if most of the record matches (at least 30 fields), it is going to be a duplicate record. The code I've written should, in theory, work as intended, but of course, it's a rather large loop and is exhausting memory. This is happening on the line that contains "array_intersect".
This is not for something I'm getting paid to do, but it is with the purpose of helping make life at work easier. I'm a data entry employee, and we are having to look at duplicate entries manually right now, which is asinine, so I'm trying to help out by making a small program for this.
Thank you so much in advance!
if (isset($_POST["submit"])) {
if (isset($_FILES["sheetupload"])) {
$fh = fopen(basename($_FILES["sheetupload"]["name"]), "r+");
$lines = array();
$records = array();
$counter = 0;
while(($row = fgetcsv($fh, 8192)) !== FALSE ) {
$lines[] = $row;
}
foreach ($lines as $line) {
if(!in_array($line, $records)){
if (count($records) > 0) {
//check array against records for dupes
foreach ($records as $record) {
if (count(array_intersect($line, $record)) > 30) {
$dupes[] = $line;
$counter++;
}
else {
$records[] = $line;
}
}
}
else {
$records[] = $line;
}
}
else {
$counter++;
}
}
if ($counter < 1) {
echo $counter." duplicate records found. New file not created.";
}
else {
echo $counter." duplicate records found. New file created as NEWSHEET.csv.";
$fp = fopen('NEWSHEET.csv', 'w');
foreach ($records as $line) {
fputcsv($fp, $line);
}
}
}
}

Answer 1

A couple of possibilities, assuming the script is reaching the memory limit or timing out. If you can access the php.ini file, try increasing the memory_limit and the max_execution_time.
If you can't access the server settings, try adding these to the top of your script:
ini_set('memory_limit','256M'); // change this number as necessary
set_time_limit(0); // so script does not time out
If altering these settings in the script is not possible, you might try using unset() in a few spots to free up memory:
// after the first while loop
unset($fh, $row);
and
//at end of each foreach loop
unset($line);