reading large csv file contain comma with php - php

Question

i have csv file that i need to read with php.I used 2 method but there is problem in theme.
methode i used are:
1- file_get_contents
2-fgetcsv
let explaine about csv file. the problem about the file is the fields contain comma that is used for delimiter and its bothering.
the first methode is fast but commas in the fiels make it work incorrectly like number seperator 14,200 . i fixed it withe a function named fixed number. but there is still random text that contain comma and doesnt follow any rule that i can fix them
the second method for large csv is very slow and i cant get out put to see that its working
the code for first methode is like:
$myFile = file_get_contents($file);
$lines = explode("\r\n",$myFile);//file to an array
while($counter <= count($lines)){
$data=$lines[$counter];
$tmp=fixnumbers($data);
$tmp=eregi_replace('"', '',$tmp);
$tmp=explode(',',$tmp);
if(count($tmp)> 0 ){
$newdata[$datacounter]=$tmp;//explode('*0*',$data);
$datacounter++;
}
$counter++;
}
the second methode is here :
$handle= fopen($file,"r");
$row=1;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($c=0; $c < $num; $c++) {
$mydata[$row][$c]=$data[$c] . "<br />\n";
}$row++;
}
print "<div class=\"longList\"><pre>";
print_r($mydata);
print "</pre></div>";
fclose($file);
}

Answer 1

So I waited the minute to download the file, grabbed the first 5 records, and used a copy/paste of the fgetcsv example in the PHP manual.
First 5 records - https://termbin.com/23ti - saved as "sm_file.csv"
<?php
if (($handle = fopen("sm_file.csv", "r")) !== FALSE) {
$data=array();
$num=0;
while (($data[] = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num++;
}
fclose($handle);
print_r($data);
}
?>
[0] => Array
(
[0] => از تاريخ وصل 01/07/1397 - با برنامه
[1] => تاريخ گزارش: 29/09/1397
[2] => شماره گزارش: (3-5)
[3] => صفحه 1
[4] => گزارش قطع و وصل فيدرهاي فشار متوسط (نمونه 3)
[5] => ملاحظات
[6] => شرايط جوي
[7] => عملكرد ريكلوزر
[8] => رله عامل
[9] => خاموشي (MWh)
[10] => بار فيدر (A)
[11] => مدت قطع
[12] => زمان وصل
[13] => تاريخ وصل
[14] => زمان قطع
[15] => تاريخ قطع
[16] => نوع اشكال بوجود آمده
[17] => فيدر فشار متوسط
[18] => پست فوق توزيع
[19] => شماره پرونده
[20] => رديف
[21] => ناحيه اسالم
[22] =>
[23] => آفتابي
[24] => ندارد
[25] => ندارد
[26] => 0.21
[27] => 3
[28] => 132
[29] => 11:30
[30] => 1397/07/04
[31] => 09:18
[32] => 1397/07/04
[33] => جهت كار در حريم شبكه
[34] => گيسوم
[35] => اسا لم
[36] => 96,042,429,972
[37] => 1
[38] => 61292.56
[39] => جمع کل بار فيدر:
[40] => 393.85
[41] => جمع کل خاموشي:
[42] => 92,725
[43] => جمع مدت قطع:
)
Looks like data element 36 is the one you are having issues with, as you can see fgetcsv handles it fine, you just need to convert from a string to a number as you process the data. Just strip the commas.
<?php
if (($handle = fopen("sm_file.csv", "r")) !== FALSE) {
$data=array();
$num=0;
while (($data[] = fgetcsv($handle, 1000, ",")) !== FALSE) {
$data[(count($data)-1)][36]=str_replace(",","",$data[(count($data)-1)][36]);
}
fclose($handle);
print_r($data);
}
?>
Which gives
[36] => 96042429972
As for how long it takes, your full file of 2k records
User time (seconds): 0.12
System time (seconds): 0.09
Percent of CPU this job got: 43%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.52
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 41820
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2448
Voluntary context switches: 18
Involuntary context switches: 55
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
on a modest i5 w/ 8gb ram. Not seeing any issues.