I have huge text file and I am trying to read and insert this line by line.
this is txt file data.
'REG','KOIL','Kohinoor Industries Ltd.','READY',4.82,2.82,3.82
'REG','EPQL','Engro Powergen Qadirpur Ltd.','READY',36.9495,33.4305,35.19
Function for insert data
$file_path =FCPATH.'uploads/text/'.$file_name;
$psx_date=$this->input->post('file_date');
$open=fopen($file_path,"r");
$i=1;
while(!feof($open)){
$line=fgets($open);
if($i>2){
$values = explode(",",$line);
$psx_symbol=str_replace('\'',null,$values[1]);
$no_of_rows=read_psx_where($psx_symbol,$psx_date);
if($no_of_rows<=0){
$psx_data=array(
'PSX_SYMBOL' => $psx_symbol,
'PSX_DATE' => $psx_date,
'PSX_HIGH' => $values[4],
'PSX_LOW' => $values[5],
'PSX_CLOSE' => $values[6],
'PSX_DATETIME' => date('Y-m-d H:i:s'),
'PSX_SATUS' => 1
);
insert_psx_data($psx_data);
}
}
$i++;
}
fclose($open);
I am first skip first two lines of test file and then I am checking if same symbol is already exist so then skip this line.
This method is working but too much slowdown and exceeding max exectution time.
Related
I am doing exporting excel file using phpExcel. I output 2000 rows and its execution time is about 10 seconds . The problem is that when I output 20,000 rows it gets error: ERROR TIME OUT . At first: max_execution_time = 30, I set it back to max_execution_time = 60 (This value is equal to the value on the main server). Now is there a way to minimize file export time, since the data is sometimes very large..Up to a few hundred thousand rows. Please help me. Thanks. Sorry my english is not good.
Here is my code :
// array example
// 20.000 rows
$data = [
[
'name' => 'hello',
'address' => 'usa',
'birthday' => '2021-04-30'
],
[
'name' => 'hello',
'address' => 'usa',
'birthday' => '2021-04-30'
],
];
// Sample processing code
$exRow = 2;
foreach ($data as $val) {
for ($i = 0; $i < 12; $i++) { // set title
$excel->getActiveSheet()->setCellValueByColumnAndRow($i, $exRow, $val);
}
}
header('Content-type: application/vnd.ms-excel');
header('Content-Disposition: attachment; filename="data.xls"');
PHPExcel_IOFactory::createWriter($excel, 'Excel2007')->save('php://output');
you just need to increase max_execution_time in php.ini file. if you don't access to do this just can make a php.ini or user.ini and put this text on there
max_execution_time=500
also you can use yield key to add excel row. this is help you render excel rows by lazy rendering
visit this link
The code I currently have as pasted below simply (or not so simply) reads an existing json file and appends a new record to the file after the '[' character, so skipping one line. The problem is this whole process of reading and writing produces very slow results. This is for a dice game, user roles the dice and the score is written to a json file, a jquery script reads the file in to a table.
function writeJson($player, $result, $bet, $betamount, $profit) {
$id=uniqid();
$response = array(
'player'=> $player,
'result'=> $result,
'bet'=>$bet,
'size'=>$betamount,
'profit'=>$profit,
'id'=>$id
);
$lines = file('dice.json');
$oldlines='';
foreach ($lines as $line_num => $line) {
$oldlines.=$line."\r\n";
if($line_num==0) { //insert new record here.
$oldlines.=json_encode($response).",\r\n";
}
}
file_put_contents('dice.json',preg_replace('/^\h*\v+/m', '', $oldlines));
clearstatcache();
}
here is the format of a json file generated by this code.
[
new records will be inserted at this location
{"player":"seang","result":"EVEN","bet":"ODD","size":"0.00000005","profit":"-0.00000005","id":"57b9ace6ce133"},
{"player":"seang","result":"EVEN","bet":"ODD","size":"0.00000005","profit":"-0.00000005","id":"57b9ace1c73f4"},
{"player":"seang","result":"ODD","bet":"ODD","size":"0.00000005","profit":"+0.00000005","id":"57b9acd8dd50a"}
]
I have a working system on which I get the data of two .csv file. And save all the data into array and then compare some of the data existing on both csv file. The system works well but later I found out that some of the rows doesn't display on the array. I think I don't use the proper code in reading a csv file. I want to edit/improve the system. This is my code on reading or getting the data from csv file.
$thedata = array();
$data = file("upload/payment.csv");
foreach ($data as $deposit){
$depositarray = explode(",", $deposit);
$depositlist = $depositarray;
$key = md5($depositlist[9] . $depositlist[10]);
$thedata[$key]['payment'] = array(
'name' => $depositlist[0],
'email' => $depositlist[1],
'modeofpayment' =>$depositlist[8],
'depositdate' => $depositlist[9],
'depositamount' => number_format($depositlist[10],2)
);
}
'<pre>',print_r($thedata),'</pre>';
//more code here for comaparing of datas...
1.) What is wrong with file("upload/payment.csv") when reading csv file?
2.) What is the best code in reading a csv file that is applicable on the
system, not changing the whole code. Should remain the foreach loop.
3.) Is fgetcsv much better for the existing code? What changes should be made?
Yes, You can use "fgetcsv" for this purpose.
The fgetcsv() function parses a line from an open file.This function returns the CSV fields in an array on success, or FALSE on failure and EOF.
check the examples given below
eg1 :
<?php
$file = fopen("contacts.csv","r");
print_r(fgetcsv($file));
fclose($file);
?>
eg 2:
<?php
$file = fopen("contacts.csv","r");
while(! feof($file))
{
print_r(fgetcsv($file));
}
fclose($file);
?>
Link : https://gist.github.com/jaywilliams/385876
I am trying to upload a CSV file and get it's contents into an array, but I am getting this an error: (Multiples of this error on each line after 10)
Notice: Undefined offset: 1 in C:\xampp\htdocs\amazon\upload_file.php on line 10
Below is a sample of my code:
if ($handle = file_get_contents($_FILES["file"]["tmp_name"])) {
$data = array();
while ($csv = array(file_get_contents($_FILES["file"]["tmp_name"]))) {
$data = array(
'order-id' => $csv[0],
'order-item-id' => $csv[1], //This is line 10.
'purchase-date' => $csv[2],
'payments-date' => $csv[3],
file() opens puts each as an array element. fgetcsv() and its family of functions are very useful when dealing with csv files.
Your code array(file_get_contents($_FILES["file"]["tmp_name"])) will only ever have one element because file_get_contents returns a string.
This issue comes if your file has only one line.
I guess you need to do this.
$row = explode(",", $csv[0]);
$data = array(
'order-id' => $fileArray[0],
'order-item-id' => $row[1], //This is line 10.
'purchase-date' => $row[2],
'payments-date' => $row[3]
);
Also, you can use functions like fgetcsv() to parse your CSV file.
I have a csv file I need to cleanup. It contains 13 fields, but I only need 7 (Business, Address, City, St, Zip, Phone, Email)
I need to run through all of the records and create a new output of just the records with email addresses.
In nutshell... I load the original file, run the for loop, explode the results, then look for the records where the $tmp[10] index is not null. I then get the rest of the rest of required fields, and do a foreach loop and fwrite the results to a new csv file.
Depending on how I tweak the code, I get either...
A text file of just email addresses.
or
A text file of just the last record with an email address.
I have been working on this too long and I just need a fresh set of eyes to point out the problem. I am new to php, and want to make this work. Thanks on advance.
<?php
// See if file exists and is readable
$file = 'uploads/AK_Accountants.csv';
$newfile = basename($file,".csv");
$newfile = $newfile.Date("Ymd").".csw";
$fileNew = fopen('uploads/AK_'.Date("Ymd").'.csv','w+');
// Read the file into an array called $sourcefile
$sourcefile = file($file);
// Loop through the array and process each line
for ($i = 0; $i < count($sourcefile); $i++) {
// Separate each element and store in a temp array
$tmp = explode('","', $sourcefile[$i]);
// Assign each element of the temp array to a named array key
if ($tmp[10] != "") {
$sourcefile[$i] = array('Business_Name' => $tmp[1], 'Address' => $tmp[3], 'City' => $tmp[4], 'State' => $tmp[5], 'Zip' => $tmp[6], 'Phone' => $tmp[7], 'Email' => $tmp[10]);
foreach($sourcefile[$i] as $key => $value);
fwrite($fileNew, $value);
}
}
?>
From a quick glance:
foreach($sourcefile[$i] as $key => $value);
fwrite($fileNew, $value);
should be
foreach($sourcefile[$i] as $key => $value){
fwrite($fileNew, $value);
}
Also, you have
$newfile = $newfile.Date("Ymd").".csw";
rather than what I assume should be
$newfile = $newfile.Date("Ymd").".csv";
Your last foreach statement is terminated by a ';' and has no code block. Once the foreach statement has finished iterating you'll get the last value written to file i.e. just the email address.
You currently have
foreach (... ) ;
fwrite(...);.
but you probably mean
foreach( ... ) {
fwrite(...) ;
}
Been there, done that :)
HTH