Convert csv to excel with PHPExcel in laravel? - php

i have found this answer ,
PHP Converting CSV to XLS - phpExcel error
but i have tried it in Laravel 4 and i am not able to get it to work , any help would be appreciated.
My Code
public function CsvExcelConverter($filename){
$objReader = Excel::createReader('CSV');
$objReader->setDelimiter(";");
$objPHPExcel = $objReader->load('uploads/'.$filename);
$objWriter = Excel::createWriter($objPHPExcel, 'Excel5');
//new file
$new_filename = explode('.',$filename);
$new_name = $new_filename[1];
$objWriter->save($new_name.'.xls');
return $new_name.'.xls';
}

thank for the answers, but for some reason we cant seem to set the delimiter on load but i have found that you can set it in the config file .
vendeor/maatwebsite/excel/src/config/csv.php
then just specify the delimiter. this way when loading the file it actually separates each entry and when converting it each entry is in its own cell.
thanks for all the help.

/* Get the excel.php class here: http://www.phpclasses.org/browse/package/1919.html */
require_once("../classes/excel.php");
$inputFile=$argv[1];
$xlsFile=$argv[2];
if( empty($inputFile) || empty($xlsFile) ) {
die("Usage: ". basename($argv[0]) . " in.csv out.xls\n" );
}
$fh = fopen( $inputFile, "r" );
if( !is_resource($fh) ) {
die("Error opening $inputFile\n" );
}
/* Assuming that first line is column headings */
if( ($columns = fgetcsv($fh, 1024, "\t")) == false ) {
print( "Error, couldn't get header row\n" );
exit(-2);
}
$numColumns = count($columns);
/* Now read each of the rows, and construct a
big Array that holds the data to be Excel-ified: */
$xlsArray = array();
$xlsArray[] = $columns;
while( ($rows = fgetcsv($fh, 1024, "\t")) != FALSE ) {
$rowArray = array();
for( $i=0; $i<$numColumns;$i++ ) {
$key = $columns[$i];
$val = $rows[$i];
$rowArray["$key"] = $val;
}
$xlsArray[] = $rowArray;
unset($rowArray);
}
fclose($fh);
/* Now let the excel class work its magic. excel.php
has registered a stream wrapper for "xlsfile:/"
and that's what triggers its 'magic': */
$xlsFile = "xlsfile://".$xlsFile;
$fOut = fopen( $xlsFile, "wb" );
if( !is_resource($fOut) ) {
die( "Error opening $xlsFile\n" );
}
fwrite($fOut, serialize($xlsArray));
fclose($fOut);
exit(0);

If you use the maatwebsite/excel library in Laravel, you can only use native PHPExcel instance methods, not static methods. To convert from CSV to excel, this code can be found at Documentation page
Excel::load($filename, function($file) {
// modify file content
})->setFileName($new_name)->store('xls');

In theory, you should create your custom class to set delimiter:
class CSVExcel extends Excel {
protected $delimiter = ';';
}
and now you could use:
CSVExcel::load('csvfilename.csv')->setFileName('newfilename')->export('xls');
But the problem is, that $delimiter isn't used in this case. Delimiter support seems to be added not long time ago, so maybe there is a bug or it needs to be used in the other way. I've added issue just in case for that: https://github.com/Maatwebsite/Laravel-Excel/issues/262

Related

Automating CSV conversion from iso-8859-2 to utf-8

I have quite a few CSV files that are unfortunately encoded with iso-8859-2 (according to Brackets). I would like to iterate over these files with PHP and convert them.
I found https://csv.thephpleague.com/9.0/converter/charset/ but the way I can use the conversion function is uncertain to me.
Their example code
use League\Csv\CharsetConverter;
$csv = new SplFileObject('/path/to/french.csv', 'r');
$csv->setFlags(SplFileObject::READ_CSV | SplFileObject::SKIP_EMPTY);
$encoder = (new CharsetConverter())->inputEncoding('iso-8859-15');
$records = $encoder->convert($csv);
This is my code so far that is part of a form to upload one file and save the contents to the database for testing. It of course saves the text in the incorrect format.
$db = ConnectDB::getConnection('address_dtb');
$sql = " ... ";
$stmt = $db->prepare($sql);
$rowCount = 0;
$temp_name = $_FILES['adresscsv']['tmp_name'];
$file_handle = fopen($temp_name, 'r');
while (($items = fgetcsv($file_handle, 1000, ';')) !== FALSE) {
if($flag) { $flag = false; continue; }
$stmt->execute($items);
$rowCount++;
}
fclose($file_handle);
ConnectDB::closeConnection($db);
What is the correct way to use the PHP CSV library above to iterate over locally saved files in a for loop to automate the process?
I ended up using iconv as hinted.
$files = glob('address/*.csv');
foreach ($files as $csv) {
$file_data = file_get_contents($csv);
$utf8_file_data = iconv('Windows-1250', 'UTF-8', $file_data);
file_put_contents($csv, $utf8_file_data);
}
You do not have to use a library. There is a function in PHP that can do that iconv

Reading large text files efficiently

I have a couple of huge (11mb and 54mb) files that I need to read to process the rest of the script. Currently I'm reading the files and storing them in an array like so:
$pricelist = array();
$fp = fopen($DIR.'datafeeds/pricelist.csv','r');
while (($line = fgetcsv($fp, 0, ",")) !== FALSE) {
if ($line) {
$pricelist[$line[2]] = $line;
}
}
fclose($fp);
.. but I'm constantly getting memory overload messages from my webhost. How do I read it more efficiently?
I don't need to store everything, I already have the keyword which exactly matches the array key $line[2] and I need to read just that one array/line.
If you know the key why don't you filter out by the key? And you can check memory usage with memory_get_usage() function to see how much memory allocated after you fill your $pricelist array.
echo memory_get_usage() . "\n";
$yourKey = 'some_key';
$pricelist = array();
$fp = fopen($DIR.'datafeeds/pricelist.csv','r');
while (($line = fgetcsv($fp, 0, ",")) !== FALSE) {
if (isset($line[2]) && $line[2] == $yourKey) {
$pricelist[$line[2]] = $line;
break;
/* If there is a possiblity to have multiple lines
we can store each line in a separate array element
$pricelist[$line[2]][] = $line;
*/
}
}
fclose($fp);
echo memory_get_usage() . "\n";
You can try this (I have not checked if it works properly)
$data = explode("\n", shell_exec('cat filename.csv | grep KEYWORD'));
You will get all the lines containing the keyword, each line as an element of array.
Let me know if it helps.
I join what user2864740 said : "The problem is the in-memory usage caused by the array itself and is not about "reading" the file"
My Solution is :
Split your `$priceList` array
Load only 1 at time a splitted Array in memory
Keep the other splitted Arrays in an intermediate file
N.B: i did not test what i've written
<?php
define ("MAX_LINE", 10000) ;
define ("CSV_SEPERATOR", ',') ;
function intermediateBuilder ($csvFile, $intermediateCsvFile) {
$pricelist = array ();
$currentLine = 0;
$totalSerializedArray = 0;
if (!is_file()) {
throw new Exception ("this is not a regular file: " . $csv);
}
$fp = fopen ($csvFile, 'r');
if (!$fp) {
throw new Exception ("can not read this file: " . $csv);
}
while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) {
if ($line) {
$pricelist[$line[2]] = $line;
}
if (++$currentLine == MAX_LINE) {
$fp2 = fopen ($intermediateCsvFile, 'a');
if (!$fp) throw new Exception ("can not write in this intermediate csv file: " . $intermediateCsvFile);
fputs ($fp2, serialize ($pricelist) . "\n");
fclose ($fp2);
unset ($pricelist);
$pricelist = array ();
$currentLine = 0;
$totalSerializedArray++;
}
}
fclose($fp);
return $totalSerializedArray;
}
/**
* #param array : by reference unserialized array
* #param integer : the array number to read from the intermediate csv file; start from index 1
* #param string : the (relative|absolute) path/name of the intermediate csv file
* #throw Exception
*/
function loadArray (&$array, $arrayNumber, $intermediateCsvFile) {
$currentLine = 0;
$fp = fopen ($intermediateCsvFile, 'r');
if (!$fp) {
throw new Exception ("can not read this intermediate csv file: " . $intermediateCsvFile);
}
while (($line = fgetcsv($fp, 0, CSV_SEPERATOR)) !== FALSE) {
if (++$currentLine == $arrayNumber) {
fclose ($fp);
$array = unserialize ($line);
return;
}
}
throw new Exception ("the array number argument [" . $arrayNumber . "] is invalid (out of bounds)");
}
Usage example
try {
$totalSerializedArray = intermediateBuilder ($DIR . 'datafeeds/pricelist.csv',
$DIR . 'datafeeds/intermediatePricelist.csv');
$priceList = array () ;
$arrayNumber = 1;
loadArray ($priceList,
$arrayNumber,
$DIR . 'datafeeds/intermediatePricelist.csv');
if (!array_key_exists ($key, $priceList)) {
if (++$arrayNumber > $totalSerializedArray) $arrayNumber = 1;
loadArray ($priceList,
$arrayNumber,
$DIR . 'datafeeds/intermediatePricelist.csv');
}
catch (Exception $e) {
// TODO : log the error ...
}
You can drop the
if ($line) {
That only repeats the check from the loop condition. If your file is 54MB, and you are going to retain every line from the file, as an array, plus the key from column 3 (which is hashed for lookup)... I could see that requiring 75-85MB to store it all in memory. That isn't much. Most wordpress or magento pages using widgets run 150-200MB. But if your host is set low it could be a problem.
You can try filtering out some rows by changing the if($line) to a if($line[1] == 'book') to reduce how much you store. But the only sure way to handle storing that much content in memory is to have that much memory available to the script.
You can try set bigger memory using this. You can change limit how you want.
ini_set('memory_limit', '2048M');
But also depents how you want that script use.

Continue Loop if Term is in an Array (odd result)

The script in question takes an excel file of language vocabulary (French to English, etc) and creates XML (zips and downloads) to format a crossword generator we use.
It works, but I've been asked to remove any duplicate terms as an enhancement. Below is the original code in full, and then the new code to skip duplicate terms. With the new code, everything runs, but it creates a corrupt ZIP. Please see the before and after code and tell me what is going on:
Full before working code:
<?php
/** Error reporting */
error_reporting(E_ALL);
ini_set('display_errors', TRUE);
ini_set('display_startup_errors', TRUE);
define('EOL',(PHP_SAPI == 'cli') ? PHP_EOL : '<br />');
/** PHPExcel */
require_once 'Classes/PHPExcel.php';
require_once 'Classes/PHPExcel/IOFactory.php';
/** Functions */
require_once 'zip.php';
require_once 'named_to_number.php';
if ($_FILES["file"]["error"] > 0) {
echo "Error: " . $_FILES["file"]["error"] . "<br>";
}
/** Create Excel Object using PHPExcel **/
$inputFileName = $_FILES["file"]["tmp_name"];
$objPHPExcel = PHPExcel_IOFactory::load($inputFileName);
$objWorksheet = $objPHPExcel->getActiveSheet();
/** Get how many rows **/
$highestRow = $objWorksheet->getHighestRow();
/** Keeps track of chapters to make new files on change **/
$chapter = -1;
/** For keeping track of the files created to zip & delete. **/
$files_to_zip = array();
/** Iterates through every row, writing data from cells into XML files. **/
for ($row = 2; $row <= $highestRow; ++$row) {
//remove spaces
$term = str_replace(' ', '', $objWorksheet->getCellByColumnAndRow(1, $row)->getValue());
//skip terms if they are too long or contain a non-alpha character.
if (strlen($term) >= 17 || !preg_match('/^\p{L}+$/ui', $term)) {
continue;
}
//translates accented characters to numbered HTML code
$term = named_to_number(htmlentities($term, ENT_SUBSTITUTE , 'UTF-8'));
/** Checks first column to see if the chapter has changed.
If it has, the current file will be closed and a new one opened. **/
if ($chapter != $objWorksheet->getCellByColumnAndRow(0, $row)->getValue()){
fwrite ($f, "</words>\n</content>");
fclose($f);
if (strlen($objWorksheet->getCellByColumnAndRow(0, $row)->getValue()) < 2) {
$filename = 'ch0' . $objWorksheet->getCellByColumnAndRow(0, $row)->getValue() . '.xml';
} else {
$filename = 'ch' . $objWorksheet->getCellByColumnAndRow(0, $row)->getValue() . '.xml';
}
$f = fopen($filename, 'a');
/** Add to the list of files to zip and delete **/
array_push($files_to_zip, $filename);
fwrite ($f, "<content>\n<words>\n");
/** Update chapter value **/
$chapter = $objWorksheet->getCellByColumnAndRow(0, $row)->getValue();
}
/** Write terms **/
$data =
"<word><entry>" . $term .
"</entry><clue>" . $objWorksheet->getCellByColumnAndRow(2, $row)->getValue() .
"</clue></word>\n";
fwrite ($f, $data);
}
fwrite ($f, "</words>\n</content>");
fclose($f);
/** Removes any blank ch.xml files **/
if(($key = array_search('ch.xml', $files_to_zip)) !== false) {
unset($files_to_zip[$key]);
unlink('chapter.htm');
}
$zip = create_zip($files_to_zip, 'crossword.zip');
foreach($files_to_zip as &$del){
unlink($del);
}
header("Content-disposition: attachment; filename=crossword.zip");
header("Content-type: application/zip");
readfile("crossword.zip");
unlink("crossword.zip");
?>
Relevant snippet where code was added (comments only related to new code):
//array to be used to hold terms to check against
$used = array();
for ($row = 2; $row <= $highestRow; ++$row) {
$term = str_replace(' ', '', $objWorksheet->getCellByColumnAndRow(1, $row)->getValue());
//Add term to running array to check against to see if it exists
array_push($used, $term);
//Added a third condition to check the array to see if the term exists. I have also tried this using isset with array_flip.
if (strlen($term) >= 17 || !preg_match('/^\p{L}+$/ui', $term) || in_array($term, $used)) {
continue;
}
Again, the odd thing is that the script runs, but it just produces a bad zip. It is definitely that third conditional that is tripping the script up. I have tried it in its own if statement (just to be sure the syntax was right), but the problem persists.
Please help!
Thanks,
Mike

Is it possible to get a resource handle from file data?

I currently have a function that takes a csv file, and returns an array of the data from it. I want to minimally alter this function to take the file data instead of the file itself.
Using the following code, I would like to get a resource handle from the passed in data, instead of from a file so that I can keep the rest of the function the same. Is this possible?
public function returnRawCSVData($filepath, $separator = ',')
{
$file = fopen($filepath, 'r');
$csvrawdata = array();
//I WANT TO CHANGE $filepath to $file_data and get a resource from it to pass into fgetcsv below.
while( ($row = fgetcsv($file, $this->max_row_size, $separator, $this->enclosure)) != false ) {
if( $row[0] != null ) { // skip empty lines
}
}
fclose($file);
return $csvrawdata;
}
It seems you're looking for a way to create a new file resource from the source text?
If so, you can create a file resource in-memory like so:
/**
* Return an in-memory file resource handle from source text
* #param string $csvtxt CSV source text
* #return resource File resource handle
*/
public static function getFileResourceFromSrcTxt($csvtxt)
{
$tmp_handle = fopen('php://temp', 'r+');
fwrite($tmp_handle, $csvtxt);
return $tmp_handle;
}
/**
* Parse csv data from source text
* #param $file_data CSV source text
* #see self::getFileResourceFromSrcTxt
*/
public function returnRawCSVData($file_data, $separator = ',')
{
$file = self::getFileResourceFromSrcTxt($file_data);
$csvrawdata = array();
while( ($row = fgetcsv($file, $this->max_row_size, $separator, $this->enclosure)) != false ) {
if( $row[0] != null ) { // skip empty lines
// do stuff
}
}
fclose($file);
}
It's worth noting that you can also use "php://memory" in place of "php://temp" -- the difference being that 'memory' ONLY stores things in memory while 'temp' will store something in memory until it reaches a given size (the default is 2 MB), then transparently switch to the filesystem.
Find out more about what the php docs say on this topic ...
If you're trying to pass around file handles, you can treat them as such:
$in_file = fopen('some_file.csv', 'r');
// Do stuff with input...
// Later, pass the file handle to a function and let it read from the file too.
$data = doStuffWithFile($in_file);
fclose($in_file);
function doStuffWithFile($file_handle)
{
$line = fgetcsv($file_handle);
return $line;
}

Export large rows to Excel document, in small memory footprint

I am using PHPExcel to create an Excel document, using data from a MySQL database. My script must execute in under 512MB of RAM, and I am running into trouble as my export reaches 200k records:
PHP Fatal error: Allowed memory size of...
How can I use PHPExcel to create large documents in as little amount of RAM as possible?
My current code:
// Autoload classes
ProjectConfiguration::registerPHPExcel();
$xls = new PHPExcel();
$xls->setActiveSheetIndex(0);
$i = 0;
$j = 2;
// Write the col names
foreach ($columnas_excel as $columna) {
$xls->getActiveSheet()->setCellValueByColumnAndRow($i,1,$columna);
$xls->getActiveSheet()->getColumnDimensionByColumn($i)->setAutoSize(true);
$i++;
}
// paginate the result from database
$pager = new sfPropelPager('Antecedentes',50);
$pager->setCriteria($query_personas);
$pager->init();
$last_page = $pager->getLastPage();
//write the data to excel object
for($pagina =1; $pagina <= $last_page; $pagina++) {
$pager->setPage($pagina);
$pager->init();
foreach ($pager->getResults() as $persona) {
$i = 0;
foreach ($columnas_excel as $key_col => $columnas) {
$xls->getActiveSheet()->setCellValueByColumnAndRow($i,$j,$persona->getByName($key_col, BasePeer::TYPE_PHPNAME));
$i++;
}
$j++;
}
}
// write the file to the disk
$writer = new PHPExcel_Writer_Excel2007($xls);
$filename = sfConfig::get('sf_upload_dir') . DIRECTORY_SEPARATOR . "$cache.listado_personas.xlsx";
if (file_exists($filename)) {
unlink($filename);
}
$writer->save($filename);
CSV version:
// Write the col names to the file
$columnas_key = array_keys($columnas_excel);
file_put_contents($filename, implode(",", $columnas_excel) . "\n");
//write data to the file
for($pagina =1; $pagina <= $last_page; $pagina++) {
$pager->setPage($pagina);
$pager->init();
foreach ($pager->getResults() as $persona) {
$persona_arr = array();
// make an array
foreach ($columnas_excel as $key_col => $columnas) {
$persona_arr[] = $persona->getByName($key_col, BasePeer::TYPE_PHPNAME);
}
// append to the file
file_put_contents($filename, implode(",", $persona_arr) . "\n", FILE_APPEND | LOCK_EX);
}
}
Still have the problem of RAM when Propel makes requests to the database, it's like Propel, does not release the RAM every time you make a new request. I even tried to create and delete the Pager object in each iteration
Propel has formatters in the Query API, you'll be able to write this kind of code:
<?php
$query = AntecedentesQuery::create()
// Some ->filter()
;
$csv = $query->toCSV();
$csv contains a CSV content you'l be able to render by setting the correct mime-type.
Since it appears you can use a CSV, try pulling 1 record at a time and appending it to your CSV. Don't try to get all 200k records at the same time.
$cursor = mysql_query( $sqlToFetchData ); // get a MySql resource for your query
$fileHandle = fopen( 'data.csv', 'a'); // use 'a' for Append mode
while( $row = mysql_fetch_row( $cursor ) ){ // pull your data 1 record at a time
fputcsv( $fileHandle, $row ); // append the record to the CSV file
}
fclose( $fileHandle ); // clean up
mysql_close( $cursor );
I'm not sure how to transform the CSV into an XLS file, but hopefully this will get you on your way.

Categories