Getting OOM from large dataset - php

I'm trying to create a spreadsheet (XLSX) from a array containing 60k entries. When it exports, it gives me a spreadsheet with a PHP OOM warning on it nothing else.
$spreadsheet = new Spreadsheet();
$spreadsheet->setActiveSheetIndex(0);
$activeSheet = $spreadsheet->getActiveSheet();
$rowIndex = 2;
foreach ($this->_values as $_val){
$activeSheet->setCellValueByColumnAndRow(1, $rowIndex, trim($_val['item1']));
$activeSheet->setCellValueByColumnAndRow(2, $rowIndex, trim($_val['item2']));
$activeSheet->setCellValueByColumnAndRow(3, $rowIndex, trim($_val['item3']));
$activeSheet->setCellValueByColumnAndRow(4, $rowIndex, trim($_val['item4']));
$activeSheet->setCellValueByColumnAndRow(5, $rowIndex, trim($_val['item5']));
$activeSheet->setCellValueByColumnAndRow(6, $rowIndex, trim($_val['item6']));
$activeSheet->setCellValueByColumnAndRow(7, $rowIndex, trim($_val['item7']));
$activeSheet->setCellValueByColumnAndRow(8, $rowIndex, trim($_val['item8']));
$activeSheet->setCellValueByColumnAndRow(9, $rowIndex, trim($_val['item9']));
$activeSheet->setCellValueByColumnAndRow(10, $rowIndex, trim($_val['item10']));
$activeSheet->setCellValueByColumnAndRow(11, $rowIndex, trim($_val['item11']));
$rowIndex += 1;
}
$spreadsheet->garbageCollect();
$writer = new Xlsx($spreadsheet);
$writer->setPreCalculateFormulas(false);
$writer->setUseDiskCaching(true);
$writer->save("php://temp");
$spreadsheet->disconnectWorksheets();
unset($spreadsheet);

An out of memory error is because your system is, well, out of memory. You can simply increase the amount of memory using something like ini_set('memory_limit', '750M'); assuming your system has the space
Or, if XLSX isn't needed (that is: if you could use a CSV instead, which Excel can still open) you could stream the data to the spreadsheet instead of loading it all into memory at once, then printing. That would look like this:
foreach ($this->_values as $_val){
for($i=1; $i<=11; $i++) {
echo trim($_val['item'.$i]);
if($i<11) echo ",";
}
echo "\n";
}
You could also include the CSV header at the top of the PHP file, so the page would prompt a download instead of showing you the contents of the CSV file. This is the CSV header cal: header("Content-type: text/csv");

Related

Not saving excel with PHPSpreadsheet in symfony 4

I'm using "phpoffice/phpspreadsheet": "^1.13", can not save updates into uploaded file. I can read data from it but can not save. The other thing to mention I am running it in the Process in background.
use PhpOffice\PhpSpreadsheet\IOFactory;
use PhpOffice\PhpSpreadsheet\Reader\Xlsx;
use PhpOffice\PhpSpreadsheet\Writer\Xlsx as XlsxWriter;
//read from file
$reader = new Xlsx();
$reader->setLoadSheetsOnly(["Gifts"]);
$spreadsheet = $reader->load($file);
$worksheet = $spreadsheet->getActiveSheet();
$highestRow = $worksheet->getHighestRow();
for ($row = 5; $row < $highestRow; $row++) {
$results[] = [
'id' => $worksheet->getCell('K'.$row)->getValue(),
];
}
//write to file
$spreadsheet = IOFactory::load($file);
$sheet = $spreadsheet->getSheetByName('Gifts');
$sheet->setCellValue('A1', 'Import Status');
$writer = new XlsxWriter($spreadsheet);
ob_end_clean();
$writer->save($file);
If not use ob_end_clean(); the file is saving corrupt and LibbreOffice cannot open it with ob_end_clean(); the file is opening but without changes.
The purpose is to create entities in db reading data from Excel, but because the file can be large I want to run it in the background, maybe it is not the best approach but the best I could do for now so the process is as follows:
upload file
start new process
Process::fromShellCommandline('php /var/www/app/bin/console app:import:excel "'.$file.'"')->start();
redirect user to another page
send an email with a report of imported stuff
All the steps are working fine except saving Excel, when I am opening the uploaded file it does not have any changes, in other question I have found advice like to add ob_end_clean(); or die() it helped at least to open the file without ob_end_clean(); it shows that it is broken.

Export data into CSV file in Symfony

I have console app made in Symfony3, where user can import CSV file (which is validate) into database. I need to put records which haven't passed validation into separate file.
I use LeagueCSV to read CSV file and I try to use it to write unvalidated records but it doesn't work.
This is my code:
$reader = Reader::createFromPath($input->getArgument('lokalizacja'));
$reader->setDelimiter(';');
$reader->setHeaderOffset(0);
$results = $reader->getRecords();
foreach ($results as $row) {
$year = $row['description'];
$isValid = false;
if ($row['qty'] > 0 && $row['price'] > 0 && !empty($row['mpn'])) {
$isValid = true;
$rok = filter_var($row['description'], FILTER_SANITIZE_NUMBER_INT);
$product = (new Produkt())
->setMpn($row['mpn'])
->setQty($row['qty'])
->setYear($year)
->setPrice($row['price']);
$this->em->persist($product); }
if ($row['qty'] == 0 || $row['price'] == 0 || empty($row['mpn'])) {
$writer = Writer::createFromPath('/path/to/saved/file.csv', 'w+');
$writer->insertOne([$row['mpn'], $row['qty'], $row['price'],
$row['description']]);
continue;
}
}
$this->em->flush();
All records which passed validation are successfully saved in the database but I have problem with others records. In new CSV file I have only first, one record which haven't passed validation and nothing more. What am I doing wrong? I tried with
$writer->insertAll($results); //using an array
Or with if...else statment but that's nothing.
Also I made ...else statement where unvalidated records are saved in other table in database and its works but I don't know how to immediately convert them into CSV file.
Don't know symfony but CSV output is pretty simple. FWIW...
Pass this an array, like a fetchall resultset.
<?php
public function outputCSV($data, $useKeysForHeaderRow = true) {
if ($useKeysForHeaderRow) {
array_unshift($data, array_keys(reset($data)));
}
$outputBuffer = fopen("php://output", 'w');
foreach($data as $v) {
fputcsv($outputBuffer, $v);
}
fclose($outputBuffer);
}
header('Content-Type: text/csv; charset=utf-8');
header('Content-Disposition: attachment; filename="FooBarFileName_' . date('Ymd') . '.csv"');
header("Pragma: no-cache");
header("Expires: 0");
$this->outputCSV($results);
If you look at the doc page for the writer you will see a warning at the top which states
When inserting records into a CSV document using League\Csv\Writer, first insert all the data that need to be inserted before starting manipulating the CSV. If you manipulate your CSV document before insertion, you may change the file cursor position and erase your data.
Your code calls
$writer = Writer::createFromPath('/path/to/saved/file.csv', 'w+');
$writer->insertOne([$row['mpn'], $row['qty'], $row['price'], $row['description']]);
In every iteration the condition is met, this appears to be overwriting or dropping your previous insert every time. You should declare the $writer once before the loop starts in order to preserve each insert.
$writer = Writer::createFromPath('/path/to/saved/file.csv', 'w+');
foreach ($results as $row) {
// do stuff with $writer
}

PHPExcel library hangs with relative "big" files

I'm trying to export some records to excel from my MySQL (webserver) and when the query returns >4k records the script hangs the web browser and temporaly the web hosting.
My PHP_version is 5.2.13-pl1-gentoo and the memory_limit configurated in php.ini is 128M
The result excel only have one column and N rows. With 100 or 200 rows the php script runs fine.
This is the php script
<? session_start();
ini_set('memory_limit', '1024M');
set_time_limit(0);
include("include/conexion.php");
require_once 'include/PHPExcel/Classes/PHPExcel.php';
require_once 'include/PHPExcel/Classes/PHPExcel/IOFactory.php';
$objPHPExcel = new PHPExcel();
$objPHPExcel->getProperties()->setCreator("Name")
->setLastModifiedBy("Name")
->setTitle("Listado")
->setSubject("Listado")
->setDescription("Listado.")
->setKeywords("Listado")
->setCategory("Listado");
$query = explode("|",stripcslashes($_POST['query']));
$objPHPExcel->getActiveSheet()->setTitle('List');
$resEmp = mysql_query ($query, $conexion ) or die(mysql_error());
$tot = mysql_num_rows($resEmp);
$num_fields = mysql_num_fields($resEmp);
$fistIndex = $objPHPExcel->getActiveSheet()->getCellByColumnAndRow(0, 1)->getColumn();
$lastIndex = $objPHPExcel->getActiveSheet()->getCellByColumnAndRow($num_campos - 1, 1)->getColumn();
//tittles
for ($e=0;$e < $num_fields;$e++){
$objPHPExcel->getActiveSheet()->setCellValueByColumnAndRow($e, 2, utf8_decode(ucwords(mysql_field_name($resEmp,$e))));
$objPHPExcel->getActiveSheet()->getColumnDimension($objPHPExcel->getActiveSheet()->getCellByColumnAndRow($e, 2)->getColumn())->setAutoSize(true);
}
//color tittles
$objPHPExcel->getActiveSheet()->getStyle( $fistIndex.'1:'.$lastIndex.'2' )->getFill()->setFillType(PHPExcel_Style_Fill::FILL_SOLID)->getStartColor()->setRGB('c5c5c7');
$objPHPExcel->getActiveSheet()->getStyle( $fistIndex.'1:'.$lastIndex.'2' )->getFont()->setBold(true);
if(isset ( $_POST ['mail'] )){
$objPHPExcel->getActiveSheet()->setCellValueByColumnAndRow(0, 2, "Email");
$emails = array();
for ($row = 0; $row < $totEmp; $row++) {
//more than one mail in field separated by ";"
$aux = explode(";", mysql_result($resEmp,$row,$col));
for($i=0; $i<count($aux); $i++){
$cleaned = utf8_encode(strtolower(trim($aux[$i])));
//filter repeated mails
if(!in_array($cleaned, $emails) && $aux[$i] != ""){
$num_rows = $objPHPExcel->getActiveSheet()->getHighestRow();
$objPHPExcel->getActiveSheet()->insertNewRowBefore($num_rows + 1, 1);
array_push($emails, $cleaned);
$objPHPExcel->getActiveSheet()->setCellValueByColumnAndRow(0, $num_rows + 1, $cleaned);
}
}
}
}
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
header('Content-type: application/vnd.ms-excel');
header("Content-Disposition: attachment; filename=".$nom_archivo.".xlsx");
// Write file to the browser
$objWriter->save('php://output');
exit();
?>
When enter to the script run a mysql query and then, iterate the result to get the mail field, if the obtained mail not exist in a array this mail is inserted in excel
I've tried to set
ini_set('memory_limit', '1024M');
set_time_limit(0);
But the problem persist.
Any idea to solve problem?
Thanks a lot
EDIT 1
I've updated the code with the recommendations and now works fine.
Anyway How can I get if occurs any error or the memory usage just before of hanging?
How can I get the max memory_limit available to set with ini_set('memory_limit', '2048M'); ?
<? session_start();
ini_set('memory_limit', '2048M');
set_time_limit(0);
include("include/conexion.php");
require_once 'include/PHPExcel/Classes/PHPExcel.php';
require_once 'include/PHPExcel/Classes/PHPExcel/IOFactory.php';
$objPHPExcel = new PHPExcel();
$objPHPExcel->getProperties()->setCreator("Name")
->setLastModifiedBy("Name")
->setTitle("Listado")
->setSubject("Listado")
->setDescription("Listado.")
->setKeywords("Listado")
->setCategory("Listado");
$activeSheet = $objPHPExcel->getActiveSheet();
$query = explode("|",stripcslashes($_POST['query']));
$activeSheet->setTitle('List');
$resEmp = mysql_query ($query, $conexion ) or die(mysql_error());
$tot = mysql_num_rows($resEmp);
$num_fields = mysql_num_fields($resEmp);
$fistIndex = $activeSheet->getCellByColumnAndRow(0, 1)->getColumn();
$lastIndex = $activeSheet->getCellByColumnAndRow($num_campos - 1, 1)->getColumn();
//tittles
for ($e=0;$e < $num_fields;$e++){
$activeSheet->setCellValueByColumnAndRow($e, 2, utf8_decode(ucwords(mysql_field_name($resEmp,$e))));
$activeSheet->getColumnDimension($activeSheet->getCellByColumnAndRow($e, 2)->getColumn())->setAutoSize(true);
}
//color tittles
$activeSheet->getStyle( $fistIndex.'1:'.$lastIndex.'2' )->getFill()->setFillType(PHPExcel_Style_Fill::FILL_SOLID)->getStartColor()->setRGB('c5c5c7');
$activeSheet->getStyle( $fistIndex.'1:'.$lastIndex.'2' )->getFont()->setBold(true);
if(isset ( $_POST ['mail'] )){
$activeSheet->setCellValueByColumnAndRow(0, 2, "Email");
$emails = array();
for ($row = 0; $row < $totEmp; $row++) {
//more than one mail in field separated by ";"
$aux = explode(";", mysql_result($resEmp,$row,$col));
for($i=0; $i<count($aux); $i++){
$cleaned = utf8_encode(strtolower(trim($aux[$i])));
//filter repeated mails
if(!in_array($cleaned, $emails) && $aux[$i] != ""){
array_push($emails, $cleaned);
}
}
}
for ($row = 0; $row < count($emails); $row++) {
$activeSheet->setCellValueByColumnAndRow(0, $row + 3, $emails[$row]);
}
}
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
header('Content-type: application/vnd.ms-excel');
header("Content-Disposition: attachment; filename=".$nom_archivo.".xlsx");
// Write file to the browser
$objWriter->save('php://output');
exit();
?>
Seems this library has serious problem in parsing large excel spreadsheets, I'd this issue already & I couldn't find a proper solution. I guess this is normal behaviour because this library is written fully in PHP that causes a lot of parsing overhead.
I strongly suggest you to use a excel parsing PHP-extension like this one.
As another thinkable solution [if its possible], you can break down your big file to several smaller files (e.g by sheets), otherwise I guess you should use a faster CPU or use another library or programming language to parse your exel files (e.g. apache-poi in java, maybe with a PHP/Java bridge).
Unfortunately, PHPExcel is not good for performing with large data because PHP is not really a good binary file processing language.
Some people export their data to XML format of excel (http://en.wikipedia.org/wiki/Microsoft_Office_XML_formats) and it can work well. However, the xml format does not have full features of excel binary file and of course it will have a bigger file size.
In order to work with the large data (import/export to binary excel file), our system now using libxl which will cost you 199$ for a license, and php_excel which is a wrapper for libxl. In effect, our system now export a excel file with more than 5k of rows in about just only some seconds using libxl and I think it's an only solution for you until now to use binary excel.
P/s: The $objPHPExcel->getActiveSheet() also have a cost, so you could store it value to a variable for reusing later which will help you to speed up your code a little bit.
I had this problem but after changed some options in php.ini and scripts, I could reduce file from 28 MB to 4 MB.
increase memory_limit=2048M in php.ini.
change max_execution_time to more seconds.
in the script yo should use Excel2007 like below:
ob_end_clean();
header('Content-Type: application/vnd.ms-excel');
header("Content-Disposition: attachment;filename=$date.xls");
header('Cache-Control: max-age=0');
ob_end_clean();
$objWriter =PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
$objWriter->save('php://output');

Exporting data to excel (multiple sheets) using php

I am trying to export data to Excel using this PHP Class, so far things are working fine and the export is being generated. But now I have a new requirement of generating multiple sheets inside a single excel file.
For example if i have two arrays, i want both to be on separate sheets.
$myarray1 = array (
1 => array ("Oliver", "Peter", "Paul"),
array ("Marlene", "Mica", "Lina")
);
$myarray2 = array (
1 => array ("Oliver", "Peter", "Paul"),
array ("Marlene", "Mica", "Lina")
);
At present both arrays are being exported on a single sheet
$xls = new Excel_XML;
$xls->addArray ( $myarray );
$xls->addArray ( $myarray2 );
$xls->generateXML ( "testfile" );
I am wondering if someone tried this before and was able to achieve it and I will appreciate any help I can get on this.
i would suggest you to use PHPExcel library.supports variety of formats, can do visual formatting and is easy to use.
You can find more about it at their webpage: http://phpexcel.codeplex.com/
You can do a lot more of course, reading excel files, setting visual styles, creating plots, expressions and lot more.
you can even use fgetcsv http://php.net/manual/en/function.fgetcsv.php
this example using PHPExcel
function exportToExcelsheets($data, $fileName){
/* Create new PHPExcel object*/
$objPHPExcel = new PHPExcel();
$sheet_index = 0;
foreach ($data as $s=>$sheet){
/* Create a first sheet, representing sales data*/
$alpha = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','V','W','X','Y','Z'];
$objPHPExcel->setActiveSheetIndex($sheet_index);
$head_keys = array_keys($sheet[0]);
foreach ($head_keys as $a=>$headval){
$objPHPExcel->getActiveSheet()->setCellValue($alpha[$a].'1', $headval);
}
$i=2;
foreach($sheet as $row) {
$index = 0;
foreach ($row as $v=>$value){
$value = isset($value)?$value:'';
$objPHPExcel->getActiveSheet()->setCellValue($alpha[$index].$i,$value);
$index++;
}
$i++;
}
/*Rename sheet*/
$objPHPExcel->getActiveSheet()->setTitle('sheet_'.$s);
/* Create a new worksheet, after the default sheet*/
$objPHPExcel->createSheet();
$sheet_index++;
}
/* Redirect output to a client’s web browser (Excel5)*/
header('Content-Type: application/vnd.ms-excel');
header("Content-Disposition: attachment; filename=\"$fileName\"");
header('Cache-Control: max-age=0');
//$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel5');
//$objWriter->save('php://output');
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel2007');
$objWriter->save('php://output');
}

PHP spreadsheet writer writing variables to excel

I am trying to write variables from an array to the excel but i cannot write in certain instances
Code is as below
$workbook =new Spreadsheet_Excel_Writer();
$workbook->send("response.xls");
$row=1;
$col=1;
$workbook->setVersion(8);
$worksheet =& $workbook->addWorksheet('My sheet ');
$Allquestions = $questions->getQuestions();
foreach ($Allquestions as $qkey=>$qval) {
foreach ($studentResponses as $rval){
$astresp= $rval->getResponse($qkey);
//$worksheet->write($row,$col,'hiii');
$worksheet->write($row, $col, $astresp);
$col++;
}
$row++;
$col=1;
}
$workbook->close();
the $astresp when changed to 'hihi' prints hihi in the excel but when it remains as $astresp it fails to open excel.

Categories