I have a report thats generated using PHPexcel. Its quite a large report 8 workbooks/tabs in total. As you can imagine it uses up quite alot of memory and im starting to run into problems with it.
I was wondering what options would be best suited for me to reduce the memory load that it takes to generate this report.
The idea I had was to create the spreadsheet with 8 empty worksheets. Save the file then load the file with the
$objReader->setLoadSheetsOnly($sheetname);
Im not sure if im heading of in the right direction with this though?
The best option to start with would be to look at the cell caching options that are explicitly written to reduce memory usage. These are described in section 4.2.1 of the developer documentation
Related
So I have a situation where I need to offer the user a multi-sheet excel document where each sheet has thousands of rows and ~10 columns. Data is coming from multiple MySQL queries.
Currently using "Laravel Excel" library to do this, but it uses up way too much memory and is giving me huge scalability problems.
So I have an idea to use MySQL OUTFILE to write a set of csv files on disk, one for each sheet, and then to create an xls document and write the previously written csv data as sheets in the xls.
Is there a method to accomplish writing the csv contents to a sheet "in bulk" per-say without iterating line-by-line through the csv or using up a large amount of memory (like writing to disk directly perhaps?)
Thanks for any help!
I had a very similar problem recently. My solution was to use the very lightweight PHP library PHP_XLSXWriter.
You can find it here: https://github.com/mk-j/PHP_XLSXWriter
It streams the output so it doesn't have to retain as much in memory.
In my usage case, I broke apart the "writeStream" method into three methods: one for each the header and footer, and one for the sheet content (i.e. the actual rows). This way I could write the header and then use Laravel's "chunking" feature, to get even more gradual with the writes.
The time increased slightly, but executing the script went from ~200Mb of RAM usage to under 15Mb!
Excel file formats (both BIFF and OfficeOpenXML) are not conducive to writing line-by-line like a CSV, because data isn't stored linearly. This means that all PHP libraries for writing native format Excel files have to work in PHP memory to manage the order of writing data to that file format, which means they will all consume large amounts of memory for larger volumes of data.
Laravel Excel is a wrapper around PHPExcel, which provides some options for reducing memory usage (eg. caching cell data to disk or SQLite database rather than holding it all in PHP memory), albeit at a cost in execution speed. What I don't know is whether Laravel Excel provides calls to enable these caching methods, though I believe some options are available allowing you to configure this.
Your alternative on a Linux platform is using non-PHP solutions like libXl or PUNO with Open/Libre Office
Using PHPExcel I can run each tab separately and get the results I want but if I add them all into one excel it just stops, no error or any thing.
Each tab consists of about 60 to 80 thousand records and I have about 15 to 20 tabs. So about 1600000 records split into multiple tabs (This number will probably grow as well).
Also I have tested the 65000 row limitation with .xls by using the .xlsx extension with no problems if I run each tab it it's own excel file.
Pseudo code:
read data from db
start the PHPExcel process
parse out data for each page (some styling/formatting but not much)
(each numeric field value does get summed up in a totals column at the bottom of the excel using the formula SUM)
save excel (xlsx format)
I have 3GB of RAM so this is not an issue and the script is set to execute with no timeout.
I have used PHPExcel in a number of projects and have had great results but having such a large data set seems to be an issue.
Anyone every have this problem? work around? tips? etc...
UPDATE:
on error log --- memory exhausted
Besides adding more RAM to the box is there any other tips I could do?
Anyone every save current state and edit excel with new data?
I had the exact same problem and googling around did not find a valuable solution.
As PHPExcel generates Objects and stores all data in memory, before finally generating the document file which itself is also stored in memory, setting higher memory limits in PHP will never entirely solve this problem - that solution does not scale very well.
To really solve the problem, you need to generate the XLS file "on the fly". Thats what i did and now i can be sure that the "download SQL resultset as XLS" works no matter how many (million) row are returned by the database.
Pity is, i could not find any library which features "drive-by" XLS(X) generation.
I found this article on IBM Developer Works which gives an example on how to generate the XLS XML "on-the-fly":
http://www.ibm.com/developerworks/opensource/library/os-phpexcel/#N101FC
Works pretty well for me - i have multiple sheets with LOTS of data and did not even touch the PHP memory limit. Scales very well.
Note that this example uses the Excel plain XML format (file extension "xml") so you can send your uncompressed data directly to the browser.
http://en.wikipedia.org/wiki/Microsoft_Office_XML_formats#Excel_XML_Spreadsheet_example
If you really need to generate an XLSX, things get even more complicated. XLSX is a compressed archive containing multiple XML files. For that, you must write all your data on disk (or memory - same problem as with PHPExcel) and then create the archive with that data.
http://en.wikipedia.org/wiki/Office_Open_XML
Possibly its also possible to generate compressed archives "on the fly", but this approach seems really complicated.
I got a database with a lot of number statistical related info, which i'm looking to generate out in an excel file.
So first of all of all, do anyone of you have experience with any popular libraries/scripts to generate excel files?
Secondly, im thinking it would be cool to display some of these very dull and boring numbers, in pie charts and graphs. Again, do anyone of you have experience with any popular libraries/scripts to generate such things inside an excel file?
Thanks a lot!
Try PHPExcel library - http://phpexcel.codeplex.com/
It satisfied all my needs, including the ones you specified, but not limited to only them.
Might need to increase memory limit a bit though since in large data cases as the one you specified it consumes a significant amount of memory.
I can second the PHPExcel library, I have used it a couple of times in the past and it works great. Unfortunately you cannot create charts with it as far as I remember. Someone else brought up this question before, maybe you will find something helpful here
Create Excel chart programmatically in PHP
I'm attempting to generate some reports dynamically, very simple HTML tables with borders.
I've tried TCPDF and it renders up to 400 rows just fine but anything more than that (about 20 pages) it can't handle it. DOMPDF can't even do that.
These reports can be thousands of rows.
Any idea on a faster library or a better plan of attack?
Try php-wkhtml2x php extension. It uses popular web engine webkit(Chrome and Safari uses that)
I use the FPDF library, the output is fast and resource-efficient. Try it out...
http://www.fpdf.org/
I don't know if these methods are the fastest, but they can certainly handle more than 20 pages.
You could use latex in combination with php:
http://www.linuxjournal.com/article/7870
or Zend_Service_LiveDocx_MailMerge
http://www.phphatesme.com/blog/webentwicklung/pdf-erzeugung-mit-dem-zend-framework/
Try DocRaptor.com. It's a web-based app that converts html to pdf. Easy to use.
It's possible that depending on the report that PHP is not the right solution you and might consider another alternative language such as perl to accomplish this. I have no experience with other server-side languages but it is something to keep in mind. Definitely follow #Pekka's advice and determine the limits and work on adjusting those.
Fact
Thousands of rows can be handled by the php.
My assumption
Most probably you will be fetching data from db and saving to an array and then you will be looping to write the rows.
This will eat memory.
My suggestion
Try to write into the pdf at time of fetching from db. remove the step of storing into an array.
check execution time and memory allocated in php ini.
At last when you generate it , think PDF can handle it or not :-)
surely It will have huge size .
I am generating PDF dynamically using html2ps PHP library. I want to decrease time of generating that PDF .I want to reduce that PDF generation time .Is there any way to reduce the time or optimize it?
Help me please.
Are you running this code on recent hardware?
While it may sound like avoiding the solution, on the Coding Horror Blog (whom happens to be the guys who made this site) they preach that you shouldn't spend time tweaking performance if your hardware is limited.
If you're doing this on a single core CPU (ex: Pentium 4), you are wasting your time worrying about what library to use or what code to change. Even the slowest Core 2 Duos and newer AMDs start at 2x faster than the best Pentium 4.
PS: I wasn't able to find the article on their site to link for you.
PPS: Most Pentium 4 motherboards support the 65nm Core 2 Duos.
One method of optimizing is to try another library. I use dompdf when I need to convert HTML to PDF, and I haven't found any need for optimizing, it's very fast, supports CSS properly and produces accurate results.
Reduce the complexity of the output.
Reduce the output quantity.
If the PDF generation is impacting other operations, delegate it to another process or server.
I'm sure you can avoid a lot of processing by skipping the HTML/CSS step(s) and go directly to PDF. Check FPDF or PDFLib