I have a report generation PHP program which used to work fine before. I have used 2 3rd party libraries in the program: Google image chart library ( returns image if I supply values in url ) and tcpdf ( for pdf generation ). I am using mysql not mysqli for queries. There are lots of queries and loops in the page.
Before it used to take less than 3 minutes to generate the report, I am using an ajax call to generate the report which gives a completed message once the file generation is done. This program saves the pdf file in a folder and I have a link with same name to download the file.
Recently when I checked its not generating properly.
Error was TCPDF unable to get the image. This was because of the google chart library not returning the image properly. When I access the chart url in browser it gives me the image without any issue but If I give it in an image src inside a php file, its not showing. So I decided to save the file in a folder using functions like file_get_contents,file_put_contents and link it in image src. This part is now working correctly I can see the image.
But now the problem is it is taking a lot of time to generate the report, even in local environment. I tried to generate the report without the chart priniting but even then its taking time. In between it was 25 minutes n all and now its close to 10 minutes to generate a 40 page pdf file.
I really don't know why its taking so much time. All of this was working fine before and now its not working. Only thing that changed was google image chart library but now even without(commented that part and checked) that also its taking time.
How do I speed this up ? Is there any way to check which part of program is slow.
Tried xdebug but its output file is more than 400 mb and webgrind is not able to process it.
Please help.
Your next step is to troubleshoot performance.
Is TCPDF doing a lot of work you don't need done? Presumably you've seen the tips from TCPDF's author on increasing performance, and put them into practice. http://www.tcpdf.org/performances.php
Are some of your MySQL queries inefficient? Obtain an interactive connection to your MySQL server, using phpMyAdmin or a similar command-line tool. While your pdf-creation process is running, repeatedly issue this command
SHOW FULL PROCESSLIST
It presents an INFO column showing the active MySQL query for each connection. It also shows each query's elapsed time in milliseconds. If you have queries that run for many hundreds of milliseconds, you might consider using MySQL's
EXPLAIN command to analyze those queries. Often adding an appropriate index to a MySQL table can dramatically speed things up.
Is the machine running your PDF program short on RAM? use a performance monitor like *nix top or Windows perfmon to take a look.
Is your 40-page report, simply put, a huge job to create? If so, you might consider switching to a faster report-generation program than PHP + TCPDF.
Sorted out.
The issue is with the database, one of the tables has more 120000 records in it. Deleted irrelevant records, not a permanent solution but now it generates the same thing in 2.1 minutes.
Now I can't do the same thing in my production server. I would love to get your inputs on how to optimize the database.
Thank You
Related
I've messed around with FFmpeg some time ago and remember using it to fetch preview images for video files. My question is, is this the correct path to be going down for the purpose of getting images from a Twitch live stream? What I need to do is get a screenshot of the end of a Twitch stream (the final scoreboard in a video game) and save that screen shot.
I'm pretty sure I could get the code written, but I want to make sure there isn't a better way of accomplishing this task because it seems like it would have gotten easier over the years. If there are any other libraries/APIs that would be more efficient than executing FFMpeg all the time.
I hope there's been some improvement in PHP video handling, but I've searched around and can't find anything.
Since it's a live stream with a variable duration then the real problem is not getting the screenshot but rather finding out when you should take the screenshot aka the exact moment when the score is shown.
If you know it's always at the end and you if can guess the stream duration then you could record the last few minutes then get the duration of the recorded video, seek to end minus x seconds and grab your screenshot.
PHP video handling usually means building a wrapper class around a tool like ffmpeg that will create an argument list and execute the binary for you.
Trying to find an efficient way to convert Graphs 50 of them to Images.
Then take those images and insert them into a PDF that is built on the server.
Problem is this process takes so long on our server as the serve must open each of the 50 graphs then take a snapshot of each one to turn them into images. Then the server take the users data and formulate a PDF that also contains these images.
Any solutions? Current process takes around 10 minutes for our clients and sometimes hangs up during conversion.
We currently use Fusion Charts as our Charting/Graph program.
PHP for our coding and conversion into PDF.
This question is probably vague or worded horribly, but I am not sure where else to go for solutions and this community is extremely helpful.
I have a very large image generated on the fly with PHP and outputted to the browser. (it's 5000px wide and 1000-2000px tall. It's a plot of the daily user activity on my site).
The problem is that nowadays the plot is too big and the PHP script gives memory exhausted errors (tough the generated PNG itself is quite small) and I can't get the image due to this.
Is there way to output this large image in multiple parts somehow using GD in PNG format?
(ps: the host where I run the site uses safe mode, so I can't modify the configuration and I think they're using the default PHP installation.)
EDIT1: It's an admin script. No users see it except me.
EDIT2: and example image can be seen here: http://users.atw.hu/calmarius/trash/wtfb2/x.png
(I also have the option to group the tracks by IP address.)
Every user+IP pair has its own 24 hour track on the plot. And every green mark denotes an user activity. As you can see this image can be output track by track. And there is no need to output and generate the whole thing all once.
This website will be an online strategy game and I want to use this graph in the future to make detecting multiaccounts easier. (Users who are trying to get advantage by registering multiple accounts over those ones who only have 1.) But this is a different problem.
I'm using PHP script because I'm too lazy to export the requestlog from the database, download it and feed the data to a program that would make the plot for me. ;)
Set the memory limit to unlimited before processing the image.
ini_set('memory_limit', '-1');
It'd help to say how you're generating the image (GD library, ImageMagick) and how you're outputting it. Are you saving the file to a directory and then using readfile() to output it? If yes, fopen / fread / echo combination is about 50%-60% faster than using readfile() to output files to the browser. Are you using gzip compression? What's the time limit on php execution? What's the exact error message you're getting?
I've got a report that can generate over 30,000 records if given a large enough date range. From the HTML side of things, a resultset this large is not a problem since I implement a pagination system that limits the viewable results to 100 at a given time.
My real problem occurs once the user presses the "Get PDF" button. When this happens, I essentially re-run the portion of the report that prints the data (the results of the report itself are stored in a 'save' table so there's no need to re-run the data-gathering logic), and store the results in a variable called $html. Keep in mind that this variable now contains 30,000 records of data plus the HTML needed to format it correctly on the PDF. Once I've got this HTML string created, I pass it to TCPDF to try and generate the PDF file for the user. However, instead of generating the PDF file, it just craps out without an error message (the 'Generating PDf...') dialog disappears and the system acts like you never asked it to do anything.
Through tests, I've discovered that the problem lies in the size of the $html variable being passed in. If the report under 3K records, it works fine. If it's over that, the HTML side of the report will print but not the PDF.
Helpful Info
PHP 5.3
TCPDF for PDF generation (also tried PS2PDF)
Script Memory Limit: 500 MB
How would you guys handle this scale of data when generating a PDF of this size?
Here is how I solved this issue: I noticed that some of the strings that I was having in my HTML output had some slight encoding issues - I ran htmlentities on those particular strings as I was querying the database for them and that cleared the problem.
Don't know if this was what was causing your problem, but my experience was very similar - when I was trying to output an HTML table that had a large size, with about 80.000 rows, TCPDF would display the page header but nothing table-related. This behaviour would be the same with different sets of data and different table structures.
After many attempts I started adding my own pagination - every 15 table rows, I would break the page and add a new table to the following page. That's when I noticed that every once and a while I would get blank pages between a lot of full and correct ones. That's when I realised that there must be a problem with those particular subsets of data, and discovered the encoding issue. It may be that you had something similar and TCPDF was not making it clear what your problem was.
Are you using the writeHTML method?
I went through the performance recommendations here: http://www.tcpdf.org/performances.php
It says "Split large HTML blocks in smaller pieces;".
I found that if my blocks of HTML went over 20,000 characters the PDF would take well over 2 minutes to generate.
I simply split my html up into the blocks and called writeHTML for each block and it improved dramatically. A file that wouldn't generate in 2 minutes before now takes 16 seconds.
TCPDF seems to be a native implementation of PDF generation in PHP. You may have better performance using a compiled library like PDFlib or a command-line app like htmldoc. The latter will have the best chances of generating a large PDF.
Also, are you breaking the output PDF into multiple pages? I.e. does TCPDF know to take a single HTML document and cut it into multiple pages, or are you generating multiple HTML files for it to combine into a single PDF document? That may also help.
I would break the PDF into parts, just like pagination.
1) Have "Get PDF" button on every paginated HTML page and allow downloading of records from that HTML page only.
2) Limit the maximum number of records that can be downloaded. If the maximum limit reaches, split the PDF and let the user to download multiple PDFs.
I have to make an image of a dynamic page i.e. the page keeps on changing in every 5 minutes.
I want to make images of that very page that keeps on changing so that i can have its records saved in the form of images.
How can i do that using php??
i have no idea about this and a little elaboration in answers will be highly appreciated!!
Two steps:
1: Create a script that captures the current data in image form.
If you provide more information about what you mean when you say "create an image of dynamic data", I can probably point you to some resources you can use. For now, just have a look at the GD library.
2: Set up a job that runs the script every 5 minutes
This can be done via Cron. I would suggest investigating if you can run the script when the data changes, instead of at specific intervals.
http://www.devarticles.com/c/a/PHP/Generating-Images-on-the-Fly-With-PHP/
http://www.thesitewizard.com/php/create-image.shtml
Getting a screenshot of a web page isn't an easy task.
You can choose one of the online services that do that for you and you can download the images from there.
Otherwise, I have found a solutions using webkit and python but you will need full access to your linux server in order to install the necessary packages, then you will be able to call that script from php and get your screenshots.