I just joined a project, and have been going over the code. We need to export a lot of data out to Excel for our internal users. There are roughly 5 people who would have access to this functionality at a given time. In order to output to Excel, here's what I found:
retrieve data from DB, store in $_SESSION
show HTML page view of data
when the user wants to export
retrieve the DB data from $_SESSION
create a string in memory of a CSV
print the HTTP Headers with Excel as the filetype
print out the CSV formatted strings
This storage in $_SESSION is happening even when the user is not going to export. This strikes me as terribly inefficient, since the $_SESSION variable could explode in size, since each of the DB table retrievals can be up to 30MB per table, and the expiration on $_SESSION is set to 24 hours. So potentially, there could up to 5 users in the system, with up to 150MB $_SESSION variables. Sound about right?
Has anyone else ever seen something like this? Is this like a bootleg Memcache? Wouldn't it be best to just write the data to a flat-file that's updated once every few hours?
I do store database some data in session, like ID or small object that I use on every page.
But when it come to larger dataset that I can't extract on the fly for each page load, I often prefer to store them in a MEMORY/HEAP table ( or a temporary file ), and just store an ID in the session so I'll be able to extract them easily.
You might want to take a look at this question about session max size:
Maximum size of a PHP session
I have seen this as well and it is a bad idea. Sometimes you want to display on screen a table of data but also make it available for export, but there is no good reason for stuffing into session memory. If the OS needs to swap and the session gets written to file then you have file IO speed issues, so in some cases it is likely slower than a fresh query to the database.
$_SESSION in your example is being used to store data which is not needed in order to ensure consistency across page views, so this is pointless. It's there to store stuff like last_page_viewed, not to duplicate the DB itself. The only reason to do it like that is if the DB calls to get the data are so hideously expensive that even with the storage inefficiency you describe, it improves performance. This is very unlikely and it sounds like lazy coding.
If they want to export, there should be a zip function that reads all the data using the same SQL and packages it into an excel file on demand. Ideally using MVC so that the same code can be fed into the HTML or the zipper function ;)
Your solution could work if your database not update frequently otherwise your users may get outdated data. (And I don't think it's worth to store in the session data anyway.)
As you explained here, I think you are going to use this in a LAN and you don't have more than 5 concurrent users. If I'm right about that why don't you just read database straightly from the database and show it on HTML(I guess you can use paging and don't want to show all 30MB data in a single HTML page) again Export all the data to Excel straight from the DB when user request it :)
Related
Good day to all,
I have a form with around 90 to 100 fields, divided into sub forms, which are loaded using ajax each time a form has to be displayed. but i would like to retain the data on the form fields every time a subform is loaded(lets say on an accidental page refresh or even if the user is switching between sub forms). What is the best way that this can be done.
I was thinking that i can store it in cookies or lets say in the database. But, storing it in the database would mean that i would have to query for the fields data every time a sub form is loaded. And again, if it was cookies, it has to read the data stored in the cookie files. I need some help with deciding what is the most efficient way, more in terms of speed.
What is the best way among these, or is there any other possibility to retain the fields data in the sub forms, every time they are loaded (which are loaded via AJAX each time.)
I am working with PHP and Codeigniter framework.
Thanks!!
A form like that needs to be durably stored. I would consider session state to smooth out the sub form loads, with writes to the database whenever the user updates something of consequence. Personally, I would start with a database-only solution, and then add session state if performance is an issue.
Cookies aren't meant to store large amounts of data. Even if it were possible, they bloat the request considerably (imagine 100 form fields all being transmitted with every request).
Local storage in the browser is also an option, though I would consider other options first.
I would first simplify it by using serialize:
$data = serialize(array_merge($_POST,$olddata));
Then that may be enough for you, but it's now super easy to store it anywhere since it is just a string. To reform it into its original state:
$data = unserialize($data);
.. wherever you end up pulling it from - database,session,etc..
Prose of database
It can also access from other computer too
You can store far far more data then cookie
Cones
If you retrive data by ajax it coukd cose more load on server
Cookie
Faster than database no query , fetch and all process .
Cones
Limited amount of space
However you can use local storage
So answer is database storage
Well, Maybe 5M is not that much, but it needs to receive a XML based on the following schema
http://www.sat.gob.mx/sitio_internet/cfd/3/cfdv3.xsd
Therefore I need to save almost all the information per row. Now by law we are required to save the information for a very long time and eventually this database will be very very veeeeery big.
Maybe create a table every day? something like _invoices_16_07_2012.
Well, I'm lost..I have no idea how to do this, but I know is possible.
On top of that, I need to create a PDF and 2 more files based on each XML and keep them on HD.
And you should be able to retrieve your files quickly using a web site.
Thats a lot of data to put into one field in a single row (not sure if that was something you were thinking about doing).
Write a script to parse the xml object and save each value from the xml in a separate field or in a way that makes sense for you (so you'll have to create a table with all the appropriate fields). You should be able to input your data as one row per xml sheet.
You'll also want to shard your database and spread it across a cluster of servers on many tables. MySQL does support this but I've only boostrapped my own sharding mechanism before.
Do not create a table per XML sheet as that is overkill.
Now, why do you need mysql for this? Are you querying the data in the XML? If you're storing this data simply for archival purposes, you don't need mysql, but can instead compress the files into, say, a tarball and store them directly on disk. Your website can easily fetch the file in this way.
If you do need a big data store that can handle 5M transactions with as much data as you're saying, you might also want to look into something like Hadoop and store the data in a Distributed File System. If you want to more easily query your data, look into HBase which can run on top of Hadoop.
Hope this helps.
I have considered about it for a long time.
I think I can't use all the API/PHP extension (e.g. memcache, APC, Xcache) that need to install something in my remote Linux server, as my web host server is a shared server, what I just can do is to place files/scripts in the httpdocs folder.
Is there any suggestion for me that can let me programmatically use caching and access the memory?
Actually what I aim at is to find a "place" to save some data, that can be accessed in higher speed than entering the DB to fetch data, and also to reduce the loading to DB.
That means, it is not a must to use memory, if someone can give any other effective suggestions. e.g. will using text file be a good choice?(actually I am just guessing it)
The PHP version of mine is 5.2.17. And I am using MySQL DB.
Hope someone can give me suggestions
Flat files will always be the EASIEST way for caching, but it will be slower than accessing data directly from memory. You can use MySQL tables that are stored in memory. you need to change the engine used by tables to memory. NOTE that this will work only if your db is on the same server as web server.
Set up an in memory table with two columns key and value. variable name will be a key and its contents are values. if you need to cache array, objects then serialize the data before storing it.
If you need to limit the size of in memory table add one more column hitCount. for each read increase the count by one. while inserting new row, check for max number of rows and if its reached a limit delete the row with lowest hitCount.
To check which one is faster (file caching or in memory cache) use following code
<?php
function getTime()
{
$a = explode (' ',microtime());
return(double) $a[0] + $a[1];
}
?>
<?php
$Start = getTime();
//Data fetching tasks comes here
$end = getTime();
echo "time taken = ".number_format(($End - $Start),2)."seconds";
?>
If possible let us know how efficient it is... Thanks
You could very easily just use flat text files as a cache if your DB queries are expensive. Just like you would use memcache with a key/value system, you can use filenames as keys and the context of the files as values.
Here's an example that caches the output of a single page in a file; you could adapt it to suit your needs: http://www.snipe.net/2009/03/quick-and-dirty-php-caching/
Flat file is the easiest way to cache business logic, queries etc on a shared server.
To cache any DB requests your best bet is to fetch the results, serialize them and store them in a file with a possible expiry date (if required). When you need to fetch those results again just pull in the file and unserialize the previously serialized data.
Also if the data is user based cookies and sessions will work too, for as long as the user stays on the application at least. If your pulling a lot of data it would still be better to go with the first option and just save the files based on a user/session id.
Depends on the size of data to cahce.
Based on the restriction of your server environment:
Use flat file( or maybe sqlite db) to cache your data for large data set (e.g., user
preference, user activity logs.)
Use share memory to cache your data for the smaller data set (e.g., system counter, system
status.)
Hope this helps.
I have a php/mysql application, part of which allows the user to upload a csv file. The steps in the process are as follows:
User uploads a file, and it gets parsed to check for validity
If the file is valid, the parsed information is displayed, along with options for the user to match columns from the csv file to database columns
Import the data - final stage where the csv data is actually imported into the database
So, the problem that I have at this point is that the same csv file gets parsed in each of the above 3 steps - so that means 3 parses for each import.
Given that there can be up to 500 rows per csv file, then this doesn't strike me as particularly efficient.
Should I rather temporarily store the imported information in a database table after step 1? If so, I would obviously run clear up routines to keep the table as clean as possible. The one downside is that the csv imports can contain between 2 and 10 columns - so I'd have to make a database table of at least 11 columns (with an ID field)...which would be somewhat redundant in most cases.
Or should I just stick with the csv parsing? Up to 500 rows is quite small...
Or perhaps there is another better alternative?
In PHP, you can store data into the Session memory for later use. This allows you to parse the CSV file only once, save it in the Session memory and use this object in all of the later steps.
See http://www.tizag.com/phpT/phpsessions.php for a small tutorial.
Let me explain a bit more.
Every time a web browser requests a page from the server, PHP executes the PHP script associated with the web page. It then sends the output to the user. This is inherently stateless: the user requests something, you give something back -> end of transaction.
Sometimes, you may want to remember something you calculated in your PHP script and use it the next time the page is requested. This is stateful, you want to save state across different web requests.
One way is to save this result in the database or in a flat file. You could even add an identifier for the currently connected user, so you use a per-user file or save the current user in your database.
You could also use a hidden form and save all of the data as hidden input fields. When the user presses "Next", the hidden input fields are sent back to the PHP script.
This is all very clumsy. There is a better way: session memory. This is a piece of memory that you can access, which is saved across different PHP calls. It is perfect for saving temporary state information like this. The session memory can be indexed per application user.
Please note that there are frameworks that take this a lot further. Java SEAM has an APPLICATION memory, a SESSION memory, a CONVERSATION memory, a PAGE memory and even a single EVENT memory.
I had to do a similar thing for importing users into a database. What I ended up doing was this:
Import and parse CSV
Assign data to an array
Next page had a bunch of hidden form fields each with the data (ex. <input type="hidden" name="data[]" value="thedata" />)
Post it over and add the data to the database
It ended up working well for me. You can also save the data to session variables.
I'd just stick with parsing it 3 times. PHP is slow anyways, as are the network latencies for using the database or sending information to the client. What's most important is that your code is maintainable and extensible. The slowest part of a computer program is the developer.
See http://en.wikipedia.org/wiki/Program_optimization#When_to_optimize
http://ubiquity.acm.org/article.cfm?id=1147993
http://benchmarksgame.alioth.debian.org/u32/which-programs-are-fastest.html
Hope that helps...
I am currently working on a web-page in HTML5/Javascript, where the user can record the value of sliders changed in a period of time. For example, when the user click Record, a timer will start and every time the user move a slider, its value will be saved in an array. When the user stop the record and play, all sliders will then be 'played' as he recorded them. I store the value in an array, something like that:
array[3] : {timeStamp, idSlider, valueSlider}
The array can actually become pretty massive as the user can then record over the already recorded settings without losing the previous one, this allow the user to change multiple sliders for the same time stamp.
I now want to be able to save this array somewhere (on server side), so the user can come back to the website later on, and just load its recorded settings, but I am not sure of the best approach to do that.
I am thinking of a dataBase, but not sure if this will be a bit slow to save and load from the server, plus my DataBase capacity is pretty small (around 25 Mo on my OVH server). I am thinking of maybe an average of 800 entries to save.
Maybe in a file (XML ?), but then I have no idea how to save that on my server-side...
Any other idea is welcome as I am a bit stuck on this one.
Thanks & sorry for any english mistakes,
Cheers,
Mat
Either serialize() or json_encode() it, save it as a longtext database record.
http://us.php.net/serialize
http://us.php.net/json-encode
http://dev.mysql.com/doc/refman/5.0/en/blob.html
I think 800 is not as massive as you think. You could just save it to a database and transfer it with JSON. I believe the database should be pretty efficient and not waste a lot of storage, but this can depend on how you setup your tables. Don't use char() columns, for example.
To see which uses more or less storage you could calculate how much space 1000 entries would take, then put them in the database and see how much space it uses and how much it's wasting.
If you really were dealing with a very large number of items then network performance will probably become your first bottleneck. For loading I would then stream javascript to the browser so that the browser doesn't have to load the whole thing into memory. Also, for sending you would want to create manageable chunks of maybe 1000 at a time and send them in these chunks. But, this is assuming you're dealing with more data.
You can try storing the whole json object as plain text in user-specific files. That would make serving the settings really easy, since it would be plain json that simple needs to be evaluated.
It seems a bit unorthodox though. When you say 800 entries do you mean
1. id : 0, timestamp : XXXXXXXXXXXXX, idSlider : X, etc
2. id : 1, timestamp : XXXXXXXXXXXXX, idSlider....
Or 800 user entries? Because you could save the whole object in the database as well and save yourself from executing lots of "costy" queries.
If you are concerned more about storage size than read/write performance you could encode the array as json string, compress it using zlib library and save it as a file.
This is a perfect use case for MongoDB.