I have 1000 objects, where each object needs a "key".
For example
$this->setItem("1", $object);
$this->setItem("2", $object);
My problem is that each time I use $this->setItem() or $this->addItem() Zend is creating a new folder with a .dat file.
I would like to create only one .dat file for all of the objects, however I am able to call it with $this->getItem("key")
Therefore, I am asking what is the difference between these two functions?
Of course I could achieve the goal with the addItem() function.
The purpose of caching is to retrieve your cached results in a fast way.
If ZF2 would aggregate all your different cache keys (and their data) in a single file it would be impossible to fetch your data fast because all the expensive file searching / splitting etc. which needs to happen.
Generating a single file for each cache key makes this process simple. ZF2 will create a MD5 hash of the cache key and can directly retrieve the file with that name from the filesystem. The different directories you see are just a substring of the hash, so the amount of directories will be limited.
setItem will always write data to the specified key(overwrite if data already exists).
addItem will only write data if there's no data present yet.
Related
Let's say I want to have a variable which should keep it's value.
Simple example:
<?php
$files = array('file1.txt','file2.txt','file3.txt'); // list of the files to process
$file = file_get_contents($files[$i]);
process_the_file($file);
$i++;
exit();
So I'd like to process only one file per script run. And in my case the index of the currently processed file is $i.
From my pont of view there are two methods available:
Store variable in a file. I can serialise (and unserialize upon initialization) any data and store in a text file. However, I should keep an eye on the script termination to be sure that the needed values will be stored before the script terminated.
Use database. It looks bulky and as previously described - I should keep an eye on storing.
So both of the methods looks too comlicated. Is there any better solution?
The solution depends on the task:
Use a cache system (memcached, redis, etc.) - when you need a runtime storage (i.e. values may be deleted and will not break an application logic).
Use DB(MySQL, Mongo) - when you need a permanent storage for values (i.e. values may not be deleted - this will break an application logic).
Use Files for both variants.
Sessions is good enough when you need to store data for one user between requests.
The best solution is the solution that the most suitable for your task. I would recommend to use either 1 or 2 variant: in projects where I am working we use memcached for a runtime storage and mongodb for a permanent storage.
We have an application that calls an API every 4 hours and gets a dump of all objects, returned in a json format which are then stored in a file.json
The reason we do this is because we need up to date data and we are not allowed to use the api directly to get small portions of this data and also that we need to do a clean up on it.
There is also another problem, we can't call for only the updated records (which is actually what we need)
The way we are currently handling this is by getting the data, storing in a file, load the previous file into memory and compare the values in order to get only the new and the updated ones, once we get the new and updated we go ahead and insert into MySQL
I am currently looking into a different option, what I was thinking is that since since the new file will contain every single record why not query for the needed objects from the file.json when needed?
The problem with that is that some of these files are larger than 50MB (each file contains one of the related tables, being 6 files in total which complete the full relation) and we can't be loading them into memory every time there is a query, does any one know of a DB system that will allow to query on a file or an easier way to replace the old data with the new one with a quick operation?
I think the approach you're using already is probably the most practical, but I'm intrigued by your idea of searching the JSON file directly.
Here's how I'd take a stab at implementing this, having worked on a Web application that used the similar approach of searching an XML file on disk rather than a database (and, remarkably, was still fast enough for production use):
Sort the JSON data first. Creating a new master file with the objects reordered to match how they're indexed in the database will maximize the efficiency of a linear search through the data.
Use a streaming JSON parser for searches. This will allow the file to be parsed object-by-object without needing to load the entire document in memory first. If the file is sorted, only half the document on average will need to be parsed for each lookup.
Streaming JSON parsers are rare, but they exist. Salsify has created one for PHP.
Benchmark searching the file directly using the above two strategies. You may discover this is enough to make the application usable, especially if it supports only a small number of users. If not:
Build separate indices on disk. Instead of having the application search the entire JSON file directly, parse it once when it's received and create one or more index files that associate key values with byte offsets into the original file. The application can then search a (much smaller) index file for the object it needs; once it retrieves the matching offset, it can seek immediately to the corresponding JSON object in the master file and parse it directly.
Consider using a more efficient data format. JSON is lightweight, but there may be better options. You might experiment with
generating a new master file using serialize to output a "frozen" representation of each parsed JSON object in PHP's native serialization format. The application can then use unserialize to obtain an array or object it can use immediately.
Combining this with the use of index files, especially if they're generated as trees rather than lists, will probably give you about the best performance you can hope for from a simple, purely filesystem-based solution.
I ended up doing my own processing method.
I got a json dump of all records which I then processed into single files with each one having all its related records in it, kind of like a join, to avoid the indexing of these files to be long I created multiple subfolders for a block of records, while creating these files I started building an index files which pointed to the directory location of the record which is a tiny file, now every time there is a query I just load the index file into memory which is under 1 MB I then check if the index key exists which is the master key of the record, if it does I then have the location of the file which I then load into memory and has all the required information to use in the application.
The query for these files ended up being a lot faster than querying the DB which works for what we need.
Thank you all for your input as it helped me decide which way to go.
I created a website, and on it the users can upload different files which they can later access through their profiles. Over time, I realized that this method of creating folders and sub-folder for users and is quite cumbersome, as it creates so many folder and sub-folders as more users join and use my website. I'm now thinking of having one folder for all the files but renaming user's files on the fly to something like USERID-FILENAME.
The problem comes when I want to display the files to their specific user; I want to search the files in the folder then grab the files that start with a particular userid e.g if its user of id 200, I want to grab all files that have 200 before the first hyphen and display them.
How can I accomplish this and is it the best method to use?
You probably want something more like a database or a key-value store for this work, rather than your file-system, which will not scale well as/if your site grows significantly.
For example, you could use something like <userid>-<filename> as a key in a key-value store, with the contents of the file set as the value, encoded as a binary blob.
On top of this, you may then need a descriptor that keeps track of all the files a user has ever uploaded, perhaps keyed directly by the userid or by something like, <userid>-files
Then when you're looking up the user, you load their <userid>-files descriptor first, using this to present a list of files they access/control/own, then lookup <userid>-<filename> when they request a specific file.
Thinking on this some more, that may not be sufficient to handle cases of a user uploading the same file multiple times -- you might want to assign IDs to the files themselves, and just keep those IDs in the <userid>-files blob. This has the added advantage that the schema for storing files is user-agnostic, so you could start extending your service to implement sharing functionality (with perhaps another hunk of data describing sharing/access-control properties).
Something like this perhaps:
foreach (glob( ((int)'$userid')'-*') as $file) {
process($file);
}
Every site should have options, changeable via a Control Panel. My new project is going to have this. However, I am curious as to which is the best method for storing these options.
Here are my known methods:
I have tested these, although not very thoroughly, as in all likelihood I will not find the long-term problems or benefits.
Using a MySQL table with fields key and value, where each column
is a new key/value pair. The downside to this is that MySQL can be
slow, and in fact, would require a loop before every page to fetch
the options from the database and parse it into an array.
Using a MySQL table with a field for each value and a single record.
The downside to this is that each new options requires a new field,
and this is not the standard use of MySQL tables, but a big benefit
is that it requires a single function to bring it into a PHP indexed
array.
Using a flat file containing the options array in serialized form,
using the PHP functions serialize and unserialize. The main
problem with this method is that I would have to first traverse to
the file, read in the whole file, and serializing can be slow, so it
would get slower as more options are created. It also offers a small
layer of obfuscation to the data.
Using an ini file. Ini parser's are rather fast, and this options
would make it easy to pass around a site configuation. However, as
above, I would have to traverse to the ini, and also, using an ini
file with PHP is generally unused.
Other formats, such as XML and JSON, have all been considered
too. However, they all require some sort of storage, and I am mostly
curious about the benefits of each kind of storage.
These are my specific idealistic requirements:
So the basic thing I am looking for is speed, security, and portability. I want the configuration to not be human readable (hence, an unencrypted flat file is bad), be easily portable (ruling out MySQL), and have almost zero but constant performance impact (ruling out most options).
I am not trying to ask people to write code for me, or anything like that. I just need a second pair of eyes on this problem, possibly bringing up points that I never factored in.
Thank you for your help
Thank you- Daniel.
Using a MySQL table with fields key and value, where each column is a
new key/value pair. The downside to this is that MySQL can be slow,
and in fact, would require a loop before every page to fetch the
options from the database and parse it into an array.
That is false. Unless you plan on storing couple hundred million configuration pairs, you will be fine and dandy. If you worry about performance using this method, simply cache the query (and wipe the cache only when you make changes inside the table).
This will also give you most flexibility, ease of use and so on.
I have considered about it for a long time.
I think I can't use all the API/PHP extension (e.g. memcache, APC, Xcache) that need to install something in my remote Linux server, as my web host server is a shared server, what I just can do is to place files/scripts in the httpdocs folder.
Is there any suggestion for me that can let me programmatically use caching and access the memory?
Actually what I aim at is to find a "place" to save some data, that can be accessed in higher speed than entering the DB to fetch data, and also to reduce the loading to DB.
That means, it is not a must to use memory, if someone can give any other effective suggestions. e.g. will using text file be a good choice?(actually I am just guessing it)
The PHP version of mine is 5.2.17. And I am using MySQL DB.
Hope someone can give me suggestions
Flat files will always be the EASIEST way for caching, but it will be slower than accessing data directly from memory. You can use MySQL tables that are stored in memory. you need to change the engine used by tables to memory. NOTE that this will work only if your db is on the same server as web server.
Set up an in memory table with two columns key and value. variable name will be a key and its contents are values. if you need to cache array, objects then serialize the data before storing it.
If you need to limit the size of in memory table add one more column hitCount. for each read increase the count by one. while inserting new row, check for max number of rows and if its reached a limit delete the row with lowest hitCount.
To check which one is faster (file caching or in memory cache) use following code
<?php
function getTime()
{
$a = explode (' ',microtime());
return(double) $a[0] + $a[1];
}
?>
<?php
$Start = getTime();
//Data fetching tasks comes here
$end = getTime();
echo "time taken = ".number_format(($End - $Start),2)."seconds";
?>
If possible let us know how efficient it is... Thanks
You could very easily just use flat text files as a cache if your DB queries are expensive. Just like you would use memcache with a key/value system, you can use filenames as keys and the context of the files as values.
Here's an example that caches the output of a single page in a file; you could adapt it to suit your needs: http://www.snipe.net/2009/03/quick-and-dirty-php-caching/
Flat file is the easiest way to cache business logic, queries etc on a shared server.
To cache any DB requests your best bet is to fetch the results, serialize them and store them in a file with a possible expiry date (if required). When you need to fetch those results again just pull in the file and unserialize the previously serialized data.
Also if the data is user based cookies and sessions will work too, for as long as the user stays on the application at least. If your pulling a lot of data it would still be better to go with the first option and just save the files based on a user/session id.
Depends on the size of data to cahce.
Based on the restriction of your server environment:
Use flat file( or maybe sqlite db) to cache your data for large data set (e.g., user
preference, user activity logs.)
Use share memory to cache your data for the smaller data set (e.g., system counter, system
status.)
Hope this helps.