I have some data in MySQL database that are static most of the time. They are mainly almost static values like cities, states, and ethnic. I want to cache them in a variable, preferebly in memory, so that I won't need to perform another query to MySQL every time a page loads.
The problem is, my hosting doesn't support memcache nor APC. The only accelerator I could find is eAccelerator, and I don't think it will do what I have in mind.
Is there any way I can do caching? It is http://www.k-disk.net
Thank you
Caching is a great example of the ubiquitous time-space tradeoff in
programming. You can save time by using space to store results. [1]
There are many ways at many levels to implement cache on a website. Lets look at them starting form the front end and moving towards the backend without getting into too much details.
HTTP caching
See How To Optimize Your Site With HTTP Caching
Application level caching
This is caching of "expensive to query" database objects.
Eg: memcache, caching pages in files etc.
Op-code cache
Eg: PHP accelerator, eAccelerator etc.
Database level cache
Optimizing the database by tuning its parameters based on the need and machine hardware.
In your case, I would recommend tweaking around with my.cnf since given enough RAM, MySQL is quite fast. Just try not to pre-optimize.
You can use the Shared Memory extension.
Look this basic sample: http://www.php.net/manual/en/shmop.examples-basic.php
You could just write a file to your server that saves a serialized php array of variables. Just plug all of your variables into an associative array then serialize and save. Though I honestly don't see why you don't save the variables to a variable table in the database. Its not an expensive operation.
$myvars = array(
'how_high_do_i_jump' => 10,
'which_tv_show_is_best' => 'Glee',
'enable_caching' => true,
'mathematical_solution' => 4534.234
);
$file_str = serialize($myvars);
//Save the file
file_put_contents('myvars.ser', $myvars);
//To reverse just do this
$file = file_get_contents('myvars.ser');
$myvars = unserialize($file);
If that doesn't work for you there is a way to get memcache on your shared host if you have SSH access. I actually did this on hostmonster. Here is a walk through on it (though this is not the article I originally used).
http://andrewpeng.net/posts/2011/06/271273-memcached-and-dreamhost-shared-tutorial.html
You can create a custom cache class storing and reading data from disk, using file_put_contents() and file_get_contents().
Related
Lets assume you're developing a multiplayer game where the data is stored in a MySQL-database. For example the names and description texts of items, attributes, buffs, npcs, quests etc.
That data:
won't change often
is frequently requested
is required on server-side
and cannot be cached locally (JSON, Javascript)
To solve this problem, i wrote a file-based caching system that creates .php-files on the server and copies the entire mysql-tables as pre-defined php variables into them.
Like this:
$item_names = Array(0 => "name", 1 => "name");
$item_descriptions = Array(0 => "text", 1 => "text");
That file contains a loot of data, will end up having a size of around 500 KB and is then loaded on every user request.
Is that a good attempt to avoid unnecessary queries; Considering that query-caching is being deprecated in MySQL 8.0? Or is it better to just get the data needed using individual queries, even if ending up with hundreds of them per request?
I suggest you to use some kind of PSR-6 compilant cache system (it could be filesystem also) and later when your requests grow you can easily swap out to a more performant cache, like a PSR-6 Redis cache.
Example for PSR-6 compatible file system cache.
More info about PSR-6 Caching Interface
Instead of making your own caching mechanism, you can use Redis as it will handle all your caching requirements.
It will be easy to implement.
Follow the links to get to know more about Redis
REDIS
REDIS IN PHP
REDIS PHP TUTORIALS
In my experience...
You should only optimize for performance when you can prove you have a problem, and when you know where that problem is.
That means in practice that you should write load tests to exercise your application under "reasonable worst-case scenario" loads, and instrument your application so you can see what its performance characteristics are.
Doing any kind of optimization without a load test framework means you're coding on instinct; you may be making things worse without knowing it.
Your solution - caching entire tables in arrays - means every PHP process is loading that data into memory, which may or may not become a performance hit in its own right (do you know which request will need which data?). It also looks like you'll be doing a lot of relational logic in PHP (in your example, gluing the item_name to the item_description). This is something MySQL is really good at; your PHP code could easily be slower than MySQL at joins.
Then you have the problem of cache invalidation - how and when do you refresh the cached data? How does your application behave when the data is being refreshed? I've seen web sites slow to a crawl when cached data was being refreshed.
In short - it's a complicated decision, there are no obvious right/wrong answers. My first recommendation is "build a test framework so you can approach performance based on evidence", my second is "don't roll your own - consider using an ORM with built-in cache support", my third is "consider using something like Redis or memcached to store your cache information".
There are many possible solutions, depends on your requirements. Possible solution could be:
File base JSON format caching. Data retrieve from database will be save to a file for next time use before the program process.
Memory base cache, such as Memcached, APC, Redis, etc. Similar the upon solution, better performance but more integrated code required.
Memory base database, such as NoSQL, MongoDB, etc. It is a memory base database.
Multiple database servers, one master write database with multiple salve for read databases, there are a synchronisation between servers.
Quick and minimise the code changes, I suggest using option B.
I'd like to make a very, very small, but persistent data structure that I can reference quickly server-side, and I'm not sure how.
Basically, what I want is an array that holds little structures that hold 3-10 strings in them. The array would be of size somewhere from 50-5,000 (expandable).
I was considering using a database, but that seems like overkill in this case. I was considering using a file that held JSON, but that just doesn't seem right (I think my server would have to load the file, parse the file, then return every time the cgi is called).
I'd like to be able to have PHP get something out of this persistent data structure in constant, fast time every time it's called.
I'm currently using just vanilla Apache and PHP.
Even without a file APC can store those data! apc_fetch and apc_store. The only problem is that the data is restricted to one server, so as soon as you will have clusters or multiple servers they don't share the data. (http://www.php.net/manual/de/ref.apc.php)
If multiple servers are involved, memcached or redis are worth a check. Redis has built-in arrays.
Edit:
Check if json_encode/json_decode are as fast as serialize/unserialize for your scenario or even faster, jsonlib can be real fast. It removes some php-specific data, which is probably unnecessary for you (object names etc).
Edit2: If the server crashes, the plain apc-solution will lose all data. That is the reason you should also write it to a file if needed. apc is inside the apache process so it will be faster than memcached or redis.
I have an array containing my reference variables, and in my scripts I need to catch one variable or two. In the current system, I have to include the entire array (and its elements) to use one element. It seems that using a database is better for two reasons:
One record is read instead of the entire array
Variables can be easily edited
However, there is a major drawback for using database: on every php run, we need to make a connection to the database.
Since simple database systems like SQLite has no server, persistent_connection is not like advanced database servers like mysql.
In action,
$db = new SQLite3('mysqlitedb.db');
takes more time (and consumes more resources) than
include 'array.php';
Is there any solution for having a basic database system (with fast connection) to be a replacement to PHP array and include file?
In other words, I need a simple database system with fast connection comparable with fopen. However, even CDB which is incredibly fast, is not fast enough on initial connection.
By including the static array file you are essentially doing what caching systems do when they pull a result from a database. You are loading a pre-digested result directly from disk.
All database connections have some overhead (certainly more than including a rendered file). You use a database when you need operational maintainability for your data, but this comes at the cost of application overhead.
If you are not worried about persistance of the data, you may want to look at using a caching system like APC, memcached or redis.
Have you considered caching the variables? You could use APC or Memcached for this purpose. They will both be faster than a database since the data is stored in the RAM, not on the disc.
It will still be slower than just including the array.
I'm creating a web service that often scrapes data from remote web pages. After scraping this data, I have a simple multidimensional array of information to use. The scraping process is fairly taxing on my server, and the page load takes a while. I was considering adding a simple cache system using a MySQL database, where I create one row per remote web page with a the array of information pulled from it stored as a JSON encoded string. Is this a good enough system? Or would something like a text file per web page be a better idea?
Since you're scraping multiple web pages, and you want to your data to be persistently cached, you have a few options -- the best of which would be to use memcache or a database such as MySQL. Using text files is not a good idea, because you would have to serialize / deserialize your data, and read from your filesystem. To query a database or a memcache is many times more efficient.
Since you're probably looking for your cache to be somewhat persistent, I would suggest going with MySQL. You would simply create a table that has an auto-incrementing primary key, which a column for each element in your parsed JSON object. (Note that MySQL currently does not support arrays. In order to emulate them, you will need to use relational tables, or serialize your array data and provide it to a text field. The former method is preferred).
Every time you scrape a page, you would run an UPDATE statement to update that individual page's information in the database. If you specify a unique index on whatever you use to uniquely identify your page (URL / etc), you will achieve optimal look-up performance.
If you're looking to store the cache locally on 1 server (e.g. if your mysql server and http server are on the same box), you might be better off using APC, which is a cache service that comes with PHP.
If you're looking to store the data remotely (e.g. a dedicated cache box) then I would go with Memcache instead of MySQL.
"When all you have is a hammer ..."
I don;'t tend to have particularly large APC configs, 64 - 128MB max. Memcache can go to a couple of gigabytes or maybe more (far more if you run multiple instances). Both are also transient - a restart of Apache, or Memcache (the the latter is slightly less likely, or often) will lose the data
It depends then, on how often you are willing to process the data to produce the cache, and how long that cache could otherwise be useful for. If it was good for weeks before you re-scraped the pages - Mysql is a entirely suitable backing store.
Potential pther options, depending on how many items are being cached & how big the data is, are, as you suggest, a file-based cache, SQlite, or other systems.
I'm using the session array to cache chunks of information retrieved from the db:
$result = mysql_query('select * from table');
array_push($_SESSION['data'],new Data(mysql_fetch_assoc($result)));
My question is, is there a limit/a sizeable amount of information that can/should be passed around in a session? Is it ill advised or significantly performance hindering to do this?
By default, $_SESSION data is stored on disk in the /tmp directory of your server. As long as you have enough room in there AND you aren't hitting your PHP memory limit, you're fine.
However, if you're attempting to cache a query that is the SAME for a larger number of users, you might want to use something like APC or memcache that isn't tied to the individual user. Otherwise, your essentially going to cache the same result 1x for each user, and not leveraging a cache across all users.
I think the answer would depend on where you are storing your data and how fast you can transfer it there.
If the data is 44 MB big, and you are on a 1000base-T network, you can expect it to take 1 second to actually transfer THERE. And 1 second to transfer back..
If you use local memory, then you have a finite amount of memory the machine.
If you use disk, then you have load/save times (disk is slow).
But also keep in mind, PHP has a finite amount of memory it allows a script to use. I think the default setting is 8 MB.
If you are talking about large blocks of data, you may want to consider Redis, Tokyo Cabinet or other key/value stores. Or even a backend interface to manipulate the data/cache it for you without transferring it through PHP.
Because Session data is stored in a file (or database record) on your server, it shouldn't matter too much how much data you store in it. I would just advise against huge objects.
You might want to look at APC or memcached to cache the results instead, as it is not a per-user cache, and it uses the memory instead of files.
The session is serialized and written to disk by default, so depending on the size and the amount of users things can become slow. However both things can be changed (read the session manual under http://php.net/session for all details) like using memcache for in-memory storage of the data. Best thing is to try it out under an environment as similar as possible tothe live system and check the resulting load and throughput.
Mmm, tricky. I think you could save it in the session. The real question is: do you want that all that information serialize and unserialize every time a client make a request?
I think it would be OK to save it in there if you will use all that information in every page of your website, but this is unprobable. It would be better if you save that information in a directory like /temptables/sometable/ and each file have the name of the session. You can use session_id to get it, and save and load the information in the pages you have to use with:
$info = unserialize(file_get_contents('/templatebles/sometable/'.session_id().'.ser'));
and saving with:
file_put_contents('/temptables/sometable/'.session_id().'.ser'), serialize($info));
But you need a cron job to clean that directory for old file. You can do it getting the session from the filename and ask for some variable, like 'itsalive', using session_start() or doing something like file_exists(session_save_path().'/sess_'.$session_name) to check if you should delete the temporary file.