First, what I intend to do is to use memory to store the most recent "user update" records for each user.
I am new to MySQL. How can I create tables in memory?
In official website, it is said that we can set ENGINE = MEMORY when creating table. But the document claims that those tables in memory are always, for read, not for write.
I have simply no idea how to do that.
I am into this problems for a few days. I can't install memcache and any PHP extension in server as I'm not using Virtual Private Server, what I can do is just transfer scripts and files in httpdocs folder... I have also tried using flat files to store data to work as buffer/cache, but I found that I cannot write/create files in server's file directory due to denied permission, and I am not allowed to change this permission.
Using MySQL to buffer may be the only choice left for me. Hope someone can give me some hints.
Thank you.
p.s. I am using Linux Apache server running PHP, with MySQL as DB.
ENGINE = MEMORY tables can be used for both read or write.
The only thing to be careful of is that all data in a memory table disappears when the server crashes, is turned off, rebooted, etc. (As you would expect for an in-memory table.)
You really should read carefully about MEMORY engine of MySQL. The data is stored in RAM so when the server is powered off, or rebooted, the RAM will be cleared, and data will be wiped. MEMORY table should be the fastest accessible table type of MySQL, but only stores temporary data, with no guarantee.
If I understood right, you are trying to make static cache of some sort of data generated from PHP, aren't you? The easiest way is to write them as solid file cache in your www directory, either HTML or JS. If you can't chmod your directory to writable, then store them in MySQL should be fine too, but only if that actually helps.
The idea of cache data is to: reduce SQL queries, reduce disk I/O, reduce code generation. But using MEMORY table costs too much memory usage. Store them in a normal MyISAM table should be fine too, and safe you a lot of background work.
However, there should be 2 things to consider: 1, if the cache does not exist when accessing; 2, if the cache is up-to-date.
Giving your result some sort of key should be a good idea, so the PHP checks for cached date first, if doesn't not exist, generate the cache, then display, or otherwise, display the cache directly.
Related
I want to use session values in my script which are stored at the server using php can any one kindly explain the process to achieve this.
I want to build a chat app for this am planning to use those session values.
Assume usera and userb are logged in and their userid is sessioned based on this scenario i want to do a chat app.
Now i had done the app but i had used setinterval function of Javascript and am calling the chats i want to avoid the database hits on every 3 mill sec.
Kindly Help me out
Thanks In Advance
You're basically attempting to use PHP session files as a file cache.
Instead, you should use an object caching system such as Memcached or Redis. If memory caching isn't an option (shared hosting, etc), then you could implement your own file cache (or you could use something like PHPFastCache, which supports file caching).
Note: File caching for a chat app may or may not speed up your application. It depends on how you implement it and a number of other factors.
Hi put the session value in input box,
<input type='hidden' id='session_value' value='<?php $_SESSION['value']?>'>
Using the id fetch the session value in script,
<script>
var session_value = document.getElementById ( "session_value" );
</script>
3ms is insanely short delay to run a polling chat system. I suggest increasing it to at least 200ms but preferably around 1000ms.
$_SESSION values are per user and not recommended for viewing a chat stream for a number of reasons. Instead it sounds like you are looking more to just update the chat feed.
The database unless it is hosted on another server and $_SESSION will be the equivalent, since the database is effectively files as well. The database will actually generally be faster than reading raw file storage since Queries are normally cached and Indexing helps lookup records quicker. In addition you won't have to worry about concurrent connections to the files either.
If anything enable OPCache and install APCu for your PHP installation, to help aid the serving of requests. OPCache will cache your compiled OP code into memory so that subsequent requests to the file won't need to be recompiled.
APCu will act as your file cache, again storing your rendered data in memory.
Additionally many Database Frameworks such as Doctrine can also utilize APC caching for query and result caching.
Instead of using a InnoDB or MyISAM storage engines for your chat messages I suggest trying the MEMORY storage engine.
So instead of accessing the File System I/O your database would instead be utilizing the Memory I/O. The general concept is few writes, many reads. Since one person writes to the database, requires everyone to read the data. Just keep in mind that the Memory storage engine is temporary and is lost if the server restarts or power is lost.
For more information see: https://dev.mysql.com/doc/refman/5.6/en/memory-storage-engine.html
Overall if you are able, I would suggest look at using Socket IO (Websockets) instead of either database or file based caching. This puts the load on the clients instead of the server, and everything occurs in real-time instead of polling for changes.
For some examples see:
Ratchet http://socketo.me/
React http://reactphp.org/
Node.js http://tutorialzine.com/2014/03/nodejs-private-webchat/
Is PHP able to handle sqlite data as inmemory DB?
I have a <50MB database and would like a php script to do SELECTs (and if possible also UPDATEs) to the sqlite without slow disk file reading or writing each time, the script is ran.
With java and c++ I know great use-cases, but how to force PHP to access the sqlite inmemory without reloading the file again and again?
There are multiple ways to do it:
Do nothing, and let the OS cache the database in disk caches / memory buffers. This is good if you have a small database (and <50 MB is small), and if you have lots of memory.
Use a tmpfs and copy your database file in it, then open it in PHP.
Use sqlite://:memory: (but you will start from a blank database).
In memory sql databases might be what you are looking for.
I am attempting my first large scale database project on my own.
I have a myisam mysql db on server 1 with a php app consuming large amount of various data.
I have mysql myisam on server 2 with php app selecting and displaying data.
I want to replicate this data on server 2.
Questions:
Should I change server 1 mysql db to innodb
Can you replicate server1 innodb to server2 myisam
I'm storing media as blobs with intention of using cache to offload stress on live server. Should I use filesystem storage and rsync.
Any general advice from other experienced people ?
Here's what I suggest based on my experience.
You can use one type of engine (MyISAM or InnoDB) for both servers.
I you mix both engines, you might get Deadlock, transaction problems etc... and time fixing them can be painful.
I had problems a little while ago with InnoDB -> MyISAM. Now I used MyISAM on all servers.
For storing media (such as images, video or documents) you can create a NFS and mount a folder like /usermedia/ that both servers access.
Therefore you don't have to rsync every time. In addition, you can save the meta data or media information to the database for reference and where the file is saved on the disk.
Note: using a blob to save files can be good depending on the media. If you have a file that is approximately 1 gig might not be a good idea to save on the database for example).
Use a caching system to retrieve data (such as memcached).
For example, if you request data and you need to display them to the user, look first in the cache. If it's not in the cache, query the database, save it to the cache and display.
Next time the same information is requested, you won't request it from the server but from memory. This solution will avoid numerous calls on the Database server which will improve performance.
Let me know if you need additional help.
I would recommend InnoDB (for transactions, row locking and not table locking) and redis as the caching very fast and efficient
I have a LAMP server with 256MB RAM (poor man's server in cloud). I have an app written to run on this machine. Currently people upload images and they go straight into mysql as BLOB.
There are concerns that this might be very memory consuming operation and we move over it to simple plain files. Can some one tell me if these concerns are valid? (Worth putting efforts into changing a lot of ode that's already written given that we will have sufficient RAM in next 6 months ?)
As a general rule when should we store images in DB and when as files?
To read a BLOB in MySQL you need three times as much memory as it takes (it gets copied into several buffers).
So yes, reading a BLOB in MySQL consumes more memory than reading a file.
You should store them in the file system for several reasons:
The images are easily accessible via other apps (shell, FTP, www, etc...),
It's less resource intensive (including memory) to read them from the file system than from a database
If the database gets corrupted, the images are safe.
You also won't have tables bump up against their size limitations (determined by OS file size limitations) which slows them down (and making them require more resources to read).
The only time you should consider storing images in the DB is when they are used in transaction processing, and even then, there are numerous workarounds to that when storing in the file system.
To summarize:
Database Storage:
Pros:
Assures referential integrity
Easier backup strategy
Easier clustering (database cluster)
Cons:
Higher cost in memory usage and storage
Hard to scale
Additional code must be written to support HTTP caching
Requires a database and associated querying code
File System Storage
Pros:
Low memory footprint (more efficient)
Storage equal to file size
Easy retrieval and storage
Allows the web server to control caching
Cons:
Referential integrity not assured
Backups are not always in sync with database backups
Requires additional backup strategy
If referential integrity of your images is important, store them in the database. The advantages is that a backup of your database will always means that your rows and images are in sync. It does mean though that it is a bit more costly resource wise to store and retrieve.
If the images themselves are not that important, store them as files. It allows for fast and simple retrieval and storage. The downside of using files though is that your backup strategy becomes more complicated and your files will not always be in sync with your database rows.
I personally always store them in the database. For me, the rewards are greater then the cost. This is hardly always the case though and you should look at your application requirements to see which is best for you.
Some large websites are using BLOBs to store their website content. Flickr's use of BLOBs is actually well documented. To answer your question though, file storage is more memory efficient than database storage.
If it is for serving on a WebPage I would user plain file system with a link to the image file name in text format on DataBase. Apache and browsers usually do an amazing job on caching static files.
Even though in theory, you could achieve similar performance serving images from a database, the amount of work you need to do for it does not justify this selection given that the only advantage I can think of is a more cohesive database (with a simple DB dump you get ALL your data: images + data).
If you have a lot of files to keep track of, or they are very large, I'd store them as files. Especially if these files are to be accessed via the web, in which case you can offload all that effort from the SQL server and let the web server handle the transfer.
A good way to track images is to name them using the primary key, and then keep track of the original file name (if you need it) in the database. This way you can always know which file connects to which row. Also, if you have many files (thousands, millions,...), you might consider 'hashing' them into directories, so that 1-1000 are stored in /1, 1001-2000 ares stored in /2, etc. Some OS's see a bit of slowdown when you get a large number of files in a single directory.
I need a simple way for multiple running PHP scripts to share data.
Should I create a MySQL DB with a RAM storage engine, and share data via that (can multiple scripts connect to the same DB simultaneously?)
Or would flat files with one piece of data per line be better?
Flat files? Nooooooo...
Use a good DB engine (MySQL, SQLite, etc). Then, for maximum performance, use memcached to cache content.
In this way, you have the ease and reliability of sharing data between processes using proven server software that handles concurrency, etc... But you get the speed of having your data cached.
Keep in mind a couple things:
MySQL has a query cache. If you are issuing the same queries repeteadly, you can gain a lot of performance without adding a caching layer.
MySQL is really fast anyway. Have you load-tested to demonstrate it is not fast enough?
Please don't use flat files, for the sanity of the maintainers.
If you're just looking to have shared data, as fast as possible, and you can hold it all in RAM, then memcached is the perfect solution.
If you'd like persistence of data, then use a DBMS, like MySQL.
Generally, a DB is better, however, if you are sharing a small, mostly static amount of data, there might be performance benefits (and simplicity) of doing it with flat files.
Anything other than trivial data sharing and I would pick a DB however.
1- Where the flat file can be usefull:
Flat file can be faster than a database, but in very specific applications.
They are faster if the data is read from start to finish without any search or write.
If the data dont fit in memory and need to be read fully to get the job done, It 'can' be faster than a database. Also if there is lot more write than read, flat file also shine, most default databases setups will need to make the read queries wait for the write to finish in order maintain indexes and foreign keys. Making the write queries usually slower than simple reads.
TD/LR vesion:
Use flat files for jobs based system(Aka, simple logs parsing), not for web searches queries.
2- Flat files pit falls:
If your going with a flat file, you will need to synchronize your scripts when the file change using custom lock mechanism. Which can lead to slowdown, corruption up to dead lock if you have a bug.
3- Ram based Database ?
Most databases have in memory cache for query results, search indexes, making them very hard to beat with a flat file. Because they cache in memory, making it run entirely from memory is most of the time ineffective and dangerous. Better to properly tune the database configuration.
If your looking to optimize performance using ram, I would first look at running your php scrips, html pages, and small images from a ram drive. Where the cache mechanism is more likely to be crude and hit the hard drive systematically for non changing static data.
Better result can be reach with a load balancer, clustering with a back plane connections up to ram based SAN array. But that's a whole other topic.
5- can multiple scripts connect to the same DB simultaneously?
Yes, its called connection pooling. In php (client side) its the function to open a connection its mysql-pconnect(http://php.net/manual/en/function.mysql-pconnect.php).
You can configure the maximum open connection in php.ini I think. Similar setting on mysql server side define the maximum of concurrent client connections in /etc/mysql/my.cnf.
You must do this in order to take advantage of parrallel processessing of the cpu and avoid php script to wait the query of each other finish. It greatly increase performance under heavy load.
There is also one connection pool/thread pool in Apache configuration for regular web clients. See httpd.conf.
Sorry for the wall of text, was bored.
Louis.
If you're running them on multiple servers, a filesystem-based approach will not cut it (unless you've got a consistent shared filesystem, which is unlikely and may not be scalable).
Therefore you'll need a server-based database anyway to allow the sharing of data between web servers. If you're serious about either performance or availability, your application will support multiple web servers.
I would say that the MySql DB would be better choice unless you have some mechanism in place to deal with locks on the flat files (and some way to control access). In this case the DB layer (regardless of specific DBMS) is acting as an indirection layer, letting you not worry about it.
Since the OP doesn't specify a web server (and PHP actually can run from a commandline) then I'm not certain that the caching technologies are what they're after here. The OP could be looking to do some sort of flying data transform that isn't website driven. Who knows.
If your system has a PHP cache (that caches compiled PHP code in memory, like APC), try putting your data into a PHP file, as PHP code. If you have to write data, there are some security issues.
I need a simple way for multiple
running PHP scripts to share data.
APC, and memcached are both good options depending on context. shared memory may also be an option.
Should I create a MySQL DB with a RAM
storage engine, and share data via
that (can multiple scripts connect to
the same DB simultaneously?)
That's also a decent option, but will probably not be as fast as APC or memcached.
Or would flat files with one piece of
data per line be better?
If this is read-only data, that's a possibility -- but may be slower than any of the options above. Especially if the data is large. Rather than writing custom parsing code, however, consider simply building a PHP array, and include() the file.
If this is a datastore that may be accessed by several writers simultaneously, by all means do NOT use a flat file! Writing to a flat file from multiple processes is likely to lead to file corruption. You can lock the file, but you risk lock contention issues, and long lock wait times.
Handling concurrent writes is the reason applications like mysql and memcached exist.