I am working on a PHP project and asked to implement a system (runs on server) which uses same memory location for every request.
To be simpler, think that there is an array in the memory (RAM) and every client ask for one element of it. Server does not create that array repeatedly. To achieve it, server must use a shared memory and returns the related elements to the clients. The question is, how can I do it? Or is there any source explaining it.
Constraints:
I don't want to use applet technology. And as much as possible, I want to implement it via PHP.
I don't want to use a database since it is too slow for our system and our data does not require to be persistent for any system down.
Data is really small (does not exceed 10MB) and fits to the memory.
Run MySQL and use the MEMORY storage engine. The table(s) will exist only in memory, will not be persisted to disk, and will not be "too slow" as operations are essentially at the speed of memory access.
Whatever you do, don't reinvent the wheel. Lots of in-memory data stores exist with PHP drivers/interfaces.
Related
I want to use session values in my script which are stored at the server using php can any one kindly explain the process to achieve this.
I want to build a chat app for this am planning to use those session values.
Assume usera and userb are logged in and their userid is sessioned based on this scenario i want to do a chat app.
Now i had done the app but i had used setinterval function of Javascript and am calling the chats i want to avoid the database hits on every 3 mill sec.
Kindly Help me out
Thanks In Advance
You're basically attempting to use PHP session files as a file cache.
Instead, you should use an object caching system such as Memcached or Redis. If memory caching isn't an option (shared hosting, etc), then you could implement your own file cache (or you could use something like PHPFastCache, which supports file caching).
Note: File caching for a chat app may or may not speed up your application. It depends on how you implement it and a number of other factors.
Hi put the session value in input box,
<input type='hidden' id='session_value' value='<?php $_SESSION['value']?>'>
Using the id fetch the session value in script,
<script>
var session_value = document.getElementById ( "session_value" );
</script>
3ms is insanely short delay to run a polling chat system. I suggest increasing it to at least 200ms but preferably around 1000ms.
$_SESSION values are per user and not recommended for viewing a chat stream for a number of reasons. Instead it sounds like you are looking more to just update the chat feed.
The database unless it is hosted on another server and $_SESSION will be the equivalent, since the database is effectively files as well. The database will actually generally be faster than reading raw file storage since Queries are normally cached and Indexing helps lookup records quicker. In addition you won't have to worry about concurrent connections to the files either.
If anything enable OPCache and install APCu for your PHP installation, to help aid the serving of requests. OPCache will cache your compiled OP code into memory so that subsequent requests to the file won't need to be recompiled.
APCu will act as your file cache, again storing your rendered data in memory.
Additionally many Database Frameworks such as Doctrine can also utilize APC caching for query and result caching.
Instead of using a InnoDB or MyISAM storage engines for your chat messages I suggest trying the MEMORY storage engine.
So instead of accessing the File System I/O your database would instead be utilizing the Memory I/O. The general concept is few writes, many reads. Since one person writes to the database, requires everyone to read the data. Just keep in mind that the Memory storage engine is temporary and is lost if the server restarts or power is lost.
For more information see: https://dev.mysql.com/doc/refman/5.6/en/memory-storage-engine.html
Overall if you are able, I would suggest look at using Socket IO (Websockets) instead of either database or file based caching. This puts the load on the clients instead of the server, and everything occurs in real-time instead of polling for changes.
For some examples see:
Ratchet http://socketo.me/
React http://reactphp.org/
Node.js http://tutorialzine.com/2014/03/nodejs-private-webchat/
I have written a php code which will get one id from database and using that id it will use some API's provided by other websites and generate a page.
here my question is where this generated page will occupy the space on the server or on the client machine?
if 10000 people will open the same page then will my server be slow down in this case.
should i store all data for that API in our MySQL-database.
what will make it fast & safe...
Please suggest me...
Thanks
I have written a php code which will get one id from database and
using that id it will use some API's provided by other websites and
generate a page.
here my question is where this generated page will occupy the space on
the server or on the client machine?
The generated page will occur on the client if you only fetch one id from your database. For this you could first do a jquery.get to fetch id from your server. Next you could get data from other API's using JSONP(JSON with padding). But for this to work the API's off course need to support JSONP, because the javascript clients can't fetch data using jquery.get because of same origin policy, but lucky JSONP can be used for that. Finally you could just easily append data to the DOM using .html. You should be carefull doing this with other API's and need to be sure these are safe API's because else you would be vulnerable to XSS. If you are not certain you should use .text instead.
should i store all data for that API in our MySQL-database.
It depends if the API's do provide JSONP.
what will make it fast & safe...
Fast
APC to cache compiled bytecode. This will speed up your website tremendously without even changing a single line in your code-base.
in memory database as redis or memcached. You can also use APC to store data in memory. This will speed up your website tremendously, because touching the disc(spinning the disc to right sector, etc) is very expensive and using memory is very fast.
The No-Framework approach will make your site fast, because PHP is dynamic language you should try to do as little as possible.
Tackle low hanging fruit only. Remember that "Premature optimization is the root of all evil". Rasmus Lerdorf teaches you how to do this in this video Simple is Hard from DrupalCon 2008. The slides are available at PHP's talks section
Safe
Read up OWASP top 10
Protect against XSS using filter
Protect against SQL-injection using PDO(prepared statements).
Protect against CSRF
It all depends on your garbage collection. The memory will be used by your server while the page is being rendered, but once the output is sent to the browser, PHP will no longer care. Now, if you have really bad garbage collection, Apache can certainly run out of memory. It has built-in garbage collection protocols but if you rely on those, you're just asking for dropped packets and page hangs.
If 10000 people access your server at the same time, it'll likely be your CPU that will be the bottleneck.
This is why tried-and-true PHP frameworks are ideal for large projects because most of them have taken all this into account and have built-in optimization implementations.
It depends really.
Factors are:-
Time taken to generate request's response
Size of the request
Concurrent connections
Web Server
Speed of the api
and many more...
You server is not likely to slow down if there are 10000 requests made of a period of time but if there are 10000 requests made every second then there is going to be a likely impact and this depends on the list given. If there are more concurrent connections to the server then each connection will use up some memory and memory overflow may halt the server. So make sure that even you get that many requests those requests are served fast and their connections and processes are not kept in the memory for long. This saves memory and from your server crashing.
However if the output for the api is going to be the same for various users then it would be wiser to keep the object in the memory as memory access is much faster than a disk access.
If 10000 people will be grabbing the same page that you're dynamically creating by manipulating another site's API, it sounds like you're pulling data from the other site, and constructing a page using PHP on your server. So yes, that consumes a small amount of memory and processing resources on your system, per hit. Memory use may be limited by the number of threads or forks your webserver is allowed to use. Processing power will not be limited artifically; it will be constrained by what your server can handle.
But back to that number of 10000 people grabbing the same page, again. If that's a possibility you would want to generate the page locally, and cache it somehow so that it only has to be generated once. It doesn't make sense to generate the same output 10,000 times when you could generate it once and let it be fetched 10,000 times instead. Then it just becomes a matter of deciding when the cache is stale.
Currently we are having a site which do a lot of api calls from our parent site for user details and other data. We are planning to cache all the details on our side. I am planning to use memcache for this. as this is a live site and so we are expecting heavier traffic in coming days(not that like FB but again my server is also not like them ;) ) so I need your opinion what issues we can face if we are going for memcache and cross opinions of yours why shouldn't we go for it. Any other alternative will also help.
https://github.com/steveyen/community-site/blob/master/db_doc/main/WhyNotMemcached.wiki
Memcached is terrific! But not for every situation...
You have objects larger than 1MB.
Memcached is not for large media and streaming huge blobs.
Consider other solutions like: http://www.danga.com/mogilefs
You have keys larger than 250 chars.
If so, perhaps you're doing something wrong?
And, see this mailing list conversation on key size for suggestions.
Your hosting provider won't let you run memcached.
If you're on a low-end virtual private server (a slice of a machine), virtualization tech like vmware or xen might not be a great place to run memcached. Memcached really wants to take over and control a hunk of memory -- if that memory gets swapped out by the OS or hypervisor, performance goes away. Using virtualization, though, just to ease deployment across dedicated boxes is fine.
You're running in an insecure environment.
Remember, anyone can just telnet to any memcached server. If you're on a shared system, watch out!
You want persistence. Or, a database.
If you really just wish that memcached had a SQL interface, then you probably need to rethink your understanding of caching and memcached.
You should implement a generic caching layer for the API calls first. Within the domain of the caching layer you can then change the strategy which backend you want to use. If you then see that memcache is not fitting you can actually switch (and/or testwise monitor how it works compared with other backends).
Even better, you can first code this build upon the filesystem quite easily (which has multiple backends, too) without the hurdle to rely on another daemon, so already get started with caching - probably file system is already enough for your caching needs?
Memcache is fast, but it also can use a lot of memory if you want to get the most out of it. Whenever you hit the disk for I/O, you're increasing the latency of your application. Pull items that are frequently accessed and put them on memcache. For my large scale deployments, we cache sessions there because DB is slow as well as filesystem session storage.
A recommendation to add to your stack is APC. It caches PHP files and lessens the overall memory usage per page.
Alternative: Redis
Memcached is, obviously, limited by your available memory and will start to jettison data when memory thresholds are reached. You may want to look redis which is as fast (faster in some benchmarks) as memcached but allows the use of both volatile and non-volatile keys, more complex data structures, and the option of using virtual memory to put Least Recently Used (LRU) key values to disk.
I need a simple way for multiple running PHP scripts to share data.
Should I create a MySQL DB with a RAM storage engine, and share data via that (can multiple scripts connect to the same DB simultaneously?)
Or would flat files with one piece of data per line be better?
Flat files? Nooooooo...
Use a good DB engine (MySQL, SQLite, etc). Then, for maximum performance, use memcached to cache content.
In this way, you have the ease and reliability of sharing data between processes using proven server software that handles concurrency, etc... But you get the speed of having your data cached.
Keep in mind a couple things:
MySQL has a query cache. If you are issuing the same queries repeteadly, you can gain a lot of performance without adding a caching layer.
MySQL is really fast anyway. Have you load-tested to demonstrate it is not fast enough?
Please don't use flat files, for the sanity of the maintainers.
If you're just looking to have shared data, as fast as possible, and you can hold it all in RAM, then memcached is the perfect solution.
If you'd like persistence of data, then use a DBMS, like MySQL.
Generally, a DB is better, however, if you are sharing a small, mostly static amount of data, there might be performance benefits (and simplicity) of doing it with flat files.
Anything other than trivial data sharing and I would pick a DB however.
1- Where the flat file can be usefull:
Flat file can be faster than a database, but in very specific applications.
They are faster if the data is read from start to finish without any search or write.
If the data dont fit in memory and need to be read fully to get the job done, It 'can' be faster than a database. Also if there is lot more write than read, flat file also shine, most default databases setups will need to make the read queries wait for the write to finish in order maintain indexes and foreign keys. Making the write queries usually slower than simple reads.
TD/LR vesion:
Use flat files for jobs based system(Aka, simple logs parsing), not for web searches queries.
2- Flat files pit falls:
If your going with a flat file, you will need to synchronize your scripts when the file change using custom lock mechanism. Which can lead to slowdown, corruption up to dead lock if you have a bug.
3- Ram based Database ?
Most databases have in memory cache for query results, search indexes, making them very hard to beat with a flat file. Because they cache in memory, making it run entirely from memory is most of the time ineffective and dangerous. Better to properly tune the database configuration.
If your looking to optimize performance using ram, I would first look at running your php scrips, html pages, and small images from a ram drive. Where the cache mechanism is more likely to be crude and hit the hard drive systematically for non changing static data.
Better result can be reach with a load balancer, clustering with a back plane connections up to ram based SAN array. But that's a whole other topic.
5- can multiple scripts connect to the same DB simultaneously?
Yes, its called connection pooling. In php (client side) its the function to open a connection its mysql-pconnect(http://php.net/manual/en/function.mysql-pconnect.php).
You can configure the maximum open connection in php.ini I think. Similar setting on mysql server side define the maximum of concurrent client connections in /etc/mysql/my.cnf.
You must do this in order to take advantage of parrallel processessing of the cpu and avoid php script to wait the query of each other finish. It greatly increase performance under heavy load.
There is also one connection pool/thread pool in Apache configuration for regular web clients. See httpd.conf.
Sorry for the wall of text, was bored.
Louis.
If you're running them on multiple servers, a filesystem-based approach will not cut it (unless you've got a consistent shared filesystem, which is unlikely and may not be scalable).
Therefore you'll need a server-based database anyway to allow the sharing of data between web servers. If you're serious about either performance or availability, your application will support multiple web servers.
I would say that the MySql DB would be better choice unless you have some mechanism in place to deal with locks on the flat files (and some way to control access). In this case the DB layer (regardless of specific DBMS) is acting as an indirection layer, letting you not worry about it.
Since the OP doesn't specify a web server (and PHP actually can run from a commandline) then I'm not certain that the caching technologies are what they're after here. The OP could be looking to do some sort of flying data transform that isn't website driven. Who knows.
If your system has a PHP cache (that caches compiled PHP code in memory, like APC), try putting your data into a PHP file, as PHP code. If you have to write data, there are some security issues.
I need a simple way for multiple
running PHP scripts to share data.
APC, and memcached are both good options depending on context. shared memory may also be an option.
Should I create a MySQL DB with a RAM
storage engine, and share data via
that (can multiple scripts connect to
the same DB simultaneously?)
That's also a decent option, but will probably not be as fast as APC or memcached.
Or would flat files with one piece of
data per line be better?
If this is read-only data, that's a possibility -- but may be slower than any of the options above. Especially if the data is large. Rather than writing custom parsing code, however, consider simply building a PHP array, and include() the file.
If this is a datastore that may be accessed by several writers simultaneously, by all means do NOT use a flat file! Writing to a flat file from multiple processes is likely to lead to file corruption. You can lock the file, but you risk lock contention issues, and long lock wait times.
Handling concurrent writes is the reason applications like mysql and memcached exist.
The site I am developing in php makes many MySQL database requests per page viewed. Albeit many are small requests with properly designed index's. I do not know if it will be worth while to develop a cache script for these pages.
Are file I/O generally faster than database requests? Does this depend on the server? Is there a way to test how many of each your server can handle?
One of the pages checks the database for a filename, then checks the server to see if it exists, then decides what to display. This I would assume would benefit from a cached page view?
Also if there is any other information on this topic that you could forward me to that would be greatly appreciated.
If you're doing read-heavy access (looking up filenames, etc) you might benefit from memcached. You could store the "hottest" (most recently created, recently used, depending on your app) data in memory, then only query the DB (and possibly files) when the cache misses. Memory access is far, far faster than database or files.
If you need write-heavy access, a database is the way to go. If you're using MySQL, use InnoDB tables, or another engine that supports row-level locking. That will avoid people blocking while someone else writes (or worse, writing anyway).
But ultimately, it depends on the data.
It depends on how the data is structured, how much there is and how often it changes.
If you've got relatively small amounts, of relatively static data with relatively simple relationships - then flat files are the right tool for the job.
Relational databases come into their own when the connections between the data are more complex. For basic 'look up tables' they can be a bit overkill.
But, if the data is constantly changing, then it can be easier to just use a database rather than handle the configuration management by hand - and for large amounts of data, with flat files you've got the additional problem of how do you find the one bit that you need, efficiently.
This really depends on many factors. If you have a fast database with much data cached in the RAM or a fast RAID system, chances are bad, that you will gain much from simple file system caching on the web server. Also think about scalibility. Under high workload a simple caching mechanism might easily become a bottle neck while a database is well designed to handle high work loads.
If there are not so much requests and you (or the operating system) is able to keep the cache in RAM, you might be able to gain some performance. But now the question arises, if it is realy neccessary to perform caching under low work load.
From plain performance perspective, it is wiser to tune the database server and not complicate the data access logic with intermediate file caches. A good database server would do the caching on its own if the results are cacheable. (I'm not sure what is teh case with mysql).
If you have performance problems, you should profile the pages to see the real bottlenecks. Even when you are -like me- a fan of the optimized codes, putting a stronger/more hardware into the equation is cheaper on the long run.
If you still need to use caches, consider using an existing solution, like memcached.