I'm looking to scale an existing phpBB installation by separating the read queries from the write queries to two separate, replicated MySQL servers. Anyone succeeded in doing this, specifically with phpBB?
The biggest concern I have so far is that it seems like the queries are scattered haphazardly throughout the code. I'd love to hear if anyone else did this, and if so, how it went / what was the process.
You could try MySQL Proxy which would be an easy way to split the queries without changing the application.
Just add more RAM. Enough RAM to hold the entire database. You'll be surprised how fast your inefficient script will fly. Memory forgives a lot of database scaling mistakes.
I know this was asked a long time ago, but I'd like to share what I experienced, in case it can help anyone.
If your problem are table locks, and knowing that the default storage engine of phpbb in that day was MyISAM, have you looked at moving to InnoDB storage engine?
Just find out which tables are most frequently locked, and convert those to InnoDB. The sessions table is the first candidate here, although you may want to look at other optimizations (such as storing session data only in memcache or something) if that is your main bottleneck.
Related
I am designing an "high" traffic application which realies mainly on PHP and MySQL database queries.
I am designing the database tables so they can hold 100'000 rows, each page loading queries the db for user data.
I can experience slow performances or database errors when there are say 1000 users connected ?
Asking because i cannot find specification on the real performance limits of mysql databases.
Thanks
If the userdata remains unchanged due loading another page, you could think about storing those information in a session.
Also, you should analyze how the read/write ratio in your database/ on specific tables is. MyIsam and InnoDB are very different when it comes to locking. Many connections can slow down your server, try to cache connections.
Take a look at http://php.net/manual/en/pdo.connections.php
if designed wrongly, one user might kill your server. you need to have performance tests, find bottle necks profiling your code. use explain for your queries...
Well designed databases can handle with tens of millions of rows, but poor designed can't.
Don't worry about performance, try to design it well.
It's just hard to say a design was good or not,you should always do some stress tests before you set up your application or website to help you see the performance,tools i often used were mysqlslap(for mysql only) and apache's ab command.you can google them for details.
i am working on a project which is a kind of social network ... studies said that we'll have the first couple of months more than 100,000 users.
the website is done using php mysql and i am searching for the fastest caching engine, since we are talking about caching user data after he sign in.
so we are talking about a huge database, huge records in same table, huge number of users and requests, huge size of caching.
please note that as first step, the website should be hosted on shared server before it will gonna be moved to dedicated server ( it's the client decision, not our)
any tip, hint or suggestion is appreciated.
thanks
1) Put a lot of thought into a sensible database schema, since changing it later will be painful. MySQL tables are good for doing fast SELECT operations, which sounds appropriate for your app.
2) Don't optimize your code prematurely, ie don't worry about a caching solution yet, instead focus on writing modular code so you can easily improve bottlenecks with caching later.
3) After 1 and 2, you need to think about caching based on what will retrieved and how often. I've seen applications that put user information into the session variable - that will reduce database hits. If that's not sufficent, look into Memcached. If you have bigger data, maybe Varnish.
I am implementing a project in PHP with mysql. Right now i don't have much data but i was wondering that in future when i have a large dataset. It will slow down my search in the table. So to decrease that searching time, i was thinking for caching techniques. Which caching i.e. client or server will be good for a large dataset?
Thanks, aby
Server, in my opinion.
A client-side cacheing technique will have one of two negative outcomes depending on how you do it:
If you cache only what the user has searched for before, the cache won't be of any use unless the user performs exactly the same search again.
If you cache the whole dataset the user will have to download the whole thing, and that will slow your site down and incur bandwidth expenses.
The easiest thing you can do is just add appropriate indexes to the table you're searching. That will be sufficient for 99% of possible applications and should be the first thing you do, before you think about cacheing at all.
Apologies if I've pitched this answer below your level, I'm not sure exactly what you're doing, what you're planning to cache or how much experience you have.
Pay close attention to indexing in your database schemas. If you do this part right, the database should be able to keep up until your data and traffic is large. The right caching scheme will depend significantly on what your usage patterns are like. You should do testing as your site grows to know where the bottlenecks are and what the best caching scheme will be for your system.
the question is very self-explained, in matter of performance and all, which way is better to store little amounts of data? MySQL or Small File in a Cache folder?
Thanks in Advance
Database because it will resolve any concurrency problem, which the file doesn't provide you with.
It's impossible to answer such vague (and possible out-of-nowhere) question.
The answer depends on circumstances (if any) which made you to put up such a problem.
For example, Conditional Get HTTP caching will do the job without wasting any space on the server in any form (and good site have to implement it anyway).
If you're taking database as possible solution, why not to use it in proper way? By proper indexing queries and using built in query cache?
If indexes themselves being a bottleneck due to frequent writes, no cache will solve the problem, slow database will make whole system slow. You have to rebuild your database structure in this case.
If it's going to be insufficient key-cache for database server, it's matter of proper tuning or hardware upgrading.
Or zillions other certain recipes to solve certain problems.
Honestly, all that "which is faster" crap has not a slightest relation to performance problems.
The only question can be accepted is "I have certain problem with [something]. Here are benchmarks and investigation results. What's the best solution?"
Otherwise no answer will do help. Especially one from usual Stackoverflow user who just heard of some technology but never used it in real. And always tends to cure a symptom, not disease.
Sorry- but your question is still quite vague in terms of what you mean by "performance".
Temporary tables can only be looked at once, but if you cache the information, you can look at it frequently.
Will you be expanding this function/site in the future? Then I suggest MySQL. It will definitely be worth getting it set up.
Also, pure disk caching can lead to issues if you don't delete the file cache and it fills up your alloted space.
Do you know memcache? Is a nice tool and very simple to use!
I'm creating and app that will rely on a database, and I have all intention on using a flat file db, is there any serious reasons to stay away from this?
I'm using mimesis (http://mimesis.110mb.com)
it's simpler than using mySQL, which I have to admit I have little experience with.
I'm wondering about the security of the db. but the files are stored as php and it seems to be a solid database solution.
I really like the ease of backing up and transporting the databases, which I have found harder with mySQL. I see that everyone seems to prefer the mySQL way - and it likely is faster when it comes to queries but other than that is there any reason to stay away from flat-file dbs and (finally) properly learn mysql ?
edit
Just to let people know,
I ended up going with mySQL, and am using the CodeIgniter framework. Still like the flat file db, but have now realized that it's way more complex for this project than necessary.
Use SQLite, you get a database with many SQL features and yet it's only a single file.
Greetings, I'm the creator of Mimesis. Relational databases and SQL are important in situations where you have massive amounts of data that needs to be handled. Are flat files superior to relation databases? Well, you could ask Google, as their entire archiving system works with flat files, and its the most popular search engine on Earth. Does Mimesis compare to their system? Likely not.
Mimesis was created to solve a particular niche problem. I only use free websites for my online endeavors. Plenty of free sites offer the ability to use PHP. However, they don't provide free SQL database access. Therefore, I needed to create a database that would store data, implement locking, and work around file permissions. These were the primary design parameters of Mimesis, and it succeeds on all of those.
If you need an idea of Mimesis's speed, if you navigate to the first page it will tell you what country you're viewing the site from. This free database is taken from the site ip2nation.com and ported into a Mimesis ffdb. It has hundreds if not thousands of entries.
Furthermore, the hit counter on the main page has already tracked over 7000 visitors. These are UNIQUE visits, which means that the script has to search the database to see if the IP address that's visiting already exists, and also performs a count of the total IPs.
If you've noticed the main page loads up pretty quickly and it has two fairly intensive Mimesis database scripts running on the backend. The way Mimesis stores data is done to speed up read and write procedures and also translation procedures. Most ffdb example scripts or other ffdb scripts out there use a simple CVS file or other some such structure for storing data. Mimesis actually interprets binary data at some levels to augment its functionality. Mimesis is somewhat of a hybrid between a flat file database and a relational database.
Most other ffdb scripts involve rewriting the COMPLETE file every time an update is made. Mimesis does not do this, it rewrites only the structural file and updates the actual row contents. So that even if an error does occur you only lose new data that's added, not any of the older data. Mimesis also maintains its history. Unless the table is refreshed the data that rows had previously is still contained within.
I could keep going on about all the features, but this isn't intended as a "Mimesis is the greatest database ever" rant. Moreso, its intended to open people's eyes to the fact that SQL isn't the ONLY technology available, and that flat files, when given proper development paradigms are superior to a relational database, taking into account they are more specialized.
Long live flat files and the coders who brave the headaches that follow.
The answer is "Fine" if you only NEED a flat-file structure. One test: Would a single simple spreadsheet handle all needs? If not, you need a relational structure, not a flat file.
If you're not sure, perhaps you can start flat-file. SQLite is a great app for getting started.
It's not good to learn you made the wrong choice, if you figure it out too far along in the process. But if you understand the importance of a relational structure, and upsize early on if needed, then you are fine.
I really like the ease of backing up
and transporting the databases, which
I have found harder with mySQL.
Use SQLite as mentioned in another answer. There is only one file to backup, or set up periodic dumps of the MySQL databases to SQL files. This is a relatively simple thing to do.
I see that everyone seems to prefer
the mySQL way - and it likely is
faster when it comes to queries
Speed is definitely a consideration. Databases tend to be a lot faster, because the data is organized better.
other than that is there any reason to
stay away from flat-file dbs and
(finally) properly learn mysql ?
There sure are plenty of reasons to use a database solution, but there are arguments to be made for flat files. It is always good to learn things other than what you "usually" use.
Most decisions depend on the application. How many concurrent users are you going to have? Do you need transaction support?
Wanted to inform that Mimesis has moved from the original URL to http://mimesis.site11.com/
Furthermore, I am shifting the focus of Mimesis from an ffdb to a key-value store. It's more sensible Given the types of information I'm storing and the methods I use to retrieve it. There was also a grave error present in the coding of Mimesis (which I've since fixed). However, I'm still in the testing phase of the new key-value store type. I've also been side-tracked by other things. Locking has also been changed from the use of file creation to directory creation as the mutex mechanism.
Interoperability. MySQL can be interfaced by basically any language that counts. Mimesis is unlikely to be usable outside PHP.
This becomes significant the moment you try to use profilers, or modify data from the outside.
You might also look at http://lukeplant.me.uk/resources/flatfile/ for the PHP Flatfile Package.
The issue with going flatfile is that in order to adjust the situation for further development you have to alter a significant amount of code in order to improve the foundation of the system. Whereas if it was a pure SQL system it would require little to no modification to proceed in the future.