We are building a website which would serve about 30k unique visitors a day.
Currently we use a simple mysql Connect > A Simple Query > mysql Close.
I'm afraid that with a dual core server running 2GB of RAM we would be able
to open about 1k mysql connection tops. is 1k a good estimate?
Is it better to make a Cron-Job output XML files and let our php files grab the data from them?
Typically XML will never be faster than MySQL for searching data (i.e. performing queries).
I don't know what kind of data you have, but XML will only be faster if you have a bunch of simple files and don't need to search, just load the files and format them.
If you need to search, then use MySQL.
MySQL does all sorts of optimizations. For example it stores KEY columns in a separate file, allowing for a much faster search.
I would suggest using Zend cache for caching MySQL query results for the the data that doesn't change frequently.
Related
I have a webpage field where it shows results as you type. Currently I am using an XML file stored on the server as the source for these results. Would it be faster if I directly query the database as letters are being typed? What is the best way to accomplish this?
I am using PHP and AJAX with MS SQL Server. Thank you.
XML files are just text files stored on your machine or another one; they need to be read, parsed and written to, and only your program can do that. They'are also really, really inefficient, because of their text nature: reading and parsing a text file is very slow, and modifying it is even worse.
XML files are good for storing configuration settings and passing data between different systems, but data storage and processing should definitely live in a proper DBMS.
Hope this suffice your query.
A more scalable solution would be to use a de-normalized table to store the results. You can use a Memory Optimized table for this if you are using SQL Server 2014 or later.
To speed up the querying, you can also create a Natively Compiled Stored Procedure, but this also supports in SQL Server 2014 or later.
Further, try and build a Full Text Search Index over the search results so that if will speed up the querying as well as give more functionality.
I need to load XML data from an external server/url into my MySQL database, using PHP.
I don't need to save the XML file itself anywhere, unless this is easier/faster.
The problem is, I will need this to run every hour or so as the data will be constantly updated, therefore I need to replace the data in my database too. The XML file is usually around 350mb.
The data in the MySQL table needs to be searchable - I will know the structure of the XML so can create the table to suit first.
I guess there are a few parts to this question:
What's the best way to automate this whole process to run every hour?
Whats the best(fastest?) way of downloading/ parsing the xml (~350mb) from the url? in a way that I can -
load it into a mysql table of my own, maintaining columns/ structure
1) A PHP script can keep running in background all the time, but this is not the best scenario or you can set a php -q /dir/to/php.php using cronos (if running on linux) or other techniques to makes server help you. (You still need access to server)
2) You can use several systems, the more linear one, less RAM consuming, is if you decide to work with files or with a modified mySQL access is opening your TCP connection, streaming smaller packages (16KB will be ok) and streaming them out on disk or another connection.
3) Moving so huge data is not difficult, but storing them in mySQL is not waste. Performing search in it is even worst. Updating it is trying to kill mySQL system.
Suggestions:
From what i can see, you are trying to synchronize or back-up data from another server. If there is just one file then make a local .xml using PHP and you are done. If there are more than one i will still suggest to make local files as most probably you are working with unstructured data: they are not for mySQL. If you work with hundreds of files and you need to search them fast perform statistics and much much more... consider to change approach and read about hadoop.
MySQL BLOOB or TEXT columns still not support more than 65KB, maybe you know another technique, but i never heard about it and I will never suggest to do so. If you are trying it just to use SQL SEARCH commands you took the wrong path.
I'm in the planning stage of building a web app for a school. I'm worried about the speed and efficiency of MySQL when multiple people are accessing it. The app will allow teachers to CRUD student records. Is it better to cache a json/xml result when a record is created/updated so that the app can quickly display it to the user (using javascript)? Or is MySQL fast enough to handle a updates and queries for the same data?
I have a program that does exactly this (plus more). Use a database, they're designed for these queries. Currently I've hit just under 100 concurrent users, and have a few thousand students, and have had no latency issues.
It's better, faster, safer to use a database.
JSON and XML is used for data exchanging between different platform/software. Like between PHP and Java or between twitter and C. Its good when you dont have any protocol defined and you use a common format like xml, json, yml. But if you have a protocol defined (between mysql and php) use that. That'll be much much faster. Besides as a database mysql can perform many other extra data manipulation operation that you can not achieve with plain xml or json or yml. So use mysql.
is mysql fast enough to handle a large number of queries that will just be retrieving data to display?
Its a lot faster. I doubt whether you'll face its slowness. I have seen many devices (not human) manipulating mysql concurrently. It produces tremendous load. And mysql still handling those data.
I'm making a website that (essentially) lets the user submit a word, matches it against a MySQL database, and returns the closest match found. My current implementation is that whenever the user submits a word, the PHP script is called, it reads the database information, scans each word one-by-one until a match is found, and returns it.
I feel like this is very inefficient. I'm about to make a program that stores the list of words in a tree structure for much more effective searching. If there are tens of thousands of words in the database, I can see the current implementation slowing down quite a bit.
My question is this: instead of having to write another, separate program, and use PHP to just connect to it with every query, can I instead save an entire data tree in memory with just PHP? That way, any session, any query would just read from memory instead of re-reading the database and rebuilding the tree over and over.
I'd look into running an instance of memcached on your server. http://www.memcached.org.
You should be able to store the compiled tree of data in memory there and retrieve it for use in PHP. You'll have to load it into PHP to perform your search, though, as well as architect a way for the tree in memcached to be updated when the database changes (assuming the word list can be updated, since there's not a good reason to store it in a database otherwise).
Might I suggest looking at the memory table type in mysql: http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html
You can then still use mysql's searching features on fast "in memory" data.
PHP really isn't a good language for large memory structures. It's just not very memory efficient and it has a persistence problem, as you are asking about. Typically with PHP, people would store the data in some external persistent data store that is optimized for quick retrieval.
Usually people use a two fold approach:
1) Store data in the database as optimized as possible for standard queries
2) Cache results of expensive queries in memcached
If you are dealing with a lot of data that cannot be indexed easily by relational databases, then you'd probably need to roll your own daemon (e.g., written in C) that kept a persistent copy of the data structure in memory for fast querying capabilities.
I noticed that ffdb in PHP are awfully slow. So how would I go about speeding up the ffdb without SQL?
Flat file will become more slow the more data it's contain. This is because in order to get a data, the engine need to get all data first into memory, then select the requested data.
SQL database have random seek feature, because the data placement is in a known location, plus it have index table. That's why any SQL database is not affected with the number of data stored in it.
You can try sqlite if you need single file database. Firefox and Thunderbird use sqlite to store bookmark and browsing history.
Tokyo Cabinet is amazingly fast for key/value lookups. There is also the search addition (tyrant) that allows for more complex queries
http://1978th.net/tokyocabinet/
PHP implementation
http://code.google.com/p/phptyrant/