I am researching the possibility of porting an application written in classic ASP with ADO record sets and an Oracle database to PHP5 and OCI8. We have lots of stored procedures and queries with bind variables for performance.
My problem is that we have become lazy from using the ADO classes and the EOF and BOF indicators along with MoveFirst, MoveNext and MovePrevious.
I can not find any similar functionality in the OCI module. Is there any hope?
This is outside my area of expertise, but I think the equivalent functionality outside of ADO would be to retrieve the dataset into an array, then use standard array navigation techniques, rather than functionality that is specific to the database API.
If you're dealing with datasets that are large enough that you don't want to load the whole thing at one time, you should try to find a way to narrow the result set in the query before you start navigating the results. For instance, if you find yourself loading a result set, then just going to the last row, it's easy enough to make the query just return the last row in the first place. If you find yourself retrieving a result set, then looping through it (or filtering) for a specific row (or set of rows), I think you'll find that letting Oracle do that for you will show significantly better performance.
The reason that you need to use arrays to do this kind of navigation with Oracle is that Oracle cursors are always forward-only (whereas with ADO, you have dynamic, keyset, and static cursors as well). If you really need to be able to navigate an entire large result set, loading the whole thing into an array is about your only choice.
Related
I have searched for a few hours already but have found nothing on the subject.
I am developing a website that depends on a query to define the elements that must be loaded on the page. But to organize the data, I must repass the result of this query 4 times.
At first try, I started using mysql_data_seek so I could repass the query, but I started losing performance. Due to this, I tried exchanging the mysql_data_seek for putting the data in an array and running a foreach loop.
The performance didn't improve in any way I could measure, so I started wondering which is, in fact, the best option. Building a rather big data array ou executing multiple times the mysql_fetch_array.
My application is currently running with PHP 5.2.17, MySQL, and everything is in a localhost. Unfortunatly, I have a busy database, but never have had any problems with the number of connections to it.
Is there some preferable way to execute this task? Is there any other option besides mysql_data_seek or the big array data? Has anyone some information regarding benchmarking testes of these options?
Thank you very much for your time.
The answer to your problem may lie in indexing appropriate fields in your database, most databases also cache frequently served queries but they do tend to discard them once the table they go over is altered. (which makes sense)
So you could trust in your database to do what it does well: query for and retrieve data and help it by making sure there's little contention on the table and/or placing appropriate indexes. This in turn can however alter the performance of writes which may not be unimportant in your case, only you really can judge that. (indexes have to be calculated and kept).
The PHP extension you use will play a part as well, if speed is of the essence: 'upgrade' to mysqli or pdo and do a ->fetch_all(), since it will cut down on communication between php process and the database server. The only reason against this would be if the amount of data you query is so enormous that it halts or bogs down your php/webserver processes or even your whole server by forcing it into swap.
The table type you use can be of importance, certain types of queries seem to run faster on MYISAM as opposed to INNODB. If you want to retool a bit then you could store this data (or a copy of it) in mysql's HEAP engine, so just in memory. You'd need to be careful to synchronize it with a disktable on writes though if you want to keep altered data for sure. (just in case of a server failure or shutdown)
Alternatively you could cache your data in something like memcache or by using apc_store, which should be very fast since it's in php process memory. The big caveat here is that APC generally has less memory available for storage though.(default being 32MB) Memcache's big adavantage is that while still fast, it's distributed, so if you have multiple servers running they could share this data.
You could try a nosql database, preferably one that's just a key-store, not even a document store, such as redis.
And finally you could hardcode your values in your php script, make sure to still use something like eaccelerator or APC and verify wether you really need to use them 4 times or wether you can't just cache the output of whatever it is you actually create with it.
So I'm sorry I can't give you a ready-made answer but performance questions, when applicable, usually require a multi-pronged approach. :-|
I'm making a website that (essentially) lets the user submit a word, matches it against a MySQL database, and returns the closest match found. My current implementation is that whenever the user submits a word, the PHP script is called, it reads the database information, scans each word one-by-one until a match is found, and returns it.
I feel like this is very inefficient. I'm about to make a program that stores the list of words in a tree structure for much more effective searching. If there are tens of thousands of words in the database, I can see the current implementation slowing down quite a bit.
My question is this: instead of having to write another, separate program, and use PHP to just connect to it with every query, can I instead save an entire data tree in memory with just PHP? That way, any session, any query would just read from memory instead of re-reading the database and rebuilding the tree over and over.
I'd look into running an instance of memcached on your server. http://www.memcached.org.
You should be able to store the compiled tree of data in memory there and retrieve it for use in PHP. You'll have to load it into PHP to perform your search, though, as well as architect a way for the tree in memcached to be updated when the database changes (assuming the word list can be updated, since there's not a good reason to store it in a database otherwise).
Might I suggest looking at the memory table type in mysql: http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html
You can then still use mysql's searching features on fast "in memory" data.
PHP really isn't a good language for large memory structures. It's just not very memory efficient and it has a persistence problem, as you are asking about. Typically with PHP, people would store the data in some external persistent data store that is optimized for quick retrieval.
Usually people use a two fold approach:
1) Store data in the database as optimized as possible for standard queries
2) Cache results of expensive queries in memcached
If you are dealing with a lot of data that cannot be indexed easily by relational databases, then you'd probably need to roll your own daemon (e.g., written in C) that kept a persistent copy of the data structure in memory for fast querying capabilities.
I am working on a project that is being built with a standard LAMP stack. Currently, I merely output the results of the query onto the page - the results are not being stored in objects at all.
In the future, I would like to edit the results of the query. I imagine that this would much easier if the results were stored in PHP objects.
Would it be more beneficial to store the objects themselves in a DB (via serialization/deserialization), or to create the objects when need be (after executing the query) and then destroying them when they are no longer needed?
You'd be better off storing a copy of the results directly in your object, rather than a serialized result handle. Serializing the result handle will NOT preserve locks, server-side variables, table state, transactions, or the data in the result set. MySQL has no provision for storing a connection handle in this fashion, so it'd be seen as a regular disconnect and resulting in outstanding queries being cleaned up, variables destroyed, transactions rolled back (or committed), etc...
As well, the data retrieved by the query is not actually fetched across the connection until you do a fetch_row()-type call, so you'd not even have that in your serialized handle.
Always create the objects in php, and destroy them later. In order to serialize you will need to use longtext or like field, which are known to be slow and you cannot index on them. If you are always doing a Select All, then go ahead, but if you ever use conditions or advanced queries, you should have all data separated.
It depends on many factors. If you are running the exact same queries again and again, then yes, store the results in your database. But why serialise them? If you tried Object-relational mapping, you could have a much easier to maintain query object, that you could store in a well organised relational database.
If you are not running the same queries very often, I would recommend caching the output in another way.
Would it be more beneficial to store the objects themselves in a DB (via serialization/deserialization), or to create the objects when need be (after executing the query) and then destroying them when they are no longer needed?
No. Somebody somewhere has done this for you. What would be beneficial is for you to use an existing ORM. It doesn't matter which one, just pick one and use it. You'll be lightyears ahead and get your project out the in a fraction of the time.
You should use a PHP framework while you're at it, many of which come coupled to an ORM.
How do the different MySQL Cursors within PHP manage memory? What I mean is, when I make a MySQL query that retrieves a large result set, and get the MySQL resource back, how much of the data that the query retrieved is stored in local memory, and how are more results retrieved? Does the cursor automatically fetch all the results, and give them to me as I iterate through the resource with fetch_array or is it a buffered system?
Finally, are the cursors for the different drivers within mysql implemented differently? There's several MySQL drivers for PHP available, mysql, mysqli, pdo, etc. Do they all follow the same practices?
That depends on what you ask php to do, for instance mysql_query() grabs all the result set (if that's 500 megabytes, goodbye) ; if you don't want that you can use :
http://php.net/manual/en/function.mysql-unbuffered-query.php
PDO, MySQLI seem to have other ways of doing the same thing.
Depending on your query, the result set may be materialized on the database side (if you need a sort, then the sort must be done entirely before you even get the first row).
For not too large result sets it's usually better to fetch it all at once, so the server can free used resources asap.
I'm working on what is turning out to be a fairly complex SELECT query. I have several hierarchical queries being nested in a single SELECT and it is getting to be quite difficult to manage.
I'm running into a few places where my inline views need to be executed in more than one place, so it seems like a reasonable idea to execute those once at the beginning of a stored procedure and then do some iteration over the results as needed.
I'm wondering if there are any reasons why I should not, or could not, execute an Oracle Stored Procedure, called via my PHP code, and return as an OUT parameter the resultset. I've tended to use SPs only to do updates/deletes/inserts but the sheer size and complexity of this query seems like it needs to be broken down.
If there aren't any technical problems with this, any comments on whether it is good or bad practice?
Im working on what is turning out to be a fairly complex SELECT query. I have several hierarchical queries being nested in a single SELECT and it is getting to be quite difficult to manage.
Ok, but why a stored procedure? Why not create a view instead?
I'm running into a few places where my inline views need to be executed in more than one place, so it seems like a reasonable idea to execute those once at the beginning of a stored procedure and then do some iteration over the results as needed.
Again - excellent use case for a view.
I'm wondering if there are any reasons why I should not, or could not, execute an Oracle Stored Procedure, called via my PHP code, and return as an OUT parameter the resultset.
If there aren't any technical problems with this, any comments on whether it is good or bad practice?
Well, I don't want to start a religous war, and I do not want to suggest the arguments against apply to your case. But here goes:
one reason why I tend to avoid stored procedures is portability - by that I mean mostly database portability. Stored procedure languages are notoriously unportable across dbs, and built-in libs like Oracle packages make things worse in that respect.
stored procedures take some additional processing power from your database server. this makes it harder to scale the application as a whole: if the capacity of your db server is exhausted due to stored procedures, and you need to upgrade harware or even buy an extra oracle software license because of that, I would not be a happy camper, especially if I could have bought cheap webserver/php boxes instead to do the computing.
Reasons where I would go for stored procedures:
language portability. If database portability is not so much an issue, but you do want to reuse logic across multiple applications, or have to ability to code in different languages, then stored procedures may save you writing language specific database invocation code.
complex permission scenarios. stored procedures give you uan extra level of permissions, since you can execute the procedure with the privileges of the definer or owner of the stored procedure. Sometimes this solves problems where a user needs to work with some tables, but cannot be allowed direct access to them.
saving rountrips: if you have to deal with complex, multistatement transactions, putting them in a stored procedures saves rountrips between the app and the db, because there is only one rountrip needed to execute the stored procedure. sometimes this can get you more performance.
I want to stress again that in all these scenarios, I would still advise to not put all your procedural logic in stored procedures. databases are best at storing and retrieving data, languages like php/java/perl/pick your poison are better at processing it.
If you are using the same inline view many times, its a good candidate for with clause
PHP can handle resultsets returned from stored procedures, by using Ref Cusrors. The Oracle+PHP Cookbook has an example.
So there are no technical impediments but as you can see from the various answers there are some philosophical aspects to your question. I think we can agree that if you are already wrapping some SQL statements in stored procedures - which you are - then you are not drastically compromising the portability of your system by extending "updates/deletes/inserts" to include selects.
The pertinent question then becomes "should you embed use a stored procedure for this particular query?" The answer to which hinges on precisely what you mean by:
the sheer size and complexity of this
query seems like it needs to be broken
down.
Deconstructing a big query into several smaller queries and then stitching results together in PL/SQL is seductive, but should be approached with caution. This can degrade the performance of your application, because PL/SQL has more overheads than SQL. Making your query more readable is not a good enough reason: you need to be certain that the complexity has a real and adverse effect on the running of your code.
A good reason for using a stored procedure rather than a view might be if you want to extend the applicability of the query by using bind variables or dynamic SQL in the body of the query.
A definitive answer to your question requires more details regarding the nature of your query and the techniques you are thinking of using to simplify it.
You could look at subquery factoring which may improve the readability of the query.
One risk of breaking up a single SQL query into a more procedural solution is you lose read consistency. As such you want to be pretty sure that someone changing data while your procedure runs won't break it. You may want to lock a table fore the duration of the procedure call. It seems drastic, but if you are pretty sure that the data is static and if there would be ugly side-effects if it wasn't, then it is a solution.
Generally if an SQL statement is complex enough, it probably isn't portable between databases anyway, so I wouldn't worry about that aspect.
Views can be a good option to hide complexity, but the downside to hiding complexity is that people start doing things that seem 'simple' but are really complex and don't work as desired. You also get another object to consider for grants etc. [Edit: As Roland commented, this applies equally to stored procedures, views, object types etc.]
If you expect to return a large resultset, you should consider a pipelined table function. That way you can avoid having the entire resultset in the Oracle session at the same time.