Lets say a have Web page with some classes. One is loaded Mysqli connect it to DB at the beginning and keep connected. Now question is:
Is good solution make in (example setting class) prepared statement for calling value from DB table 'settings' and keep it open (statement) until finish (at footer close statement and connection) or just load all data from 'settings' DB table to array() in php and just call it from array not binding it from DB.
Second question is if I have statement open may I open another statement for another class (example class for calling text from DB) and do it same like in previous example? And than, of course close it at finish page.
Is there any performance or security problem, you can see there...
As far as I know, nobody is doing it this way. Mostly because the real benefit from the multiple execution is not that grand as some imagine, and just doesn't worth the trouble. For the short primary key lookups run in small numbers (several dozens at max) you'll hardly be able to tell the difference.
(However, there are no arguments against such practice either - you can make it this way, with single statement prepared/multiple executions, if you wish).
Yet single query that is fetching no more than couple hundreds of records still would be faster than separate queries (even prepared) to get the same amount. So, as long as your settings keep at moderate amount, it's better to get them all at once.
Yes, of course you can have as many statements prepared as you need.
(The only problem could be with fetching results. You have to always get_result/store_result, to make sure there are no results left pending and preventing other queries to run, either regular or prepared).
The statement executes as one SQL statement over your DB connection. It's not going to keep going back to the database and grabbing a single row one at a time, so don't worry about that.
In general, you should be loading everything into some data structure. If your query is returning more data than you need, then that's something you need to fix in your query. Don't run SQL that returns a huge set of data, then rely on PHP to go through it row by row and perform some hefty operations on it. Just write SQL that gets what you need in the first place. I realize this isn't always possible, but when people talk about optimizing their website, query optimization is usually at/near the top of that list, so it's pretty important.
You're definitely supposed to execute multiple statements. It's silly to keep opening and closing entire db connections before getting any data.
Related
This is odd. I'm running a query with just a single INSERT, preceded by a SET statement. The query looks something like this:
SET #discount:=(SELECT discount * :applyDiscount FROM fra_cus WHERE customerID=:customerID AND franchiseID=:franchiseID);
INSERT INTO discounts_applied (unitID, franchiseID, customerID, amount)
VALUES(:unitID, :franchiseID, :customerID, #discount * :price);
It appears that if I prepare these as two separate PDO queries, lastInsertID() works fine... but if I prepare them and execute them in the same statement, lastInsertID() returns nothing.
It's not the end of the world, but it's annoying. Anyone know why this would be the case? For the record, there's a reason I need to define #discount as a variable (pertains to triggers on one of the tables). Also this is all taking place within a larger transaction.
First of all, I would strongly recommend to run every query in a distinct API call. This is how an Application Programming Interface is intended to work.
It won't only prevent situations like this but also will make your code a multitude times more readable and maintainable.
And it will make your code much safer too. You can run multiple statements in a single call only at the expense of the native prepared statements. However virtual this vulnerability is, why taking chances at all?
Why not to make a regular SELECT query instead of SET, get the resulting value into a PHP variable and then use it among other variables, just through a placeholder? I don't see any reason why there should be such a complex way to deal with simple data.
In case I failed to convince you, the reason is simple. You are running two queries, and the first one doesn't trigger any insert ids. And obviously, you need this query's metadata (errors, affected rows, whatever), not the other one's first. So you get it. And to get the second'query's metadata you have to ask a database for it. The process is explained in my article: Treating PHP delusions - The only proper PDO tutorial: Running multiple queries with PDO. Basically PDOStatement::nextRowset() is what you need.
I am building a Web Application, that will query a lot of data, with the emphasis being on the user to customise their query down to the nth degree.
As an example I have built a form, where the user can define anywhere between 0 - 50 criteria, each tied to a specific database value. if they define nothing they get all, and so on.
I then allow the user to define once they define the criteria what columns they would like to see, this too allows anywhere from 1 - 50 columns. default around 5 - 10.
I am trying to keep as much of the app client side as possible, limiting the server side strain. But what is less effort on the server;
PHP If statement around every column, if the user doesnt want it, then dont POST it as part of SQL Statement. i.e. Check if user wants it and filter SELECT statement.
Or
Have the SQL Statement rigid, and return all columns, and filter once returned client side.
I know it might side cut and dry, but the way i see it, 50 PHP IF statements, is a lot to check? but returning all Rows and all Columns is also quite a large ask?
Any help would be appreciated.
Conditional statements actually improve performance in a lot of cases, because the script is allowed to skip over code blocks. The only time an if statement would have a significant impact on code execution, is if it were executing a function or doing some processing as part of the conditional, like:
if(isTrue()) {
doThis();
}
Of course, I haven't seen your code; There are several profiling tools, like Webgrind, if you want to compare the speed yourself, then you can make a more informed decision.
I cringed when Sebastien stated he was disconnecting & reconnecting between each use of mysqli_multi_query() # Can mysqli_multi_query do UPDATE statements? because it just didn't seem like best practice.
However, Craig # mysqli multi_query followed by query stated in his case that it was faster to disconnect & reconnect between each use of mysqli_multi_query() than to employ mysqli_next_result().
I would like to ask if anyone has further first-hand knowledge or benchmark evidence to suggest an approximate "cutoff" (based on query volume or something) when a programmer should choose the "new connection" versus "next result" method.
I am also happy to hear any/all concerns not pertaining to speed. Does Craig's use of a connecting function have any bearing on speed?
Is there a speed difference between Craig's while statement:
while ($mysqli->next_result()) {;}
- versus -
a while statement that I'm suggesting:
while(mysqli_more_results($mysqli) && mysqli_next_result($mysqli));
- versus -
creating a new connection for each expected multi_query, before running first multi_query. I just tested this, and the two mysqli_multi_query()s were error free = no close() needed:
$mysqli1=mysqli_connect("$host","$user","$pass","$db");
$mysqli2=mysqli_connect("$host","$user","$pass","$db");
- versus -
Opening and closing between each mysqli_multi_query() like Sebastien and Craig:
$mysqli = newSQL();
$mysqli->multi_query($multiUpdates);
$mysqli->close();
- versus -
Anyone have another option to test against?
It is not next_result() to blame but queries themselves. The time your code takes to run relies on the time actual queries take to perform.
Although mysqli_multi_query() returns control quite fast, it doesn't mean that all queries got executed by that time. Quite contrary, by the time mysqli_multi_query() finished, only first query got executed. While all other queries are queued on the mysql side for the asynchronous execution.
From this you may conclude that next_result() call doesn't add any timeout by itself - it's just waiting for the next query to finish. And if query itself takes time, then next_result() have to wait as well.
Knowing that you already may tell which way to choose: if you don't care for the results, you may just close the connection. But in fact, it'll be just sweeping dirt under the rug, leaving all the slow queries in place. So, it's better to keep next_result() loop in place (especially because you have to check for errors/affected rows/etc. anyway) but speed up the queries themselves.
So, it turns out that to solve the problem with next_result() you have to actually solve the regular problem of the query speed. So, here are some recommendations:
For the select queries it's usual indexing/explain analyze, already explained in other answers.
For the DML queries, especially run in batches, there are other ways:
Speaking of Craig's case, it's quite much resembling the known problem of speed of innodb writes. By default, innodb engine is set up into very cautious mode, where no following write is performed until engine ensured that previous one were finished successfully. So, it makes writes awfully slow (something like only 10 queries/sec). The common workaround for this is to make all the writes at once. For insert queries there are plenty of methods:
you can use multiple values insert syntax
you can use LOAD DATA INFILE query
you can wrap all the queries in a transaction.
While for updating and deleting only transaction remains reliable way. So, as a universal solution such a workaround can be offered
$multiSQL = "BEGIN;{$multiSQL}COMMIT;";
$mysqli->multi_query($multiSQL);
while ($mysqli->next_result()) {/* check results here */}
If it doesn't work/inapplicable in your case, then I'd suggest to change mysqli_multi_query() for the single queries run in a loop, investigate and optimize the speed and then return to multi_query.
To answer your question:
look before you jump
I expect your mysqli_more_results() call (the look before you jump), doesn't speed up things: If you have n results, you'll do (2*n)-1 calls to the database, whereas Craig does n+1.
multiple connections
multi_query executes async, so you'll just be adding connection overhead.
opening and closing db
Listen to Your Common Sense ;-) But don't loose track of what you're doing. Wrapping queries in a transaction, will make them atomic. That means, they all fail, or they all succeed. Sometimes that is required to make the database never conflict with your universe of discourse. But using transactions for speedups, may have unwanted side-effects. Consider the case where one of your queries violates a constraint. That will make the whole transaction fail. Meaning that if they weren't a logical transaction in the first place and most queries should have succeeded, that you'll have to find out which went wrong and which will have to be reissued. Costing you more instead of delivering a speedup.
Sebastien's queries actually look like they should be part of some bigger transaction, that contains the deletion or updates of the parents.
Instead, try and remember
there is no spoon
In your examples, there was no need for multiple queries. The INSERT ... VALUES form takes multiple tuples for VALUES. So instead of preparing one prepared statement and wrap its repeated executions in a transaction like Your Common Sense suggest. You could prepare a single statement and have it executed and auto-committed. As per mysqli manual this saves you a bunch of roundtrips.
So make a SQL statement of the form:
INSERT INTO r (a, b, c) VALUES (?, ?, ?), (?, ?, ?), ...
and bind and execute it. mysqldump --opt does it, so why don't we? The mysql reference manual as a section on statement optimization. Look in its DML section for insert and update queries. But understanding why --opt does what it does is a good start.
the underestimated value of preparing a statement
To me, the real value of prepared statements is not that you can execute them multiple times, but the automatic input escaping. For a measly single extra client-server round-trip, you save yourself from SQL injection. SQL injection is a serious point of attention especially when you're using multi_query. multi_query tells mysql to expect multiple queries and execute them. So fail to escape properly and you're in for some fun:
So my best practise would be:
Do I really need multiple queries?
If I do, escape them well, or prepare them!
In PHP I'm using mysqli_fetch_assoc() in a while-loop to get every record in a certain query.
I'm wondering what happens if the data is changed while running the loop (by another process or server), so that the record doesn't match the query any more. Will it still be fetched?
In other words, is the array of records that are fetched fixed, when you do query()? Or is it not?
Update:
I understand that it's a feature that the resultset is not changed when the data is changed, but what if you actually WANT that? In my loop I'm not interested in records that are already updated by another server. How do I check for that, without doing a new query for each record that I fetch??
UPDATE:
Detailed explanation:
I'm working on some kind of searchengine-scraper that searches for values in a database. This is done by a few servers at the same time. Items that have been scraped shouldn't be searched anymore. I can't really control which server searches which item, I was hoping I could check the status of an item, while fetching the recordset. Since it's a big dataset, I don't transfer the entire resultset before searching, I fetch each record when I need it...
Introduction
I'm wondering what happens if the data is changed while running the loop (by another process or server), so that the record doesn't match the query any more. Will it still be fetched?
Yes.
In other words, is the array of records that are fetched fixed, when you do query()? Or is it not?
Yes.
A DBMS would not be worth its salt were it vulnerable to race conditions between table updates and query resultset iteration.
Certainly, as far as the database itself is concerned, your SELECT query has completed before any data can be changed; the resultset is cached somewhere in the layers between your database and your PHP script.
In-depth
With respect to the ACID principle *:
In the context of databases, a single logical operation on the data is called a transaction.
User-instigated TRANSACTIONs can encompass several consecutive queries, but 4.33.4 and 4.33.5 in ISO/IEC 9075-2 describe how this takes place implicitly on the per-query level:
The following SQL-statements are transaction-initiating
SQL-statements, i.e., if there is no current SQLtransaction, and an
SQL-statement of this class is executed, then an SQL-transaction is
initiated, usually before execution of that SQL-statement proceeds:
All SQL-schema statements
The following SQL-transaction statements:
<start transaction statement>.
<savepoint statement>.
<commit statement>.
<rollback statement>.
The following SQL-data statements:
[..]
<select statement: single row>.
<direct select statement: multiple rows>.
<dynamic single row select statement>.
[..]
[..]
In addition, 4.35.6:
Effects of SQL-statements in an SQL-transaction
The execution of an SQL-statement within an SQL-transaction has no
effect on SQL-data or schemas [..]. Together with serializable
execution, this implies that all read operations are repeatable
within an SQL-transaction at isolation level SERIALIZABLE, except
for:
1) The effects of changes to SQL-data or schemas and its contents
made explicitly by the SQL-transaction itself.
2) The effects of differences in SQL parameter values supplied to externally-invoked
procedures.
3) The effects of references to time-varying system
variables such as CURRENT_DATE and CURRENT_USER.
Your wider requirement
I understand that it's a feature that the resultset is not changed when the data is changed, but what if you actually WANT that? In my loop I'm not interested in records that are already updated by another server. How do I check for that, without doing a new query for each record that I fetch??
You may not.
Although you can control the type of buffering performed by your connector (in this case, MySQLi), you cannot override the above-explained low-level fact of SQL: no INSERT or UPDATE or DELETE will have an effect on a SELECT in progress.
Once the SELECT has completed, the results are independent; it is the buffering of transport of this independent data that you can control, but that doesn't really help you to do what it sounds like you want to do.
This is rather fortunate, frankly, because what you want to do sounds rather bizarre!
* Strictly speaking, MySQL has only partial ACID-compliance for tables other than those with the non-default storage engines InnoDB, BDB and Cluster, and MyISAM does not support [user-instigated] transactions. Still, it seems like the "I" should remain applicable here; MyISAM would be essentially useless otherwise.
I am creating a website with content generated by php and mysql. All over the page, you will see stuff like comments, user profile information, announcements, etc that require the use of mysql queries and php.
As you can see there will be lots of mysql queries and row fetching being done throughout the whole page.
Would it be faster if I call all queries, fetch all rows and put them into arrays at the start of the php file? Or would it be faster if I just dispersed them throughout the document? e.g. Put the comment queries, comment row fetches and comment array definitions where the comment section is.
If I placed all the queries at the start of the document, would it be faster to combine all the queries into one huge query? I'm worried about the memory usage here.
After defining arrays using values fetched from mysql queries, I usually free the results of the query. Is this beneficial or just a waste of time?
1) You should separate application logic from presentational code. So in this sense, yes, you should perform the queries required to build a page before actually rendering the page. It is not 'faster', but it makes your application easier to understand, maintain and extend.
2) It is always worth performing as few queries as possible, so long as your results sets don't start needing large amounts of post-processing by PHP to be usable by the application.
3) It's a waste of time.
What I normally do to make php run faster;
Put all variables at the top, use if, elseif etc, and else, then order your queries in the order you think which will be most popular. That way if the query is unlikely then it will not execute as it is further down the elseif list.
I would free the result, especially when checking if a variable already exists or not in the db.
Some more tips...
Use session_write_close where possible.
Place all arrays at top of page.
Sometimes using flush() can help, you would have to try it out.