I cringed when Sebastien stated he was disconnecting & reconnecting between each use of mysqli_multi_query() # Can mysqli_multi_query do UPDATE statements? because it just didn't seem like best practice.
However, Craig # mysqli multi_query followed by query stated in his case that it was faster to disconnect & reconnect between each use of mysqli_multi_query() than to employ mysqli_next_result().
I would like to ask if anyone has further first-hand knowledge or benchmark evidence to suggest an approximate "cutoff" (based on query volume or something) when a programmer should choose the "new connection" versus "next result" method.
I am also happy to hear any/all concerns not pertaining to speed. Does Craig's use of a connecting function have any bearing on speed?
Is there a speed difference between Craig's while statement:
while ($mysqli->next_result()) {;}
- versus -
a while statement that I'm suggesting:
while(mysqli_more_results($mysqli) && mysqli_next_result($mysqli));
- versus -
creating a new connection for each expected multi_query, before running first multi_query. I just tested this, and the two mysqli_multi_query()s were error free = no close() needed:
$mysqli1=mysqli_connect("$host","$user","$pass","$db");
$mysqli2=mysqli_connect("$host","$user","$pass","$db");
- versus -
Opening and closing between each mysqli_multi_query() like Sebastien and Craig:
$mysqli = newSQL();
$mysqli->multi_query($multiUpdates);
$mysqli->close();
- versus -
Anyone have another option to test against?
It is not next_result() to blame but queries themselves. The time your code takes to run relies on the time actual queries take to perform.
Although mysqli_multi_query() returns control quite fast, it doesn't mean that all queries got executed by that time. Quite contrary, by the time mysqli_multi_query() finished, only first query got executed. While all other queries are queued on the mysql side for the asynchronous execution.
From this you may conclude that next_result() call doesn't add any timeout by itself - it's just waiting for the next query to finish. And if query itself takes time, then next_result() have to wait as well.
Knowing that you already may tell which way to choose: if you don't care for the results, you may just close the connection. But in fact, it'll be just sweeping dirt under the rug, leaving all the slow queries in place. So, it's better to keep next_result() loop in place (especially because you have to check for errors/affected rows/etc. anyway) but speed up the queries themselves.
So, it turns out that to solve the problem with next_result() you have to actually solve the regular problem of the query speed. So, here are some recommendations:
For the select queries it's usual indexing/explain analyze, already explained in other answers.
For the DML queries, especially run in batches, there are other ways:
Speaking of Craig's case, it's quite much resembling the known problem of speed of innodb writes. By default, innodb engine is set up into very cautious mode, where no following write is performed until engine ensured that previous one were finished successfully. So, it makes writes awfully slow (something like only 10 queries/sec). The common workaround for this is to make all the writes at once. For insert queries there are plenty of methods:
you can use multiple values insert syntax
you can use LOAD DATA INFILE query
you can wrap all the queries in a transaction.
While for updating and deleting only transaction remains reliable way. So, as a universal solution such a workaround can be offered
$multiSQL = "BEGIN;{$multiSQL}COMMIT;";
$mysqli->multi_query($multiSQL);
while ($mysqli->next_result()) {/* check results here */}
If it doesn't work/inapplicable in your case, then I'd suggest to change mysqli_multi_query() for the single queries run in a loop, investigate and optimize the speed and then return to multi_query.
To answer your question:
look before you jump
I expect your mysqli_more_results() call (the look before you jump), doesn't speed up things: If you have n results, you'll do (2*n)-1 calls to the database, whereas Craig does n+1.
multiple connections
multi_query executes async, so you'll just be adding connection overhead.
opening and closing db
Listen to Your Common Sense ;-) But don't loose track of what you're doing. Wrapping queries in a transaction, will make them atomic. That means, they all fail, or they all succeed. Sometimes that is required to make the database never conflict with your universe of discourse. But using transactions for speedups, may have unwanted side-effects. Consider the case where one of your queries violates a constraint. That will make the whole transaction fail. Meaning that if they weren't a logical transaction in the first place and most queries should have succeeded, that you'll have to find out which went wrong and which will have to be reissued. Costing you more instead of delivering a speedup.
Sebastien's queries actually look like they should be part of some bigger transaction, that contains the deletion or updates of the parents.
Instead, try and remember
there is no spoon
In your examples, there was no need for multiple queries. The INSERT ... VALUES form takes multiple tuples for VALUES. So instead of preparing one prepared statement and wrap its repeated executions in a transaction like Your Common Sense suggest. You could prepare a single statement and have it executed and auto-committed. As per mysqli manual this saves you a bunch of roundtrips.
So make a SQL statement of the form:
INSERT INTO r (a, b, c) VALUES (?, ?, ?), (?, ?, ?), ...
and bind and execute it. mysqldump --opt does it, so why don't we? The mysql reference manual as a section on statement optimization. Look in its DML section for insert and update queries. But understanding why --opt does what it does is a good start.
the underestimated value of preparing a statement
To me, the real value of prepared statements is not that you can execute them multiple times, but the automatic input escaping. For a measly single extra client-server round-trip, you save yourself from SQL injection. SQL injection is a serious point of attention especially when you're using multi_query. multi_query tells mysql to expect multiple queries and execute them. So fail to escape properly and you're in for some fun:
So my best practise would be:
Do I really need multiple queries?
If I do, escape them well, or prepare them!
Related
This is odd. I'm running a query with just a single INSERT, preceded by a SET statement. The query looks something like this:
SET #discount:=(SELECT discount * :applyDiscount FROM fra_cus WHERE customerID=:customerID AND franchiseID=:franchiseID);
INSERT INTO discounts_applied (unitID, franchiseID, customerID, amount)
VALUES(:unitID, :franchiseID, :customerID, #discount * :price);
It appears that if I prepare these as two separate PDO queries, lastInsertID() works fine... but if I prepare them and execute them in the same statement, lastInsertID() returns nothing.
It's not the end of the world, but it's annoying. Anyone know why this would be the case? For the record, there's a reason I need to define #discount as a variable (pertains to triggers on one of the tables). Also this is all taking place within a larger transaction.
First of all, I would strongly recommend to run every query in a distinct API call. This is how an Application Programming Interface is intended to work.
It won't only prevent situations like this but also will make your code a multitude times more readable and maintainable.
And it will make your code much safer too. You can run multiple statements in a single call only at the expense of the native prepared statements. However virtual this vulnerability is, why taking chances at all?
Why not to make a regular SELECT query instead of SET, get the resulting value into a PHP variable and then use it among other variables, just through a placeholder? I don't see any reason why there should be such a complex way to deal with simple data.
In case I failed to convince you, the reason is simple. You are running two queries, and the first one doesn't trigger any insert ids. And obviously, you need this query's metadata (errors, affected rows, whatever), not the other one's first. So you get it. And to get the second'query's metadata you have to ask a database for it. The process is explained in my article: Treating PHP delusions - The only proper PDO tutorial: Running multiple queries with PDO. Basically PDOStatement::nextRowset() is what you need.
Lets say a have Web page with some classes. One is loaded Mysqli connect it to DB at the beginning and keep connected. Now question is:
Is good solution make in (example setting class) prepared statement for calling value from DB table 'settings' and keep it open (statement) until finish (at footer close statement and connection) or just load all data from 'settings' DB table to array() in php and just call it from array not binding it from DB.
Second question is if I have statement open may I open another statement for another class (example class for calling text from DB) and do it same like in previous example? And than, of course close it at finish page.
Is there any performance or security problem, you can see there...
As far as I know, nobody is doing it this way. Mostly because the real benefit from the multiple execution is not that grand as some imagine, and just doesn't worth the trouble. For the short primary key lookups run in small numbers (several dozens at max) you'll hardly be able to tell the difference.
(However, there are no arguments against such practice either - you can make it this way, with single statement prepared/multiple executions, if you wish).
Yet single query that is fetching no more than couple hundreds of records still would be faster than separate queries (even prepared) to get the same amount. So, as long as your settings keep at moderate amount, it's better to get them all at once.
Yes, of course you can have as many statements prepared as you need.
(The only problem could be with fetching results. You have to always get_result/store_result, to make sure there are no results left pending and preventing other queries to run, either regular or prepared).
The statement executes as one SQL statement over your DB connection. It's not going to keep going back to the database and grabbing a single row one at a time, so don't worry about that.
In general, you should be loading everything into some data structure. If your query is returning more data than you need, then that's something you need to fix in your query. Don't run SQL that returns a huge set of data, then rely on PHP to go through it row by row and perform some hefty operations on it. Just write SQL that gets what you need in the first place. I realize this isn't always possible, but when people talk about optimizing their website, query optimization is usually at/near the top of that list, so it's pretty important.
You're definitely supposed to execute multiple statements. It's silly to keep opening and closing entire db connections before getting any data.
How to take advantage of prepared statements for performance? I understand that something like this might benefit if I put it in a loop:
SELECT `Name` FROM `Hobbits` WHERE `ID` = :ID;
I've read that looping with prepared statements is faster than looping without, but otherwise prepared statements would slightly decrease performance. So - how big may that loop be?
If I run a complex SQL query at the beginning of my code and repeat it with one different parameter at the end - will the second query run faster? (We are using a single connection for each page load). Is there a limit on cached queries, so I better repeat my queries right away?
What about executing the entire script twice with the exact same parameters (reload the page or 2 users)?
A prepared query is given to the SQL server, which parses it and possibly already prepares an execution plan. You're then basically given an id for these allocated resources and can execute this prepared statement by just filling in the blanks in the statement. You can run this statement as often as you like and the database will not have to repeat the parsing and execution planning, which may bring a speed improvement.
As long as you do not throw away the statement, there's no hard timeout for how long the statement will "stay prepared". It's not a cache, it's an allocated resource on the SQL server. At least as long as your database driver uses native prepared statements in the SQL API. PDO for example does not do so by default, unless you set PDO::ATTR_EMULATE_PREPARES to false.
At the end of the script execution though, all those resources will always be deallocated, they do not persist across different page loads. Beyond that, the SQL server may or may not cache the query and its results for some time regardless of the client script.
How long are prepared mysql queries cached?
This is not actually "a cache". Prepared statement lasts as little as during script execution.
If I run a complex SQL query at the beginning of my code and repeat it with one different parameter at the end - will the second query run faster?
The more complex a query, the less effect you will see. Frankly, prepared statement saves you only parsing, while if execution involves temporary or filesort, or table scan - prepared statement would speed up none of them.
On the other hand, for the simple primary-key lookups, which involve no complex query parsing nor building sophisticated query plans, the benefit would be negligible to none.
So - how big may that loop be?
The more iterations it gets - the more benefit. However, in a sane web-application one have to avoid looping queries at all.
What about executing the entire script twice with the exact same parameters (reload the page or 2 users)?
As I said above, there will be no benefit from a prepared statement at all. A classical query cache, however, most likely would fire.
How to take advantage of prepared statements for performance?
Noway. Not in web-serving PHP, at least. In a some long-running cli-based script - may be.
However, prepared statements ought to be used anyway, for the purpose of producing syntactically correct queries.
FYI, from
http://dev.mysql.com/doc/refman/5.7/en/statement-caching.html
The max_prepared_stmt_count system variable controls the total number of statements the server caches. (The sum of the number of prepared statements across all sessions.)
I'm sorry, this is a very general question but I will try to narrow it down.
I'm new to this whole transaction thing in MySQL/PHP but it seems pretty simple. I'm just using mysql not mysqli or PDO. I have a script that seems to be rolling back some queries but not others. This is uncharted territory for me so I have no idea what is going on.
I start the transaction with mysql_query('START TRANSACTION;'), which I understand disables autocommit at the same time. Then I have a lot of complex code and whenever I do a query it is something like this mysql_query($sql) or $error = "Oh noes!". Then periodically I have a function called error_check() which checks if $error is not empty and if it isn't I do mysql_query('ROLLBACK;') and die($error). Later on in the code I have mysql_query('COMMIT;'). But if I do two queries and then purposely throw an error, I mean just set $error = something, it looks like the first query rolls back but the second one doesn't.
What could be going wrong? Are there some gotchas with transactions I don't know about? I don't have a good understanding of how these transactions start and stop especially when you mix PHP into it...
EDIT:
My example was overly simplified I actually have at least two transactions doing INSERT, UPDATE or DELETE on separate tables. But before I execute each of those statements I backup the rows in corresponding "history" tables to allow undoing. It looks like the manipulation of the main tables gets rolled back but entries in the history tables remain.
EDIT2:
Doh! As I finished typing the previous edit it dawned on me...there must be something wrong with those particular tables...for some reason they were all set as MyISAM.
Note to self: Make sure all the tables use transaction-supporting engines. Dummy.
I'd recommend using the mysqli or PDO functions rather than mysql, as they offer some worthwhile improvements—especially the use of prepared statements.
Without seeing your code, it is difficult to determine where the problem lies. Given that you say your code is complex, it is likely that the problem lies with your code rather than MySQL transactions.
Have you tried creating some standalone test scripts? Perhaps you could isolate the SQL statements from your application, and create a simple script which simply runs them in series. If that works, you have narrowed down the source of the problem. You can echo the SQL statements from your application to get the running order.
You could also try testing the same sequence of SQL statements from the MySQL client, or through PHPMyAdmin.
Are your history tables in the same database?
Mysql transactions only work using the mysqli API (not the classic methods). I have been using transactions. All I do is deactivate autocommit and run my SQL statements.
$mysqli->autocommit(FALSE);
SELECT, INSERT, DELETE all are supported. as long as Im using the same mysqli handle to call these statements, they are within the transaction wrapper. nobody outside (not using the same mysqli handle) will see any data that you write/delete using INSERT/DELETE as long as the transaction is still open. So its critical you make sure every SQL statement is fired with that handle. Once the transaction is committed, data is made available to other db connections.
$mysqli->commit();
According to the documentation, a prepared query provides a significant performance benefit if you're running a query multiple times because the overhead of the MySQL server parsing the query only happens once. I'm wondering what exactly they mean by "multiple times" there.
I.e., say you have a web page that runs a query one time. Now say that page gets called 50 times per second. Does it make more sense from a performance standpoint to prepare() the query (thereby requiring two roundtrips to the DB server; one to prepare the query, one to run it) or to just send the query normally (which only requires one roundtrip)? Is MySQL and/or the PHP mysqli driver smart enough to realize when a query was prepare()'d in a previous invocation?
No. PHP is a "shared nothing" architecture, so every resource associated with one request (one page view) is discarded at the end of that request. Prepared queries are not available to a subsequent database connections.
The scenario in which you would get benefit from a prepared query is when you prepare it and execute it many times during the same PHP request.