I have an issue concerning PDO persistent connection. Now this may not be an actual problem, but I can't seem to find any post addressing this behavior.
I'm using the good old PDO in a persistent connection mode for my web app. Now I'm creating a new connection via new PDO(...).
When I run this script a new connection (C#1) is getting established and a MySql process (P#1) to accommodate the persistent connection.
So, I run the script again creating a new conction (C#2) and expecting C#2 to use the P#1 from the last connection. Every time I run this script a new process appears while the last one is still alive (in sleep mode).
On my production server there are about 350 prossers (in sleep) at any given time from 3 defrent users (all users connect from the same apache server).
The question: is this situation valid?
found my answer
They cause the child process to simply connect only once for its entire lifespan, instead of every time it processes a page that requires connecting to the SQL server. This means that for every child that opened a persistent connection will have its own open persistent connection to the server. For example, if you had 20 different child processes that ran a script that made a persistent connection to your SQL server, you'd have 20 different connections to the SQL server, one from each child.
http://php.net/manual/en/features.persistent-connections.php
Related
Question:
What are the rules/logic behind persistent connection management when using PDO?
Environment:
Web Server
Windows 7 x64
Dual-core with 16GB RAM
Apache 2.2.17
PHP 5.3.5
Connecting through DSN string with IP address, port, service name, etc...
No ODBC for DB conn (been trying to create one for 2 hours now, thanks Oracle!)
DB Server
Oracle 10g on Linux
Multi-core with 4GB RAM
Username specifically created for my web app (yes, it's fake)
user: webuser
My understanding/observations:
Non-persistent connections
<?php
// Open a new connection
// Session created in Oracle
$dbh = new PDO('DSN', 'webuser', 'password');
// webuser is active in v$session with a SID=1
$dbh = NULL;
// webuser removed from v$session
// Manually calling $dbh = NULL; will remove the session from v$session
// OR
// Wait for script EOL so a kill-session command is sent to Oracle?
?>
Script reliably takes about ~.09 seconds to execute with framework overhead, etc...
Persistent connections
<?php
// Open a new connection and make it persistent
// Session created in Oracle
// Is Apache maintaining some sort of keep-alive with Oracle here?
// because I thought php.exe is only alive for the duration of the script
$dbh = new PDO('DSN', 'webuser', 'password', array(PDO::ATTR_PERSISTENT => TRUE));
// webuser is active in v$session with a SID=1
$dbh = NULL;
// webuser is still active in v$session with a SID=1
$dbh = new PDO('DSN', 'webuser', 'password', array(PDO::ATTR_PERSISTENT => TRUE));
// webuser is still active in v$session with a SID=1
// Manually calling $dbh = NULL; does not kill session
// OR
// Script EOL does not kill session
// ^^ this is good, just as expected
?>
Script takes ~.12 seconds to execute upon initial visit with framework overhead, etc...
Sub-sequent executes take ~.04
The issue:
I visit the page and webuser gets a SID=1
My colleague visits the page and webuser gets an additional SID=2 <- rinse, repeat, and increment SID for new computers visiting this page
Shouldn't a new visitor be re-using SID=1?
All answers, suggestions, requests for alternate testing, links to reading material are welcomed.
I have RTFM'ed for a while and Googling has only produced meager Advantages of Persistent vs. Non-persistent blogs.
Apaches point of view
Apache has one parent process. This process creates child processes that will handle any requests coming to the web server.
The initial amount of child processes being started when the web server starts is configured by the StartServers directive in the apache configuration. The number goes up as needed with a raising amount of requests hitting the web server until ServerLimit is reached.
PHP and persistent connections
If PHP (ran as mod_php, as CGI all resources are freed at the end of script execution) is now being told to establish a persistent connection with a database for a request, this connection is hold even after the script finishes.
The connection being now hold is a connection between the apache child process which the request was handled by and the database server and can be re-used by any request that is being handled by this exact child process.
If, for some reason (do not ask me exactly why), the child process is being occupied longer than the actual request and another request comes in, the parent apache process redirects this request to a (new) child process which may has not established a connection to the database up to this time. If it has to during the execution of the script, it raises the SID as you have observed. Now there are two connections be hold by two different child processes of apache.
Keep in mind that...
It is important to know, that this can also cause a lot of trouble.
If there is an endless loop or an aborted transaction or some other may be even unpredictable error during the script execution, the connection is blocked and can not be re-used.
Also it could happen that all of the available connections of the database are used, but there is another child process of the apache server trying to access the database.
This process is blocked for the time being until a connection is freed by the database or apache (timeout or voluntarily by termination).
Any further information about this topic on this page: http://www.php.net/manual/en/features.persistent-connections.php
I hope I got all that we have discussed in our comment conversation summarized correctly and did not forget anything.
If so, please, leave me a hint and I will add it. :)
Edit:
I just finished reading the article #MonkeyZeus mentioned in this comment.
It describes the process I summarized above and provides useful information on how to optimize your apache server to work better together with persistent connections.
It can be used with or without oracle database backends, though.
You should give a look: http://www.oracle.com/technetwork/articles/coggeshall-persist-084844.html
Advantages
From the manual page of php for persistent connections on this link:
Persistent connections are links that do not close when the execution of your script ends. When a persistent connection is requested, PHP checks if there's already an identical persistent connection (that remained open from earlier) - and if it exists, it uses it. If it does not exist, it creates the link.
The reason behind using persistent connections is, of course, reducing the number of connections which are rather expensive; Even though they are much faster with MySQL than with most other databases.
Issues
There are some issues with table locking while using persistent connections.
if the script for whatever reason cannot release the lock, then subsequent scripts using the same connection will block indefinitely and may require that you either restart the httpd server or the database server.
Another is that when using transactions by mysql commit.
A transaction block will also carry over to the next script which uses that connection if script execution ends before the transaction block does. In either case, you can use register_shutdown_function() to register a simple cleanup function to unlock your tables or roll back your transactions.
I suggest you read this question about disatvantages of persistent connections.
PDO is kinda funny that way. Even the same user/visitor can cause a second or even third instance to be created. The same thing happened to me on my local machine, while testing the performance of my db queries.
That is nothing to worry about, because these instances will timeout sooner or later, the exact timeout depends on your server configuration.
Why that happens? If the current instance is busy, then a new instance will be created and the older one will timeout sooner or later. At least that seems logical to me.
Here are my observations as I recently experienced an issue similar to yours.
MySQL server kept opening new connections and eventually maxed out the number of concurrent connections to MySQL server even though there were a lot of Idle connections that can be used.
Setting PDO::ATTR_PERSISTENT => true does re-use available idle connections. It may not seem like this at first glance when you try to monitor MySQL process because in the time that the report is sent back to you, that idle connection may have become active by another process.
Overall, you should notice a drop in the number of idle connections as opposed to not using a persistent connection. As regards the table locking issue, I decided to use InnoDB Storage Engine for my tables as it uses row-level locking as opposed to table locking with MyISAM Storage Engine.
I have not had an issue yet with concurrency when using this combination of InnoDB Storage Engine and PDO persistent Connection.
Also, as a safeguard on badly executed queries locking tables, keep queries within a try-catch Block.
A framework or an application automatically connect the database and we have to just use the database object for DB related operation. In CMS or framework, a term "connection pooling" is very popular. You can opt CMS or framework of PHP.
What is connection pooling?
Can someone describe this with an example?
What is the advantage of connection pooling?
Without connection pooling:
Every time you want to talk to the database, you have to open a connection, use it, then close it again.
With connection pooling:
The connections are kept open all the time (in a pool). When you want to talk to the database, you take an already connection, that isn't already in a use, use it, then put it back.
This is more efficient then opening and closing them all the time.
Connection pooling generally refers to, well, having a pool of connections which is being reused. To contrast this with non-pooled connections: typically every program instance connects to the database by itself every time it is run. In a PHP program, you just have the line $db = new PDO(...), which connects to the database. If you have 100 simultaneous visitors, 100 separate instances of that script will be run simultaneously, and 100 separate connections will be established to the database simultaneously. This may be very inefficient and/or temporarily overwhelm the database server.
A connection pool works by establishing, say, 50 permanent connections to the database which stay open the whole time. A PHP script would then simply pick one of these open connections to talk to the database and drop it back into the pool when it's done. If suddenly more than 50 PHP scripts try to use connections from this pool at once, the first 50 will succeed, and the rest will have to wait in line until an unused connection becomes available. This is more efficient, because connections aren't opened and torn down all the time, and it doesn't overwhelm the database server when sudden spikes occur.
I've read a ton about persistent database connections between PHP and MySQL (mysql_connect vs. mysql_pconnect). Same with PDO and MySQLi. It's definitely just my lack of understanding on this one, but how can a database connection be persistent between webpages? In this code:
$conn = mysql_pconnect( $server , $user, $pass );
mysql_select_db( $dbname );
If two users load this page at the same time, with two different $dbname variables, will PHP only make one connection to the database or two? I am fairly certain that
$conn = mysql_connect( $server , $user, $pass );
would make two connections.
If pconnect reuses the connection opened by the first user, will the mysql_select_db call work for the second user?
Ideally, what I am looking for is a way to have fewer database connections but still be able to set the default database in each PHP script. I have clients who all use the same PHP scripts, but the data is stored in their own client database (hence, $dbname is always different, but the MySQL connection parameters are the same - same mysql ip address, user and password).
Hope that makes sense. We can use MySQL, MySQLi or PDO, just need to know how to accomplish this the best way without having the possibility for clients to accidently write data to someone else's database! Thanks in advance.
The persistence is done by the copy of the PHP that's embedded in the webserver. Ordinarily you'd be right- if PHP was running in CGI mode, it would be impossible to have a persistent connection, because there'd be nothing left to persist when the request is done and PHP shuts down.
However, since there's a copy of PHP embedded in the webserver, and the webserver itself keeps running between requests, it is possible to maintain a pool of persistent connections within that "permanent" PHP.
However, note that on Apache multi-worker type server models, the connection pools are maintained PER-CHILD. If you set your pool limit to 10, you'll have 10 connections per Apache child. 20 children = 200 connections.
Persistent connections will also lead to long-term problems with deadlocks and other hard-to-debug problems. Remember - there's no guarantee that a user's HTTP requests will be serviced by the SAME apache child/mysql connection. If a script dies part-way through a database transaction, that transaction will NOT be rolled back, because MySQL does not see the HTTP side of things - all it sees is that the mysql<->apache connection is still open and assumes all's well.
The next user to hit that particular apache/mysql child/connection combination will now magically end up in the middle of that transaction, with no clue that the transaction is open. Basically, it's the Web equivalent of an unflushed toilet - all the "garbage" from the previous user is still there.
With non-persistent connections, you're guaranteed to have a 'clean' environment each time you connect.
From my reading of documentation and comments, I see:
Docs on mysql_pconnect (deprecated method)
Second, the connection to the SQL server will not be closed when the execution of the script ends. Instead, the link will remain open for future use ( mysql_close() will not close links established by mysql_pconnect()).
and a comment on that page
Persistent connections work well for CGI PHP managed by fastCGI, contrary to the suggestion above that they only work for the module version. That's because fastCGI keeps PHP processes running between requests. Persistent connections in this mode are easily made immune to connection limits too, because you can set PHP_FCGI_CHILDREN << mysql's max_connections <<< Apache's MaxClients. This also saves resources.
Docs on mysqli_connect (new method)
Prepending host by p: opens a persistent connection. mysqli_change_user() is automatically called on connections opened from the connection pool.
Docs for mysqli_change_user:
Changes the user of the specified database connection and sets the current database.
So my understanding is as follows: pconnect keeps the connection open after a script ends but while a process (or maybe group of processes) is still alive (like in a server with FCGI set up). Only one script at a time uses a connection, and when a new script grabs that connection the user and database are updated.
Thus if you use FCGI and persistent connections you can reduce the number of db connections open, but scripts running simultaneously will not be sharing the same connection. There is no problem with the connection being confused as to which database is selected.
I am having a problem with a website, connecting to a MySQL database using two types of connection on different parts: some PDO, some mysql_connect().
The first part of the website is requesting MySQL using the very classic 'mysql_query()' PHP function. This part makes some heavy queries on geographical data. Some of these requests (already optimized) take a long time.
Another part of the site is more recent, and made using Doctrine via a PDO connection.
The problem is, when one of the big processes is being ran in one browser page (can take around 1minute to process and return the page), if a user opens another page the PDO connection is in sleep mode, and holds the whole page from loading. After 60s (wait_timeout of mysql) the connection is killed, and the PDO gets an exception "The MySQL Server has gone away".
What is strange is that other pages with only classical mysql_connect() and mysql_query() can be run without a problem in parallel, only PDO queries are holding back and eventually dying.
Any input would be really appreciated.
Closing this question, it was in fact related to the php session being held up on write, preventing the other process from running. session_write_close() resolved it.
There are various reasons that a connection gets closed.
Reference:
https://dev.mysql.com/doc/refman/5.0/en/gone-away.html
I too faced the similar problem on using PDO where the hosting administrator kills the connection if it sleeps more than a minute. Hence I came up with my own class which will wrap the PDO class. This will detect whether a connection is closed and will try to reconnect on query execution.
Answer Below
PDO: MySQL server has gone away
Apache/PHP/MySQL persistent connections have such a bad reputation because Apache handles each request as a child php process, each having 1 persistent connection. When visitors scale, MySQL reachs max connections limit from all the apache/php child processes, each with 1 persistent connection. There's also the issues of temporary tables, user variables, charsets, transactions and last-insert-id.
In my situation, we don't have to deal with the latter issues, because we are only READING from MySQL. There are no updates to the db: these are handled by another set of scripts on a different machine. The scripts we want to have persistent connections are the server-end of AJAX connections, returning JSON data.
Each pageview has 5 AJAX requests, so 6 different php child processes are started on the server for each page requested (5 ajax, 1 html). Ideally, I could have ONLY 1 connection from the php/ajax server to MySQL. This connection would be shared by all php child processes.
So, how can I do this?
Should I use another webserver software other than apache? ngynx?
cheers
UPDATE: In this situation, the right way to connect to the MySQL server (http://bit.ly/15rBDP):
using mysql native driver (mysqlnd)
and mysqli
each client reconnecting using the
mysql-change-user function
using
MYSQLI-NO-CHANGE-USER-ON-PCONNECT in
php config ('coz we don't need to
cleanup).
UPDATE 2:
To clarify my question, what i want to have is: ALL php client processes, connecting through only ONE persistent connection. This connection is defined, ran and stored some how (my question), but all new php client processes know it and can use it. The problem with apache/php is that each php client process has 1 connection. If I serve 20,000 pages per minute, there will be 20,000 persistent connections. I want the 20,000 php child processes connecting to 1 unique, central, persistent connection to mysql.
You do realize that having only one (persistent) connection for all your requests, effectively serializes all requests to your server. So request C has to wait for request B to finish, which has to wait for request A to finish etc.
So having one connection turns your multi-threaded/multi-process webserver into a single-threaded application.
Read the accepted answer on this post : Which is better: mysql_connect or mysql_pconnect
Simply, using mysql persistent connections may be good or bad, depending on the hardware resources that you have as well as the way you code your applications.
A php native mysql connector is included in php 5.3 which has improved support for persistent connections.
http://dev.mysql.com/downloads/connector/php-mysqlnd/
http://blog.ulf-wendel.de/?p=211