I am currently accessing a SOAP Web Service through a PHP script using SoapClient. My script calls multiple subscripts (~30 a second) that each send a request and then pushes the response to a MySQL Database. My process attempts to emulate an "asynchronous" request/response mechanism.
In my subscript I connect to mysql and close the connection once it is complete. I'm running about 30 subscripts per second. I'm running into an issue where I am maxing out my MySQL connections.
I don't want to increase the maximum number of connections as I feel this is bad practice. Is there a better way to approach this problem? I am thinking I can somehow share a single mysql connection between the subscript and script.
If all subscripts are run in sequence, in one thread, then you can connect to MySQL once and pass this connection to all of them.
If subscripts are run in parallel, then it depend whether your MySQL library is thread safe or not. If it is, then you can pass one connection to all of them. But if not, you have no choice than one connection per script. This information should be mentioned in it's documentation.
If you need to run only some of the scripts in parallel and some can wait a while, then you can prepare pool of few connections (10 or so) and run only 10 scripts at once. When one script ends, you launch next and reuse old connection.
You can try connection pooling. I am not sure whether this is possible in php and whether there are frameworks already available for that.
If its not available You can use a Singleton class which contains a list of connections. Let the connections be closed by this class if its idle for N seconds. This means that your 30 subscripts can reuse connections which are not used by other scripts.
Did you try mysqli_pconnect? How do you spawn your sub processes? Can't you open the database connection in the main process and pass it to the sub processes? Any code examples of what you are doing?
Related
I read some threads here about PDO::lastInsertId() and its safety. It returns last inserted ID from current connection (so it's safe for multiuser app while there is only one connection per user/script run).
I'm just wondering if there is a possibility to get invalid ID if there is only one DB connection per one long script (lots of SQL requests) in multicore server system? The question is more likely to be theoretical.
I think PHP script run is linear but maybe I'm wrong.
PDO itself is not thread safe. You must provide your own thread safety if you use PDO connections from a threaded application.
The best, and in my opinion the only maintainable, way to do this is to make your connections thread-private.
If you try to use one connection from more than one thread, your MySQL server will probably throw Packet Out of Order errors.
The Last Insert ID functionality ensures multiple connections to MySQL get their own ID values even if multiple connections do insert operations to the same table.
For a typical php web application, using a multicore server allows it to handle more web-browser requests. A multicore server doesn’t make the php programs multithreaded. Each php program, to handle each web request, allocates is own PDO connections. As you put it, each php script run is “linear”. The multiple cores allow multiple scripts to run at the same time, but independently.
Last Insert ID is designed to be safe for that scenario.
Under some circumstances a php program may leave the MySQL connection open when it's done so another php program may use it. This is is called a persistent connection or connection pooling. It helps performance when a web site has many users connecting to it. The generic term for a reusable connection is "serially reusable resource.*
Some php programs may use threads. In this case the program must avoid allowing more than one thread to use the same connection at the same time, or get the dreaded Packet Out of Order errors.
(Virtually all machines have multiple cores.)
I run PHP via FCGI - that is my web server spawns several PHP processes and they keep running for like 10,000 requests until they get recycled.
My question is - if I've a $mysqli->connect at the top of my PHP script, do I need to call $mysqli->close in when I'm about to end running the script?
Since PHP processes are open for a long time, I'd image each $mysqli->connect would leak 1 connection, because the process keeps running and no one closes the connection.
Am I right in my thinking or not? Should I call $mysqli->close?
When PHP exits it closes the database connections gracefully.
The only reason to use the close method is when you want to terminate a database connection that you´ll not use anymore, and you have lots of things to do: Like processing and streaming the data, but if this is quick, you can forget about the close statement.
Putting it in the end of a script means redundancy, no performance or memory gain.
In a bit more detail, specifically about FastCGI:
FastCGI keeps PHP processing running between requests. FastCGI is good at reducing CPU usage by leveraging your server's available RAM to keep PHP scripts in memory instead of having to start up a separate PHP process for each and every PHP request.
FastCGI will start a master process and as many forks of that master process as you have defined and yes those forked processes might life for a long time. This means in effect that the process doesn't have to start-up the complete PHP process each time it needs to execute a script. But it's not like you think that your scripts are now running all the time. There is still a start-up and shutdown phase each time a script has to be executed. At this point things like global variables (e.g. $_POST and $_GET) are populated etc. You can execute functions each time your process shuts down via register_shutdown_function().
If you aren't using persistent database connections and aren't closing database connections, nothing bad will happen. As Colin Schoen explained, PHP will eventually close them during shutdown.
Still, I highly encourage you to close your connections because a correctly crafted program knows when the lifetime of an object is over and cleans itself up. It might give you exactly the milli- or nanosecond that you need to deliver something in time.
Simply always create self-contained objects that are also cleaning up after they are finished with whatever they did.
I've never trusted FCGI to close my database connections for me. One habit I learned in a beginners book many years ago is to always explicitly close my database connections.
Is not typing sixteen keystrokes worth the possible memory and connection leak? As far as I'm concerned its cheap insurance.
If you have long running FastCGI processes, via e.g. php-fpm, you can gain performance by reusing your database connection inside each process and avoiding the cost of opening one.
Since you are most likely opening a connection at some point in your code, you should read up on how to have mysqli open a persistent connection and return it to you on subsequent requests managed by the same process.
http://php.net/manual/en/mysqli.quickstart.connections.php
http://php.net/manual/en/mysqli.persistconns.php
In this case you don't want to close the connection, else you are defeating the purpose of keeping it open. Also, be aware that each PHP process will use a separate connection so your database should allow for at least that number of connections to be opened simultaneously.
You're right in your way of thinking. It is still important to close connection to prevent memory/data leaks and corruption.
You can also lower the amount of scipts recycled each cycle, to stop for a connection close.
For example: each 2500 script runs, stop and close and reopen connection.
Also recommended: back up data frequently.
Hope I helped. Phantom
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
close mysql connection important?
How important is it close an mysql connection in a website and why?
Every database engine has a limit on a maximum number of simultaneous connections. So if you don't close the connection, mysql can run out of available connections (default max_connections is 100). In addition, each connection you hold consumes server's resources (memory, a thread to listen, might be open filehandles).
CAVEAT
This does NOT hold true if the only things opening connection are web apps from ONE server and they use pooled connections. In such a case, you don't risk opening more and more new connections (since every time your app needs a new one, it picks the ones available from the pool); and closing and re-opening the pool's connections just wastes resources.
I don't know about other languages, but php closes the connection automatically on the end of script execution.
In general case, you close connection so that sql server doesn't have to wait for more commands from the website (which it won't receive, because the script has finished execution), and doesn't hit connection quota (if it has any).
PHP will automatically close the connection when the script exits, so I wouldn't normally worry about it too much.
The database server will have a finite possible number of simultaneous connections, so on a very heavily loaded site it might help to free the connection as soon as possible.
If you don't properly close your connections you could get into trouble. You will probably receive Too many connections errors. See here
Like so often, it depends.
What could happen:
The connection could stay open and you can run out of the maximum open connections
Your changes aren't committed
Depending on program language, middle ware, ...
Because there is a limit to the number of sockets connections that can be opened.
I'm not THE programmer, but I think it's better to close it as soon as you don't need it. Say, for example, you open the connection, execute the query and for some reason, working with that data takes a long time. If you haven't closed the connection yet (and the program/script hasn't ended either, so it hasn't automatically closed the connection -if there's such a chance), there will be a busy resource which is doing nothing in MySQL. A connection is open but you're not using it, that means another client using the service probably will see the great "Error: Can't connect to the DB" message.
So, in my humble opinion, it's better to connect to the DB, gather the requiered data, close the connection and then process the data. That at least will leave one available connection to future clients (which is really important if you have high-concurrency apps).
Nevertheless, that kind of behaviour depends on the programming language but all I know can retrieve data to a variable and it's connection-independent, so I can just close it and keep working with the data while leaving the connection available to another client.
Apache/PHP/MySQL persistent connections have such a bad reputation because Apache handles each request as a child php process, each having 1 persistent connection. When visitors scale, MySQL reachs max connections limit from all the apache/php child processes, each with 1 persistent connection. There's also the issues of temporary tables, user variables, charsets, transactions and last-insert-id.
In my situation, we don't have to deal with the latter issues, because we are only READING from MySQL. There are no updates to the db: these are handled by another set of scripts on a different machine. The scripts we want to have persistent connections are the server-end of AJAX connections, returning JSON data.
Each pageview has 5 AJAX requests, so 6 different php child processes are started on the server for each page requested (5 ajax, 1 html). Ideally, I could have ONLY 1 connection from the php/ajax server to MySQL. This connection would be shared by all php child processes.
So, how can I do this?
Should I use another webserver software other than apache? ngynx?
cheers
UPDATE: In this situation, the right way to connect to the MySQL server (http://bit.ly/15rBDP):
using mysql native driver (mysqlnd)
and mysqli
each client reconnecting using the
mysql-change-user function
using
MYSQLI-NO-CHANGE-USER-ON-PCONNECT in
php config ('coz we don't need to
cleanup).
UPDATE 2:
To clarify my question, what i want to have is: ALL php client processes, connecting through only ONE persistent connection. This connection is defined, ran and stored some how (my question), but all new php client processes know it and can use it. The problem with apache/php is that each php client process has 1 connection. If I serve 20,000 pages per minute, there will be 20,000 persistent connections. I want the 20,000 php child processes connecting to 1 unique, central, persistent connection to mysql.
You do realize that having only one (persistent) connection for all your requests, effectively serializes all requests to your server. So request C has to wait for request B to finish, which has to wait for request A to finish etc.
So having one connection turns your multi-threaded/multi-process webserver into a single-threaded application.
Read the accepted answer on this post : Which is better: mysql_connect or mysql_pconnect
Simply, using mysql persistent connections may be good or bad, depending on the hardware resources that you have as well as the way you code your applications.
A php native mysql connector is included in php 5.3 which has improved support for persistent connections.
http://dev.mysql.com/downloads/connector/php-mysqlnd/
http://blog.ulf-wendel.de/?p=211
Is it crucial to close mysql connections efficiency wise, or does it automatically close after php file has run?
From the documentation:
Note: The link to the server will be closed as soon as the execution of the script ends, unless it's closed earlier by explicitly calling mysql_close().
If your script has a fair amount of processing to perform after fetching the result and has retrieved the full result set, you definitely should close the connection. If you don't, there's a chance the MySQL server will reach it's connection limit when the web server is under heavy usage. If you can't close the MySQL connection until near the end of the script, it's cleaner though unnecessary to do so explicitly.
I'm not certain how fastcgi affects things. One page claims that a build of PHP that supports fastcgi will create persistent connections, even for mysql_connect. This contradicts the documentation in that the connection is closed when the process, rather than the script, ends. Rather than testing it, I'm going to recommend using mysql_close(). Actually, I recommend using PDO, if it's available.
Is it crucial? Not so much
Is it considered to be a good practice to follow? Yes.
I don't see why you wouldn't want to close it.
When using something like cgi, it's completely unnecessary to close your mysql connections since they close automatically at the end of script execution. When using persistent technologies like mod_perl and others, which maintain your connections between requests, then it's important to keep track of connections, global variables, etc..
Basically, for persistent data, clean up after yourself. For trivial, non-persistent data, it'll all go away when the request finishes anyway. Either way, best practice is to always close your connections.
Gets closed as soon as the script completes execution. Unless you've opened a persistent connection.
Ideally you should release a resource (a connection here) as soon as you are done with it. Unless there is a good chance that you will be needing it again very soon in the execution.
Connection pooling or using persistent connections (if that's what you meant) is a good idea if you are behind a single database server.
However if there are more servers and you are load balancing, it might hurt the distribution of work. Typically some clients run heavy queries while others run lighter ones. So if the same connection is used over n over, some servers would hit heavy load while others would be under utilized.
Consider using smaller ttls and variable connection pool size.
Most CMSs close the MySQL connection at the end of the request, which is really meaningless, because PHP will do it anyway.
However, if you have a script where the connection is no longer needed say towards the middle of the script, and then other heavy activities take place, then it's a good idea to explicitly close the connection. This will free some resources.
Now, much has been said about the benefits of closing a connection, but nearly nothing has been said about the benefits of not closing it. Essentially, if you do not close the connection at the end of a script, then you really are saving some resources. Imagine a web application (or any application) receiving 100 pageviews/second. So, every second, you will need to invoke mysqli_close 100 times - which means that in every second, you have 100 unnecessary roundtrips to the database server to close the open connections. From a performance perspective, this is pure overhead, since PHP will check for open connections when the script is finished anyway and will close those connections, and it might be that, because everything happens so quickly, that PHP doesn't see that you have closed those connections and will try to close them again.
Note: the answer above assumes that you are not using persistent connections (persistent connections are not used in any of the major CMSs).