I have a very expensive query which gets executed from php and it can take a while to execute. Is there a way, in php, to detect if a user disconnects prior to the query being done and cancel it?
A possible solution is to use pg_send_query(), that function sends a query to the database and returns immediatly without blocking. Then you can poll to see if the user disconnected before the query finished. See this:
ignore_user_abort(false);
$db = pg_connect(DATABASE_DSN);
pg_send_query($db, "SELECT pg_sleep(10000)"); // long running query
while(pg_connection_busy($db)) {
/* You need to output something to the browser so PHP can know when the
browser disconnected. The 0 character will be ignored.
*/
echo chr(0);
/* Need to do both flushes to make sure the chr is sent to the browser */
ob_flush();
flush();
usleep(100000); // 0.1s to avoid starving the CPU
if(connection_status() != CONNECTION_NORMAL || connection_aborted()) {
// Browser disconnected, cleanup an die
pg_cancel_query($db);
pg_query($db, "ROLLBACK");
pg_close($db);
die();
}
}
// At this point the query finished and you can continue fetching the rows
This approach works but has a big problem: you really need to send something to the browser to detect the browser disconnection. If you don't, connection_status() and connection_aborted() will not work. This seems to be an old PHP bug, see here: https://bugs.php.net/bug.php?id=30301
So this method doesn't work when, for example, you query Postgres in the middle of a PDF generation routine. In that case the needed chr(0) will break the generated binary file.
You would want to use connection_aborted to detect if the user has disconnected it returns 1 if the client has disconnected otherwise it returns 0. There is some documentation here, however its usage is self documenting and you should have no problem using it.
Related
The problem is that for a long process the PHP script keeps on executing whether or not the client browser is currently connected or not. Is there any possibility that if the client has terminated the Ajax call to a script then the script also terminates on server?
As pointed out by #metadings php does have a function to check for connection abort named connection_aborted(). It will return 1 if connection is terminated otherwise 0.
In a long server side process the user may need to know if the client
is disconnected from the server or he has closed the browser then the
server can safely shutdown the process.
Especially in a case where the application uses php sessions then if we left the long process running even after the client is disconnected then the server will get unresponsive for this session. And any other request from the same client will wait until the earlier process executes completely. The reason for this situation is that the session file is locked when the process is running. You can however intentioanlly call session_write_close() method to unlock it. But this is not feasible in all scenarios, may be one need to write something to session at the end of the process.
Now if we only call connection_aborted() in a loop then it will always
return 0 whether the connection is closed or not.
0 means that the connection is not aborted. It is misleading. However, after re-search and experiments if have discovered that the output buffer in php is the reason.
First of all in order to check for aborts the developer in a loop must send some output to the client by echoing some text. For example:
print " ";
As the process is still running, the output will not be sent to the client. Now to send output we then need to flush the output buffer.
flush ();
ob_flush ();
And then if we check for aborts then it will give correct results.
if (connection_aborted () != 0) {
die();
}
Following is the working example, this will work even if you are using PHP session:
session_start ();
ignore_user_abort ( TRUE );
file_put_contents ( "con-status.txt", "Process started..\n\n" );
for($i = 1; $i <= 15; $i ++) {
print " ";
file_put_contents ( "con-status.txt", "Running process unit $i \n", FILE_APPEND );
sleep ( 1 );
// Send output to client
flush ();
ob_flush ();
// Check for connection abort
if (connection_aborted () != 0) {
file_put_contents ( "con-status.txt", "\nUser terminated the process", FILE_APPEND );
die ();
}
}
file_put_contents ( "con-status.txt", "\nAll units completed.", FILE_APPEND );
EDIT 07-APR-2017
If someone is using Fast-Cgi on Windows then he can actually terminate
the CGI thread from memory when the connection is aborted using following code:
if (connection_aborted () != 0) {
apache_child_terminate();
exit;
}
Check out PHP connection_aborted() function. While doing your processing, you can sometimes check for the aborted connection to gracefully cancel the progress, as one would do in an interactive threading model.
One way I've found which is the easiest way to handle this time-out issue is as such:
1: set a value on the server as 'processing'. Start an independent thread to do the processing.
2: The initial ajax call returns a success
3: The javascript on the page goes into 'waiting mode' which sends a new ajax request every 10 or 30 or 60 seconds or five or ten minutes or whatever (depending on your situation) to find out whether the value on the server is still set to 'processing'.
4: The independent thread completes. It sets the value on the server to 'done'.
5: The javascript on the page makes its next waiting-mode query, and returns 'done' and the appropriate data.
4b: If an obscene amount of time goes by without a 'done', it registers as a failure. How much time is obscene depends upon your situation. Send an ajax call updating the value from 'processing' to 'cancel'.
5b: The independent thread periodically checks the status to make sure it's still set to 'processing'. If it sees a mode-shift to 'cancel' it cancels itself.
This is what you're looking for:
http://php.net/manual/en/function.ignore-user-abort.php
Stopping a script in the middle of execution can lead to unexpected results, so caveat emptor
I started using push in HTML5 using the JavaScript EventSource object. I was totally happy with a working solution in PHP:
$time = 0;
while(true) {
if(connection_status() != CONNECTION_NORMAL) {
mysql_close()
break;
}
$result = mysql_query("SELECT `id` FROM `table` WHERE UNIX_TIMESTAMP(`lastUpdate`) > '".$time."'");
while($row = mysql_fetch_array($result)) {
echo "data:".$row["id"].PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();
}
$time = time();
sleep(1);
}
But suddenly my WebApp wasn't reachable anymore with an MySQL error "too many connections".
It turned out that the MySQL connection doesn't close after closing the event source in JavaScript:
window.onload = function() {
sse = new EventSource("push.php");
sse.onmessage = function(event) {
console.log(event.data.split(":")[1]);
}
}
window.onbeforeunload = function() {
sse.close();
}
So I guess that the PHP script does not stop to execute. Is there any way to call in function (like die();) before the clients connection disconnects? Why doesn't my script terminate after calling .close(); on the EventSource?!
Thanks for help! —
First off: when you wrap your code in a huge while(true) loop, your script will never terminate. The connection with the DB would have been closed when your script "ran out of code to execute", but since you've written a deadlock, that's not going to happen... ever.It's not an EventSource issue, that merely does what it's supposed to do. It honestly, truly, and sadly is your fault.
The thing is: a user connects to your site, and the EventSource object is instantiated. A connection to the server is established and a request for the return value of push.php is requested. Your server does as requested, and runs the script, that -again- is nothing but a deadlock. There are no errors, so it can just keep on running, for as long as the php.ini allows it to run. The .close() method does cancel the stream of output (or rather it should), but your script is so busy either performing its infinite loop, or sleeping. Assuming the script to be stopped because a client disconnects is like assuming that any client could interfere with what the server does. Security-wise this would be the worst that could happen. Now, to answer your actual question: What causes the issue, how to fix it?The answer: HEADERS
Look at this little PHP example: the script isn't kept alive server-side (by infinite loops), but by setting the correct headers.
Just as a side-note: Please, ditch mysql_* ASAP, it's being deprecated (finally), use PDO or mysqli_* istead. It's not that hard, honestly.
I had exactly the same issue and as far as I understood it the reason was that apache didn't terminate the php script until I tried to write data trough the closed socket connection again.
I didn't have any changes in my database so there was no output.
To fix this I simply echo a event source comment in my while loop like:
echo ": heartbeat\n\n";
ob_flush();
flush();
This causes the script to be terminated if the socket connection is closed.
you might want to try to add this to the loop:
if(connection_status() != CONNECTION_NORMAL)
{
break;
}
but php should stop when the client disconnected.
1.
window.onbeforeunload = function() {
sse.close();
}
this is not required, EventSource will be closed at page unloading time.
even if you can not find a way to detect disconnected client on PHP, you can just stop execution after N seconds, EventSource will reconnect automatically.
http://www.php.net/manual/ru/function.connection-aborted.php
comments on this page says, that u should use "flush" before connection_status
anyway, you should drop connection after N seconds, because u cant detect "bad" disconnects.
3.
echo "retry:1000\n"; // use this to tell EventSource the reconnection delay in ms
$time = 0;
while(true) {
if(connection_status() != CONNECTION_NORMAL) {
mysql_close()
break;
}
$result = mysql_query("SELECT id FROM table WHERE UNIX_TIMESTAMP(lastUpdate) > '".$time."'");
while($row = mysql_fetch_array($result)) {
$rect[] = $row;
}
for($i-0;$i echo "data:".implode('',$disp)."\n\n"; $time = time(); sleep(1);
}
As Elias Van Ootegem pointed out, your script never terminates and hence you have a bunch of MySQL connections active. More concerningly, you have a bunch of resources being used in the form of PHP loops running with no end in sight!
Unfortunately, the solution to keeping the is not "headers". In the example at this link referenced by Elias, the php script terminates, and the client javascript actually has to re-open a connection. You can test this by using the following code
source.addEventListener('open', function(e) {
// Connection was opened.
console.log("Opening new connection");
}, false);
If you follow his example on github, in implementation the author actually employs a loop that terminates after X seconds. See the do loop and comments at bottom of the script.
The solution is to give your loop a lifespan.
By defining a limit to the loop, IE X itterations or after X amount of time to trigger the end of the loop or die();, you will make it so that if the client disconnects there is a limit to how long a script will remain active. If the client is still there after X amount of time, they will simply re connect.
I have a ob_start() and a corresponding ob_flush(). I would like to flush a portion of data and continue executing the rest. Using ob_flush() didn't help. Also if possible rest needs to happen without showing loading in browser.
EDIT:
I don't want to use ajax
I have done this in the past and this is how I solved it:
ob_start();
/*
* Generate your output here
*/
// Ignore connection-closing by the client/user
ignore_user_abort(true);
// Set your timelimit to a length long enough for your script to run,
// but not so long it will bog down your server in case multiple versions run
// or this script get's in an endless loop.
if (
!ini_get('safe_mode')
&& strpos(ini_get('disable_functions'), 'set_time_limit') === FALSE
){
set_time_limit(60);
}
// Get your output and send it to the client
$content = ob_get_contents(); // Get the content of the output buffer
ob_end_clean(); // Close current output buffer
$len = strlen($content); // Get the length
header('Connection: close'); // Tell the client to close connection
header("Content-Length: $len"); // Close connection after $len characters
echo $content; // Output content
flush(); // Force php-output-cache to flush to browser.
// See caveats below.
// Optional: kill all other output buffering
while (ob_get_level() > 0) {
ob_end_clean();
}
As I said in a couple of comments before, you should watch out for gzipping your content, since that will alter the length of your content, but not change the header about it. It also can buffer your output, so it won't get send to the client instantly.
You could try letting apache know to not gzip your content by using apache_setenv('no-gzip', '1');. But this will not work if you use rewrite-rules to go to your page, since then it will also modify those environment variables. At least, it did so for me.
See more caveats about flushing your content to the user in the manual.
ob_flush writes the buffer. In other words, ob_flush tells PHP to give Apache (or nginx/lighttpd/whatever) the output and then for PHP to forget about it. Once Apache has the output, it does whatever it wants with it. (In other words, after ob_flush it's out of your control whether or not it gets immediately written to the browser).
So, short answer: There's no guaranteed way to do that.
Just a guess, you're likely looking for AJAX. Whenever people are trying to manipulate when page content loads as you're doing, AJAX is almost always the correct path.
If you want to continue a task in the background, you can use ignore_user_abort, as detailed here, however, that is often not the optimal approach. You essentially lose control over that thread, and in my opinion, a web server thread is not where heavy processing belongs.
I would try to extract it out of the web facing stuff. This could mean a cron entry or just spawning a background process from inside of PHP (a process that though started from inside of script execution will not die with the script, and the script will not wait for it to finish before dying).
If you do go that route, it will mean that you can even make some kind of status system if necessary. Then you could monitor the execution and give the user periodic updates on the progress. (Technically you could make a status system with a ignore_user_abort-ed script too, but it doesn't seem as clean to me.)
this is my function
function bg_process($fn, $arr) {
$call = function($fn, $arr){
header('Connection: close');
header('Content-length: '.ob_get_length());
ob_flush();
flush();
call_user_func_array($fn, $arr);
};
register_shutdown_function($call, $fn, $arr);
}
wrap the function to be executed in the end, after php close the connection. and of course the browser will stop buffering.
function test() {
while (true) {
echo 'this text will never seen by user';
}
}
this is how to call the function
bg_process('test');
first argument is callable,
second argument is an array to be passed to 'test' function with an indexed array
Note : I don't use ob_start() at the beginning of the script.
I have an article explaining how this can be achieved using apache/mod_php on my blog here: http://codehackit.blogspot.com/2011/07/how-to-kill-http-connection-and.html Hope this helps, cheers
If you are using PHP-FPM:
ignore_user_abort(true);
fastcgi_finish_request();
Above two functions are the key factors which ignore_user_abort prevents error and fastcgi_finish_request closes client connection.
fastcgi_finish_request
This function flushes all response data to the client and finishes the request. This allows for time consuming tasks to be performed without leaving the connection to the client open.
not working on Apache.(PHP 5 >= 5.3.3, PHP 7)
Use:
header("Content-Length: $len");
..where $len is the length of the data to be flushed to the client.
I don't have the background to know when and where this is going to work, but I tried on a few browsers, and all returned instantly with:
<?PHP
header("Content-length:5");
echo "this is more than 5";
sleep(5);
?>
edit: Chrome, IE, and Opera showed this, while FireFox showed this is more than 5. All of them closed the request after that though.
I have a simple file upload service, written out in PHP, which also includes a script that controls download speeds by sending limited-sized packets when a user requests a download from this site.
I want to implement a system to limit parallel/simultaneous downloads to 1 per user if they are not premium members. In the download script above, I can use a MySQL database to store a record that has: (1) the user ID; (2) the file ID; (3) when the download was initiated; and (4) when the last packet was sent, which is updated each time this is done (if DL speed is limited to 150 kB/sec, then after every 150 kB, this record is updated, etc.).
However, thus far, the database record will only be deleted once the download has successfully completed — at the end of the script, after the download has been fully served, the record is deleted from the table:
insert DB record;
while (download is being served) {
serve packet of data;
update DB record with current date/time;
}
// Download is now complete
delete DB record;
How will I be able to detect when a download has been cancelled? Would I just have to have a Cron job (or something similar) detect if an existing download record is more than X minutes/hours old? Or is there something else I can do that I'm missing?
I hope I've explained this well enough. I don't think posting specific code is required; I'm interested more in the logistics of how/whether this can be done. If specific is needed, I will gladly provide it.
NOTE: I know how to detect if a file was successfully downloaded; I need to know how to detect if it was cancelled, aborted, or otherwise stopped (and not just paused). This will be useful in stopping parallel downloads, as well as preventing a situation where the user cancels Download #1 and tries to initiate Download #2, only to find that the site claims he is still downloading file #1.
EDIT: You can find my download script here: http://codetidy.com/1319/ — it already supports multi-part downloads and download resuming.
<?php
class DownloadObserver
{
protected $file;
public function __construct($file) {
$this->file = $file;
}
public function send() {
// -> note in DB you've started
readfile($this->file);
}
public function __destruct() {
// download is done, either completed or aborted
$aborted = connection_aborted();
// -> note in DB
}
}
$dl = new DownloadObserver("/tmp/whatever");
$dl->send();
should work just fine. No need for a shutdown_function or any funky self-built connection observation.
You will want to check out the following functions: connection_status(), connection_aborted() and ignore_user_abort() (see the connection handling section of the PHP manual for more info).
Although I can't guarantee the reliability (it's been a while since I've played around with it), with the right combination you should be able to accomplish what you want. There are a few caveats when working with these though, the big one being that if something goes wrong you could end up with stranded PHP scripts running on the server requiring you to kill Apache to stop them.
The following should give you a good idea of how to do it (adapted from the PHP code examples and a couple of the comments):
<?php
//Set PHP not to cancel execution if the connection is aborted
//and drop the time limit to allow for big file downloads
ignore_user_abort(true);
set_time_limit(0);
while(true){
//See the ignore_user_abort() docs re having to send data
echo chr(0);
//Make sure the data gets flushed properly or the connection check won't work
flush();
ob_flush();
//Check then connection status and exit loop if aborted
if(connection_status() != CONNECTION_NORMAL || connection_aborted()) break;
//Just to provide some spacing in this example
sleep(1);
}
file_put_contents("abort.txt", "aborted\n", FILE_APPEND);
//Never hurts to ensure that the script halts execution
die();
Obviously for how you would be using it the data being sent would simply be the download data chunk (just make sure you flush the buffer properly to ensure the data is actually sent). As far as I'm aware, there is no way of making a distinction between pausing and aborting/stopping. Pause/resume functionality (and multi-part downloading - i.e. how download managers accelerate downloads) relies on the "Range" header, basically requesting byte x to byte y of the file. So if you want to allow resumable downloads you'll have to deal with that too.
There is no HTTP "cancel" signal that is sent by default. So, it looks like you will need to decide on a timeout, the length of time a connection can sit without sending/receiving another packet. If you are sending rather small packets (as I presume you are) keep the timeout short for best effect.
In your while condition you will need to check the age of the last timestamp update, if its too old, stop sending the file.
I have a long-running script that seems to occasionally report the following NOTICE-level error:
pg_send_query(): Cannot set connection to blocking mode
It seems to continue to send queries afterward, but it's unclear if it successfully sends the query that generates the error.
What is this a symptom of?
Edit: There are no entries in the postgres log at the time the error occurred, suggesting this is solely a connection error, not something going wrong on postgres' side (e.g. probably not the result of postgres crashing and restarting or something)
Edit: As far as I can tell, my INSERT statements are succeeding, one way or another, when this error is triggered.
Edit: Looks like this may have been fixed in June 2013: https://bugs.php.net/bug.php?id=65015
It is a symptom of pg_send_query() not being able to successfully switch the connection back to blocking mode. Looking at the source code in PHPs pgsql.c, you can find:
/* {{{ proto bool pg_send_query(resource connection, string query)
Send asynchronous query */
PHP_FUNCTION(pg_send_query)
{
<... snipped function setup stuff ...>
if (PQ_SETNONBLOCKING(pgsql, 1)) {
php_error_docref(NULL TSRMLS_CC, E_NOTICE, "Cannot set connection to nonblocking mode");
RETURN_FALSE;
}
<... snipped main function execution stuff ...>
if (PQ_SETNONBLOCKING(pgsql, 0)) {
php_error_docref(NULL TSRMLS_CC, E_NOTICE, "Cannot set connection to blocking mode");
}
RETURN_TRUE;
}
So the error gets raised at the end of the function, after the main work is done. This fits with your observation that your INSERT statements get executed.
The whole purpose of the two PQ_SETNONBLOCKING calls is to put the connection in non blocking mode to allow asynchronous execution and putting it back to the default blocking behaviour afterwards. From the documentation of PQsetnonblocking: (PQ_SETNONBLOCKING is just an alias defined for that function):
Sets the nonblocking status of the
connection.
int PQsetnonblocking(PGconn *conn, int arg);
Sets the state of the connection to nonblocking if arg is 1,
or blocking if arg is 0. Returns 0 if
OK, -1 if error.
In the nonblocking state, calls to
PQsendQuery, PQputline, PQputnbytes,
and PQendcopy will not block but
instead return an error if they need
to be called again.
Note that PQexec does not honor
nonblocking mode; if it is called, it
will act in blocking fashion anyway.
Looking further at the source of PQsetnonblocking (in PostgeSQLs fe-exec.c), there are two possible reasons why the call could fail:
/* PQsetnonblocking:
* sets the PGconn's database connection non-blocking if the arg is TRUE
* or makes it non-blocking if the arg is FALSE, this will not protect
* you from PQexec(), you'll only be safe when using the non-blocking API.
* Needs to be called only on a connected database connection.
*/
int
PQsetnonblocking(PGconn *conn, int arg)
{
bool barg;
if (!conn || conn->status == CONNECTION_BAD)
return -1;
barg = (arg ? TRUE : FALSE);
/* early out if the socket is already in the state requested */
if (barg == conn->nonblocking)
return 0;
/*
* to guarantee constancy for flushing/query/result-polling behavior we
* need to flush the send queue at this point in order to guarantee proper
* behavior. this is ok because either they are making a transition _from_
* or _to_ blocking mode, either way we can block them.
*/
/* if we are going from blocking to non-blocking flush here */
if (pqFlush(conn))
return -1;
conn->nonblocking = barg;
return 0;
}
So either the connection got lost somehow, or pqFlush did not finish successfully, indicating leftover stuff in the connection output buffer.
The first case would be harmless, as your script would certainly notice the lost connection for later calls and react to that (or fail more noticeable).
This leaves the second case, which would mean you have a connection in the non default, non blocking state. I do not know if this could affect later calls that would reuse this connection. If you want to play it safe, you'd close the connection in this case and use a new/other one.
It sounds like you're trying to use the pg_send_query() function for sending asynchronous queries to PostgreSQL. The purpose of this function is to allow your PHP script to continue executing other code while waiting for PostgreSQL to execute your query and make a result ready.
The example given in the docs for pg_send_query() suggest that you shouldn't send a query if PostgreSQL is already chewing on another query:
if (!pg_connection_busy($dbconn)) {
pg_send_query($dbconn, "select * from authors; select count(*) from authors;");
}
Is there a reason you're using pg_send_query() instead of pg_query()? If you can allow your script to block waiting for query execution, I'm guessing (admittedly without having tried it) that you won't see these errors.
I've recently had the same problem, and with the help from Henrik Opels answer realized that PHP does not actually wait for the buffer to flush before setting the connection back to blocking mode.
The 'cannot set connection to blocking mode' is trivially repeatable with large enough queries to fill the send buffer (padding with spaces at the end is enough). With smaller queries I imagine it is dependent on load, and rather intermittent.
if you do actually need asynchronous mode then try the patch at https://bugs.php.net/bug.php?id=65015
This could occur if you are using threads and the connection is being reused.
If is this the case you could use the PGSQL_CONNECT_FORCE_NEW like this:
pg_connect("...", PGSQL_CONNECT_FORCE_NEW)
This will force a new database connection resource but be advised: you could run out of connections clients, so be carefully using this inside threads so don't forget to use pg_close().
I encountered same error message with PHP 5.6.9
It occurs when persistent connection made by pg_pconnect() is lost and
pgsql.auto_reset_persistent is set to Off.
Connection might get lost when:
PHP Session expires
Connection to DB timeouts
Webserver / DB server is restarted
You can check PHP.ini for pgsql.auto_reset_persistent and set it to On.
With pgsql.auto_reset_persistent enabled, each time pg_pconnect() is being called, connection link is checked, if it is still valid. This requires a little overhead, but fixies error message when conncetion is lost.
I got that error too. I solve my problem by restarting the web server (Apache).