I am creating a page that executes a shell script on a remote server to scan a website and outputs the results on the screen. The output can sometimes take awhile to get depending on the size of the site being scanned. Currently the script works and does what it's supposed to but the problem is when I scan larger sites it stalls and on the platform the website is being hosted on has a timeout of 30 seconds that I cannot alter.
I am wondering what the best way to keep the connection alive whether it just be sending dots to the screen or maybe something else just to keep the connection alive.
Here is my script
$ssh = new Net_SSH2('hostname');
if (!$ssh->login('username', 'password')) {
exit('Login Failed');
}
$ansi = new File_ANSI();
$ssh->enablePTY();
$ssh->setTimeout(60);
$ssh->exec("./test.sh | awk 'NR >= 16 {print}'\n");
$ansi->appendString($ssh->read());
echo $ansi->getHistory();
Any help or guidance is deeply appreciated.
You should rather let the page load and e.g. run an AJAX request that will wait for a reply/listen on a port than trying to keep the connection alive.
So on the user's side, it would run an ajax request (javascript) to the php url, then on success you display the result.
$.ajax({
url: "/thescript.php":,
type: "POST",
datatype: "POST"
success: function(){
//do display stuff
}
});
Would probably add a reasonable timeout.
The timeout you are referring to is most likely a script execution time limit [very common on shared hosting]
And there is not much you can do about that sadly.
However what you can do is [if you have control over the server where the script is called from] is
send the request to the remote server [including a webhook callback url]
Have the script do it's thing
Have the script run the webhook to do the processing of the results
offcourse this also has implications on how the processing/displaying should be handled but i do not have enough information to go into specifics in this answer.
Do not attempt to execute long running scripts in a web page.
If you need a response from another system and you have SSH access then seperate the invocation of the task and the collection of the results into 2 seperate steps (see link above and the discussion it links to for some hints on how to do the invocation). Put a timed redirect on the first page to the second.
Related
I have problem with two simultaneous AJAX requests running. I have a PHP script which is exporting data to XSLX. This operation take a lot of time, so I'm trying to show progress to the user. I'm using AJAX and database approach. Actually, I'm pretty sure it used to work but I can't figure out why, it's no longer working in any browser. Did something change in new browsers?
$(document).ready(function() {
$("#progressbar").progressbar();
$.ajax({
type: "POST",
url: "{$BASE_URL}/export/project/ajaxExport",
data: "type={$type}&progressUid={$progressUid}" // unique ID I'm using to track progress from database
}).done(function(data) {
$("#progressbar-box").hide();
clearInterval(progressInterval);
});
progressInterval = setInterval(function() {
$.ajax({
type: "POST",
url: "{$BASE_URL}/ajax/progressShow",
data: "statusId={$progressUid}" // the same uinque ID
}).done(function(data) {
data = jQuery.parseJSON(data);
$("#progressbar").progressbar({ value: parseInt(data.progress) });
if (data.title) { $("#progressbar-title").text(data.title); }
});
}, 500);
});
the progress is correctly updating in database
the JS timer is trying to get the progress, I can see it in console, but all these request are loading the whole duration of the first script, as soon as the script ends, these ajax progress calls are loaded
So, why is the second AJAX call waiting for the first one to finish?
Sounds like a session blocking issue
By default PHP writes its session data to a file. When you initiate a session with session_start() it opens the file for writing and locks it to prevent concurrent edits. That means that for each request going through a PHP script using a session has to wait for the first session to be done with the file.
The way to fix this is to change PHP sessions to not use files or to close your session write like so:
<?php
session_start(); // starting the session
$_SESSION['foo'] = 'bar'; // Write data to the session if you want to
session_write_close(); // close the session file and release the lock
echo $_SESSION['foo']; // You can still read from the session.
After a bit of hair-pulling, I found one other way that these non-parallel AJAX requests can happen, totally independent of PHP session-handling... So I'm posting it here just for anyone getting here through Google with the same problem.
XDebug can cause this, and I wouldn't be surprised if Zend Debugger could too.
In my case, I had:
XDebug installed on my local LAMP stack
xdebug.remote_autostart enabled
My IDE accepting inbound debugger-connections, even though no breakpoints were active
This caused all my AJAX tests to run sequentially, no matter what. In retrospect it makes a lot of sense (from the standpoint of debugging things) to force sequential processing, but I simply hadn't noticed that my IDE was still interacting behind-the-scenes.
After telling the IDE to stop listening entirely, parallel runs resumed and I was able to reproduce the race-condition I had been looking for.
Be aware, that session_write_close()(answer of chrislondon) may not resolve the problem if you have enabled output buffering (default in PHP 7+). You have to set output_buffering = Off in php.ini, otherwise session won't be closed correctly.
When working with APIs, you sometimes need to issue multiple AJAX requests to different endpoints. Instead of waiting for one request to complete before issuing the next, you can speed things up with jQuery by requesting the data in parallel, by using jQuery's $.when() function:
Run multiple AJAX requests in parallel
a.php generates a main HTML page that contains two simultaneous AJAX calls to b.php and c.php. In order for b.php and c.php to share session variables, the session variables must exist BEFORE the first AJAX call. Provided this is true, a.php and b.php can change the value of the session variables and see each other's values. Therefore, create the session variables with a.php while generating the HTML page. At least that's how it works with Rogers shared web hosting.
You could also set
async: true,
The apache server I am using to develop my system will not respond to request while the scripts that control the polling of messages is being run. This only happends on a domain level meaning that I can send an http request to any other apps hosted localy and get a response. When I do eventually get a response from this its about a minute later.
Here is the Js
window.fetch_messages = function ()
{
var last_message = $("div.message:last").attr('data-ai_id');
var last_message_status = $("p.message_status:last").text();
var project_id = getParameterByName('project-id');
$.ajax({
url:'/project_messages',
type:'POST',
data:{ project_id:project_id, latest_message:last_message, status:last_message_status },
timeout:50000,
async: true,
success:new_messages, // This upon completion also resends the request
error:function(data){ console.log(data); setTimeout(fetch_messages(),50000); }
});
}; // When On the page that uses this I call this function to start polling
Here is the server side code
do
{
// Check for status change
$status_change = $this->mentor_model->query_status($this->project_id, $this->last_message_id, $this->last_message_status, $_SESSION['user']);
// Check for new messages
$messages = $this->mentor_model->query_messages($this->project_id, $this->last_message_id);
// If there is a status update or new message.
if($messages || $status_change)
break;
usleep(1000000);
}
while(empty($messages) && empty($status_change));
echo json_encode(array("messages"=>$messages, "status"=>$status_change));
exit;
While this action is being run The server takes a long time to handle any request weather it be a GET, POST or another AJax request. Iv also tried changing both code sets to no avail as long as its long polling, the server will take a long time to handle.
Do I have this wrong or is there some apache setting I'm suppose to change. Using xamp on windows 8.1 also tried wamp with no change
Thanks to steven for this. Ansewer taken straight from the source of php manual page
for session_write_close();
You can have interesting fun debugging anything with sleep() in it if
you have a session still active. For example, a page that makes an
ajax request, where the ajax request polls a server-side event (and
may not return immediately).
If the ajax function doesn't do session_write_close(), then your outer
page will appear to hang, and opening other pages in new tabs will
also stall.
I have Wamp setup on my windows 8.1 machine which I am using for development. My problem is Apache wont serve me pages from the specific web app I am working on in a reasonable time while I am running the script that is doing the polling. Here is the script and its back end implementation
window.fetch_messages = function () // I call this when my page is loaded
{
var last_message = $("div.message:last").attr('data-ai_id');
var project_id = getParameterByName('project-id'); // Another one of my helpers
$.ajax({
url:'project_messages',
type:'POST',
data:{ project_id:project_id, latest_message:last_message },
timeout:50000,
success:new_messages,
error:function(data){ console.log(data); setTimeout(fetch_messages(),50000); }
});
};
And the backend
do
{
$messages = $this->mentor_model->query_messages($this->project_id,$this->viewer, $this->last_message_id);
if($messages)
break;
usleep(25000);
}
while(empty($messages));
echo json_encode($messages);
exit;
This all works but I cant work properly if apache is not responding to my other request to go to another page or something in a reasonable time. I have other web apps on the machine and they will work fine while polling but the web app itself wont respond to other requests in reasonable time and this only happends when I'm on the page that uses this script. As a note I also made sure mysql was not giving the issues here by visiting another wapp(coining) on the localhost that uses mysql and it responds fine.
What's apache's deal ? is there some setting or something I have to change. It should be able to handle this fine since its just me testing.
This is more than less a resource handle problem. The all round use of sessions were being blocked because the script in question was not allowing session data use while it was running (because it was using the data).
A simple session_write_close() placed in the loop just before calling usleep()/sleep() on the script solved my problem.
Placing it anywhere after you have done using the session data should solve yours.
I have php script which can take quite a lot of time (up to 3-5 minutes), so I would like to notify user how is it going.
I read this question and decided to use session for keeping information about work progress.
So, I have the following instructions in php:
public function longScript()
{
$generatingProgressSession = new Zend_Session_Namespace('generating_progress');
$generatingProgressSession->unsetAll();
....
$generatingProgressSession->total = $productsNumber;
...
$processedProducts = 0;
foreach($models as $model){
//Do some processing
$processedProducts++;
$generatingProgressSession->processed = $processedProducts;
}
}
And I have simple script for taking data from session (number of total and processed items) which return them in json format.
So, here is js code for calling long script:
$.ajax({
url: 'pathToLongScript',
data: {fileId: fileId, format: 'json'},
dataType: 'json',
success: function(data){
if(data.success){
if(typeof successCallback == "function")
successCallback(data);
}
}
});
//Start checking progress functionality
var checkingGenerationProgress = setInterval(function(){
$.ajax({
url: 'pathToCheckingStatusFunction',
data: {format: 'json'},
success: function(data){
console.log("Processed "+data.processed+" items of "+data.total);
if(data.processed == data.total){
clearInterval(checkingGenerationProgress);
}
}
});
}, 10000)
So, long scripted is called via ajax. Then after 10 seconds checking script is called one time, after 20 second - second time etc.
The problem is that none of requests to checking script is completed until main long script is complete. So, what does it mean? That long script consumes too many resources and server can not process any other request? Or I have some wrong ajax parameters?
See image:
-----------UPD
Here is a php function for checking status:
public function checkGenerationProgressAction()
{
$generatingProgressSession = new Zend_Session_Namespace('generating_progress');
$this->view->total = $generatingProgressSession->total;
$this->view->processed = $generatingProgressSession->processed;
}
I'm using ZF1 ActionContext helper here, so result of this function is json object {'total':'somevalue','processed':'another value'}
I'd
exec ('nohup php ...');
the file and send it to background. You can set points the long running script is inserting a single value in DB to show it's progress. Now you can go and check every ten or whatever seconds if a new value has been added and inform the user. Even might be possible to inform the user when he is on another page within your project, depending on your environment.
Yes, it's possible that the long scripts hogs the entire server and any other requests made in that time are waiting to get their turn. Also i would recommend you to not run the check script every 10 seconds no matter if the previous check has finished or not but instead let the check script trigger itself after it has been completed.
Taking for example your image with the requests pending, instead of having 3 checking request running at the same time you can chain them so that at any one time only one checking request is run.
You can do this by replacing your setInterval() function with a setTimeout() function and re-initialize the setTimeout() after the AJAX check request is completed
Most likely, the following calls are not completing due to session locking. When one thread has a session file open, no other PHP threads can open that same file, as it is read/write locked until the previous thread lets go of it.
Either that, or your Server OR Browser is limiting concurrent requests, and therefore waiting for this one to complete.
My solution would be to either fork or break the long-running script off somehow. Perhaps a call to exec to another script with the requisite parameters, or any way you think would work. Break the long-running script into a separate thread and return from the current one, notifying the user that the execution has begun.
The second part would be to log the progress of the script somewhere. A database, Memcache, or a file would work. Simply set a value in a pre-determined location that the follow-up calls can check on.
Not that "pre-determined" should not be the same for everyone. It should be a location that only the user's session and the worker know.
Can you paste the PHP of "pathToCheckingStatusFunction" here?
Also, I notice that the "pathToCheckingStatusFunction" ajax function doesn't have a dataType: "json". This could be causing a problem. Are you using the $_POST['format'] anywhere?
I also recommend chaining the checks into after the first check has completed. If you need help with that, I can post a solution.
Edit, add possible solution:
I'm not sure that using Zend_namespace is the right approach. I would recommend using session_start() and session_name(). Call the variables out of $_SESSION.
Example File 1:
session_name('test');
session_start();
$_SESSION['percent'] = 0;
...stuff...
$_SESSION['percent'] = 90;
Example File 2(get percent):
session_name('test');
session_start();
echo $_SESSION['percent'];
Recently, I am going to make a instant-notification system for my website. I heard COMET is an essential in such cases.
I have been searching about PHP & Comet for a while already, however, the guides & articles I have found seems like just ajax requests in a loop. For example, there is a basic javascript code which gets the value from PHP file every 2 seconds and outputs to HTML. As far as I know, it should be COMET pushing new values to HTML, hence, the loop should be on server side, not client. Half of the articles in my native language was using setInterval() and contact PHP file every X seconds.
So, I have some questions to ask you.
Is there any guides or examples, which doesn't use any external framework like XAJAX/NOLOH that is easy to understand?
What is the performance difference between using COMET in server side, or requesting value from ajax.php every X seconds?
The timed requests I mentioned above can be called as COMET? (ex. Long Polling using jQuery and PHP)
Do I need any extensions to run COMET serverside? (My webhost is using Apache, I personally use Nginx)
You have to use a client-side script (AJAX), because the server has to be polled. The server cannot simply send messages to someone's browser without an open connection.
I'm not too familiar with HTML5 websockets, but I believe this allows you can have a persistent connection with the server, however HTML5 browsers aren't used widely to use this as a solution on a 'public' website.
How long polling works is that an asynchronous request is sent from the browser with a long time-out time (e.g. 30 seconds), when the request arrives at the server, it goes and checks for new messages, but when there are now messages to be displayed, instead of directly outputting the result, it goes into an infinite loop, polling the database e.g. every second (using sleep to postpone the queries), until a message has been found. When a message has been found it terminates the loop and outputs the result. If there have been no messages after 30 seconds, the script times out and sends back an empty request.
So the request can be sent back between 0 and 30 seconds. As soon as the request arrives in the browser, it is handled and a new 30 second request is sent.
As for your questions;
You will need a client-side framework for doing the polling
You cannot use Comet only on server-side. Using longpolling over normal polling (e.g. polling every second) is significant because you make much less server requests
To my understanding; yes
You can use any server-side language, as long as it can keep the connection open while querying for messages.
Also take a look at http://nodejs.org/
I don't know what exactly COMMET is mean. But for this purpose you have many solutions.
One, as you mentioned is long-polling by ajax. is simple. and not requeire new browsers only (HtML5).
One more option is "server-sent -event". It's require browser with HTML5 but it keep connection alive without polling:
client:
if (window.EventSource) {
window.onload = function() {
window.scrollTo(0,1);
setTimeout(
function() {
var source = new EventSource("events.php");
source.onmessage = function (event) {
document.body.innerHTML += event.data + "<br>";
};
}, 1000);
};
} else {
document.write("Please visit this page in a browser that supports EventSource to see the test");
}
server:
if ($_SERVER['HTTP_ACCEPT'] === 'text/event-stream') {
header('Content-Type: text/event-stream');
echo "data: This is the first event\n\n";
flush();
$i = 5;
while (--$i) {
sleep(1);
$time = date('r');
echo "data: The server time is: {$time}\n\n";
flush();
}
} else {
echo 'This demo is for use with an EventSource compatible browser.';
}
goodluck.