Requirement:
I need to run a background process (per a user request) that takes about 30 to 60 seconds to complete. I'd like to give the user some status feedback. Note: Toly is right, 'Background' is not required.
What's working:
The process prints about 20 status messages during this time and I retrieve them with a proc_open and listening on a read pipe using fgets. I can save those messages into a session var and using timestamps (to help debug) I can see that the session array is getting written to with these messages as the process progresses.
The Trouble:
My plan was to poll the server with ajax calls (every sec) to retrieve these session vars for display in the DOM. The bottleneck seems to be that the server cannot service the ajax request while it's still running the background process. Everything dumps out at once when the background process completes. From what I can tell, the issue is not with output buffering because using (debugging) timestamps saved with each process message shows the server is writing to the session var sequentially, so that's how I know the proc_open and pipe reads are working as I expect. The issue appears to be the server not being able to give the AJAX request it's JSON object until it is done with the process; or, probably more accurately, done with the loop that is reading the pipe.
Obvious Misconception:
I thought sending a process to the background (using &) might give me a solution here. Apparently I do not know the difference between a background process and a forked process. What benefit is gained - if any - by running a process in the background when doing so appears to make no difference to me in this scenario?
Possible Solutions:
I do not expect the user initiated process that runs this
process/scenario to be that heavy, but if there's something I can
build into this solution that would help a heavy load then I would
like to do that now.
Is this a multi-threading (pthreads) or a
multi-process (fork) solution?
Or, should I save a process id,
let go polling it with a while( .. fgets ..) statement and then
come back to the process after the server has serviced the ajax
request?
I suppose I could run fake status messages and then
response accurately when the results come back after completion.
The time to process the request is not dependent upon the user, so
my fake timing could be pretty accurate. However, I would like to
know what the solution would be to provide real-time feedback.
After google-ing one day for a technique to get the same behavior you are describing here I come up with an easy solution for my project.
A bit of important theory:
- session_start () and a set like $_SESSION["x"] = "y" will always lock the session file.
Case scenario:
- A - process.php - running through an ajax call
- B - get_session.php - a second ajax call;
The main problem is/was, that even if you set a $_SESSION inside a process that is being run through an AJAX it will always have to wait the for the session file to get unlocked and it will result into a sync between the two processes (A. + B.) - both finishing at the same time!
So, the easiest way to fix this matter and get a good result is by using session_write_close() after each set. E.g.:
%_SESSION["A"] = "B";
$_SESSION["x"] = "y";
session_write_close();
PS: Best approach is to have a customed set of functions to handle the sessions.
Sorry for the mark-up. I just created an stack account.
Why would you think that you need a background process? Also, where did you get the idea that you needed one?
A normal php script, with sufficient time out set, with flush() function used every step of the way will give you the output you need for your AJAX.
What's even easier, since you use sessions - AJAX request to a separate handler, that will just check what's in session, and if there is smth new - will return you the new part.
$_SESSION['progress'] = array();
inside process.php
$_SESSION['progress'][] = 'Done 5%';
// complete some commands
$_SESSION['progress'][] = 'Done 10%';
inside ajax.php
if(count($_SESSION['progress']) > $_GET['laststep']) {
// echo the new messages
}
inside your normal page
$.ajax('ajax.php', 'GET', 'laststep=1', success: function(data){ show(data);})
Something like that should work.
Related
I've got a name lookup box that operates by your typical ajax requests. Here's the general flow of the Javascript that fires every time a letter is pressed:
If ajax request already open, then abort it.
If timeout already created, destroy it.
Set new timeout to run the following in half a second:
Send string to 'nameLookup.php' via ajax
Wait for response
Display results
The issue is that nameLookup.php is very resource heavy. In some cases up to 10,000 names are being pulled from an SQL database, decrypted, and compared against the string. Under normal circumstances requests can take anywhere from 5 to 60 seconds to return.
I completely understand that when you abort a request on the client side the server is still working on things and sends back the result. That it's just the client side that knows to ignore the response. But the server is getting so hung up on working on all of these requests.
So if you do:
Request 1
Abort Request 1
Request 2
Abort Request 2
Request 3
Wait for response from Request 3
The server is either not even working on Request 3 until it's finished with 1 and 2... or it's just so hung up on working on Request 1 and 2 that Request 3 is taking an extra long amount of time.
I need to know how to tell the server to stop working on Request 1 and 2 so I can free up resources for it to work on Request 3.
I'm using Javascript & jQuery on the client side. PHP/Apache and SQL on the server side.
Store a boolean value in the DB in a table, or in the session.
Have your resource intensive script check periodically that value to see if it should continue or not. If the DB says to stop, then your script cancels itself (by calling return; in the current function for example).
When you want to cancel, instead of calling abort();, make an AJAX request to set that value to false.
Next time the resource checks that value it will see that it has to stop.
Potential limitations:
1. Your script does not have a way of checking periodically the DB.
2. Based on how often the script checks the DB, it might take a few seconds to effectively kill the script.
I think there is something missing from this question. What are the triggers for doing the requests? You might be trying to solve the wrong problem.
Let me elaborate on that. If you lookup box is actually doing autocompletion of some kind and is doing a new search everytime the user presses a key, then you are going to have the issue you describe.
The solution in that case is not killing all the process. The solution lies in not starting them. So, you might make some decisions like not trying to search if there is only one character to search with - lets say we go for three. We then might say we want to wait until we can be reasonable sure the user has finished typing before sending off the request. Lets say we wait 1 second.
Now, someone looking for all the Paul's in you list of names will send off one search when they type 'pau' and pause for 1 second, instead of three searches for 'p' then 'pa' then 'pau'... so no need to kill anything.
I've come up with an awesome solution that I've tested and it's working beautifully. It's just a few lines of PHP code to put in whatever files are being resource intensive.
This solution utilizes the Process Identifier (PID) from the server. We can use two PHP function: posix_getpid() to get the current PID and posix_kill() to kill another PID. This also assumes that you already have called session_start() somewhere else.
Here's the code:
//if any existing PIDs to kill, go through each
if ($_SESSION['pid']) foreach ($_SESSION['pid'] as $i => $pid) {
//if posix_kill returns true, unset this PID from the session so we don't waste time killing it again
if(posix_kill($pid,0)) unset($_SESSION['pid'][$i]);
}
//now that all others are killed, we can store the current PID in the session
$_SESSION['pid'][]=posix_getpid();
//close the session now, otherwise the PID we just added won't actually be saved to the session until the process ends.
session_write_close();
Couple things to note:
posix_kill has two values. The first is the pid, and the second is supposed to be one of the signal constants from this list. Nothing there made any sense to me, other people seemed to have success just using 0, and when I use 0 it returns true. So whatever works!
calling session_write_close() before the resource intensive things start happening is crucial. Otherwise the new PID that has been saved to the session won't ACTUALLY be saved to the session until all of the page's processing is done. Which means the next process won't know to cancel the one that's still going on and taking forever.
I am making a notification system for my website. I want the logged in users to immediately noticed when a notification has made. As many people say, there're only a few ways of doing so.
One is writing some javascript code to ask the server "Are there any new notifications ?" at a given time interval. It's called "Polling" (I should be right).
Another is "Long Polling" or "Comet". As wikipedia says, long polling is similar to polling. Without asking everytime for new notifications, when new notifications are available, server sends them directly to the client.
So how can i use Long Polling with PHP ? (Don't need full source code, but a way of doing so)
What's its architecture/design really ?
The basic idea of long-polling is that you send a request which is then NOT responded or terminated by the server until some desired condition. I.e. server-side doesn't "finish" serving the request by sending the response. You can achieve this by keeping the execution in a loop on server-side.
Imagine that in each loop you do a database query or whatever is necessary for you to find out if the condition you need is now true. Only when it IS you break the loop and send the response to the client. When the client receives the response, it immediately re-sends the "long-polling" request so it wouldn't miss a next "notification".
A simplified example of the server-side PHP code for this could be:
// Set the loop to run 28 times, sleeping 2 seconds between each loop.
for($i = 1; $i < 29; $i++) {
// find out if the condition is satisfied.
// If YES, break the loop and send response
sleep(2);
}
// If nothing happened (the condition didn't satisfy) during the 28 loops,
// respond with a special response indicating no results. This helps avoiding
// problems of 'max_execution_time' reached. Still, the client should re-send the
// long-polling request even in this case.
You can use (or study) some existing implementations, like Ratchet. There are a few others.
Essentially, you need to avoid having apache or the web server handle the request. Just like you would with a node.js server, you can start PHP from the command line and use the server socket functions to create a server and use socket_select to handle communications.
It could technically work throught the web server by keeping a loop active. However, the memory overhead of keeping a php process active per HTTP connection is typically too high. Creating your own server allows you to share the memory between connections.
I used long polling for a chat application recently. After doing some research and playing it with a while here are some things I would recommend.
1) Don't long poll for more than about 20 seconds. Some browsers will timeout. I normally set my long poll to run about 20 seconds and send back an empty response at that point. Then you can use javascript to restart the long poll.
2) Every once in a while a browser will hang up. To help add a second level of error checking, I have a javascript timer run for 30 seconds and if no response has come in 30 seconds I abandon the ajax call and start it up again.
3) If you are using php make sure you use session_write_close()
4) If you are using ajax with Jquery you may need to use abort()
You can find your answer here. More detail here . And you should remember to use $.ajaxSetup({ cache:false }); when working with jquery.
I need to provide interaction between js on html-page and php-script.
- Use AJAX.
Ok. But the problem is that php-script executing for a long time, and i need to know the state of this script processing (e.g. 60% complete)
What should i do? Create 2 php-scripts (client&server) and do ajax-request to client.php which will do requests to server.php via sockets or smth?
Are there more elegant solutions?
What if you had the script doing the processing write its status to a file once in awhile. Make a second script that will read the file and return the status of the original one.
You should never have a long-running process being executed entirely within an HTTP session.
A simple and common approach to this problem is message queuing. Basically, you have your UI queue up the request into a database table and then have external daemon(s) process the queue.
To provide feedback, have the daemon periodically update the table with the status for the row it's currently working on. Then, your javascript code can make AJAX requests to a script that retrieves the status for that work item from the database and displays it to the user.
See: Dealing with long server-side operations using ajax?
Ajax call php script and return information that script is runing.
Main script create lock.file.
Script called from cron is checking if lock.file exists and run the correct script.
The correct script saves the current progress into progress.txt.
Ajax is reading progress.txt and when progress is 100% then return information that script processing is finished.
edited: Thanks to Justin for poiting the timeout problem ;)
If you want to be really fancy, write output from the php script to stdout, and capture it via a pipe. This would require running the php script using exec() or proc_open() (http://php.net/manual/en/function.proc-open.php) and pipe the output to a file (or if you want to be extra-extra fancy, use node.JS to listen for that data)
There are quite a few ways to accomplish this:
Node.JS
An Ajax query every x seconds
A META/Javascript page reload
An iframe that is routinely reloading with the status in it.
Good luck!
You could use PHP's output buffering (see ob_flush) to flush the contents at certain points in your script and tailor your JavaScript so that it uses the flushed contents. I believe readyState in your AJAX call won't be set to 4 on flushes so that's where you'll have to handle that yourself (see this article). I think its a much nicer way than writing to a file and checking the contents of that.
on your process.php:
// 1st task
$_SESSION['progress'] = 0;
// your code for the first task here ...
// 2nd task
$_SESSION['progress'] = 10;
// you code for 2nd task ...
// 3rd task
$_SESSION['progress'] = 17;
// continue ...
// everything finished?
$_SESSION['progress'] = 100;
on your progress.php:
// simply output
echo $_SESSION['progress'];
now from your client-side, just make a request to your progress.php, receive the number and give it to your progress bar ...
didn't check that by myself, but hope that it works! :)
I need to create an event listener. I'm a novice so be kind :)
Basically I am on page1.php (php file); I want inside a loop to go check page2.xml (xml file) for some information which should be received at some point. Either check it all the time, or wait and every 5 minutes or so to see if some information has been received there. Either of them work for me.
If no info has been received after a few minutes, then I want to run again the loop (until it is received), otherwise, move forward and do something with my newly received information. This part I have no problem with, just the event listener itself. I couldn't find the function I should be using anywhere. :( I only need to check and retrieve the content of the xml file every so often.
I am not so sure how I should go about this if there isn't just a function which does this, but I couldn't find much when I searched for "event listener php".
Any help would be appreciated: reference to tutorials/sample code/even just telling me what keywords I should be looking for or what I need to learn first in order to do this.
Thanks!
Well, first you should understand the terminology you're using. PHP is not an event-driven language, it is a request-driven language. A request comes into the web-server, PHP parses it and a response is sent back to the requester. At no point are there events triggered that you can process or handle. You can implement your own "event system" but ultimately this is much more work than what your use-case entails.
Your best bet is likely utilizing AJAX and continuously making requests to your PHP script until you return the data that you are looking for. Ultimately you will need to learn about the XMLHttpRequest JavaScript object. After you understand how to make asynchronous requests utilizing JavaScript you can look at the setInterval() method for how to repeatedly make a request.
Once you can repeatedly make asynchronous requests it should be a relatively simple process of creating a webpage where you can trigger the AJAX requests to be sent.
There is no need for a loop in your PHP code. The loop is effectively done on the other end. Here's a textual workflow that you might follow:
Go to a site designed to trigger your AJAX calls and trigger them.
Make your async request to your PHP script.
Inside your PHP script open up the XML file and check for the necessary content.
Return a response in the form of a JSON object. One response can mean the data wasn't updated, the other response means the data was updated.
Parse the response, if the data was not updated repeat from step (2). If the data was updated continue to step (6).
Display a celebratory greeting that your data was updated or a notice that we are still waiting for the data to be updated. Perhaps you can have the number of tries as well, off to the side.
I did the following:
Automatically saved to database every time something new came in.
Then ran a php loop that every few minutes checked to see if there is something new in the database which fits the parameters of this new event (including that it happenned within the timeframe of the past few minutes). I used flush(); and then sleep(120); in the loop to get the loop to keep running every few minutes, until the new info came in in which case it will break(); or die();.
I did something like this writing an inbox parser in PHP. You're best option is to:
Code page1.php in which you just need to do 2 things: read XML from page2.xml and
if there is something "new" just execute the data-parsing
code.
Setup a Cron job (if you're under linux) to execute every 5 minutes or so (Cron command is something like: php /path/to/page1.php). In the same way, if you're running Windows you can setup a scheduled task and execute the same command. Be aware that the full path to your PHP installation should be in PATH environment variable.
I am working in a tool in PHP that processes a lot of data and takes a while to finish. I would like to keep the user updated with what is going on and the current task processed.
What is in your opinion the best way to do it? I've got some ideas but can't decide for the most effective one:
The old way: execute a small part of the script and display a page to the user with a Meta Redirect or a JavaScript timer to send a request to continue the script (like /script.php?step=2).
Sending AJAX requests constantly to read a server file that PHP keeps updating through fwrite().
Same as above but PHP updates a field in the database instead of saving a file.
Does any of those sound good? Any ideas?
Thanks!
Rather than writing to a static file you fetch with AJAX or to an extra database field, why not have another PHP script that simply returns a completion percentage for the specified task. Your page can then update the progress via a very lightweight AJAX request to said PHP script.
As for implementing this "progress" script, I could offer more advice if I had more insight as to what you mean by "processes a lot of data". If you are writing to a file, your "progress" script could simply check the file size and return the percentage complete. For more complex tasks, you might assign benchmarks to particular processes and return an estimated percentage complete based on which process has completed last or is currently running.
UPDATE
This is one suggested method to "check the progress" of an active script which is simply waiting for a response from a request. I have a data mining application that I use a similar method for.
In your script that makes the request you're waiting for (the script you want to check the progress of), you can store (either in a file or a database, I use a database as I have hundreds of processes running at any time which all need to track their progress, and I have another script that allows me to monitor progress of these processes) a progress variable for the process. When the process begins, set this to 1. You can easily select an arbitrary number of 'checkpoints' the script will pass and calculate the percentage given the current checkpoint. For a large request, however, you might be more interested in knowing the approximate percent the request has completed. One possible solution would be to know the size of the returned content and set your status variable according to the percentage received at any moment. I.e. if you receive the request data in a loop, each iteration you could update the status. Or if you are downloading to a flat file you could poll the size of the file. This could be done less accurately with time (rather than file size) if you know the approximate time the request should take to complete and simply compare against the script's current execution time. Obviously neither of these are perfect solutions, but I hope they'll give you some insight into your options.
I suggest using the AJAX method, but not using a file or a database. You could probably use session values or something like that, that way you don't have to create a connection or open a file to do anything.
In the past, I've just written messages out to the page and used flush() to flush the output buffer. Very simple, but it may not work correctly on every web server or with every web browser (as they may do their own internal buffering).
Personally, I like your second option the best. Should be reliable and fairly simple to implement.
I like option 2 - using AJAX to read a status file that PHP writes to periodically. This opens up a lot of different presentation options. If you write a JSON object to the file, you can easily parse it and display things like a progress bar, status messages, etc...
A 'dirty' but quick-and-easy approach is to just echo out the status as the script runs along. So long as you don't have output buffering on, the browser will render the HTML as it receives it from the server (I know WordPress uses this technique for it's auto-upgrade).
But yes, a 'better' approach would be AJAX, though I wouldn't say there's anything wrong with 'breaking it up' use redirects.
Why not incorporate 1 & 2, where AJAX sends a request to script.php?step=1, checks response, writes to the browser, then goes back for more at script.php?step=2 and so on?
if you can do away with IE then use server sent events. its the ideal solution.