I have a Flex program that gets a JSON array from a PHP script. The PHP script doesn't contain just a simple JSON array but it grabs data from Activecollab and do some work on the data before encoding the data.
The first test involve a small JSON array that took a short time to encode by PHP. However, when I try to scale up the test, the Flash movie will crash trying to load the JSON data from PHP. There's no code difference between the tests, just the amount of data and amount of time it takes PHP to encode. Am I looking at a memory problem or a time out problem?
PS: When I call the PHP script in Firefox, it doesn't time out and still return a JSON array. It just took awhile to return the array.
I'm assuming that hitting your php service in a browser does not time out. If it does, then you need to change your php settings to allow the script to execute longer.
Otherwise you may try a different strategy altogether like this: Have flex call your php service and tell it to start data processing, have your php service return a token id to flex to use for polling. Have another php service track the progress of the processing (receiving the token for tracking the job). This second service will return a progress report with each request until the processing is done. When the processing is done, it returns the data on the next request.
Have php generate an id used to track the 'job' so that flex can poll and retrieve the data when the job is done. This will at least eliminate the wait time that Flex is dealing with while server-side processing happens.
Of course you'll need to store your pre-preocessed output server side somewhere while waiting for the request with the matching token. You'll also need a cleanup mechanism that clears this map/cache occasionally with a timeout.
You can check your memory usage on the client by using a flash memory profiler. Is your total memory usage going up a lot when you try to load the data?
Related
I'm trying to figure out the best way to send a response to an incoming API POST to our system, without returning.
We receive an XML post from another service, and consume it, and then our response to them is in XML also. What we are currently doing is digesting the incoming post, do some stuff on our end, then do a php return with the XML.
I would like to change this so that we can respond to their call with the XML, but then do some processing after the fact, without making some type of exec/background call.
What's the best way to send a response in PHP without returning? If we do an "echo" will that close the connection and allow us to process more afterwards without the other server "waiting?
Calling PHP's echo will not close the connection, in fact you can call echo multiple times in your PHP script and the output will be added to the response. The connection will only close when
The end of the script is reached
exit() or the alias die() are called
A fatal/parse error or an uncaught Exception occurs or your server runs out of memory
The maximum script execution time which you can set in php.ini is exceeded
Usually, the calling client code will also have some kind of timeout, so if your 'digesting' code could take longer and you want to take care about this as well as point 4 in the list, you can store the request data for later processing, for example in a database or serialized in files. Having successfully stored the data, you then have basically 2 options to go:
Option 1: Spawn a background PHP process
To spawn a background PHP process that will survive the livecycle of the calling script, use exec and nohup. The basic usage could look like this:
exec('RESOURCE_ID=123 nohup /path/to/your/php/executable your_script.php > /dev/null');
Within the first segment of the command, RESOURCE_ID=123, you can pass a unique identifier of the previously stored request data, maybe a database entry id or the storage filename, to the background script. Use getenv('RESOURCE_ID') in your background script to retrieve the variable.
[EDIT] > /dev/null for output redirection is crucial for running the process in the background, otherwise the parent script will wait for the output of the background process. I also suggest to write the output as well as error output to an actual file like &> my_script.out, this will have the same effect. You could also get the process id of the background process by appending & echo $! to the command, exec() will then return it.
After starting the background script, you can send your 'OK' response and exit the parent script.
Option 2: Cronjob for processing, as suggested by Jim Panse
The more complex your system grows, you probably need more control over the execution of your 'digesting' code. Perhaps you want to balance server load peaks, or restart failed tasks, or throttle malicious usage of your API. If you need this kind of control, you are better off with this option.
Since i guess you want your system-to-system communication synchronously there are multiple things you can consider.
Even though time consuming requests you usually still want a fast response.
To satisfy this you can't process the request immediately.
So, just save the request and process it later (give the client a 202 response back). Systems like queues are very popular to save time consuming jobs for running them later. Another time controlled script (cronjob) could then do polling and process the stacked messages/data.
If you want to provide the results to the client too, return them a unique resource id on the initial rest call and implement another resource with exactly this parameter as the input. If your system finished processing, the result will appear there.
Spawning a process from within another php script isn't very handy since it's very difficult to debug and error-prone.
I personally would't go for this solution.
I need to recieve a big amount of data from external source. The problem is that external source sends data very slow. The workflow is like this:
The user initiates some process from app interface (common it is fetching data from local xml file). This is quite fast process.
After that we need to load information connected with fetched data from external source(basically it is external statistics for data from xml). And it is very slow. But user needs this additional inforamtion to continue work. For example he may perform filtering according to external data or something else.
So, we need to do it asynchronously. The main idea is to shows external data as it becomes available. The question is how could we organise this async process? Maybe some quess or something else? We`re using php+mysql as backend and jquery at front-end.
Thanks a lot!
Your two possible strategies are:
Do the streaming on the backend, using a PHP script that curls the large external resource into a database or memcache, and responds to period requests for new data by flushing that db row or cache into the response.
Do the streaming on the frontend, using a cross-browser JavaScript technique explained in this answer. In Gecko and WebKit, the XmlHttpRequest.onreadystatechange event fires every time new data is received, making it possible to stream data slowly into the JavaScript runtime. In IE, you need to use an iframe workaround, also explained at Ajax Patterns article linked in the above SO post.
One possible solution would be to make the cURL call using system() with the output being redirected in a file. Thus PHP would not hang until the call is finished. From the PHP manual for system():
If a program is started with this function, in order for it to continue running in the background, the output of the program must be redirected to a file or another output stream. Failing to do so will cause PHP to hang until the execution of the program ends.
This would split the data gathering from the user interface. You could then work with the gathered local data by several means, for example:
employ an iFrame in the GUI that would refresh itself in some intervals and fetch data from the local stored file (and possibly store it in the database or whatever),
use jQuery to make AJAX calls to get the data and manipulate it,
use some CGI script that would run in the background and handle the database writes too and display the data using one of the above from the DB directly,
dozens more I can't think of now...
Here's the situation. I have a php file. I can get the value in one of the variables array in this php file really fast. But further processing on the data inside the array takes too long resulting into timeout. I would like to use AJAX so that when the page is loaded, the processed result is displayed and updated as soon as individual data inside the variable array is processed.
Any example would be highly appreciated.
An easier way to do this might be to turn off buffering, have a look at the PHP flush method, the comments explain a neat method you could try
You can then do your processing at the bottom of your page (after flushing the output) after the closing </html> tag and update your results with javascript when the processing has completed
Steps to follow (I would call this a "standard way" of doing what you need):
request that generates page stores to DB data that should be processed
cron job runs in background and
periodically checks if mentioned DB
has any data to process; if yes, it
starts processing data; after
processing it stores results to DB
ajax script that's located in
page periodically (say, once in 15
seconds; depends on how long it
takes to process the data) connects
to server to check if data is
already processed; if it is, it
loads results, if no, it may load
some status information - like
number of jobs in queue or something
like "15% done"
You might also think about using some flash client and java server to communicate over plain TCP socket connection to get the data faster and with smaller overhead.
Ok, I didn't really now how to formulate this question, and especially not the title. But i'll give it a try and hope i'm being specific enough while trying to keep it relevant to others.
I you want to run a php script in the background (via ajax) every X seconds that returns data from a database, how do you do this the best way without using to much of the server resources?
My solution looks like this:
A user visits a webpage, ever x seconds that page runs a javascript. The javascript calls a PHP script/file that calls the database, retrieves the data and returns the data to the javascript. The javascript then prints the data to the page. My fear is that this way of solving it will put a lot of pressure on the server if there is a lot (10 000) simultaneous visitors on the page. Is there another way to do this?
That sounds like the best way, given the spec/requirement you set out.
Another way is to have an intermediary step. If you are going to have a huge amount of traffic (otherwise this does not introduce any benefit, but to the contrary may overcomplicat/slow the process), add another table that records the last time a dataset was pulled, and a hard file (say, XML) which if the 'last time' was deemed too long ago, is created from a new query, this XML then feeds the result returned to the user.
So:
1.Javascript calls PHP script (AJAX)
2.PHP pings DB table which contains last time data was fully output
3.If time is too great, 'main' query is rerun and XML file is regenerated from output
ELSE skip to 4
4.Fetch the XML file and output as appropriate for returned AJAX
You can do it the other way, contacting the client just when you need it and wasting less resources.
Comet it's the way to go for this option:
Comet is a programming technique that
enables web servers to send data to
the client without having any need for
the client to request it. This
technique will produce more responsive
applications than classic AJAX. In
classic AJAX applications, web browser
(client) cannot be notified in real
time that the server data model has
changed. The user must create a
request (for example by clicking on a
link) or a periodic AJAX request must
happen in order to get new data fro
the server.
I am working in a tool in PHP that processes a lot of data and takes a while to finish. I would like to keep the user updated with what is going on and the current task processed.
What is in your opinion the best way to do it? I've got some ideas but can't decide for the most effective one:
The old way: execute a small part of the script and display a page to the user with a Meta Redirect or a JavaScript timer to send a request to continue the script (like /script.php?step=2).
Sending AJAX requests constantly to read a server file that PHP keeps updating through fwrite().
Same as above but PHP updates a field in the database instead of saving a file.
Does any of those sound good? Any ideas?
Thanks!
Rather than writing to a static file you fetch with AJAX or to an extra database field, why not have another PHP script that simply returns a completion percentage for the specified task. Your page can then update the progress via a very lightweight AJAX request to said PHP script.
As for implementing this "progress" script, I could offer more advice if I had more insight as to what you mean by "processes a lot of data". If you are writing to a file, your "progress" script could simply check the file size and return the percentage complete. For more complex tasks, you might assign benchmarks to particular processes and return an estimated percentage complete based on which process has completed last or is currently running.
UPDATE
This is one suggested method to "check the progress" of an active script which is simply waiting for a response from a request. I have a data mining application that I use a similar method for.
In your script that makes the request you're waiting for (the script you want to check the progress of), you can store (either in a file or a database, I use a database as I have hundreds of processes running at any time which all need to track their progress, and I have another script that allows me to monitor progress of these processes) a progress variable for the process. When the process begins, set this to 1. You can easily select an arbitrary number of 'checkpoints' the script will pass and calculate the percentage given the current checkpoint. For a large request, however, you might be more interested in knowing the approximate percent the request has completed. One possible solution would be to know the size of the returned content and set your status variable according to the percentage received at any moment. I.e. if you receive the request data in a loop, each iteration you could update the status. Or if you are downloading to a flat file you could poll the size of the file. This could be done less accurately with time (rather than file size) if you know the approximate time the request should take to complete and simply compare against the script's current execution time. Obviously neither of these are perfect solutions, but I hope they'll give you some insight into your options.
I suggest using the AJAX method, but not using a file or a database. You could probably use session values or something like that, that way you don't have to create a connection or open a file to do anything.
In the past, I've just written messages out to the page and used flush() to flush the output buffer. Very simple, but it may not work correctly on every web server or with every web browser (as they may do their own internal buffering).
Personally, I like your second option the best. Should be reliable and fairly simple to implement.
I like option 2 - using AJAX to read a status file that PHP writes to periodically. This opens up a lot of different presentation options. If you write a JSON object to the file, you can easily parse it and display things like a progress bar, status messages, etc...
A 'dirty' but quick-and-easy approach is to just echo out the status as the script runs along. So long as you don't have output buffering on, the browser will render the HTML as it receives it from the server (I know WordPress uses this technique for it's auto-upgrade).
But yes, a 'better' approach would be AJAX, though I wouldn't say there's anything wrong with 'breaking it up' use redirects.
Why not incorporate 1 & 2, where AJAX sends a request to script.php?step=1, checks response, writes to the browser, then goes back for more at script.php?step=2 and so on?
if you can do away with IE then use server sent events. its the ideal solution.