I'm been a long time reading how to solve my problem, but I can't find the solution.
I'm working with symfony, and I have a long time process to execute when an user calls an action. Can I process the data when the request has finished? The purpose is launch a polling process from client with jQuery and wait until the process finish to redirect to another action.
Now, I'm doing that with a ContainerAwareCommand, but it waits until the background process finish.
Please, could you help me?
Thanks in advance.
Yes, it's possible to do some background process in Symfony after Response was sent to the user.
You need to write a listener for kernel.terminate event.
And define your long-running process inside of callback.
Just be aware of few things:
This techniek does not work if Response is send in encoded gzip format. So, you should force apache/nginx not to use gzip for this particular response.
It's pretty complex to set any session data during this request, because session will be set only after your long-running process is finished. It means, that you need to find an alternative to flashbag messages.
This is a good case for a queue such as RabbitMQ or Redis. Put a message into a queue as each file is uploaded. One or more PHP daemons read out of the queue, process each file, and update status for the user (e.g. update a row in a database).
This lets you run your processing on servers separate from your web requests and scales easily. With a queue you can also break your processing up into multiple concurrent tasks, if what you need to do allows it.
Related
this time I come with a question that I hope you can guide me to solve.
I have created a PHP script that allows loading a CSV file with a large amount of data (to load it I use the AJAX request). This script extracts the data from the file, then checks that this data is not already stored in the database, makes use of another script to obtain information of each data that is extracted from the file and finally saves the data that has passed successfully. all that validation process in a BD table.
It is a process that can last a few seconds or many minutes, because there are files that I can upload that contain more than 100 thousand data, so I would not like to leave the browser open all the time the process lasts.
What I want to know is how I could leave this process running internally on the server when I close the browser. Something like putting it in queue and let it continue running when I close my browser.
Once I reopen the browser and open the page of the script that shows me how the process is currently going. The idea is that the data processing is not interrupted when I close my browser.
Any suggestions or examples you could give me to achieve this?
Based on your description, I think you'd better run a dedicated daemon (either a 3rd party one or one written by yourself) yourself which does the background stuff.
The rationale behind why I don't think it right to do that in your PHP code is:
If you fork it from your server code, you have to install something else and since it is a folk, that process you are gonna spawn will inherit some data not useful at all from the parent process
With a dedicated daemon, it's easier for you to track the status of each job and more importantly, not a bunch of processes will be spawned if you just fork a new process for each job in the server code.
I have a web application written in PHP using a Postgres database.
The next phase of development is for background batch processes to be built that will need to be executed once a day (or adhoc as requested) for each user of the app. The process will query, await response and process the response from third-party services to feed information into the user's account within the web application.
Are there good ways to do this?
How would batches be triggered every day at 3am for each user?
Given there could be a delay in the response, is this a good scenario to use something like node.js?
Is it best to have the output of the batch process directly update the web application's database
with the appropriate data?
Or, is there some other way to handle the output?
Update: The process doesn't have to run at 3am. The key is that a few batch processes may need to run for each user. The execution of batches could be spread throughout the day.. I want this to be a "background" process separate to the app.
You could write a PHP script that runs through any users that need to be processed, and set up a cron job to run your script at 3am. Running as a cron job means you don't need to worry so much about how slow the third party call is. Obviously you'd need to store any necessary data in the database.
Alternatively, if the process is triggered by the user doing something on the site, you could use exec() to trigger the PHP script to process just that user, right away, without the user having to wait. The risk with this is that you can't control how rapidly the process is triggered.
Third option is to just do the request live and make the user wait. But it sounds like this is not an option for you.
It really depends on what third party you're calling and why. How long does the third party take to respond, how reliable they are, what kind of rate limits they might enforce, etc...
I have a PHP app on Heroku, the app does lots of communication with external APIs, which in turn trigger jobs on the database, the results are then displayed in a Facebook app...
Currently I have 2 worker processes and a web process. The web process triggers the workers and monitors database flags to know when each worker job is complete...I know...this setup isn't great, ideally I'd like to get notified in my web process when each worker process is finished, but this doesn't seem to be possible...
Is there a better way to approach this in Heroku using PHP?
Maybe a PHP app on Heroku isn't the best solution, but I've written lots of PHP that I'd rather not re-write....
Thanks in advance...
I can think of two relatively straightforward things you can do without ditching PHP (though I have to mention that PHP doesn't have much to recommend it, and you would likely be better off with Python/Django, Python/Flask or Ruby/Rails):
One is that you can switch to Redis for managing your workers instead of using your database. The advantage to this is that Redis has a pub/sub system where you can subscribe to signals while you hold a connection open. This means that if a connection is open, for instance from a web process, you will be notified of a change immediately without having to poll.
Two is that you can switch to using ajax so that you don't block the loading of your page while you're waiting. Load your page immediately and then use javascript to hit a separate PHP page to periodically check for updates on the status of your job, and then use javascript to render the results on the page in place when the results are available.
Even better, use ajax long polling. Render your page immediately and then use javascript to send a request back. Then when your php page receives the second request, register a subscription with Redis and then also manually check for updates (if you're not using Redis, just check for updates). If there are no updates, then just wait until the subscription receives a message, or wait for 30 seconds, whichever. (To be honest, I've never done Redis subscriptions in PHP so I'm not sure how implement that -- if you can't do it easily then just poll every couple of seconds instead.) If the 30 second timer expires, return json that says there are no results and have the javascript retry immediately. If you do receive results within that time, return the results and have the javascript render them.
I'm not an expert on http request so this question might be trivial for some. I'm sending a request to a php script which takes a lot of time to process a file and return a response. Is there a way to send a response before this script finishes its task to let the user know about the process status? Since this task can take up to several minutes I'd like to notify the user when key parts of the process are finished.
Note: I cannot break this request into several others
I might not have the correct approach here if so do you have other ideas how this could be handled?
Technically yes, but it would require you to have fine grained control of the http-stack, which you may or may not have in a typical php setup. I would suggest you look into other solutions (E.g. Make request to start the task - then poll to get an update on the progress)
http://www.redips.net/javascript/ajax-progress-bar/
here's a great article that goes over creating ajax a progress bar to use with php.
let me know if it doesn't make sense!
I think the best way for long proccessing requests is cron jobs. You can send request which will create 'task' and catch the task by cron job. Cron job can change task status while working and you can check task status via interval requests. I can't imagine another way to inform users about request proccessing. As soon as you response your headers are sent and PHP stops.
EDIT: it should be noted that Cron jobs are only available on Linux servers. windows servers would require access to the task scheduler, which most web hosts will not allow.
Is there any sane way to make a HTTP request asynchronously in PHP without throwing out the response? I.e., something similar to AJAX - the PHP script initiates the request, does it's own thing and later, when the response is received, a callback function/method or another script handles the response.
One approach has crossed my mind - spawning a new php process with another script for each request - the second script does the request, waits for the response and then parses the data and does whatever it should, while the original script goes on spawning new processes. I have doubts, though, about performance in this case - there must be some performance penalty from having to create a new process every time.
Yes, depending on the traffic of your site, spawning a separate PHP process for running a script could be devastating. It would be more efficient to use shell_exec() to start a background process that saves the output to a filename you already know, but even this could be resource intensive.
You could also have a request queue stored in a database. A single, separate background process would pull the job, execute it, and save the output, possibly setting a flag in the DB that your web process could check.
If you're going to use the DB queue approach, use curl_multi* class of functions to send all queued requests at once. This will limit the execution time of each iteration in your background process to the longest request time.
V5 may not be threaded, but you can create applications that exploit in-process multitasking.
Check out the following article: "Develop multitasking applications with PHP V5" from
IBM DeveloperWorks. You can find it here http://www.ibm.com/developerworks/web/library/os-php-multitask/