I'm using Celery to schedule asynchronous tasks (processing time= 2-3 minutes) (the tasks are implemented in python, but they are being delegated from a PHP code using celery-php), and it was working fine, until I needed to process some PHP code automatically after a task finishes, so I can't just send a request to Celery to fetch the results.
However, I did a workaround to do that by sending an HTTP request by the ending of the processing from the celery task, which I know it's bad because it might create a bottle-neck on the PHP server (not to talk about the security issues that will require some maintaining).
My question is: is there any way to invoke PHP code from Celery using the same backend it is using and having PHP code as a passive participant? if there is not, is there a way to make this call scheduled in the same way the task itself is scheduled to avoid performance overhead?
This probably answer your question http://comments.gmane.org/gmane.comp.python.amqp.celery.user/3954 .
I would also suggest, if possible, to consider the possibility to rewrite the PHP code in Python.
Related
Would appreciate some help understanding typical best practices in carrying out a series of tasks using Gearman in conjunction with PHP (among other things).
Here is the basic scenario:
A user uploads a set of image files through a web-based interface. The php code responding to the POST request generates an entry in a database for each file, mostly with null entries in the columns, queues a job for each to do analysis using Gearman, generates a status page and exits.
The Gearman worker gets a job for a file and starts a relatively long-running analysis. The result of that analysis is a set of parameters that need to be inserted back into the database record for that file.
My question is, what is the generally accepted method of doing this? Should I use a callback that will ultimately kick off a different php script that is going to do the modification, or should the worker function itself do the database modification?
Everything is currently running on the same machine; I'm planning on using Gearman for background scheduling, rather than for scaling by farming out to different machines, but in any case any of the functions could connect to the database wherever it is.
Any thoughts appreciated; just looking for some insights on how this typically gets structured and what might be considered best practice.
Are you sure you want to use Gearman? I only ask because it was the defacto PHP job server about 15 years ago but hasn't been a reliable solution for quite some time. I am not sure if things have drastically improved in the last 12 months, but last time I evaluated Gearman, it wasn't production capable.
Now, on to the questions.
what is the generally accepted method of doing this? Should I use a callback that will ultimately kick off a different php script that is going to do the modification, or should the worker function itself do the database modification?
You are going to follow this general pattern with any job queue:
Collect a unit of work. In your case, it will be 1 of the images and any information about who that image belongs to, user id, etc.
Submit the work to the job queue with this information.
Job Queue's worker process picks up the work, and starts processing it. This is where I would create records in the database as you can opt to not create them on job failure.
The job queue is going to track which jobs have completed and usually the status of completion. If you are using gearman, this is the gearmand process. You also need something pickup work and process that work, I will refer to this as the job worker. The job worker is where the concurrency happens which is what i think you were referring to when you said "kick off a different php script." You can just kick off a PHP script at an interval (with supervisord or a cronjob) for a kind of poll & fork approach. It's not the most efficient approach, but it doesn't sound like it will really matter for your applications use case. You could also use pcntl_fork or pthreads in PHP to get more control over your concurrent processes and implement a worker pool pattern, but it is much more complicated than just firing off a script. If you are interested in trying to implement some concurrency in PHP, I have a proof-of-concept job worker for beanstalkd available on GitHub that implements a worker pool with both fork and pthreads. I have also include a couple of other resources on the subject of concurrency.
Job Worker (pthreads)
Job Worker (fork)
PHP Daemon Example
PHP IPC Example
I have a web-app, written in PHP, and I am using Symfony2 as the core framework.
I need to regularly run thousands of small network requests every 10 minutes or so and I am trying to find the best solution for asynchronously running these jobs, without conflicting or doubling up.
Currently I have a very basic and inelegant solution where a cron job executes a PHP command script. The command synchronously works through each entry in the database and sends a network request. When that request completes (or fails), it moves on to the next one. When it has iterated over all entries, it exists, to be executed again from the cron job.
For the rewrite, I have looked at php-resque and pcntl_fork as solutions for running jobs in parallel, thereby speeding up the execution significantly. I have also looked at running multiple non-blocking socket requests from PHP but, so far, have preferred the simplicity of isolated jobs.
PHP can't do threading in the traditional sense, so what you're trying to do isn't really possible in the way that you're thinking. You've looked at the best solution (probably) in pcntl_fork, but that still won't be truly asynchronous. Have you considered using cron to accomplish this instead?
http://php.net/pthreads
http://github.com/krakjoe/pthreads
Examples on github... and included in distribution, use code in github, it contains fixes not yet released.
I'm looking for a job queue manager in node.js which can be invoked by php. This is for a web application which needs to send emails, create pdf files and so on which I'd like to perform asynchronous of the php process.
Example of the process:
User requests a php page
Php invokes the job queue manager and adds a task
Task is executed in node.js asynchronously of php, preferably when it's a bit more quiet
Task is to execute a php script
Why this "complex" system?
We write all our web-applications in php (Zend Framework)
We'd like to start learning node.js
We need a asynchronous process (fast response!)
The "real" task should be a php script as well, to utilize already written php classes, to have easy access to database connections and be as much DRY as possible
Use cases of this system:
User registers himself, system will send welcome email
User completes ecommerce order, system will send invoice
In the end, we'd like to use node-cron as well, to perform non-system wide cron tasks (very application specific). Node-cron will invoke the job queue manager, which will subsequently run a php script.
Is there such an application already in node?
In such a case I would prefer a message queue like RabbitMQ and client side libraries like node-amqp and php-amqp. Then simply send your job from your PHP script in the queue and let nodejs pick up the job from the queue. A big advantage is that it is extensible and it is widely used and tested in the enterprise market.
One possible options is node-jobs, which uses Redis.
I am writing a Wordpress plug in php in and next step is some kind of add on to this plug in.
The add on would scrape data from web, sending forms etc. I have this part almost ready from the time before I had any thoughts about Wordpress plugin - it's coded in ruby using mechanize. I haven't found anything similar to mechanize in php anyway.
But I do not know what is the best way to call my ruby script from Wordpress. Some tasks will be managed by cron. What about the ones based on user request?
php script only triggers the ruby script. It won't wait/require anything from ruby's output
Wordpress plugin is fully portable and functional without ruby script. Ruby adds on something more. If somebody requires it.
everything will be running on my linux server where I have a root access
A WordPress plugin that depends on Ruby isn't going to be portable. That's OK if you're the only one who will be using it, though.
If the Ruby script needs to return a result that will be used immediately by the PHP script that's calling it, then something like exec() is the only way. Make sure you escape any arguments you pass to the Ruby script; otherwise you'll be vulnerable to injection attacks.
If the Ruby script doesn't need to return a result immediately (e.g. some background processing, such as thumbnail generation) then I think the best way would be for the PHP script to insert a row into a MySQL database or something similar. The Ruby script can work in the background or run from cron, check the database periodically for new jobs, and do whatever processing it needs to do. This approach avoids the performance overhead and security issues of exec(), and it's arguably also more scalable. (A similar approach would have the Ruby script listen on a socket, and your PHP scripts would connect to the socket. But this requires more work to get it right.)
If i were you i would handle all the ruby stuff from the cron. Make a queue in the DB to hand user requests then make the script (in ruby?) invoked by cron grab all the unprocessed jobs from the queue and start running them, then remove the job from the queue (or set some kind of flag for it being done). This way you dont have to call exec which in most cases is going to be off limits unless the user is running on VPS/Dedicated server where they have root access.
You could also make this a seperate job and have it poll the DB for unprocessed jobs more regularly than the primary job... if necessary.
Still, this begs the question... why use ruby in a php blog/cms app??????
Use exec() to run the ruby interpreter, giving it the path to your ruby script.
http://php.net/manual/en/function.exec.php
I am creating a web application using zend, here I create an interface from where user-A can send email to more than one user(s) & it works excellent but it slow the execution time because of which user-A wait too much for the "acknowledged response" ( which will show after the emails have sent. )
In Java there are "Threads" by which we can perform that task (send emails) & it does not slow the rest application.
Is there any technique in PHP/Zend just like in Java by which we can divide our tasks which could take much time eg: sending emails.
EDIT (thanks #Efazati, there seems to be new development in this direction)
http://php.net/manual/en/book.pthreads.php
Caution: (from here on the bottom):
pthreads was, and is, an experiment with pretty good results. Any of its limitations or features may change at any time; [...]
/EDIT
No threads in PHP!
The workaround is to store jobs in a queue (say rows in a table with the emails) and have a cronjob call your php script at a given interval (say 2 minutes) and poll for jobs. When jobs present fetch a few (depending on your php's install timeout) and send emails.
The main idea to defer execution:
main script adds jobs in the queue
cron script sends them in tiny slices
Gotchas:
make sure u don't send an email without deleting from queue (worst case would be if a user rescieves some spam at 2 mins interval ...)
make sure you don't delete a job without executing it first ...
handle bouncing email using a score algorithm
You could look into using multiple processes, such as with fork. The communication between them wouldn't be as simple as with threads (but then, it won't come with all of its pitfalls either), but if you're just sending emails, it might not be necessary to communicate much, if at all.
Watch out for doing forks on an Apache process. You may get some behaviors that you are not expecting. If you are looking to do any kind of asynchronous execution it should be via some kind of queuing mechanism. Gearman is one. Zend Server Job Queue is another. I have some demo code at Do you queue? Introduction to the Zend Server Job Queue. Cron can be used, but you'll have the problem of depending on your cron scheduler to run tasks whereas asynchronous computing often needs to be run immediately. Using a queuing system allows you to do that without threading.
There is a Threading extension being developed based on PThreads that looks promising at https://github.com/krakjoe/pthreads
There is pcntl, which allows you to create sub-processes, but php doesn't work very well for this kind of architecture. You're probably better off creating a long-running script (a daemon) and spawning multiple of them.
As of PHP there are no threads in it. However for php, you can have a look at this roundabout way
http://www.alternateinterior.com/2007/05/multi-threading-strategies-in-php.html
You may want to use a queue system for your email sending and send the email from another system which supports threads. PHP is just a tool and you should the tool that is best fitted for the job.
PHP doesn't include threading as part of the language, there are some methods that can emulate it but they aren't foolproof.
This Google search shows a few potential workarounds