Here is my problem : I have an application that allows users to synchronize their data with social networks. The problem is that each synchronization can take up to 10 seconds. So I would like to trigger the synchronization in "background" task, and if possible parallelize all synchronisation.
Currently, my script looks like :
$user = user::login($username, $password);
/* Do some treatments, display the home page */
$user->synchronize();
/* END OF LOGIN SCRIPT */
and in my user.class.php, I have something like
public function synchronize(){
$Networks = socialNetworks::getByUserId($this->userid);
foreach($Networks as $n) $n->synchronize();
}
and finally in the socialNetworks.class.php, I have implemented all the synchronization scripts for each social network (fb, linkedin, google+, ...).
Note that I don't need to wait for the result of the synchronization when logging in.
So I have 2 problems here :
when calling $user->synchronize(); the login script is blocked until the end of the loop
the loop itself is very long, and I would like to launch parallel "threads", to make it faster when calling the synchronize() method (I have some Ajax triggering this action).
What would you suggest to optimize this ?
Thanks.
One option is to use Gearman, and offload this work to it's workers.
You basically create a PHP script that does some work. Then you tell it to do that work from your login script, and the work will be done in the background outside of the request.
Her is an blog post about asynchronous processing with php:
http://robert.accettura.com/blog/2006/09/14/asynchronous-processing-with-php/
<?php
include 'AsynchronousProcessing.php'
launchBackgroundProcess('php /path/to/task1.php');
launchBackgroundProcess('php /path/to/task2.php');
$class->SomeWork();
$class->SomeOtherWork();
?>
There are 2 ways.
Use exec() to make multiple syscalls like exec("php /path/to/script?networkId=1 > /dev/null 2>/dev/null &")
use pcntl_fork() function which works only in Linux.
Related
I have a script that I run for multiple clients.
Same script, I'm just using a different GET variable to load the client credentials.
eg.
example.com/script.php?client=lego
example.com/script.php?client=nike
example.com/script.php?client=stackoverflow
I've setup multiple crons to hit the script at midnight, with each cron having a different client GET variable.
What would be the best way to run a single CRON but process all clients? So I don't need to setup a CRON each time for each client.
There can be various solutions but without knowing the code what comes to my mind is.
Delete all crons and setup just one.
example.com/script.php
Inside script.php wrap whatever you earlier had in a function, create an array of clients and call that function for every client by passing username. For example
<?php
// if you have lots of clients and script can exhaust time limit
ini_set('max_execution_time', 0);
$clients = ['lego', 'nike', 'stackoverflow'];
foreach ($clients as $client) {
myScript($client);
}
function myScript($client)
{
// Whatever you had in script.php earlier replacing $_GET['client'] with $client.
}
Hope it answers your question.
I'm building an integration that communicates data to several different systems via API (REST). I need to process data as quickly as possible. This is a basic layout:
Parse and process data (probably into an array as below)
$data = array( Title => "Title", Subtitle => "Test", .....
Submit data into service (1) $result1 = $class1->functionservice1($data);
Submit data into service (2) $result2 = $class2->functionservice2($data);
Submit data into service (3) $result3 = $class3->functionservice3($data);
Report completion echo "done";
Run in a script as above I'll need to wait for each function to finish before it starts the next one (taking 3 times longer).
Is there an easy way to run each service function asynchronously but wait for all to complete before (5) reporting completion. I need to be able to extract data from each $result and return that as one post to a 4th service.
Sorry if this is an easy question - I'm a PHP novice
Many thanks, Ben
Yes, there are multiple ways.
The most efficient is to use an event loop that leverages non-blocking I/O to achieve concurrency and cooperative multitasking.
One such event loop implementation is Amp. There's an HTTP client that works with Amp, it's called Artax. An example is included in its README. You should have a look at how promises and coroutines work. There's Amp\wait to mix synchronous code with async code.
<?php
Amp\run(function() {
$client = new Amp\Artax\Client;
// Dispatch two requests at the same time
$promises = $client->requestMulti([
'http://www.google.com',
'http://www.bing.com',
]);
try {
// Yield control until all requests finish
list($google, $bing) = (yield Amp\all($promises));
var_dump($google->getStatus(), $bing->getStatus());
} catch (Exception $e) {
echo $e;
}
});
Other ways include using threads and or processes to achieve concurrency. Using multiple processes is the easiest way if you want to use your current code. However, spawning processes isn't cheap and using threads in PHP isn't really a good thing to do.
You can also put your code in another php file and call it using this :
exec("nohup /usr/bin/php -f your script > /dev/null 2>&1 &");
If you want to use asynchronicity like you can do in other languages ie. using threads, you will need to install the pthreads extension from PECL, because PHP does not support threading out of the box.
You can find an explaination on how to use threads with this question :
How can one use multi threading in PHP applications
Hi i'm trying to execute a LONG RUNNING request (action) in background.
function actionRequest($id){
//execute very long process here in background but continue redirect
Yii::app()->user->setFlash('success', "Currently processing your request you may check it from time to time.");
$this->redirect(array('index', 'id'=>$id));
}
What i'm trying to achieve is to NOT have the user waiting for the request to be processed since it generally takes 5-10min, and the request usually goes to a timeout, and even if I set the timeout longer, waiting for 5-10 min. isn't a good user experience.
So I want to return to the page immediately notifying the user that his/her request is being processed, while he can still browse, and do other stuff in the application, he/she can then go back to the page and see that his/her request was processed.
I've looked into Yii extensions backjob, It works, the redirect is executed immediately (somehow a background request), but when doing other things, like navigating in the site, it doesn't load, and it seems that the request is still there, and i cannot continue using the application until the request is finished.
A similar extension runactions promises the same thing, but I could not even get it to work, it says it 'touches a url', like a fire and forget job but doesn't work.
I've also tried to look into message queuing services like Gearman, RabbitMQ, but is really highly technical, I couldn't even install Gearman in my windows machine so "farming" services won't work for me. Some answers to background processing includes CRON and AJAX but that doesn't sound too good, plus a lot of issues.
Is there any other workaround to having asynchronous background processing? I've really sought hard for this, and i'm really not looking for advanced/sophisticated solutions like "farming out work to several machines" and the likes. Thank You very much!
If you want to be able to run asynchronous jobs via Yii, you may not have a choice but to dabble with some AJAX in order to retrieve the status of the job asynchronously. Here are high-level guidelines that worked for me. Hopefully this will assist you in some way!
Setting up a console action
To run background jobs, you will need to use Yii's console component. Under /protected/commands, create a copy of your web controller that has your actionRequest() (e.g. /protected/commands/BulkCommand.php).
This should allow you to go in your /protected folder and run yiic bulk request.
Keep in mind that if you have not created a console application before, you will need to set up its configuration similar to how you've done it for the web application. A straight copy of /protected/config/main.php into /protected/config/console.php should do 90% of the job.
Customizing an extension for running asynchronous console jobs
What has worked for me is using a combination of two extensions: CConsole and TConsoleRunner. TConsoleRunner uses popen to run shell scripts, which worked for me on Windows and Ubuntu. I simply merged its run() code into CConsole as follows:
public function popen($shell, $redirectOutput = '')
{
$shell = $this->resolveCommandLine($shell, false, $redirectOutput);
$ret = self::RETURN_CODE_SUCCESS;
if (!$this->displayCommands) {
ob_start();
}
if ($this->isWindows()) {
pclose(popen('start /b '.$shell, 'r'));
}
else {
pclose(popen($shell.' > /dev/null &', 'r'));
}
if (!$this->displayCommands) {
ob_end_clean();
}
return $ret;
}
protected function isWindows()
{
if(PHP_OS == 'WINNT' || PHP_OS == 'WIN32')
return true;
else
return false;
}
Afterwards, I changed CConsole's runCommand() to the following:
public function runCommand($command, $args, $async = false, &$outputLines = null, $executor = 'popen')
{
...
switch ($executor) {
...
case 'popen':
return $this->popen($shell);
...
}
}
Running the asynchronous job
With the above set up, you can now use the following snippet of code to call yiic bulk request we created earlier.
$console = new CConsole();
$console->runCommand('bulk request', array(
'--arg1="argument"',
'--arg2="argument"',
'--arg3="argument"',
));
You would insert this in your original actionRequest().
Checking up on the status
Unfortunately, I'm not sure what kind of work your bulk request is doing. For myself, I was gathering a whole bunch of files and putting them in a folder. I knew going in how many files I expected, so I could easily create a controller action that verifies how many files have been created so far and give a % of the status as a simple division.
I'm trying a create a script (in preference in PHP but Python should be fine too) which have the following behaviour :
We register a call back function which should start as soon as we receive a signal with an argument. Then, we create an infinite loop (this script should never stop!) to poll a webservice with a session (we got a logout very 15 minutes and we didn't want to be disconnected!).
Here is a the behaviour in pseudo-code :
function CALLBACK($arguments)
{
CURL(URL, {ARGUMENTS : $arguments});
}
add_handler(SIGNAL, ARGUMENTS, CALLBACK);
$last_poll = time();
while(true)
{
if (time() - $last_poll > 600)
{
CURL(URL_TO_POLL);
$last_poll = time();
}
sleep(1);
}
How can I do that ?
maybe its help you.
Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates. A few strong points about Gearman:
more on offical site http://gearman.org/
I have a foreach loop that calls a function to set values to an array. Sometimes it takes hours to complete depending on how many times it has to run thru the function to complete.
What I would like to have is a progress bar or at least a 1/1000 completed type progress indicator.
Is this possible? If so how could I implement this into my code? Would it be in the function or in the foreach loop? Been researching and found some examples using for and $i++ but I am not really sure how to implement that since I am already using a foreach loop.
Thanks much.
function scrape_amazon($links) {
//my code runs here to set all values in $ret array.
}
foreach($links as $link) {
$ret = scrape_amazon($link);
}
PHP probably isn't really the right tool for this task, however what you could do is:
Launch the slow code as a background process, and output progress to a file.
Have a PHP script that polls that file for progress information (either by page refresh or AJAX)
Launching the background process can be done in several ways, including:
Launch via cron every 60 seconds, and poll for new jobs spooled in some readable area
Launch via a fork/exec mechanism from a web page
Launch as a daemon at system startup
It will take some effort to avoid problems with multiple executions and/or overlap.
I use this, which well, not an ajax, do only flushing, but not so ugly.
I place an image
<img src='progress.gif' height=18 width=0 name=probar>
Then set on every event done on server a echo a line, then flush:
echo "<script language='JavaScript'>\ndocument.probar.width=".(($sys["probar_width"]/$task_all)*$task_i).";\n</script>\n";
flush();
If your server (eg. apache) use caching (eg. gzip is enabled) it won't work well.