I'm making a large request to the brightcove servers to make a batch change of metadata in my videos. It seems like it only made it through 1000 iterations and then stopped - can anyone help in adjusting this code to prevent a timeout from happening? It needs to make about 7000/8000 iterations.
<?php
include 'echove.php';
$e = new Echove(
'xxxxx',
'xxxxx'
);
// Read Video IDs
# Define our parameters
$params = array(
'fields' => 'id,referenceId'
);
# Make our API call
$videos = $e->findAll('video', $params);
//print_r($videos);
foreach ($videos as $video) {
//print_r($video);
$ref_id = $video->referenceId;
$vid_id = $video->id;
switch ($ref_id) {
case "":
$metaData = array(
'id' => $vid_id,
'referenceId' => $vid_id
);
# Update a video with the new meta data
$e->update('video', $metaData);
echo "$vid_id updated sucessfully!<br />";
break;
default:
echo "$ref_id was not updated. <br />";
break;
}
}
?>
Thanks!
Try the set_time_limit() function. Calling set_time_limit(0) will remove any time limits for execution of the script.
Also use ignore_user_abort() to bypass browser abort. The script will keep running even if you close the browser (use with caution).
Try sending a 'Status: 102 Processing' every now and then to prevent the browser from timing out (your best bet is about 15 to 30 seconds in between). After the request has been processed you may send the final response.
The browser shouldn't time out any more this way.
Related
I am trying to update StoreRocket using their REST API in a cron job. However, they have a 60-requests per minute limit. But if I put a 1-second timer after every request, the function time out at 2 minutes because the max_execution_time is set to 120. I have no way to update the max_execution_time because I do not have access to it. Is there another way around to bypass this timeout issue?
function cron_repeat_function () {
$remote_api_url = "StoreRocket api url";
$request_args = "setup request arguments";
foreach ( $locations as $location ) {
$storerocket_post_request = wp_remote_post( $remote_api_url, $request_args );
sleep(1);
}
}
This question already has answers here:
curl: (6) Could not resolve host: google.com; Name or service not known
(7 answers)
Closed 9 months ago.
Ok so I am a little stuck with this issue. I have a foreach loop (usually 50 results) that queries an API using Guzzle via Laravel Http and I am getting really inconsistent results.
I monitor the inserts in the database as they come in and sometimes the process seems slow and other times the process will fail with the following after x number of returned results.
cURL error 6: Could not resolve host: api.coingecko.com
The following is the actual code im using to fetch the results.
foreach ($json_result as $account) {
var_dump($account['name']);
$name = $account['name'];
$coingecko_id = $account['id'];
$identifier = strtoupper($account['symbol']);
$response_2 = Http::get('https://api.coingecko.com/api/v3/coins/'.urlencode($coingecko_id).'?localization=false');
if($response_2->successful()){
$json_result_extra_details = $response_2->json();
if( isset($json_result_extra_details['description']['en']) ){
$description = $json_result_extra_details['description']['en'];
}
if( isset($json_result_extra_details['links']['twitter_screen_name']) ){
$twitter_screen_name = $json_result_extra_details['links']['twitter_screen_name'];
}
}else {
// Throw an exception if a client or server error occurred...
$response_2->throw();
}
$crypto_account = CryptoAccount::updateOrCreate(
[
'identifier' => $identifier
],
[
'name' => $name,
'identifier' => $identifier,
'type' => "cryptocurrency",
'coingecko_id' => $coingecko_id,
'description' => $description,
]);
//sleep(1);
}
Now I know I am within the API rate limit of 100 calls a minute so I don't think that is the issue. I am wondering if this is a server/api issue which I don't really have any control over or if it related to my code and how Guzzle is implemented.
When I do single queries I don't seem to have a problem, the issue seems to be when it is inside the foreach loop.
Any advice would be great. Thanks
EDIT
Ok to update the question, I am now wondering if this is Guzzle/Laravel related. I changed the API to now point to the Twitter API and I am getting the same error after 80 synchronous requests.
I think it's better to use Asynchronous Request directly with Guzzle.
$request = new \GuzzleHttp\Psr7\Request('GET', 'https://api.coingecko.com/api/v3/coins?localization=false');
for ($i=0; $i < 50 ; $i++) {
$promise = $client->sendAsync($request)
->then(function ($response) {
echo 'I completed! ' . $response->getBody();
});
$promise->wait();
}
more information on Async requests: Doc
I have a similar problem as yours.
I doing the HTTP requests in the loop, and the first 80 requests are okay.
But the 81st start throwing this "Could not resolve host" exception.
It's very strange for me because the domain can be resolved perfectly fine on my machine.
Thus I start digging into the code.
End up I found that Laravel's Http facades keep generate the new client.
And I guess this eventually trigger the DNS resolver's rate limit?
So I have the workaround as following:
// not working
// as this way will cause Laravel keep getting a new HTTP client from guzzle.
foreach($rows as $row) {
$response = Http::post();
}
// workaround
$client = new GuzzleHttp\Client();
foreach($rows as $row) {
$response = $client->post();
// don't forget use $response->getBody();
}
i believe it's because $client will cached the DNS resolve result, thus it will reduce the call to DNS resolver and not trigger the rate limit?
I'm not sure whether it was right. BUT it's working for me.
I have been trying to implement multi-threading in php to achieve multi-upload using pthreads php.
From my understanding of multi-threading, this is how I envisioned it working.
I would upload a file,the file will start uploading in the background; even if the file is not completed to upload, another instance( thread ) will be created to upload another file. I would make multiple upload requests using AJAXand multiple files would start uploading, I would get the response of a single request individually and I can update the status of upload likewise in my site.
But this is not how it is working. This is the code that I got from one of the pthread question on SO, but I do not have the link( sorry!! ).
I tested this code to see of this really worked like I envisioned. This is the code I tested, I changed it a little.
<?php
error_reporting(E_ALL);
class AsyncWebRequest extends Thread {
public $url;
public $data;
public function __construct ($url) {
$this->url = $url;
}
public function run () {
if ( ($url = $this->url) ){
/*
* If a large amount of data is being requested, you might want to
* fsockopen and read using usleep in between reads
*/
$this->data = file_get_contents ($url);
echo $this->getThreadId ();
} else{
printf ("Thread #%lu was not provided a URL\n", $this->getThreadId ());
}
}
}
$t = microtime (true);
foreach( ["http://www.google.com/?q=". rand () * 10, 'http://localhost', 'https://facebook.com'] as $url ){
$g = new AsyncWebRequest( $url );
/* starting synchronized */
if ( $g->start () ){
printf ( $url ." took %f seconds to start ", microtime (true) - $t);
while ($g->isRunning ()) {
echo ".";
usleep (100);
}
if ( $g->join () ){
printf (" and %f seconds to finish receiving %d bytes\n", microtime (true) - $t, strlen ($g->data));
} else{
printf (" and %f seconds to finish, request failed\n", microtime (true) - $t);
}
}
echo "<hr/>";
}
So what I expected from this code was it would hit google.com, localhost and facebook.com simultaneously and run their individual threads. But every request is waiting for another request to complete.
For this it is clearly waiting for first response to complete before it is making another request because time the request are sent are after the request from the previous request is complete.
So, This is clearly not the way to achieve what I am trying to achieve. How do I do this?
You might want to look at multi curl for such multiple external requests. Pthreads is more about internal processes.
Just for further reference, you are starting threads 1 by 1 and waiting for them to finish.
This code: while ($g->isRunning ()) doesn't stop until the thread is finished. It's like having a while (true) in a for. The for executes 1 step at a time.
You need to start the threads, add them in an array, and in another while loop check each of the threads if it stopped and remove them from the array.
I have the following code which executes a piece of code on the MongoDb's side:
$mongoCode = new MongoCode('/* Some JS code */');
$db->execute($mongoCode, array(
'socketTimeoutMS' => 1000000,
));
As you see I have tried to set timeout for the code's execution by setting the socketTimeoutMS value in second parameter of execute() function. But it does not work. Documentations in PHP website indicate that the second parameter of execute() command is sent to code as arguments.
How can I set timeout for MongoDB::execute()? Please note that I am using version 1.5 of MongoDB driver for php and MongoCursor::$timeout is deprecated and does not work anymore.
You can set the socketTimeoutMS on MongoClient:
$mongo = new MongoClient("mongodb://localhost:27017",
array(
"socketTimeoutMS" => 100000
)
);
The args parameters for the execute method are passed to the code not to the driver.
You can also set a timeout just when executing the command:
$result = $mongo->dbname->command(
['eval' => $code],
['socketTimeoutMS' => 1]
);
Alternatively, if you're not executing commands, you can set the timeout on the cursor:
$cursor = $collection->find([]);
$cursor->timeout(10000);
This will obviously not work on the execute command, because that command doesn't return a cursor.
You want the MongoDB::command implementation for this which actually accepts the argument:
<?php
$mongo = new MongoClient('mongodb://192.168.2.3/test');
$db = $mongo->test;
$code = new MongoCode( 'sleep(100); return "hello";' );
try {
$res = $db->command(
array("eval" => $code),
array( 'socketTimeoutMS' => 1 )
);
echo var_dump( $res );
} catch (Exception $e) {
echo 'Caught exception: ', $e->getMessage(), "\n";
}
?>
Note that even though the exception will be thrown for the timeout, this does not not actually stop the code running on the server. That you would have to handle yourself.
Look into the killOp() and currentOP() methods, with their usage and implementation for a way to control and processes left running after your timeout expires on this operation.
Really try to look for other approaches rather than executing JavaScript on the server like this.
Very simply, I have a program that needs to perform a large process (anywhere from 5 seconds to several minutes) and I don't want to make my page wait for the process to finish to load.
I understand that I need to run this gearman job as a background process but I'm struggling to identify the proper solution to get real-time status updates as to when the worker actually finishes the process. I've used the following code snippet from the PHP examples:
do {
sleep(3);
$stat = $gmclient->jobStatus($job_handle);
if (!$stat[0]) // the job is known so it is not done
$done = true;
echo "Running: " . ($stat[1] ? "true" : "false") . ", numerator: " . $stat[2] . ", denomintor: " . $stat[3] . "\n";
} while(!$done);
echo "done!\n";
and this works, however it appears that it just returns data to the client when the worker finished telling the job what to do. Instead I want to know when the literal process of the job finished.
My real-life example:
Pull several data feeds from an API (some feeds take longer than others)
Load a couple of the ones that always load fast, place a "Waiting/Loading" animation on the section that was sent off to a worker queue
When the work is done and the results have been completely retrieved, replace the animation with the results
This is a bit late, but I stumbled across this question looking for the same answer. I was able to get a solution together, so maybe it will help someone else.
For starters, refer to the documentation on GearmanClient::jobStatus. This will be called from the client, and the function accepts a single argument: $job_handle. You retrieve this handle when you dispatch the request:
$client = new GearmanClient( );
$client->addServer( '127.0.0.1', 4730 );
$handle = $client->doBackground( 'serviceRequest', $data );
Later on, you can retrieve the status by calling the jobStatus function on the same $client object:
$status = $client->jobStatus( $handle );
This is only meaningful, though, if you actually change the status from within your worker with the sendStatus method:
$worker = new GearmanWorker( );
$worker->addFunction( 'serviceRequest', function( $job ) {
$max = 10;
// Set initial status - numerator / denominator
$job->sendStatus( 0, $max );
for( $i = 1; $i <= $max; $i++ ) {
sleep( 2 ); // Simulate a long running task
$job->sendStatus( $i, $max );
}
return GEARMAN_SUCCESS;
} );
while( $worker->work( ) ) {
$worker->wait( );
}
In versions of Gearman prior to 0.5, you would use the GearmanJob::status method to set the status of a job. Versions 0.6 to current (1.1) use the methods above.
See also this question: Problem With Gearman Job Status