How to split php cronjob into parts - php

Question

There is a PHP script. It gets data from external API and import(update/delete) data into WordPress database (products for Woocommerce). There are a lot of products... To import all of them the script needs about 2-3 hours.
The problem is that when the script executes, the memory is not cleaned which leads to its overflow. After that, the script just silently dies without any error.
In short, the script looks like this:
$products = getProductsFromApi();
foreach ($products as $key => $product) {
$this->import($product);
}
The idea is to split the cronjob script into parts: if $currentMemory > 100Mb then stop the script and run it again but not from the beginning, from the moment it stopped.
How can this be realized? If there is a restriction on a server: only 1 cronjob script per 2 hours.
Any other ideas?

Answer 1

You can use a tool such as Gearman to create a queue and workers for importing processes. You can program each worker to process a certain amount of products that would take time less than the server's maximum execution time.
Gearman will also allow you to control how many workers can run simultaneously. Therefore, the importing process would be faster and you'll make sure the server resources aren't being totally consumed by workers.

Answer 2

You can serilize the $products array when $currentMemory > 100Mb to a file and then execute the script again:
$limit = 100*1000*1000;
$store = 'products.bin';
$products = [];
if ( !file_exists($store)) {
$products = getProductsFromApi();
} else {
$products = unserialize(file_get_contents($store));
}
foreach ($products as $key => $product) {
$this->import($product);
unset($products[$key]);
if (memory_get_usage() > $limit) {
file_put_contents($store,serialize($products));
exec('nohup /usr/bin/php -f myscript.php');
exit(1);
}
}
unlink ($store);

Answer 3

You can use sleep function
For example
$products = getProductsFromApi();
$i=0;
foreach ($products as $key => $product) {
// you can use your condition here instead of this
if($i%10==0){// run ten times then sleep for 100 second
sleep(100);
}
$this->import($product);
$i++;
}
https://php.net/manual/en/function.sleep.php