Schedule php wordpress script to run in background - php

Question

I hope I am not expecting too much. Any help would be extremely useful for me, because I am stuck for days now.
I created a relatively simple wordpress plugin in php.
What is the plugin supposed to do ?
-The plugin is supposed to communicate with external api and import product data in json file.
Import is started by pressing "start import" button that is created by pluin in the wordpress menu - import products.
Example request:
curl -X GET
-H 'X-Api-Key: [api-key]'
https://example.com/products/[PRODUCT ID]
[PRODUCT ID] is supposed to range from 1 to 10000
Plugin receives json file with product feed for every single request - every single [PRODUCT ID]
Plugin creates a woocommerce product and attaches imported information.
Does the plugin work ?
Yes and no, It imports first 100 products (in about 1min) correctly and then it just stops, sometimes resulting in error related to the request taking too much time to finish.
I know that the plugin doesn't work because the import script is executed in the browser and gets timed out.
I also know that I should do this process in the background, split it into batches and queue execution. The thing is I tried many ways to do so but failed miserably.
I have composer and action scheduler installed.
Unfortunately everytime I try to split this into batches, use action scheduler It just doesn't work or imports first product and stops.
I know that I'm dealing with large amount of products, I don't have error handling, checking if imported product exists etc, but I really have to run this import once
so there is no need to make this plugin very refined. This has to run, import products and I can get rid of it.
I do have wp debug on, so my attepmts on using action scheduler didn't create any errors or fatal errors, but it just didn't work properly.
I have 1500MB Ram available, this is shared hosting server, I/O 1MB, 60 available processes, 88000/600000 Inodes used, 5/200GB disc space used.
I increased php maxExecutionTime to 3000, memorylimit 1536M, maxInputTime 3000, but that didn't change anything.
I'm attaching my working code below. This is the version without my poor attemts on using action scheduler, splitting it into batches and running it in the backgroud.
I feel like this is going to be easier to read.
This code below runs in the web browser and works, but gets timed out.
I will be extremely grateful for any help with running it in the background.
Is it possible to just run this script from SSH linux terminal so it doesn't get timed out ?
`
<?php
/**
* Plugin Name: Product Importer
* Description: Imports products from an external API
* Version: 1.0
* Author: me
* Author URI: http://www.example.com
*/
// Include the Autoscheduler library
require_once '/home/user/vendor/woocommerce/action-scheduler/action-scheduler.php';
add_action('admin_menu', 'add_import_button');
function add_import_button() {
add_menu_page('Import Products', 'Import Products', 'manage_options', 'import_products', 'import_products_page');
}
function import_products_page() {
echo '<h1>Import Products</h1>';
echo '<form method="post">';
echo '<input type="submit" name="start_import" value="Start Import">';
echo '</form>';
if (isset($_POST['start_import'])) {
import_function();
}
}
function import_function() {
$product_id = 1;
while($product_id < 10000){
$product_id++;
$api_key = 'my-api-key';
$headers = array(
'X-Api-Key: ' . $api_key,
);
$url = 'https://example.com/products/';
$product_url = $url . $product_id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $product_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
$product_data = json_decode($response, true);
// post array etc
// Set other product data as required.
}
}
}
`

Answer 1

One of the ways to solve this issue is using recursion where the code can run in the background. Take a look at the example below
require_once '/home/user/vendor/woocommerce/action-scheduler/action-scheduler.php';
add_action('admin_menu', 'add_import_button');
function add_import_button() {
add_menu_page('Import Products', 'Import Products', 'manage_options', 'import_products', 'import_products_page');
}
function import_products_page() {
echo '<h1>Import Products</h1>';
echo '<form method="post">';
echo '<input type="hidden" name="product_id" value="1">'; // optional
echo '<input type="submit" name="start_import" value="Start Import">';
echo '</form>';
if (isset($_POST['start_import'])) {
import_function();
}
}
// AJAX function
add_action( 'wp_ajax_nopriv_import_function', 'import_function' );
add_action( 'wp_ajax_import_function', 'import_function' );
function import_function() {
$product_id = ( ! empty( $_POST['product_id'] ) ) ? $_POST['product_id'] : 1;
$url = 'https://example.com/products/';
$args = array(
'headers' => array(
'Content-Type' => 'application/json',
'X-Api-Key' => 'apikey12345'
)
);
// this call the function and return the body
$results = wp_remote_retrieve_body(wp_remote_get($url . $product_id, $args));
// convert to array
$results = json_decode( $results );
// Stop the code execution on this conditions
if( ! is_array( $results ) || empty( $results ) ){
return false;
}
// Do your product creation here...
$product_id++; // increase $product_id
wp_remote_post( admin_url('admin-ajax.php?action=import_function'), [
'blocking' => false, // needed for the script to continue running on the background
'sslverify' => false, // needed if working on localhost.
'body' => [
'product_id' => $product_id
]
] );
}

Answer 2

OK, I managed to get it to work. I assume that it's dumb way to do this, but since it works it's fine for me. The only thing that is problematic now is the fact that the code for image import that I used previously (when code ran in browser) now makes the plugin stuck. Without it it works great and fast. Here is my current code:
<?php
/**
* Plugin Name: Product Importer
* Description: Imports products from an external API
* Version: 3.0
* Author: me
* Author URI: http://www.example.com
*/
register_activation_hook( __FILE__, 'schedule_import' ); //plugin starts when activated, fine for me
function schedule_import() {
wp_schedule_single_event( time(), 'import_products' );
}
add_action( 'import_products', 'import_function' );
function import_function() {
$batch_size = 50; //when split into batches it worked faster than one products after another, so ok
$batch_delay = 60; //give it some time to finish batch, to avoid problems
$last_imported_product_id = get_option( 'last_imported_product_id', 1 ); //need it for incrementation and rescheduling
$api_key = 'apikey123456789';
$headers = array(
'X-Api-Key: ' . $api_key,
);
$url = 'https://example.com/products/';
for ( $i = 0; $i < $batch_size; $i++ ) {
$product_id = $last_imported_product_id + $i;
$product_url = $url . $product_id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $product_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
$product_data = json_decode($response, true);
$post = array(
'post_title' => $product_data['name'],
'post_content' => $product_data['description'],
'post_status' => 'publish',
'post_type' => 'product',
'meta_input' => array(
'_virtual' => 'yes',
'_regular_price' => $product_data['price'],
),
);
$post_id = wp_insert_post($post);
update_post_meta($post_id, '_sku', $product_data['Id']);
wp_set_object_terms($post_id, 'external', 'product_type');
$external_url = 'https://example.external.url';
update_post_meta( $post_id, '_product_url', $external_url );
update_option( 'last_imported_product_id', $product_id + 1 ); //incrementation of product_id for rescheduling
wp_schedule_single_event( time() + $batch_delay, 'import_products' ); //rescheduling
}
}
This above works well. However when I add my old code for image imports it imports nothing or 1 product (without image lol) and becomes stuck. After a minute it imports the same product over and over again. Old code for image import:
// Get the product images
$images = $product_data['images']['screenshots'];
$image_ids = [];
foreach ($images as $image) {
// Download the image
$image_url = $image['url'];
$response = wp_remote_get($image_url);
if (is_array($response)) {
$image_data = $response['body'];
$filename = basename($image_url);
$file_array = array(
'name' => $filename,
'tmp_name' => download_url($image_url),
);
// Insert the image into the media library
$attach_id = media_handle_sideload($file_array, $post_id);
if (!is_wp_error($attach_id)) {
array_push($image_ids, $attach_id);
}
}
// Set the product image gallery
update_post_meta($post_id, '_product_image_gallery', implode(',', $image_ids));
// Set the product featured image
update_post_meta($post_id, '_thumbnail_id', $image_ids[0]);
And no, I don't put it after rescheduling, but after $post array.
Honestly no idea why it would crash, tried smaller batches, longer wait between batches etc. I assume the code for images import is just very poorly written. Any solutions for this problem are going to be very appreciated ! :)
PS: I am totally fine with importing the first image available (there is around 6) and setting it up as thumbnail for product. I can give up on the gallery since It's going to slow down the process anyway.
EDIT: here is the fragment of json file received from api. (for better context of importing images):
"images":{
"screenshots":[
{
"url":"https://example.jpg",
"thumbnail":"https://example.jpg"
},
{
"url":"https://example.jpg",
"thumbnail":"https://example.jpg"
},
{
"url":"https://example.jpg",
"thumbnail":"https://example.jpg"
etc....