Scraping GPU data seems taking too long - php

Question

so in my spare time I wanted to make a web to track the GPU price on a e-commerce. I am using PHP and the library Simple HTML DOM to parse the target HTML and it happen every hour from CRON Job.
(Yes, I knew I can make it in Selenium or others to scrape data more efficiently, but in this case just to challenge myself while learning it).
How it work is : Grab data and store it into database. Next, in other table it matches data from database : When the new price of a GPU is the same as latest price, it just update the date and time; If the new price is different with the latest, it make the latest price into old price and update some other things.
The scraping things is coded for a specific e-commerce website;
These variables placement are still scattered a little bit because I tried other
things;
It grab data every hour and logs the seconds on average 40-50, so my assumption is this processing time.
My question is : How can I make the code more efficient compared to my current method?
This is the code to grab the data :
<?php
error_reporting(E_ALL ^ E_WARNING);
require_once 'simple_html_dom.php';
// Database variables here
// ...
try {
$conn = new PDO("mysql:host=$servername;$dbname", $username, $password);
// set the PDO error mode to exception
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// Get the URL List
$stmt = $conn->prepare("SELECT id,url FROM url_list");
$stmt->execute();
$url_list = $stmt->fetchAll(PDO::FETCH_COLUMN|PDO::FETCH_UNIQUE);
} catch(PDOException $e) {
echo "Connection failed: " . $e->getMessage();
}
// Scrap the data from a website then return as array
function get_gpu_info(string $targeturl, int $gpu_id)
{
$results = array();
$html = new simple_html_dom();
$html->load_file($targeturl);
if (!empty($html)) {
$div_class = $price = $stock = "";
$div_class = $html->find("#main-pdp-container", 0);
$out_of_stock = $html->find(".css-1igct5v-unf-quantity-editor__input[disabled]", 0);
$price = $div_class->find(".price", 0)->innertext;
$price_int = intval(preg_replace('/[^\d\,]+/', '', $price));
$stock = ($div_class->find(".css-1a29oke p b", 0)->innertext) ?: 0;
if (!empty($price)) {
$results = array(
'GPUID' => $gpu_id,
'PRICE' => $price,
'PRICEINT' => $price_int,
'STOCK' => $stock
);
} else {echo "Price not found";}
} else {echo "URL Not Found";}
return $results;
}
// Scrap every single data from the URL list found
$gpu_data = array_map('get_gpu_info', array_values($url_list), array_keys($url_list));
try {
$time = date("H:i:s");
$date = date("Y-m-d");
$stmt = $conn->prepare("INSERT INTO price_history (gpu_id, price, price_int, stock, update_time, update_date)
VALUES (:insert_gpu_id, :insert_price, :insert_price_int, :insert_stock, :insert_update_time, :insert_update_date)");
$stmt->bindParam(':insert_gpu_id', $insert_gpu_id);
$stmt->bindParam(':insert_price', $insert_price);
$stmt->bindParam(':insert_price_int', $insert_price_int);
$stmt->bindParam(':insert_stock', $insert_stock);
$stmt->bindParam(':insert_update_time', $time);
$stmt->bindParam(':insert_update_date', $date);
foreach ($gpu_data as $data => $val) {
$insert_gpu_id = $val['GPUID'];
$insert_price = $val['PRICE'];
$insert_price_int = $val['PRICEINT'];
$insert_stock = $val['STOCK'];
$stmt->execute();
$stmt2 = $conn->prepare("SELECT COUNT(gpu_id) FROM gpu_data WHERE gpu_id = :gpu_id");
$stmt2->bindValue(':gpu_id', $val['GPUID'], PDO::PARAM_INT);
$stmt2->execute();
$count = (int)$stmt2->fetchColumn();
if($count) {
$stmt4 = $conn->prepare("SELECT old_price, old_price_int, latest_price, latest_price_int, latest_update_time, latest_update_date FROM gpu_data WHERE gpu_id = :gpu_id");
$stmt4->bindParam(':gpu_id', $val['GPUID']);
$stmt4->execute();
$old_data = $stmt4->fetch(PDO::FETCH_ASSOC);
$old_price_int = $old_data['old_price_int'];
$old_latest_price_int = $old_data['latest_price_int'];
$old_price = $old_data['old_price'];
$get_date = $old_data['latest_update_date'];
$get_time = $old_data['latest_update_time'];
$combined_old_date_time = date('Y-m-d H:i:s', strtotime("$get_date $get_time"));
if($old_price_int == $insert_price_int) {
//print_r("Same price");
$stmt3 = $conn->prepare("UPDATE gpu_data SET
stock = :stock,
latest_update_time = :update_time,
latest_update_date = :update_date
WHERE gpu_id = :gpu_id");
} else {
//print_r("Different price");
$stmt3 = $conn->prepare("UPDATE gpu_data SET
old_price = :old_price,
old_price_int = :old_price_int,
old_datetime = :old_datetime,
latest_price = :price,
latest_price_int = :price_int,
stock = :stock,
latest_update_time = :update_time,
latest_update_date = :update_date
WHERE gpu_id = :gpu_id");
$stmt3->bindParam(':old_price', $old_price);
$stmt3->bindParam(':old_price_int', $old_price_int);
$stmt3->bindParam(':old_datetime', $combined_old_date_time);
$stmt3->bindParam(':price', $insert_price);
$stmt3->bindParam(':price_int', $insert_price_int);
print_r("Old price updated");
}
$stmt3->bindParam(':update_time', $time);
$stmt3->bindParam(':update_date', $date);
$stmt3->bindParam(':stock', $insert_stock);
$stmt3->bindParam(':gpu_id', $val['GPUID']);
$stmt3->execute();
//print_r("GPU Data with the same record found and has been updated");
} else {//print_r("ERROR: No GPU Data with that GPU ID has been found");
}
}
//print_r("Price record/s updated successfully");
} catch(PDOException $e) {
echo $sql . "<br>" . $e->getMessage();}
$conn = null;
?>
Thanks in advance!

Answer 1

It's likely you're taking a lot of time to load each page you're scraping. Probably some pages are a lot slower than others. Try doing something like this, to time your load_file() operations, to figure that out.
$loadStartTime = date();
$html->load_file($targeturl);
$loadEndTime = date();
echo $targeturl . ': ' . $loadEndTime - $loadStartTime . ' seconds to load.';
Your dom-romping code looks straightforward enough.
It seems doubtful you have many thousands of rows in your table, so your database stuff should be fast enough.