Scrape shipping cost which is based on country - dynamiclly build - php

Question

i have been trying to grab the product price and shipping cost
on aliexpress.com
The price is set and fixed and thus - easy...
However, the shipping cost is loaded after the site determines
which country you are from.
I viewed the source and it has a hidden input field which is populated (probably) after checking my location or ip.
How can i use CURL to "fool" the site and get the shipping cost to my country - aka scraping it using PHP?
The CURL i got:
$html = curl_download($producturl, $browserAgent);
$dom = new DOMDocument();
$dom->validateOnParse = true;
#$dom->loadHTML($html);
$xpath = new DOMXPath($dom);
libxml_clear_errors();
// get and clean product price
$price = $dom->getElementById('product-price');
$price = $price->nodeValue;
$clnprice = currency_string_remover($price);
$clnprice = explode(' ', $clnprice);
$clnprice = array_filter(array_map('trim',$clnprice),'strlen');
$clnprice = array_values($clnprice)[0];
$currency = currency_string_extractor($price);
// get and clean shipping price
// >> this is empty until page determines location! PROBLEM
$shipprice = $dom->getElementById('shipping-cost');
$shipprice = $shipprice->nodeValue;
echo '<pre>SPRICE';
print_r($shipprice);
echo '</pre>';
$shipprice = explode('-', $shipprice);
$shipprice = $shipprice[0];
$shipprice = currency_string_remover($shipprice);
echo '<div id="sitename">aliexpress</div>';
echo '<div id="price">'.$clnprice.'</div>';
echo '<div id="shipprice">'.$shipprice.'</div>';
echo '<div id="currency">'.$currency.'</div>';
Does anyone have any ideas? Pointers? Helping links?

Answer 1

I've checked the site. It works for several languages and countries. With Russia (my case) at a product page a main price includes a shipping cost. So this dom html item remains empty: <span id="shipping-cost"></span>
By the way, it's not in a form (in my case).
If you suspect it's ajax (javascript) populated you better check all js files for the shipping-cost key word. I've done it with Chrome dev-tools and in my case I found no appearing of it in any js-files (incl. the source html file). So most likely it's not javascript (ajax) updated but rather this field is originally on server generated and might be served empty.
Your browser watches the site from certain country and the server where you run php code (curl scraper) does it from totally different country (IP). Thus the Aliexpress will respond with different page content. So I recommend you a free proxy service hola.org to change/rotate country (IP) thru proxying for debugging. Thus you might check this site with different country based IPs to see this field.
You might need to check other field (product-info-shipping) to see shipping conditions. http://joxi.ru/xAe8Wy1hGDgq2y
If you really want to request the web page with the shipping-cost populated as in certain country (IP) then you need to use a proxy service to proxy your curl requests.