Call scrapy from PHP/Delphi with references list - php

Question

I'm building a scrapy spider that checks if there are stock of some products in an online web shop.
The idea is to call this spider from PHP/Delphi code, passing a list of products (3500 references). Then, the spider returns another list with stock information.
This is my spider:
import scrapy
from scrapy.crawler import CrawlerProcess
class Spider(scrapy.Spider):
name = "Spider"
start_urls = ['https://www.url.net/Administration/Account/Login']
def parse(self, response):
return scrapy.FormRequest.from_response(
response,
formdata={'UserName': 'username', 'Password': 'password'},
callback=self.after_login
)
def after_login(self, response):
yield scrapy.Request(url="https://www.url.net/Home/Home/ShowPriceDetail?articleNo=" + REFERENCE, callback=self.parse_stock)
def parse_stock(self, response):
print("STOCK" + response.selector.xpath('//*[#id="priceDetails"]/form/div[8]/div[1]/span/span[2]/text()').extract_first())
print("Date" + response.selector.xpath('//*[#id="priceDetails"]/form/div[8]/div[1]/span/span[1]/i/#style').extract_first())
So... What is the correct way to do this? I know that you can pass arguments to spider using something like:
def __init__(self, product=None, *args, **kwargs):
super(Spider, self).__init__(*args, **kwargs)
And I know that you can execute a spider from another python script with CrawlerProcess. Also, I know that you can call a python script from PHP using:
<?php
$command = escapeshellcmd('/home/myscript.py');
$output = shell_exec($command);
echo $output;
?>
But I don't know how to merge all of this methods...
Thanks in advance.

Answer 1

You have to use some data storage to transfer your data.
So in your other programming language you save the data you have in some file or database, e.g. csv or json and then in you pass the file name to your scrapy spider via command argument. Finally in your spider you can iterate through the file contents to generate requests.
For example if we have this json:
{ "items": [
{ "url": "http://example1.com" },
{ "url": "http://example2.com" }
]}
We would use something like:
class MySpider(scrapy.Spider):
name = 'myspider'
def __init__(self, *args, **kwargs):
super(self, *args, **kwargs)
self.filename = kwargs.get('filename', None)
def start_requests(self):
if not self.filename:
raise NotImplementedError('missing argument filename')
with open(self.filename, 'r') as f:
data = json.loads(f.read())
for item in data['items']:
yield Request(item['url'])