Scraping php-generated html tables in R - php

Question

I want to scrape data from this website http://demo.istat.it/bilmens2012gen/index02.html
On the left there's a webform which passes the parameters to a php page which in turn outputs the resulting html tables and in a frame in the same page.
From the the first drop-down list there are 107 cities and from the second 12 months so I should manualy run 1.284 queries to collect the desired data.
Any suggestion for automating this process?
I used R and rvest library to scrape static html tables but since these tables are generated by the form parameters I don't know how to do. Wish I could the combination of the parameters (like "city1" "month1") and retrieve the html and later do my stuff to join the data.

Answer 1

This is a fairly straightforward scraping job. When you select buttons on the page, the browser just requests some html from the server and puts it into the main frame. The request is just encoded in the url in this format:
Province (1 - 107) Period (1 - 12)
| |
v v
http://demo.istat.it/bilmens2012gen/query1.php?lingua=ita&Pro=1&allrp=4&periodo=1&submit=Tavola
So you can do this to get all the urls:
urls <- do.call("c",
lapply(1:107,
function(x) paste0("http://demo.istat.it/bilmens2012gen/",
"query1.php?lingua=ita&Pro=", x,
"&allrp=4&periodo=", 1:12,
"&submit=Tavola")
)
)
Of course, you still need to scrape the data from these pages. Here's an example of a function that will get the data from each link:
get_table <- function(url)
{
df <- xml2::read_html(url) %>%
html_nodes("table") %>%
`[`(2) %>% html_table()
df <- df[[1]]
breaks <- which(df[,1] == "CodiceComune")
output <- df[(breaks[1] + 2):(breaks[2] - 1),]
output <- setNames(output, paste(df[1,], df[2,]))
for(i in 3:8) output[[i]] <- as.numeric(as.character(output[[i]]))
dplyr::as_tibble(output)
}
So I can get the first period of the first region like this:
get_table(urls[1])
#> # A tibble: 315 x 11
#> `CodiceComune T~ `Comuni Totale` `Popolazioneini~ `Nati Vivi Tota~ `Morti Totale`
#> <chr> <chr> <dbl> <dbl> <dbl>
#> 1 001269 Strambino 6314 1 5
#> 2 001270 Susa 6626 2 10
#> 3 001271 Tavagnasco 812 0 1
#> 4 001272 Torino 869312 749 1011
#> 5 001273 Torrazza Piemo~ 2833 2 4
#> 6 001274 Torre Canavese 592 1 1
#> 7 001275 Torre Pellice 4514 4 8
#> 8 001276 Trana 3877 2 5
#> 9 001277 Trausella 132 0 1
#> 10 001278 Traversella 351 0 0
#> # ... with 305 more rows, and 6 more variables: `SaldoNaturale Totale` <dbl>, `Iscritti
#> # Totale` <dbl>, `Cancellati Totale` <dbl>, `Saldomigratorio e per altri motivi Totale` <chr>,
#> # `Unità inpiù/menodovute avariazioniterritoriali Totale` <chr>, `Popolazionefine periodo
#> # Totale` <chr>
Of course, you would want to set up a loop to get all the pages and glue the data frames together, perhaps like this:
result_list <- list()
for(i in seq_along(urls))
{
cat("Getting url", i, "of", length(urls), "\n")
result_list[[i]] <- get_table(urls[i])
}
result_df <- do.call(rbind, result_list)
Obviously I have not tested this as it is likely to take about an hour to download and process all the tables.