Is it possible to repeat file_get_html() in a php loop? - php

Question

I am having some trouble with a fairly simple scraping program. Hopefully the working parts help someone set up a news aggregator if they're interested. I like to get specific stuff so I find grabbing the content and filtering by keywords helps me avoid the annoying issue of headlines about things that don't pertain to my life.
I would really appreciate any help or suggestions you all might have. This one has been bugging me for a couple days and I haven't come across a good solution.
Basically, the program fetches a set of items from a table to move to another table, and in the process should take the URL component of the table entry from table A that it is currently processing, and calls another program using the include. Program B uses it to grab content from that entry and save it to a file, which is named based on other attributes of the current item from table A.
The file is then saved to a folder, and the dom variable from program B is cleared so that the loop can continue.
Here is program A:
<?php
$db = 'newsfeed';
$zeta = 0;
$beta = 0;
// connect to RDS instance MySQL Database Newsfeed
include_once('/var/www/dbfunctions/mysqli_connectdb.php');
// set content source table
$sourcetable = 'feedsources';
$mastertable = 'mastertable';
// set date to remove results older than
date_default_timezone_set("UTC");
$datenow = date_timestamp_get(date_create());
$offset = "86400";
$deldate = $datenow - $offset;
//begin cycling through content data
//delete all "old" entries from the mastertable
//get number of source items present
$itemquery = "SELECT id,name FROM $sourcetable";
$itemresult = mysqli_query($conn, $itemquery);
while ($row = mysqli_fetch_assoc($itemresult)) {
$sourceid = $row['id'];
$sourcename = $row['name'];
// cycle throught the data tables
$dataquery = "SELECT * FROM $sourcetable WHERE id = $sourceid;";
$dataresult = mysqli_query($conn, $dataquery);
while ($row = mysqli_fetch_assoc($dataresult)) {
$table = $row['datatable'];
}
// copy all data from the targetted table into the master table
//loop through the targetted table and copy to mysql
$getdata = "SELECT * FROM ".$table.";";
$datareturn = mysqli_query($conn, $getdata);
while ($row = mysqli_fetch_assoc($datareturn)) {
$date = $row['datecreated'];
$title = addslashes($row['title']);
$url = addslashes($row['url']);
$tags = addslashes($row['tags']);
$titleid = $row['id'];
//get content and place in html file in /var/www/html/nuzr/content/
//check whether the item already exists in the table
$checkquery = "select id from ".$mastertable." where title = '".$title."';";
$checkcheck = mysqli_query($conn, $checkquery);
if(mysqli_num_rows($checkcheck) > 0){
}else{
require_once("getcontent.php");
$copy = "INSERT INTO ".$mastertable." VALUES ('NULL','$table','$sourcename','$date','$title','$url','$tags','$filename');";
mysqli_query($conn, $copy);
echo "Beta is ".$beta;
$beta = $beta + 1;
}
}
// clean the master table
$delquery = 'DELETE FROM '.$mastertable.' WHERE datecreated < '.$deldate.';';
mysqli_query($conn, $delquery);
}
function clear()
{
$this->dom = null;
$this->parent = null;
$this->parent = null;
$this->children = null;
}
?>
Program B
<?php
//Check Start
//echo "Program Starts";
// Include the library
include('/var/www/tools/dom/simple_html_dom.php');
$source = $url;
$content = array();
$header1 = array();
$header2 = array();
$i = 0; $y = 0;
// Retrieve the DOM from a given URL
$html = file_get_html($source);
//grab headers in case initial title is a header
foreach($html->find('h1') as $e){
$header1[$i] = $e->outertext;
//echo $e->outertext;
$i = $i + 1;
}
$i = 0;
foreach($html->find('h2') as $e){
$header2[$i] = $e->outertext;
//echo $e->outertext;
$i = $i + 1;
}
//reset counter
$i = 0;
// Find all paragraph tags and print their content into a text file
foreach($html->find('p') as $e){
$content[$i] = $e->outertext;
//echo $e->outertext;
$i = $i + 1;
}
//create the content storage file
$filename = "/var/www/html/nuzr/content/".$table.$titleid.".html";
echo "The filename is".$filename;
$file = fopen($filename,"a");
// write header and link to original article
$titleblurb = "<b>Original article courtesy of <a href='".$url."'>".$sourcename."</a></b>";
fwrite($file, $titleblurb);
// set site specific parameters based on header / footer size
if($sourcename == "The Globe and Mail"){
//Set indexing parameters
$z = $i - 13; $y = 2;
//Add Header content
$text = $header1[0];
fwrite($file, $text);
$text = $header2[1];
fwrite($file, $text);
}elseif($sourcename == "CNN Money"){
//Set indexing parameters
$z = $i - 3; $y = 1;
//Add header content
$text = $header1[0];
fwrite($file, $text);
$text = $header2[1];
fwrite($file, $text);
}elseif($sourcename == "CNN Markets"){
//Set indexing parameters
$z = $i - 3; $y = 1;
//Add header content
$text = $header1[0];
fwrite($file, $text);
//$text = $header2[1];
//fwrite($file, $text);
}elseif($sourcename == "BBC Business"){
//Set indexing parameters
$z = $i - 9; $y = 1;
//Add header content
$text = $header1[0];
fwrite($file, $text);
//$text = $header2[1];
//fwrite($file, $text);
}elseif($sourcename == "BBC Politics"){
//Set indexing parameters
$z = $i - 0; $y = 1;
//Add header content
$text = $header1[0];
fwrite($file, $text);
//$text = $header2[1];
//fwrite($file, $text);
}else{
echo $sourcename;
}
do{
$text = $content[$y];
fwrite($file, $text);
$y = $y +1;
}while($y<$z);
echo "Zeta is".$zeta;
$zeta = $zeta +1;
//close the content file
fclose($file);
//echo "File end.";
$html->clear();
unset($html);
?>
So long story short when I run this with all these output echos as update points it appears that the include(program B) only runs the first iteration and then it stops running. I was getting an issue with the file_get_html() until I added the clear.

Answer 1

You need to place the code from PROGRAM B inside
while ($row = mysqli_fetch_assoc($datareturn)) {
fetchURL($url);
}
so that for every entry, when you have the $url, you can invoke program B. I suggest placing PROGRAM B as function and calling that function within the while loop.
something like this
function fetchURL($url) {
// Place PROGRAM B here. You can make the `include` be part of the program A at the top.
}