Is there something in this code block that would stop execution? - php

Question

I am parsing a webpage and following the links in order to map links from one page to another. I am only pulling the title of the page the link is on, the URL used to link the pages, and the title of the page the URL leads to.
My code works smoothly to discover the links I am interested in and descends subpages to find additional product links. There are a few hundred of these across at least a hundred pages, so it's several HTML files of parsing. I am building an in the form $products[index] contains is an array array(['url'] => URLToPage, ['title'] => TitleOfPage, ['link_title'] => TitleOfLinkedPage) as I hope this demonstrates.
The script works fine until I add this snippet, after which the script will stop execution with no errors, warnings, notices or anything; it simply never reaches the end of the script. I have included set_time_limit(0) to prevent execution time from expiring as this script takes some time to complete. This code is executed after the $products array has been populated, if any links found, and $products is always an array, and I have outputted the $link_html_strings in test cases to verify that the pages are being retrieved as expected. This is the offending code:
// Populate the destination link titles
if ( isset( $products ) && count( $products ) > 0 )
{
foreach( $products as $id => $product )
{
$from_this_page = $product['url'];
if ( $DEBUG ) echo 'Parsing ' . $from_this_page . '.<br />';
$link_html_string = file_get_contents( $from_this_page, NULL, NULL, NULL, 500 );
$string_parts = explode( '<title>', $link_html_string );
$string_parts = explode( '</title>', $string_parts[1] );
$products[$id]['link_title'] = $string_parts[0];
if ( $DEBUG ) echo 'Found title: ' . $products[$id]['link_title'] . '<br />';
ob_flush();
flush();
}
}
There should never really be 500 characters needed, however, I had some concern with memory usage when reading the entire files so I reduced the load (I think) by limiting the read. I thought perhaps the script was using up all the allocated memory for PHP. When this is included, it will iterate through this loop several times but at some point stop execution, this point is not also exactly the same. I will get several echo for what file is being parsed.
This is the full code for the script, included to answer questions regarding contents of $products in comments.
<?php
// PHP HTML DOM Parser from http://simplehtmldom.sourceforge.net/
require_once( 'includes/simple_html_dom.php' );
//error_reporting( E_ALL );
set_time_limit( 0 );
// Debugging flag
$DEBUG = false;
function reportProducts( $category, $products )
{
echo '<table width="90%" align="center"><tr><th colspan="3">';
echo $category . ' has ' . count( $products ) . ' products listed, or in subpages.';
echo '</th></tr>';
echo '<tr><td bgcolor="#777777" width="30%">This page</td>
<td bgcolor="#bbbbbb" width="30%">links with</td>
<td bgcolor="#777777" width="30%">to this page</td></tr>';
foreach( $products as $product )
{
echo '<tr><td bgcolor="#777777">' . $product['title'] . '</td>
<td bgcolor="#bbbbbb"><a href="' . $product['url'] . '">' . $product['url'] .
'</a></td><td bgcolor="#777777">' . $product['link_title'] . '</td></tr>';
}
echo '</table><br />';
ob_flush(); // Server may buffer again, preventing incremental display
flush();
}
function parseProductsForPage( $page_to_parse )
{
global $DEBUG;
$failed = false;
$product_id = 0;
$page_dom = new simple_html_dom();
$page_html_string = #file_get_contents( $page_to_parse->href );
$load_state = #$page_dom->load( $page_html_string );
if ( $load_state === NULL )
{
// Find any direct product pages for this page
if ( $DEBUG ) echo $page_to_parse->href . ' being checked for products... ';
$possible = $page_dom->find( 'a[onclick]' );
foreach( $possible as $link )
{
if ( $link->innertext == "[ Add to cart ]" )
{
$products[$product_id]['url'] = $link->href;
$titles = $page_dom->find( 'title' );
$products[$product_id]['title'] = $titles[0]->innertext;
$product_id++;
}
}
if ( $DEBUG )
{
if ( isset( $products ) )
{
echo count( $products ) . ' found on page.<br />';
} else
{
echo '0 found on page.<br />';
}
}
// Find subpages...
if ( $DEBUG ) echo $page_to_parse->href . ' being checked for links... ';
$subpages = $page_dom->find( 'a[class=buy]' );
if ( $DEBUG ) echo count( $subpages ) . ' found.<br />';
// ... and parse
foreach( $subpages as $subpage )
{
$subpage_dom = new simple_html_dom();
$subpage_html_string = #file_get_contents( $subpage->href );
$load_state = #$subpage_dom->load( $subpage_html_string );
if ( $load_state === NULL )
{
// Find any direct product pages for this page
if ( $DEBUG ) echo $subpage->href . ' being checked for products... ';
$possible = $subpage_dom->find( 'a[onclick]' );
foreach( $possible as $link )
{
if ( $link->innertext == "[ Add to cart ]" )
{
$products[$product_id]['url'] = $link->href;
$titles = $page_dom->find( 'title' );
$products[$product_id]['title'] = $titles[0]->innertext;
$product_id++;
}
}
if ( $DEBUG )
{
if ( isset( $products ) )
{
echo count( $products ) . ' found on page.<br />';
} else
{
echo '0 found on page.<br />';
}
}
$subpage_dom->clear();
} else
{
$failed[] = $subpage->href;
}
$subpage_dom->clear();
unset( $subpage_dom );
}
// Populate the destination link titles
if ( isset( $products ) && count( $products ) > 0 )
{
foreach( $products as $id => $product )
{
// $from_this_page = $product['url'];
// if ( $DEBUG ) echo 'Parsing ' . $from_this_page . '.<br />';
// $link_html_string = file_get_contents( $from_this_page, NULL, NULL, NULL, 500 );
// $string_parts = explode( '<title>', $link_html_string );
// $string_parts = explode( '</title>', $string_parts[1] );
// $products[$id]['link_title'] = $string_parts[0];
// if ( $DEBUG ) echo 'Found title: ' . $products[$id]['link_title'] . '<br />';
// ob_flush();
// flush();
}
}
} else
{
$failed[] = $page_to_parse->href;
}
$titles = $page_dom->find( 'title' );
if ( isset( $products ) ) reportProducts( $titles[0]->innertext, $products );
$page_dom->clear();
unset( $page_dom );
return $failed;
}
// Initialize the object
$html = new simple_html_dom();
$html->load_file( 'index.html' );
// Start output buffer
ob_start();
// Find all product categories listed on the website
if ( $DEBUG ) echo '<h1>Collecting links from LHN...</h1>';
$sidelinks = $html->find( 'a[class=sidelink_main]' );
$html->clear();
unset( $html );
echo '<h1>Found ' . count( $sidelinks ) . ' categories.</h1><br />';
ob_flush(); // Server may buffer output, preventing incremental display
flush();
// Find links and products for each category
foreach( $sidelinks as $sidelink )
{
if ( $DEBUG ) echo 'Sending ' . $sidelink->href . ' to parser.<br />';
$parse_failed = parseProductsForPage( $sidelink );
if ( $parse_failed )
{
foreach( $parse_failed as $failure )
{
$failures[] = $failure;
}
}
}
echo count( $failures ) . ' pages failed to parse.<br />';
echo '<br />FIN!<br />'; // Easily searched to verfiy end of script was reached, also
// celebratory.
ob_end_flush(); // Clear output buffer
flush();
?>

Answer 1

Are you sure that set_time_limit has any effect (when running php with safe_mode on it will not have any effect )?
Also be sure that $string_parts = explode( '<title>', $link_html_string ); gives an result(there may be no title-element or the tagName may be used uppercase)