My Link Crawler Coes Not Work on Certain Webpages - php

Question

I wrote a program in PHP to find and print all links present on a web page. It also goes inside any links it found and does the same. My problem is that in some sites (like Youtube) it won't print the links, or follow inside them.
Here is my main code:
function echo_urls($site_address){
if(check_valid_url($site_address)){
$site = new site();
$site->address = $site_address;
$site->full_address = "$site_address";
$site->depth = 0;
$queue = new queue();
$queue->push($site);
array_push($queue->seen,$site->address);
$depth = 0;
while(($site = $queue->get_first())){
$depth++;
echo $site->depth." : ".$site->full_address."<br>";
$queue = push_links($site->address,$queue,$depth);
}
}
else;
}
function push_links($site_address,$queue,$depth){
if($depth<4){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$site_address);
curl_setopt($ch, CURLOPT_TIMEOUT, 30); //timeout after 30 seconds
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);
if( $result ){
preg_match_all( '/<a\s[^>]*href=([\"\']??)([^\" >]*?)\\1[^>]*>(.*)<\/a>/siU', $result, $list);
$list = $list[0];
foreach( $list as $item ) {
if(!(empty($item)))
if($result = get_all_string_between($item,"href=\"","\"")){
if((array_search($result[0],$queue->seen))==false){
$site = new site();
$site->address = $result[0];
$site->full_address = $item;
$site->depth = $depth;
$queue->push($site);
array_push($queue->seen,$site->address);
}
}
}
}
}
return $queue;
}

Answer 1

It's hard to tell by looking at a couple of functions, but my guess is:
YouTube is blocking you
This part if($depth<4){ is stopping push_links from executing because it might be returning FALSE
Also, don't use RegEx for this. Use something like The DOMDocument class

Answer 2

I usually use PHPQuery for crawling the sites. It's very simple
http://code.google.com/p/phpquery/