Organising an array of URLs by hierarchy - php

Question

I have built a web scraper that recursively gets all URLs from a specific website and stores them in an array
Example below:
$array = [
'http://site.test',
'http://site.test/blog',
'http://site.test/blog/blog1',
'http://site.test/blog/blog2',
'http://site.test/services',
'http://site.test/services/service1',
'http://site.test/services/service2',
'http://site.test/services/service2/sub-service',
'http://site.test/product',
'http://site.test/product/product1',
'http://site.test/product/product1',
];
I am looking for some sort of way to organise this array into a multidimensional array so that I can see what pages are child pages and of which section something like the below structure
ie:
Home
----blog
--------article1
--------article2
----services
--------service1
--------service2
------------sub-service1
-----product
--------product1
--------product2
I have tried looping through and extracting certain segments of each string but cannot seem to get the desired result.
Ideally I would like to have the result in an array or even in displayed in a multi-level list for display purposes.
Any guidance would be much appreciated!

Answer 1

Let's try) we have an array of links
$array = [
'http://site.test',
'http://site.test/blog',
'http://site.test/blog/blog1',
'http://site.test/blog/blog2',
'http://site.test/services',
'http://site.test/services/service1',
'http://site.test/services/service2',
'http://site.test/services/service2/sub-service',
'http://site.test/product',
'http://site.test/product/product1',
'http://site.test/product/product2',
];
For creating a tree we should create the Node class
class Node
{
private array $childNodes;
private string $name;
public function __construct(string $name)
{
$this->name = $name;
$this->childNodes = [];
}
public function getName(): string
{
return $this->name;
}
public function addChildNode(Node $node): void
{
$this->childNodes[$node->getName()] = $node;
}
public function hasChildNode(string $name): bool
{
return array_key_exists($name, $this->childNodes);
}
public function getChildNode(string $name): Node
{
return $this->childNodes[$name];
}
public function getChildNodes(): array
{
return $this->childNodes;
}
}
And Tree class, that used Node class.
Method appendUrl parses URL and builds nodes chain.
class Tree
{
private Node $head;
public function __construct()
{
$this->head = new Node('Head');
}
public function getHead(): Node
{
return $this->head;
}
public function appendUrl(string $url): void
{
$parsedUrl = parse_url($url);
$uri = sprintf('%s//%s', $parsedUrl['scheme'], $parsedUrl['host']);
$keys = array_filter(explode('/', $parsedUrl['path'] ?? ''));
$keys = [$uri, ...$keys];
$node = $this->head;
foreach ($keys as $key) {
if (!$node->hasChildNode($key)) {
$prevNode = $node;
$node = new Node($key);
$prevNode->addChildNode($node);
} else {
$node = $node->getChildNode($key);
}
}
}
}
Now we create ConsoleTreeDrawer class that draw our tree to console
class ConsoleTreeDrawer
{
public function draw(Tree $tree): void
{
$node = $tree->getHead();
$this->drawNode($node);
}
private function drawNode(Node $node, int $level = 1): void
{
$prefix = implode('', array_fill(0, 2 * $level, '-'));
print("{$prefix}{$node->getName()}\n");
foreach ($node->getChildNodes() as $childNode) {
$this->drawNode($childNode, $level + 1);
}
}
}
And let`s use our classes
$tree = new Tree();
foreach ($array as $url) {
$tree->appendUrl($url);
}
$drawer = new ConsoleTreeDrawer();
$drawer->draw($tree);
And we drew the tree
--Head
----http//site.test
------blog
--------blog1
--------blog2
------services
--------service1
--------service2
----------sub-service
------product
--------product1

Answer 2

Algorithm:
Remove the prefix http:// for now as it is useless for our requirement. You can add it later on again.
Next is to sort all the elements using usort. Here, based on length obtained from exploding based on /.
Now, we can be assured that all parents are before child in the array.
Next is to assign version number/rank to each link. Naming is as follows:
'http://site.test' => 1
'http://site.test/blog' => 1.1
'http://site.test/services' => 1.2
'http://site.test/blog/blog1' => 1.1.1
Above is the strategy in which version numbers will be assigned.
Now, we just need to sort the array based on this version numbers using uasort
and you are done.
Snippet:
<?php
$array = [
'http://site.test',
'http://site.test/blog',
'http://site.test/blog/blog1',
'http://site.test/blog/blog2',
'http://site.test/services',
'http://site.test/services/service1',
'http://site.test/services/service2',
'http://site.test/services/service2/sub-service',
'http://site.test/product',
'http://site.test/product/product1',
];
// remove http://
foreach($array as &$val){
$val = substr($val,7);
}
// sort based on length on explode done on delimiter '/'
usort($array, function($a,$b){
return count(explode("/",$a)) <=> count(explode("/",$b));
});
$ranks = [];
$child_count = [];
// assign ranks/version numbers
foreach($array as $link){
$parent = getParent($link);
if(!isset($ranks[$parent])){
$ranks[$link] = 1;
}else{
$child_count[$parent]++;
$ranks[$link] = $ranks[$parent] . "." . $child_count[$parent];
}
$child_count[$link] = 0;
}
function getParent($link){
$link = explode("/",$link);
array_pop($link);
return implode("/",$link);
}
// sort based on version numbers
uasort($ranks,function($a,$b){
$version1 = explode(".", $a);
$version2 = explode(".", $b);
foreach($version1 as $index => $v_num){
if(!isset($version2[$index])) return 1;
$aa = intval($v_num);
$bb = intval($version2[$index]);
if($aa < $bb) return -1;
if($bb < $aa) return 1;
}
return count($version1) <=> count($version2);
});
// get the actual product links that were made as keys
$array = array_keys($ranks);
print_r($array);// now you can attach back http:// prefix if you like
Note: Current algorithm removes duplicates as well as there is no point in keeping them.
#Update:
Since you need a multidimensional hierarchical array, we can keep track of parent and child array link references and insert children into their respective parents.
<?php
$array = [
'http://site.test',
'http://site.test/blog',
'http://site.test/blog/blog1',
'http://site.test/blog/blog2',
'http://site.test/services',
'http://site.test/services/service1',
'http://site.test/services/service2',
'http://site.test/services/service2/sub-service',
'http://site.test/product',
'http://site.test/product/product1',
];
foreach($array as &$val){
$val = substr($val,7);
}
usort($array, function($a,$b){
return count(explode("/",$a)) <=> count(explode("/",$b));
});
$hier = [];
$set = [];
foreach($array as $link){
$parent = getParent($link);
if(!isset($set[$parent])){
$hier[$link] = [];
$set[$link] = &$hier[$link];
}else{
$parent_array = &$set[$parent];
$parent_array[$link] = [];
$set[$link] = &$parent_array[$link];
}
}
function getParent($link){
$link = explode("/",$link);
array_pop($link);
return implode("/",$link);
}
print_r($hier);