Check whether website is using sitemap and robots files [closed] - php

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
How do I know whether the website is using robot.txt and sitemap.txt? I have done extracting keyword, description, title; however I am unable to find the way to code to check whther the website is using robot.txt and sitemap.txt.
I am doing something like this http://www.seoptimer.com/report/loadster.in/5553240531d12

Answer 1

Use file_get_contents:
$robotsContents = file_get_contents("http://targetdomain.com/robots.txt");
$sitemapContents = file_get_contents("http://targetdomain.com/sitemap.xml");
Check if contents are false, false will mean 404 Not Found, then check if it's not HTML contents (because some sites redirect every URL) with strpos($robotsContents, '<html') === false, if there is no tag, that mean it can be txt ou xml file.
So:
function pathExistsAndIsNotHtml($path) {
$contents = #file_get_contents($path);
return ! empty($contents) && strpos($contents, '<html') === false;
}
if(pathExistsAndIsNotHtml("http://targetdomain.com/robots.txt")) {
echo 'http://targetdomain.com/robots.txt';
} else {
echo 'There is no robots.txt';
}
if(pathExistsAndIsNotHtml("http://targetdomain.com/sitemap.xml")) {
echo 'http://targetdomain.com/sitemap.xml';
} else {
echo 'There is no sitemap.xml';
}