Google, bing and other web search engines have tons of uris in their indexes my site does not contain. Let's say something like http://www.mydomain.com?key=apple+banana+orange .
Despite there is no direct link to this uri in my site, it displays a good page according to my own search engine results. (php, mysql and other stuff). The problem is that bots are eating my server resources because of heavy access spidering thousands and thousands of uris like that one. Even worst, there are indexed lot of odd strings (cannot say words). All this is lowering performance and (I suspect) lowering site ranking.
I only want to keep all those that really exist as links in my site like
http://www.mydomain.com?key=apple or
http://www.mydomain.com?key=banana or
http://www.mydomain.com?key=orange (one simple word key)
and remove the others (the combinations, like the first uri).
I have created a google sitemap a year ago.
I need a solution according to google rules. The only thing I have in mind is
if(strstr($_SERVER['QUERY_STRING'],'+') then redirect to index.php
Thank you
If you have your index page look at the query string and return a 404 Not Found for key's that don't actually exist, that should get them out of the index. Redirecting can be an indication that the URLs are actually valid.
There are two ways, I can think of, to attack this issue:
1. create a sitemap.xml (google it)
2. Open an account in google webmaster tool: http://www.google.com/webmasters/ and claim ownership (5 mins process). after you're verified as the website owner, login your webmaster account, go to: site configuration -> sitelinks
and there you'll have the option to demote specific links you want google to ignore.
You could use a "robots.txt" file to give instructions about your site to web robots.
You can read about how to set it up here.
Edit
Google talks about robots.txt as well here.
Related
I've been searching for this for hours, and can't seem to find anything remotely close to it. Maybe, I don't have the right search queries, so I thought I would ask here:
Situation
I have a list of about a thousand URLs in a mySQL DB that are all mixed types of sites (blog/article, forum, resources page, directory etc). I would like to take these URLs and determine what kind of site it is without visiting every single site. Is there a way that this can be done?
The only possible solution I can think of is to search for common URL structures like /forums, /blog or the subdomains and filter that way. The problem with this is that I will be missing a lot of other potentials and will have to manually go through them.
Does anyone have any creative ideas or possibly a resource/script that can provide more information on this?
First search the URL for keywords as you mentioned.
Next if none are found, follow up with a site search using something like strpos() or a regex on the contents of the page which you get get easily with file_get_contents(). That may get some false positives, if a blog mentions a forum, likely you would just search on a set of keywords and the first result would be the one to go with. You might be able to get a little more information out of the content, but it would quickly get complicated.
One example of more complicated, search for admin interface based on the base url, such as wordpress being www.example.com/wp-admin.
Im not talking about extracting a text, or downloading a web page.
but I see people downloading whole web sites, for example, there is a directory called "example" and it isnt even linked in web site, how do I know its there? how do I download "ALL" pages of a website? and how do I protect against?
for example, there is "directory listing" in apache, how do I get list of directories under root, if there is a index file already?
this question is not language-specific, I would be happy with just a link that explains techniques that does this, or a detailed answer.
Ok so to answer your questions one by one; how do you know that a 'hidden' (unlinked) directory is on the site? Well you don't, but you can check the most common directory names, whether they return HTTP 200 or 404... With couple of threads you will be able to check even thousands a minute. That being said, you should always consider the amount of requests you are making in regards to the specific website and the amount of traffic it handles, because for small to mid-sized websites this could cause connectivity issues or even a short DoS, which of course is undesirable. Also you can use search engines to search for unlinked content, it may have been discovered by the search engine on accident, there might have been a link to it from another site etc. (for instance google site:targetsite.com will list all the indexed pages).
How you download all pages of a website has already been answered, essentially you go to the base link, parse the html for links, images and other content which points to a onsite content and follow it. Further you deconstruct links to their directories and check for indexes. You will also bruteforce common directory and file names.
Well you really effectively can't protect against bots, unless you limit user experience. For instance you could limit the number of requests per minute; but if you have ajax site, a normal user will also be producing a large number of requests so that really isn't a way to go. You can check user agent and white list only 'regular' browsers, however most scraping scripts will identify themselves as regular browsers so that won't help you much either. Lastly you can blacklist IPs, however that is not very effective, there is plenty of proxies, onion routing and other ways to change your IP.
You will get directory list only if a) it is not forbidden in the server config and b) there isn't the default index file (default on apache index.html or index.php).
In practical terms it is good idea not to make it easier to the scraper, so make sure your website search function is properly sanitized etc. (it doesn't return all records on empty query, it filters % sign if you are using LIKE mysql syntax...). And of course use CAPTCHA if appropriate, however it must be properly implemented, not a simple "what is 2 + 2" or couple of letters in common font with plain background.
Another protection from scraping might be using referer checks to allow access to certain parts of the website; however it is better to just forbid access to any parts of the website you don't want public on server side (using .htaccess for example).
Lastly from my experience scrapers will only have basic js parsing capabilities, so implementing some kind of check in javascript could work, however here again you'd also be excluding all web visitors with js switched off (and with noscript or similar browser plugin) or with outdated browser.
To fully "download" a site you need a web crawler, that in addition to follow the urls also saves their content. The application should be able to :
Parse the "root" url
Identify all the links to other pages in the same domain
Access and download those and all the ones contained in these child pages
Remember which links have already been parsed, in order to avoid loops
A search for "web crawler" should provide you with plenty of examples.
I don't know counter measures you could adopt to avoid this: in most cases you WANT bots to crawl your websites, since it's the way search engines will know about your site.
I suppose you could look at traffic logs and if you identify (by ip address) some repeating offenders you could blacklist them preventing access to the server.
I am making a simple web spider and I was wondering if there is a way that can be triggered in my PHP code that I can get all the webpages on a domain...
e.g Lets say I wanted to get all the webpages on Stackoverflow.com . That means that it would get:
https://stackoverflow.com/questions/ask
pulling webpages from an adult site -- how to get past the site agreement?
https://stackoverflow.com/questions/1234214/
Best Rails HTML Parser
And all the links. How can I get that. Or is there an API or DIRECTORY that can enable me to get that?
Also is there a way I can get all the subdomains?
Btw how do crawlers crawl websites that don't have SiteMaps or Syndication feeds?
Cheers.
If a site wants you to be able to do this, they will probably provide a Sitemap. Using a combination of a sitemap and following the links on pages, you should be able to traverse all the pages on a site - but this is really up to the owner of the site, and how accessible they make it.
If the site does not want you to do this, there is nothing you can do to work around it. HTTP does not provide any standard mechanism for listing the contents of a directory.
You would need to hack the server sorry.
What you can do is that, if you own the domain www.my-domain.com, you can put a PHP file there, that you use as a request on demand file. That php file you will need to code some sort of code in that can look at the Folders FTP Wise. PHP can connect to a FTP server, so thats a way to go :)
http://dk1.php.net/manual/en/book.ftp.php
You can with PHP read the dirs folders and return that as an array. Best i can do.
As you have said, you must follow all the links.
To do this, you must start by retrieving stackoverflow.com, easy: file_get_contents ("http:\\stackoverflow.com").
Then parse its contents, looking for links: <a href="question/ask">, not so easy.
You store those new URL's in a database and then parse that those after, which will give you a whole new set of URL's, parse those. Soon enough you'll have the vast majority of the site's content, including stuff like sub1.stackoverflow.com. This is called crawling, and it is quite simple to implement, although not so simple to retrieve useful information once you have all that data.
If you are only interested in one particular domain, be sure to dismiss links to external sites.
No, not the way you are asking.
However, provided you have a clear goal in mind, you may be able to:
use a "primary" request to get the objects of interest. Some sites provide JSON, XML, ... apis to list such objects (e.g SO can list questions this way). Then use "per-object" requests to fetch information specific to one object
fetch information from other open (or paid) sources, e.g. search engines, directories, "forensic" tools such as SpyOnWeb
reverse engineer the structure of the site, e.g. you know that /item/<id> gets you to the page of item whose ID is <id>
ask the webmaster
Please note that some of these solutions may be in violation of the site's termes of use. Anyway these are just pointers, on top of my head.
You can use WinHTTPTack/. But it is a polite not to hammer other peoples web sites.
I just use it to find broken links and make a snap shot.
If you do start hammering other peoples sites they will take measures. Some of them will not be nice (i.e. hammer yours).
Just be polite.
I have some pages names index2.php, index3.php,..
they are copies of my index.php, for testing purposes
should I disallow these pages in robots.txt because of the double content?
or it doesn't matter because these pages are not linked to on my website?
if I get punished by google for the double content, how bad is it?
If you never link to the test pages or share the link with anyone (who might post it elsewhere), and they don't appear in any sitemap then crawlers will never find the pages.
I wouldn't recommend adding them to your robots.txt because anyone can view your robots.txt and find out the location of your test pages. Usually you don't want the public to have access to test pages.
If you wanted to ensure no one was able to view the test pages other than you, then add an IP check, or some other security such as a login.
Dupe content or anything that can knock you down, can be very hard to recover from. Overall I would say it depends on what your doing with your site/service. If its all about content, then yea its going to matter to some extent or another. Once google and other engines list you in a pool of domains for dupe content or other possible flags that would be an attempt to falsely boost your rank, getting out of that pool is tough. Its like trust, once you lose it for someone its hard for them to gain it back.
Then again its hard to say, Ive known some who have been plauged for months on end trying to fix similar issues, and I know some that fixed it within a week.
Again it all boils down to what your site/service is doing, and how its doing it. So many factors, that not just one alone will kill..
As for that whole robots text thing, if your that concerned go for it, theres no harm in it. Engines like google pay mind to the robots text and respect it enough. Without it, despite not being linked directly google and or other engines sometimes find a way to find it. Ive had whole sub domains never seen by the public end up getting indexed in search engines, I have had thee most obsucre file names never linked publicly end up in search engines.. its hit or miss
To be on the safe side, you should disallow them and it would be awesome if you completely remove them.
And about the punishment, it's just that if you have duplicate content on different pages those pages will start competing each other for higher rankings. And you would not want your own pages fighting each other for rankings.
I have a community site which has around 10,000 listings at the moment. I am adopting a new url strategy something like
example.com/products/category/some-product-name
As part of strategy, I am implementing a site map. Google already has a good index of my site, but the URLs will change. I use a php framework which accesses the DB for each product listing.
I am concerned about the perfomance effects of supplying 10,000 new URLs to google, should I be?
A possible solution I'm looking at is rendering my php-outputted pages to static HTML pages. I already have this functionality elsewhere on the site. That way, google would index 10,000 html pages. The beauty of this system is that if a user arrives via google to that HTML page, as soon as they start navigating around the site, they jump straight back into the PHP version.
My problem with this method is that I would have to append .html onto my nice clean URLs...
example.com/products/category/some-product-name.html
Am I going about this the wrong way?
Edit 1:
I want to cut down on PHP and MySQL overhead. Creating the HTML pages is just a method of caching in preparation of a load spike as the search engines crawl those pages. Are there better ways?
Unless I'm missing something, I think you don't need to worry about it. I'm assuming that your list of product names doesn't change all that often -- on a scale of a day or so, not every second. The Google site-map should be read in a second or less, and the crawler isn't going to crawl you instantly after you update. I'd try it without any complications and measure the effect before you break your neck optimizing.
You shouldnt be worried about 10000 new links, but you might want to analyze your current google traffic, to see how fast google would crawl them. Caching is always a good idea (See: Memcache, or even generate static files?).
For example, i have currently about 5 requests / second from googlebot, which would mean google would crawl those 10,000 pages in a good half hour, but, consider this:
Redirect all existing links to new locations
By doing this, you assure that links already indexed by google and other search engines are almost immediatelly rewritten. Current google rank is migrated to the new link (additional links start with score 0).
Google Analytics
We have noticed that google uses Analytics data to crawl pages, that it usually wouldn't find with normal crawling (javascript redirects, logged in user content links). Chances are, google would pick up on your url change very quickly, but see 1).
Sitemap
The rule of thumb for the sitemap files in our case is only to keep them updated with the latest content. Keeping 10,000 links, or even all of your links in there is pretty pointless. How will you update this file?
It's a love & hate relationship with me and Google crawler theese days, since most used links by users are pretty well cached, but the thing google crawler crawls usually are not. This is the reason google causes 6x the load in 1/6th the requests.
Not an answer to your main question.
You dont have to append .html. You can leave the URLs as they are. If you cant find a better way to redirect to the html file (which does not have ot have an .html suffix), you can output it via PHP with readfile.
I am concerned about the perfomance effects of supplying 10,000 new URLs to google, should I be?
Performance effects on Google's servers? I wouldn't worry about it.
Performance effects on your own servers? I also wouldn't worry about it. I doubt you'll get much more traffic than you used to, you'll just get it sent to different URLs.