What is a Robots.txt File?
This post is another in the spirit of my Sitemaps post, where I explained how to create an XML sitemap for inclusion into Google and Bing Webmaster Tools. Today I want to talk about the robots.txt file, which is an extremely powerful and oft-misunderstood part of websites. I misunderstood it myself recently. Overview The robots.txt […]