This is a file. One that’s particularly important for SEO. It notifies the search engines which areas of your blog are restricted for them. Restricting search engines from accessing some of your pages might not sound that tempting at first but in fact it’s a valuable thing. First of all, you can exclude all your admin pages from indexing (for example, pages in the wp-admin section of your WordPress blog). You can also use it to prevent search engines from seeing duplicate content on your site. What you do, for example, is prevent them from accessing the category listings on your blog, so only the tag listings are indexed, or the other way around. Very useful file. You can (and should) learn more about it here:
« Back to Glossary Index « Back to Glossary Index