10 Robots.txt Files Worth to Have a Look at
Even a junior SEO knows what a robots.txt file is. While performing my marketing job I have come across a lot of interesting cases connected with these files and their usage. So I compiled a list of robots files for you to have a look and enjoy.
Google itself has a robots file!
User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images Disallow: /catalogs Allow: /catalogs/about Allow: /catalogs/p? Disallow: /catalogues Disallow: /news Allow: /news/directory Disallow: /nwshp Disallow: /setnewsprefs? Disallow: /index.html? Disallow: /? Allow: /?hl= Disallow: /?hl=*& Disallow: /addurl/image? Disallow: /pagead/ Disallow: /relpage/ Disallow: /relcontent Disallow: /imgres Disallow: /imglanding Disallow: /sbd Disallow: /keyword/ Disallow: /u/ Disallow: /univ/ Disallow: /cobrand Disallow: /custom
2) Nike store
Nike is famous for its motivation campaign. They decided to extend it in a funny way.
Humor again! Or do you expect Google to be serious? No way…
These guys also included a funny line. And this file is rather complicated. If you try to help your SEO with Twitter, you should read it to know what Twitter wants to hide from search engines.
It is known that many big online stores are closing their websites from Chinese bots indexation. But here we see that a huge and popular Chinese store has closed all of its pages from Baidu spiders.
6) The Pirate Bay
This file is all in all confusing and shows an error when being checked by validation tools. We don’t know whether it was made purposely or not.
So, they included the quote of a famous science fiction author to amuse the robots. They must be bored to death by reading instructions only.
Some more fun! Also here we can see they use the Noarchive instruction which is, however, not official, but is claimed to be working.
Hiring with the help of robots.txt? Easy and effective.
Hate cats? No way!
Did you come across funny or extremely confusing robots.txt files?
Data from: backlink checker.