The Only World-Standard SEO Software

Download Now
SEO PowerSuite
SEO PowerSuite Hot-new version
Supported OS

10 robots.txt files worth to have a look at

July 8th, 2013 | | Posted in category On Page SEO Search Engine Optimization

Even a junior SEO knows what a robots.txt file is. While performing my marketing job I have come across a lot of interesting cases connected with these files and their usage. So I compiled a list of robots files for you to have a look and enjoy.

1)      Google

www.google.com/robots.txt

Google itself has a robots file!

User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Allow: /catalogs/about
Allow: /catalogs/p?
Disallow: /catalogues
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /sbd
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom

2)    Nike store

http://store.nike.com/robots.txt

Nike is famous for its motivation campaign. They decided to extend it in a funny way.

3)      YouTube

https://www.youtube.com/robots.txt

Humor again! Or do you expect Google to be serious? No way…

4)      Twitter

https://twitter.com/robots.txt

These guys also included a funny line. And this file is rather complicated. If you try to help your SEO with Twitter, you should read it to know what Twitter wants to hide from search engines.

5)      Taobao

http://www.taobao.com/robots.txt

It is known that many big online stores are closing their websites from Chinese bots indexation. But here we see that a huge and popular Chinese store has closed all of its pages from Baidu spiders.

6)      The Pirate Bay

thepiratebay.sx/robots.txt

This file is all in all confusing and shows an error when being checked by validation tools. We don’t know whether it was made purposely or not.

7)      Yelp

http://yelp.com/robots.txt

So, they included the quote of a famous science fiction author to amuse the robots. They must be bored to death by reading instructions only.

8)      Etsy.com

http://www.etsy.com/robots.txt

Some more fun! Also here we can see they use the Noarchive instruction which is, however, not official, but is claimed to be working.

9)      tripadvisor.com

http://www.tripadvisor.com/robots.txt

Hiring with the help of robots.txt? Easy and effective.

10)      http://catmoji.com

http://catmoji.com/robots.txt

Hate cats? No way!

Did you come across funny or extremely confusing robots.txt files?

Image Credit



back to SEO blog


  • Gordon Campbell

    Great post but I think this one - http://explicitly.me/robots.txt - really deserves to be there :)

    Link-Assistant.Com Like or Dislike: Thumb up 0 Thumb down 0

    Thanks, Gordon, it is a good one!

  • http://www.business-online-guidance.com/ Steven Fabian

    Hi there, really funny post! The one with Asimov's 3 rules is my favorite. :-) I was hoping to find something similar over at http://www.link-assistant.com/robots.txt :)

    Link-Assistant.Com Like or Dislike: Thumb up 0 Thumb down 0

    Thanks Steven!

    You're right, we will definitely think on that!=)

  • http://www.incapsula.com/ Igal-Incapsula

    Nice list. From an optimization POV, how would respond to inclusion of ASCII messages (like Nike Store above) in your robots.txt? Is it a fun Easter-egg or just a few Bytes that should be shaved off?

    Link-Assistant.Com Like or Dislike: Thumb up 0 Thumb down 0

    It's all about fun, because everything hidden behind ###s is ignored by robots.

    Igal-Incapsula Like or Dislike: Thumb up 0 Thumb down 0

    Obviously. I`m talking about the extra bytes this adds to the robots.txt... I know a lot of SEO sticklers - myself excluded - who will spend hours on cleaning the HTML code (i.e. getting rid of redundant s) I`m just asking if this is the same scenario - un-needed HTML chars...

    Link-Assistant.Com Like or Dislike: Thumb up 0 Thumb down 0

    https://plus.google.com/+JohnMueller/posts/Wbk17p6bMSe
    Check out this G+ post, as the question was covered there, great discussion too!

Get burning industry news
from SEO PowerSuite team!
  • Crucial industry changes and Google updates
  • Top SEO trends and pros' recommendations
  • Cutting edge SEO software developments

Enter your email and name below to stay on top of things: