robots.txt, An Online Marketers Friend or Foe?

robots.txt is possibly the most miss understood file that a
website can contain.

Many people think that by using a robots.txt file on their
website they are protecting pages and folders from thieves
and hackers. In fact it is totally the opposite! robots.txt
opens up an enormous security hole that hackers and theives
will use to easily gain access to the parts of your
website that you don't want them to.

What is robots.txt?

robots.txt is a file that you create and upload to your
websites root directory that is used by search engine
spiders to determine which parts of your website they should
index and which folders/pages that you, the website owner,
don't want listed in search engine indexes.

Why would not want pages indexed?

There are many reasons why you might not want search engines
to index pages on your website, such as private membership
pages or exclusive training pages and such like.

If you are an Internet Marketer selling your own ebook or
other digital product, you wouldn't want your thank you
pages indexed either!

And this is where the misunderstanding comes to the fore,
and robots.txt becomes your foe.

Many online marketers who provide ebooks or other digital
products for instant download will list their download thank
you pages in the robots.txt file because they obviously
don't want those pages indexed in search engines.

By using robots.txt this way though, you will be opening up
your product to anyone who has a slight bit of knowledge
about how the file works.

robots.txt is easily readable by any human that opens a
browser and types in http://www.yourdomain.com/robots.txt
and if you have listed your thank you pages, all they have
to do is go to that url and take your product(s)!

It's that easy!

And I'm living proof that this works, as this is exactly
what happened to me. I had listed my thank you pages in
robots.txt and thought that they were safe from hackers and
thieves, then one day I was checking my web site stats and
BAM, someone had been to every single thank you page, and
taken everything.

The moral is, don't list any URL in robots.txt that you
don't want humans to have free access to. Use robots.txt
with great caution and secure your thank you pages using
dedicated software.

Resource


Secure your thank you pages with this single click
expiring link generator.