Increasing SEO activity is an inexpensive way to drive traffic and sales. Search Engine Optimization (SEO) is comprised of a number of techniques used to increase the Natural Search referrals from Google, MSN, Yahoo and the other major search engines.
While the major search engines will send spiders to crawl your site, it is best to spell out exactly what areas you want crawled and what areas you want left alone. The best way to do this is to create a robots.txt file.
“All search engines, or at least all the important ones”, states Thomas Brunt, “now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.”
A robots.txt file (notice it is plural with an “s”) helps spiders understand which parts of your site to crawl . . . well actually, it helps spiders understand which parts NOT to crawl.
The robots.txt file is a text file that you place in your main directory that instructs spiders to NOT crawl certain parts of your site. The idea is that by doing so, you will reduce the amount of “drag” these crawlers put on your site and therefore, ensure that your sites response time stays at its’ peak.
An example of how it would be placed in your domain file would be as follows: http://www.YourDomainName.com/robots.txt
Remember two things: (1) it is the plural form of “robots” [not just robot]; and (2) it needs to be in the lower case and not Robots.TXT.
Setting is up can be very easy. It is just an ASCII file. You can use your Notepad to create the page (Start / Programs / Accessories / Desktop). The format you need is like this:
User-Agent: *
Disallow : /Name_of_Directory_to_NOT_Crawl
You can find examples by adding “/robots.txt” to most popular sites (for example, check out how Lowes does it by clicking here).
But here’s the real tip. While the robots.txt file is really meant to keep spiders from crawling specific pages, it is a great opportunity to add links to your sitemap.
Check out how Amazon does it (click here). If you track the site map provided, you will find that Amazon allows the spiders to find every SKU page within 3 clicks.
1) http://www.amazon.com/sitemap_index_4.xml
2) http://www.amazon.com/sitemap_page_4044.xml.gz
3) http://www.amazon.com/10-Wally-Joiner-Autographed-Baseball/dp/B001H9R37Q
Following the 3 click rule, Amazon ensures that they will receive improved natural search results for each item – even this baseball that is autographed by Wally Joiner.
Until next time,
Dan
(originally posted on www.Spellman.Typepad.com)



Merchandising 101

