Skip to main content
#
SiteBuilder SEO
site map
contact

 Adding a Robots Text File 

What is a Robots Text File? 

A robots text file instructs the web bots/crawlers on exactly what to index and what not to index.  For instance if you do not want certain content indexed, you can disallow access to the spiders. This will keep that content of the specified page/ directory out of the search engines results pages. 
You should have the correctly formatted Robots Text file uploaded to your HTML directory.

Here is an example:

User-Agent: *
Sitemap: http://www.mysite.com/sitemap.xml
Disallow: /page-I-want-to-block.html
Disallow: /directory-I-want-to-block

This example can be saved to a text file using notepad and edited an necessary.  It should be saved as robots.txt and uploaded to your HTML directory using file manager or FTP. 
You can also use Google Webmaster Tools to generate and test your robots.txt file.  Log on and go to Site Configuration, Crawler Access, Test or Generate.

Here's what the commands mean.

User-Agent: is the bot/crawler. ( they all have names) i.e Googlebot, Yahoo is Slurp, Bing is MSNBot,  * means all bots/crawlers are allowed. 
Sitemap: Directs the bot to your XML sitemap
Disallow: Blocks bots from accessing pages and directories.  You can also block particular bots that follow the correct protocol.

For more information on the Robots Text file please see http://www.robotstxt.org/
Main Navigation

Sitebuilder SEO
3267 Bee Cave Rd.
Suite 107-65
Austin, TX 78746
281-554-9557
khickman@sitebuilderseo.net


Site Mailing List 
Generate Targeted Traffic to YOUR Web site.