# User-agent: webcrawler
# Disallow:

# User-agent: lycra
# Disallow: /

# User-agent: *
# Disallow: /tmp
# Disallow: /logs

# Lines starting with '#' specify a comment.

# The first paragraph specifies that the robot called 'webcrawler' 
# has nothing disallowed: it may go anywhere. 

# The second paragraph indicates that the robot called 'lycra' 
# has all relative URLs starting with '/' disallowed. Because 
# all relative URL's on a server start with '/', this means the 
# entire site is closed off. 

# The third paragraph indicates that all other robots should 
# not visit URLs starting with /tmp or /logs. Note the '*' is 
# a special token, meaning "any other User-agent"; you cannot
# use wildcard patterns or regular expressions in either 
# User-agent or Disallow lines. 

# Two common errors: 
# Wildcards are _not_ supported: 
# 	instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp/'. 
# You shouldn't put more than one path on a Disallow line (this 
# may change in a future version of the spec) 

# If you include the tag: 
# 	<META NAME="ROBOTS" CONTENT="NOINDEX">
# in your HTML document, that document won't be indexed.
# If you include the tag: 
# 	<META NAME="ROBOTS" CONTENT="NOFOLLOW">
# the links in that document will not be parsed by the robot.

# Aug-2009 - extract from /_Logs/09/06 awstats_files/awstats_002.htm
#    Robots shown here gave hits or traffic "not viewed" by visitors, 
#     so they are not included in other charts. 
#    Numbers after + are successful hits on "robots.txt" files.
#          - added /KCL and /yabb to Disallow - what effect will this have?

User-agent: *
Disallow: /site/
Disallow: /KCL/
Disallow: /yabb/
Disallow: /Brook/
Disallow: /SiteTEST/
Disallow: /WebTest6/