# robots.txt - the following is a list opf useragents # Inktomi - Slurp # Excite - ArchitextSpider # Altavista - Scooter # Lycos - Lycos_Spider_(T-Rex) # Google - Googlebot # Alltheweb - FAST-WebCrawler/ # # The following allows all robots to visit all files. User-agent: * Disallow: