# All robots will spider the domain User-agent: * Disallow: User-agent: MSNBot Disallow: Crawl-delay: 120