# Tells Scanning Robots Where They Are And Are Not Welcome # # User-agent: can also specify by name (Googlebot, Scooter, etc) - "*" is for everyone # Disallow: disallow if this matches first part of requested path (file or directory) # # Note that if a specific bot is called out, the bot will only respect that group so be prepared to duplicate all relevant directives. User-Agent: * Disallow: /