User-agent: Mediapartners-Google Disallow: Crawl-delay: 20 User-agent: * Disallow: /cgi-bin Crawl-delay: 20