User-agent: gsa-crawler Disallow: / #User-agent: MSNBot User-agent: Slurp Disallow: / User-agent: * Disallow: /ntsc/doc/judgements/ Disallow: /ntsc/doc/sentencing_remarks/ Disallow: /d/news/ Disallow: /d/surveytool/ Disallow: /d/Primary_industry/admin/ Disallow: /scheduled_maintenance/