Blocking user-agents

The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

To block the entire site, use a forward slash.
Disallow: /To block a directory and everything in it, follow the directory name with a forward slash.
Disallow: /junk-directory/ To block a page, list the page.
Disallow: /private_file.html

To remove a specific image from Google Images, add the following:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

To remove all images on your site from Google Images:
User-agent: Googlebot-Image
Disallow: /

To block files of a specific file type (for example, .gif), use the following:

User-agent: Googlebot
Disallow: /*.gif$

To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages,

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /