Blocking user-agents

The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

To block the entire site, use a forward slash.
Disallow: /To block a directory and everything in it, follow the directory name with a forward slash.
Disallow: /junk-directory/ To block a page, list the page.
Disallow: /private_file.html

To remove a specific image from Google Images, add the following:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

To remove all images on your site from Google Images:
User-agent: Googlebot-Image
Disallow: /

To block files of a specific file type (for example, .gif), use the following:

User-agent: Googlebot
Disallow: /*.gif$

To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages,

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Do the Little Things in SEO Make a Big Difference?

Many companies and professionals in the SEO sphere often make “best practices” recommendations to clients that include items like:

* Switch from underscores to hyphens to separate words in the URLs
* Employ the H1 tag for page headlines
* Use H2s and H3s for subheadlines
* Use the keyword you’re targeting on the page in bold/strong at least once
* Change your internal links to contain the anchor text you’re targeting on the page
* Move the keywords to the front of the title tag
* Reduce the code to content ratio
* Apply relevant alt tags to images & photos
* Nofollow links to pages like login, register and legal disclosures