Robot Files
1. To allow your site to search engines crawlers, create robots.txt file and place on root with

following content:
User-agent: *
Disallow:
2. To remove your site from search engines and prevent all robots from crawling it in the future, place the following robots.txt file in your server root:
User-agent: *
Disallow: /
The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).
* To block the entire site, use a forward slash.
Disallow: /
* To block a directory and everything in it, follow the directory name with a forward slash.
Disallow: /private_directory/
* To block a page, list the page.
Disallow: /private_file.html
4. Block or remove pages using meta tags
Rather than use a robots.txt file to block crawler access to pages, you can add a tag to an HTML page to tell robots not to index the page.
To prevent all robots from indexing a page on your site, you'd place the following meta tag into the section of your page:
5. To allow robots to index the page on your site but instruct them not to follow outgoing links, you'd use the following tag:

Source: Google.com

No comments: