1. To allow your site to search engines crawlers, create robots.txt file and place on root with
2. To remove your site from search engines and prevent all robots from crawling it in the future, place the following robots.txt file in your server root:
The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).
* To block the entire site, use a forward slash.
* To block a directory and everything in it, follow the directory name with a forward slash.
* To block a page, list the page.
4. Block or remove pages using meta tags
Rather than use a robots.txt file to block crawler access to pages, you can add a tag to an HTML page to tell robots not to index the page.
To prevent all robots from indexing a page on your site, you'd place the following meta tag into the section of your page:
5. To allow robots to index the page on your site but instruct them not to follow outgoing links, you'd use the following tag: