A Robot communicate with Search Engine

Search engine spider crawl your
periodically. How to limit the access of the spider? So the search
engine only crawled the page pages you want to show and leave others
ignored. What we need is a file called “robots.txt”. The robot text file
is used to disallow the search engine spider’s access to folders or
pages that you don’t want to be indexed.

Why you need “robots.txt”?

For example, you
may create some personal pages or you may have some company web pages,
which is for internal use only. You may also do not want to expose the
image folder.

How to use “robots.txt”?

You just need to upload “robots.txt” to the root of your web site.

How to create “robots.txt”?

Use a text editor (e.g. NotePad) to create a new file named “robots.txt”. The syntax is

User-agent: spiders_name_here
Disallow: filename_here

User-agent
You can use * acts as a wildcard and disallows all spiders.
Visit http://www.robotstxt.org/wc/active/html/ to find the name of user
agent.

Disallow
To disallow an entire directory use
Disallow: /directory_name/

To disallow an individual file use
Disallow: /filename

For multiple files, you need to write in separate lines.
Disallow:/file1.htm
Disallow:/file2.htm

Yes, it’s simple.
All you need is the above, and you can restrict the access of the spider.

ChequeSystem Cheque Printing and Management Softwar

No bulky cheque writer. Fill cheque data & print it with ChequeSystem software.