Evinco Logo- Provides Innovative Business Software
Evinco Knowledge Sharing

A Robot communicate with Search Engine

Search engine spider crawl your
periodically. How to limit the access of the spider? So the search
engine only crawled the page pages you want to show and leave others
ignored. What we need is a file called “robots.txt”. The robot text file
is used to disallow the search engine spider’s access to folders or
pages that you don’t want to be indexed.

Why you need “robots.txt”?

For example, you
may create some personal pages or you may have some company web pages,
which is for internal use only. You may also do not want to expose the
image folder.

How to use “robots.txt”?

You just need to upload “robots.txt” to the root of your web site.

How to create “robots.txt”?

Use a text editor (e.g. NotePad) to create a new file named “robots.txt”. The syntax is

User-agent: spiders_name_here
Disallow: filename_here

You can use * acts as a wildcard and disallows all spiders.
Visit http://www.robotstxt.org/wc/active/html/ to find the name of user

To disallow an entire directory use
Disallow: /directory_name/

To disallow an individual file use
Disallow: /filename

For multiple files, you need to write in separate lines.

Yes, it’s simple.
All you need is the above, and you can restrict the access of the spider.

ChequeSystem Cheque Printing and Management Softwar
Cheque management & keep cheque records. Provides account report, monthly statement & more

One Response to “A Robot communicate with Search Engine”

  1. Hae Cardon

    We had been attempting to start your website with the latest google android mobile and for some reason the menu would start blinking but I wasn’t able to choose any of the options. Do you have any kind of idea what could be the issue?