Evinco Logo- Provides Innovative Business Software
Evinco Knowledge Sharing


Search engine spider crawl your
periodically. How to limit the access of the spider? So the search
engine only crawled the page pages you want to show and leave others
ignored. What we need is a file called “robots.txt”. The robot text file
is used to disallow the search engine spider’s access to folders or
pages that you don’t want to be indexed.

Why you need “robots.txt”?

For example, you
may create some personal pages or you may have some company web pages,
which is for internal use only. You may also do not want to expose the
image folder.

How to use “robots.txt”?

You just need to upload “robots.txt” to the root of your web site.

How to create “robots.txt”?

Use a text editor (e.g. NotePad) to create a new file named “robots.txt”. The syntax is

User-agent: spiders_name_here
Disallow: filename_here

User-agent
You can use * acts as a wildcard and disallows all spiders.
Visit http://www.robotstxt.org/wc/active/html/ to find the name of user
agent.

Disallow
To disallow an entire directory use
Disallow: /directory_name/

To disallow an individual file use
Disallow: /filename

For multiple files, you need to write in separate lines.
Disallow:/file1.htm
Disallow:/file2.htm

Yes, it’s simple.
All you need is the above, and you can restrict the access of the spider.

RoboMail 電郵推廣軟件
發送 HTML/TEXT 電郵, 宣傳你的產品. 立即下載 RoboMail 試用.

One Response to “A Robot communicate with Search Engine”

  1. Hae Cardon

    We had been attempting to start your website with the latest google android mobile and for some reason the menu would start blinking but I wasn’t able to choose any of the options. Do you have any kind of idea what could be the issue?