Questions/Issues/Wishlists for BFP

Selenay · 01-01-2010, 03:39 AM

It's a "bot" index tool for search engines.

Example:

Yahoo! Slurp is the Yahoo! web indexing robot. The Yahoo! Slurp web crawler collects documents from the Web to build a searchable index for search services using the Yahoo! Search engine. These documents are discovered and crawled because other webpages contain links to these documents.
As part of the crawling effort, the Yahoo! Slurp crawler takes robots.txt standards into account to ensure that we do not crawl and index content from pages that you do not want to be included in Yahoo! Search results. If a page is disallowed to be crawled by robots.txt standards, Yahoo! does not read or use the contents of that page.

Clicky
Clicky

01-01-2010, 03:39 AM	#11
Selenay Member How Do You Identify?: A soul for a compass and a heart for a pair of wings. Preferred Pronoun?: All I ask of living is to have no chains on me. Relationship Status: All I own are the strides I spend to the finish line. Join Date: Nov 2009 Location: Somewhere in between here and gone. Posts: 662 My Photo Gallery Thanks: 110 Thanked 1,447 Times in 369 Posts Rep Power: 6344716	It's a "bot" index tool for search engines. Example: Yahoo! Slurp is the Yahoo! web indexing robot. The Yahoo! Slurp web crawler collects documents from the Web to build a searchable index for search services using the Yahoo! Search engine. These documents are discovered and crawled because other webpages contain links to these documents. As part of the crawling effort, the Yahoo! Slurp crawler takes robots.txt standards into account to ensure that we do not crawl and index content from pages that you do not want to be included in Yahoo! Search results. If a page is disallowed to be crawled by robots.txt standards, Yahoo! does not read or use the contents of that page. Clicky Clicky __________________ Two or three things I know for sure, And one is that I would rather go naked Than wear the coat the world has made for me.