 | Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl. How can they tell the difference? You have to tell them, and you have to speak their language, which is an industry standard called the Robots Exclusion Protocol. Dan Crow has written about this on the Google Blog recently, including an introduction to setting up your own rules for robots and a description of some of the more advanced options. |
Controlling how search engines access and index your website
The Robots Exclusion Protocol
Stay tuned for the next installment.
While we're on the topic, I'd also like to point you to the robots section of our help center and our earlier posts on this topic:
Debugging Blocked URLs
All About Googlebot
Using a robots.txt File
L I N K S
Windup Gallery
Robots Inc.biz
Gallery
Time Tunnel
Damon Soule
www.thesneeze.com
www.sdbarber.com
www.strenco.de
www.shootinggallerysf.com
www.mmoderngallery.com
www.coreyhelfordgallery.com
www.nathanspoor.com
www.home.earthlink.net/~anselmyew
www.spacevacuum.com
www.laluzdejesus.com
www.loficustoms.com
www.grantcorley.com
www.willowtoons.com
www.jonahburlingame.com
www.classictintoy.com
www.artofrobertcraig.com
www.the3dxperience.com
www.heebink.com
www.annenicholls.com
www.artwithbyte.com
www.wotartist.com
www.RobotsInc.biz
www.windupgallery.com
www.trifectagallery.com
www.happytrailsanimation.com
www.millergallery.com
www.visualbabel.blogspot.com
www.d-six.org/cartoon_drawing/index.asp
www.dwellingspaces
www.home.comcast.net/~caia.com
