Search engine spiders (also called “crawlers” or “bots”) are programs that browse the Internet for web pages. They index and rank each page they find, based on subject matter and a number of other factors. The spiders work on unique algorithms programmed by software engineers for the various engines.