subject: What Are Search Engine "Spiders" and How Do They Work? [print this page] What Are Search Engine "Spiders" and How Do They Work?
Although the question "What Are Search Engine "Spiders" and How Do They Work?" may seem a geeky one it is actually an important question for all internet marketers and one that you should have at least a basic understanding of the answer to.
Spiders are automated software agents (often known as "bots") that search the internet on a regular and controlled basis in order to discover new sites, pages and information (this process is commonly known as "crawling")
They carry out these searches by following the hyperlinks they find within sites and pages (hence the importance of site links and backlinks - they are the signposts that the spiders follow whilst they are crawling the web).
The search engines use spiders not only to discover new sites and pages to be included in their search results but also to discover the number of links that there are for a particular site or page. The discovery of these backlinks normally constitutes an important part of how a page is ranked within the search results.
When a spider visits or crawls a site it first looks for a "robots.txt" file as this contains important instructions for visiting robots that tell them which sections of the site to crawl, how often to visit and other important information - it is therefore important that you have an up to date robots.txt file that is in spider readable form - there are tools available to check this.
As spiders look for links having a sitemap is really useful as it provides all the links in a one stop shop for the spiders to access in an easy fashion. Anything that facilitates the efficient spidering of your site is to be recommended and sitemaps certainly fall into this category. Again there are tools available to generate and validate your sitemaps.
