subject: Crawlers, Spiders, Robots These Algorithms Help Search Engines Work [print this page] For site owners, having a business operational is not enough as they need to find place in the top few searches displayed by search engines. But have you ever wondered how these search engines find out the required information from millions of websites on internet? They also need some tools to find information that is presented to the users within microseconds. While the process is not as complex as it may seem, the search engines make use of some algorithms that provide fast solutions and help in filtering information. This is where the role of web crawlers, web spiders and search engine robots come into play. They perform various functions which are discussed here.
Web Spiders or Crawlers are sent to the World Wide Web by the search engines to look out for the required information. Like real spiders, these bots also crawl through numerous websites available and rank them as per the relevance of information and index these for databases of search engines. You may also be interested to know as to how they start this whole process. While the web spiders start by looking at the popular sites on WWW. After reaching to these sites, they index the words on web pages and scan through all the links in websites. This is how they begin the process which is followed by reaching out to other websites available. Their main role is to create key search words in order to help users in finding the data they are looking for.
They basically aim to store a copy of all the pages visited for the purpose of later processing for a search engine. This will provide data from these stored pages and reduce time spent for data processing and search by users. These spiders browse the whole World Wide Web by following methodological, automated approach in an orderly manner.
Search Engine Robots play the role of agents that perform different tasks automatically or under the guidance of some one. These travel through the web to search for information from different sites. An interesting and an extremely relevant point here is that they do not give attention to pictures or images and sort the information as per the textual data. This may be one of the reasons for visually appealing websites not being able to figure in the top searches! But these crawlers do read all the text given on various links or tabs on web pages. These bots behave differently with various search engines as they are set up with certain rules and guidelines by respective search engines and hence they index the web pages accordingly.
Once the website appears in the search engine directory, the spiders keep on revisiting them for any updates. Also, popular sites with heavy traffic are also visited very frequently and are more likely to improve their ranking. Many users may not have even heard of them before but they have been specially created to make a users search easier and more meaningful!
