Search Engines

There are billions of web-pages on the world-wide-web, which contain information on every conceivable topic on earth. So how can you find the one piece of information that you are looking for? Well, you go to a "Search Engine", or your favorite "Portal". But how do they keep track of all the information? Well, they use a multitude of ideas to help you find the right piece of information.

A Search Engine is a program which is used to search the desired information on the world wide web. A typical search engine has three parts.

A spider (also called a "crawler" or a "bot") that goes to a web page and reads it's content; and uses the hypertext links on each page to discover and read a site's other pages. Typically spiders will ask for permission from the server containing the site before reading the pages from the site.

A program that creates a huge index (sometimes called a "catalog") from the pages that have been read.

A program that receives your search request, compares it to the entries in the index, and returns results to you.

An alternative to using a Search Engine is to explore a structured directory of topics. Yahoo, which also lets you use its Search Engine, is the most widely-used directory on the Web. A number of Web sites offer both the Search Engine and Directory approaches for finding information. Often these sites are called "portals" because they give the user a good place to "enter" the web.

Web-sites follow different approaches to help you find what you are looking for, For example:

Major Search Engines such as Yahoo, AltaVista, Lycos, and Google index the content of a large portion of the Web and provide results that can run for pages - and consequently overwhelm the user.

Specialized content Search Engines are selective about what part of the Web is crawled and indexed. For example, TechTarget sites for products such as the AS/400 (http://www.search400.com) and Windows NT (http://www.searchnt.com) selectively index only the best sites about these products and provide a shorter but more focused list of results.

Ask Jeeves (http://www.askjeeves.com) provides a general search of the Web but allows you to enter a search request in natural language, such as "What's the temperature in New York now".

Special tools such as WebFerret (from http://www.softferret.com) let you use a number of search engines at the same time and compile results for you in a single list. Ask Jeeves is also a "meta-search engine", since it searches other search engines.

Individual Web sites, especially larger corporate sites, may use a search engine to index and retrieve the content of just their own site. Some of the major search engine companies license or sell their search engines for use on individual sites.