Les moteurs de recherche et annuaires web
Avant le Web
1988, WAIS (Wide Area Information Servers) permet de se connecter (via Internet à partie de 1991) à un serveur annuaire pour :
accéder à une base de données de serveurs de documents ;
faire des recherches et consulter ces documents.
À la naissance du Web
1991, WWW Virtual Library est une page sur le site du Cern contenant une liste de liens, que Tim Berners-Lee tient à jour. On pouvait lui envoyer un email pour demander à être ajouté.
Exemple : Yahoo! : premier annuaire élargi avec moteur de recherche
1994, Yahoo! propose un annuaire initialement nommé "Jerry and David's Guide to the World Wide Web". À l'origine, l'annuaire est constitué par du travail humain (consulter, trier et filtrer), tandis que des programmes informatiques exécutés sur le serveur permettent de gérer les recherches.
Exemple : AltaVista : intégration de l'indexation automatique
1995, Lycos et AltaVista intègrent l'indexation automatique des pages : des programmes (robots) explorent le Web en sautant de lien en lien et gèrent automatiquement les filtres et classements. AltaVista est racheté par Yahoo! en 2003.
Exemple : Google : PageRank
1998, Google se différencie grâce à PageRank, un algorithme de classement qui ordonne la présentation des résultats de recherche. La fonction est devenue déterminante avec la croissance du nombre de sites Web.
Complément : La recherche et le biais publicitaire, par les fondateurs de Google en 1998
Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is "The Effect of Cellular Phone Use Upon Driver Attention", a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.
Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. A good example was OpenText, which was reported to be selling companies the right to be listed at the top of the search results for particular queries [Marchiori 97]. This type of bias is much more insidious than advertising, because it is not clear who "deserves" to be there, and who is willing to pay money to be listed. This business model resulted in an uproar, and OpenText has ceased to be a viable search engine. But less blatant bias are likely to be tolerated by the market. For example, a search engine could add a small factor to search results from "friendly" companies, and subtract a factor from results from competitors.