The Major Search Engines And Their Differences

By Wyatt Galt, 22 Jan 23:00

This article needs to be updated

There are true “engines” that are powered by software that searches the net and then there are “directories” that are big lists, often compiled in a different way. Most people refer to both as search engines but they are different animals.

Directories (Big Lists) and True Search Engines.
Directories: Yahoo is a big list and for the most part, it is created and maintained by humans. Webmasters submit their sites to Yahoo and hope that the Yahoo Gods like the site and list it favorably. This is a double edge sword. Changes you make within your site have nothing to do with the listing result although they might help you get reviewed at a later date.

Search Engines: Search engines, such as Google, produce search results on the fly. The software of the search engine company crawls the Internet and refreshes their internal index. When you request a search, the search engine looks to its index and delivers the results to your browser. These listings are often unfiltered by humans. When Websites change these search engines periodically find the changes and it affects how results come back to the user.

Specific parts of Web pages have a significant impact on how the page gets listed in terms of ranking, such as titles, meta tags, body copy, location of text, frequency etc. Google has recently produced a directory so they are in both categories now, a search engine and a directory.

The Hybrids that combine both: More and more search sites combine directories and crawling the Web.

Components of a Search Engine
1. Spider: The technology that searches the Web, often called a spider or crawler. This “spider” of technology lands on a Website and follows within that site, performing a deeper crawl. Once the Website is indexed, the spider will come back periodically to look for changes and update its listing.

2. Index: The spider finds data and sends it to the search engine database or index or catalog. This takes time, from a few days to weeks.

3. Ranking: Once a user on the search engine Website requests a search, the search engine uses its special software to analyze the index and produces results in the order it thinks they are most relevant, i.e. ranking the results.

The Majors
They are many search engines but the following are all you need to concern yourself with at this point in time. Direct your efforts with these and worry about the rest later: (not in order of importance)

AOL Search http://search.aol.com/
Users can search Web content across the full Internet. AOL uses Open Directory for its directories (see below) and the Inktomi engine (see below) for its crawler-based search.

AltaVista http://www.altavista.digital.com/
AltaVista is a large search site and has excellent coverage plus a diversity of power searching commands.

Ask Jeeves http://www.ask.com/
Ask Jeeves is powered by humans for the most part, that the site attempts to point you to the exact page that answers your question. We do not think it does this particularly well. In their defense, this is a difficult task. If the search fails, this site tries to send you elsewhere on the Web.

Direct Hit http://www.directhit.com/
The site is kind of a popularity engine in that it measures what people click on in the results of searches on its site and that of its partners. The popular sites rise to the top of the listings. They partner with HotBot. Direct Hit provides search results for HotBot and can also be seen on MSN. They are owned by Ask Jeeves.

Excite http://www.excite.com/
Very popular search engine with a large index. They also make available other information such as sports. Excite owns Magellan and WebCrawler which still are separate sites

FAST Search http://www.fastsearch.com/
Fast Search is on a mission to index the entire Internet, a daunting task to say the least. It used to be called All The Web. It is huge and claims to be the first search engine to break the 200 million Web page index milestone. Based in Norway

Go / Infoseek http://www.go.com/l
Formerly Infoseek. Go is part of Disney. It attempted to be a portal site with personalization, customized services, free E-mail and more. InfoSeek was integrated into the site. It used to use an ESP search algorithm which produces good results and had a directory. On March 15th 2001 they announced they are shutting down. They now have paid listings from GoTo.com

GoTo.com http://www.goto.com/
Not related to Go.com. You pay to play on GoTo, in other words, you buy your way in. The more you pay the higher you are listed. When results come back that are non paying sites, GoTo uses the Inktomi engine. The University of Colorado-based World Wide Web Worm became GoTo a few years ago but was replaced by Inktomi technology. When sites pay to be listed on GoTo, they show up elsewhere such as AltaVista, AOL Search, Lycos, HotBot and Netscape Search.

Google http://www.google.com/
Critically acclaimed Google separates itself from the pack by high relevancy, which is largely determined by the link popularity of the listed sites. This works well when users are searching for generic terms such as home or vacations.
Google’s index is very large and also services searches for Netscape and Yahoo.

HotBot http://www.hotbot.com/
Known as a power searching favorite, HotBot splits search results with the top page being provided from Direct Hit and the rest of the results from Inktomi. There is also a directory, which is sourced from the Open Directory project (see below). Lycos purchased HotBot from Wired Digital.

IWon http://www.iwon.com/
Win a million bucks while you search. Financed by CBS among others when venture capital money flowed like water, iWon produces a directory powered by Inktomi. The site actually gives away money, a strategy to buy eyeballs (visitors). The jury is still out on their business model, which probably cannot continue. Search fast on this site while they are still around. If they don’t make it, they can reinvent themselves with a pay search site, “I-Lost” or “I’ll search if you pay me.”

Inktomi
From its roots at UC Berkeley, the founders left to go out on their own and the company got hot. This is not a search site but technology that powers search so we thought you should at least know about it.

LookSmart http://www.looksmart.com/
LookSmart is a powered by smart looking humans that make a directory of the Web. They provide their services to Excite and Microsoft, among others. They use Inktomi when they do not produce a review of a site. Formerly part of Reader's Digest.

Lycos http://www.lycos.com/
Once a spider that crawled all over the Web, it transformed into a new animal, a directory of the Internet jungle. Its main menu is dished up from the Open Directory project and back up results via FAST Search and Direct Hit. The old spider gobbled up HotBot but let it live for the moment on its own. Nice spider.

MSN Search http://www.msn.com/
We can not do much without the 400 pound gorilla rearing its head. Microsoft's MSN Search service is powered by LookSmart with a back up of Inktomi. RealNames and Direct Hit data is also there. Within five years, all data of the Universe will be controlled, filtered and approved by Microsoft and you will not need search at all because Microsoft will just tell you what you need to know. (a little joke…)

Netscape Search http://www.netscape.com/
Netscape uses Open Directory and its own "Smart Browsing" technology and does a pretty good job. Back-up data comes from Google. Owned by AOL-Time Warner so watch for entertainment plugs popping up. Do a search, get free popcorn.

Northern Light http://www.northernlight.com/
Northern Light charges you for top search results. If you are a researcher with a budget great, otherwise it is not a great choice. They have special collection of documents that spiders can not see readily.

Open Directory http://dmoz.org/
Believe it or not, this is done by volunteers. Used to be known as NewHoo as opposed to YaHoo. Netscape bought it and it works through an open license arrangement. Netscape was an early licensee as were Lycos and AOL Search.

Raging Search http://ragingsearch.altavista.com/
Owned by AltaVista and uses the same index and algorithms. It does not have the bells and whistles of Alta Vista.

Yahoo http://www.yahoo.com/
Yahoo is a giant - just look at the chart above. They started with humans evaluating Websites and people liked the quality results. Google does assist in searching but it is dominantly controlled by Yahooites. Yahoo was a pioneer out of Stanford in 1994.

WebCrawler http://www.Webcrawler.com/
Now owned by Excite but continues to run on its own. Uses crawler-based searching.

WebTop http://www.Webtop.com/
A crawler-based engine with a huge index that is supplemented with content news sources and more. Has a Webcheck discovery tool and is owned by Bright Station.

Tags: search engines