A new breed of souped-up engines

Search engines are not doing their jobs. Consequently, new strains of for intelligence. the technology are being developed to challenge companies such as Yahoo, Excite and Altavista.

The portal sites, such as Netscape Netcenter, know that a better search engine means more visitors. As a result, companies are trying to improve their search capabilities, turning to projects such as Google, Clever and Direct Hit.

Following in the footsteps of its predecessors, Yahoo and Excite, a third search engine start-up has emerged from Silicon Valley's Stanford University.

Google is based on three years' research by two graduate students, Larry Page and Sergey Brin, who take a new approach to relevance ranking.

Their product looks at the way websites link to one another and then orders matches based on their 'importance' - how often they are pointed to by other 'important' websites.

"You're the sum of the importance of the things that point to you," said Page. He added that other search engines use link relationships to build indexes, but do not do a good job of prioritising results based on link relationships.

Google currently has more than 25 million indexed pages and hopes to raise that to 100 million. Researchers estimate there are currently more than 300 million publicly indexable pages on the web.

Google is not the only start-up developing new ways to search the web.

Clever is being developed at IBM's Almaden Research Center. Like Google, it ranks pages by calculating links between them and measuring their importance,

but it does not crawl the web.

The system, based on an algorithm developed by Jon Kleinberg, a Cornell University researcher, looks at the links between pages, ranking them in order of importance.

"The principle of Clever is to exploit the work of millions of participants on the web, who are all over the world creating pages without any centrally directed motivation," said Prabhakar Raghavan, a researcher at the research centre. "The Clever system exploits these distributed judgments and aims to extract one consensus view on any given topic."

A third contender comes from venture capital backed Direct Hit, which has a system that records users' behaviour and documents popular results.

The company has produced technology that tracks millions of internet searches and records which pages users visit from a list of results. The data is then used to determine which pages are popular and they are ranked accordingly.

Direct Hit also factors in the frequency with which websites have been visited by previous internet searches. By keeping track of the outcome of previous searches, the company plans to create a market for that kind of information.

"The efforts of all these millions of searchers out there is actually a by-product that search engines aren't capturing," said Direct Hit inventor Gary Culliss, a former patent agent and graduate of Harvard Law School.

While all these new projects are aiming for the top spot, it is unlikely that a single search technology is going to serve everybody's needs. Search engine expert, Carl Frappaolo, of Delphi Consulting Group, said we can expect a new breed of search engines in the future.

"As the web grows, it'll become even harder for one search engine to cover everything," he said. "If you need medical information, you'll use one engine; if you need financial data, you'll use another."

Julia Pickar, an analyst at Zona Research, added that, although the major search companies are trying to imbue some human logic in these searches and help users get more relevant results, it will remain a hurdle because of the dynamic nature of the web.

While search engines continue to index the tens to hundreds of millions of web pages, the desire for better intelligence results is likely to be met by companies such as Google, Clever and Direct Hit.

SEEK THEM HERE

The working Google site can be found at:

http://google.stanford.edu/

- There is not currently a version of Clever in operation, but more info is available from IBM at:

http://www.almaden.ibm.com/cs/k53/clever.html

- To use the Direct Hit search engine or to find info on Direct Hit integrated into Hotbot and MacOS 8.5 point your browser at:

http://www.directhit.com/.