Linkrot: A growing problem

Linkrot has been a growing problem since the conception of the world wide web, and the problem has attracted attention from internet watchers in very high places.

Some believe that the only solution will be the evolution of a 'semantic' internet which allows machines to process and 'understand' data rather than merely display it.

Everyone has experienced linkrot: clicking on a promising hyperlink only to be confronted by 'this page has been moved' or 'that page no longer exists' or the frustrating '404 error'.

And the problem is growing. The last proper survey on linkrot, carried out in 1997 by the University of Georgia, revealed that around six per cent of all links are bad.

This figure was up 50 per cent from the year before and, although there isn't the research to confirm it, if that rate continued then around 25 to 30 per cent of all links on the internet could suffer.

In his Webword.com weblog this week John Rhodes put forward the idea for a self repairing internet. An integrated application would sit on a web server capable of doing reverse lookups on links to pages using referrer logs or even by piggybacking on search engines.

Once broken incoming links were found the software would send repair information to the server at fault. But aside from ensuring secure connections, the big problem with this is the extra traffic that is generated. A site like Yahoo would drown from bandwidth demand.

But the idea isn't as crazy as it seems. Tim Berners-Lee, the father of the world wide web, and director of the World Wide Web Consortium, has been working for years on a semantic web with a small group of developers.

Last year Berners-Lee wrote a feature for Scientific American in which he described the semantic web as "not a separate web but an extension of the current one in which information is given well-defined meaning, better enabling computers and people to work in co-operation ... as machines become much better able to process and 'understand' the data that they merely display at present".

He explained that most of the web's content today is designed for humans to read, not for computer programs to manipulate meaningfully.

"Computers can adeptly parse web pages for layout and routine processing: here a header, there a link to another page. But, in general, computers have no reliable way to process the semantics," said Berners-Lee.

The semantic web intends to make up for this. Berners-Lee thinks that, since its creation, the web has developed into a medium of documents for people, rather than for data and information that can be processed automatically.

And so the aim of the SemanticWeb.org community is to "bridge the gap between the one end of the scale where we have everything from the five-second TV commercial to poetry, and at the other end where we have databases, programs and sensor output".

It promises to be a hot topic at the European and National conferences on artificial intelligence in the summer, where the group will be holding workshops.

Last month SymantecWeb.org released Triple, a language that allows other languages such as XML, based on a Resource Description Framework, to be defined with rules.