How technology is revolutionising spying

Data mining is a vital tool

Former Whitehall intelligence and security co-ordinator Sir David Omand has laid out in a policy paper how technology is fundamentally changing the role and method of the intelligence community.

Traditionally interception and deciphering of communications was the poor relation to information from human sources.

But in the internet age, vast quantities of information about target groups and countries are available on the internet, according to Omand.

"Intelligence targets also use the internet, as seen by the imaginative use of websites by Takfiri jihadists to promote radicalisation and recruitment, " says Omand's report for the Institute of Public Policy Research. (IPPR)

And as well as information online, personal data is another equally important part of the new intelligence mix.

The report says this personal information – known as protected information or "protint" - is mainly found in public and private sector databases, such as advance passenger information, airline bookings, passport and biometric data, immigration, identity and border records, criminal records, financial, telephone and email records.

"Access to such information, and in some cases the ability to apply data mining and pattern recognition software to databases, might well be the key to effective pre-emption in future terrorist cases," says Omand.

Contrary to popular opinion, this information has always been available to the intelligence community. But whereas traditional methods would involve tapping a particular line of communication or searching for a particular record, new data mining software can proactively sift personal information on databases to look for suspicious patterns.

"Application of modern data mining and processing techniques does involve examination of the innocent as well as the suspect to identify patterns of interest for further investigation," says Omand.

Omand's report expresses no doubt of the imperative for the intelligence community of pursuing these privacy infringing methods.

"Obtaining international agreement on the sharing of such data will become increasingly important in order to ensure access to these vital sources," he says.

"Finding out other people’s secrets is going to involve breaking everyday moral rules."

Key to ensuring public confidence in the taking of such measures would be ensuring "greater recognition that members of the intelligence community do, as part of their everyday professional life, follow a set of ethical norms set firmly within the framework of human rights".

Making the public aware that there are clear principles of sufficient cause, proportion and oversight to such analysis of public data is vital.

"Even a violent business such as war can have its ethical guidelines," Oman's report says.

The paper also recognises the vulnerability of the national infrastructure to cyber warfare as an increasing threat to the nation – one that justifies the entire methodical approach of the security services.

"The justification for the adoption of an anticipatory approach resides in the nature of the risks to our societies themselves, and flows from a recognition that advanced societies are more vulnerable to disruption as they become more networked and IT-dependent," the report says.

"Even relatively small-scale attacks can lead to significant cascading failures in interconnected networked systems. In the future such attacks may well be delivered through cyberspace."