The government is proceeding with plans to compel communications service providers [CSPs] to retain electronic data beyond that required for commercial purposes, and make it available to the security services, police and other public authorities, despite substantial opposition.

The Home Office admitted a tiny majority — just 53 per cent — of those consulted back the approach and a large minority — 38 per cent — are opposed to any enhancement of surveillance powers. The plan will see CSPs retaining details of all emails, phone calls, texts and other electronic communications – but not their content.

The proposals are expected to cost £2bn to implement over 10 years, with no indication of any payments to CSPs to offset costs.

Officials also rejected inserting an independent judgment into the process – similar to the requirement for police to obtain a magistrates' signature on a search warrant - claiming it would "impair the effectiveness of the techniques in question" without protecting privacy.

The document spelling out the outcome of a consultation on the proposals complained they had been "widely misrepresented", insisting communications data "is a vital tool for public authorities who protect the public" and must not be limited to dealing with the terrorist threat but extend to crime, public health and public safety as well.

The Home Office insisted the plan does not extend to the interception of the actual communications, which is subject to a stricter regime. The government has always maintained it is not interested in the content of messages but rather in details about the messages – who contacted whom, the time of contact and so on.

The document included a number of instances involving the use of communications data: to rescue a man lost with his dog on a remote moor on Lewis through his mobile phone; to prove members of a South London gang were at the scene of the knife murder of a teenager caught in a love triangle; to secure convictions of the drug-dealing, semi-automatic-armed "Gooch Gang" in Manchester; and the seizure of 16 kilos of heroin imported from Afghanistan through Birmingham airport.

Security minister David Hanson spelled out the government's approach in a written statement to Parliament defending the "middle way" decided earlier this year when ministers rejected initial proposals for a huge national database to retain all communications data, seen as a Big Brother threat.

Hanson said responses to the consultation welcomed the decision against a central database but claimed "there was a recognition of the importance of communications data and agreement that the capability of communications data to protect the public should be maintained".

He said the government would develop the approach requiring CSPs to hold and supply the data, but promised "to work closely with CSPs in order to minimise as far as possible any impact on them" and "include strong safeguards to minimise the potential for abuse and ensure the security and integrity of the data".

The government claimed existing legislation including the EU Data Retention Directive are inadequate. It defended the use of deep packet inspection to extract information and carry out "lawful interception", denying this blurred the distinction between communications data and interception of content, and said work on dealing with encryption will continue.

In July, The Information Commissioner's Office expressed concerns over the collection of communications data proposed in the government's programme.