Home secretary Jacqui Smith has ruled out the creation of a national communications database to record details of all UK emails, phone calls and web activity.

The controversial proposal was originally expected to be included in the Queen’s Speech detailing forthcoming legislation last November, but was dropped due to a widespread outcry over the plan. The idea was mooted as part of the Interception Modernisation Programme designed to examine how to maintain the UK's communications data capability in the light of the challenges arising from the rapidly changing technological environment.

Launching a consultation on the use and retention of communications data today, Smith said that while action must be taken, a single store of information is not the way to proceed.

“Used in the right way, and subject to important safeguards to protect individuals’ right to privacy, communications data can play a critical role in keeping all of us safe,” Smith said in her introduction to the consultation report, called Protecting the public in a changing communications environment.

“To do nothing in the face of these developments – thus allowing the capability to use communications data to degrade – could lead to more crimes left unsolved and more cases where public authorities could not protect people from harm,” she said.

“I also know that the balance between privacy and security is a delicate one, which is why this consultation explicitly rules out the option of setting up a single store of information for use in relation to communications data. My intention is to find a model which avoids the dangers of these two extreme positions, and which strikes the right balance between maximising public protection and minimising intrusion into individuals’ private lives.”

The report states that a single database is still considered to be “the most effective technical solution“ but acknowledges that the privacy implications make this impossible.

But the government is now proposing that communications data be retained by service providers such as ISPs, mobile operators and telecoms companies instead, especially to tackle the problem of overseas services on the internet, a situation that could include social media sites such as Facebook and Twitter.

“There are two alternative ways to address the challenges which both aim to strike the right balance between privacy and security,” says the report.

“They would require legislation to ensure that the data required by public authorities to protect the public is collected and retained by the communications service providers.”

The first option identified is for communications companies to store the data they already collect, in addition to information from overseas providers, but this alternative would still be fragmented and make it slower for the authorities to piece it together, says the report. The other option would be for communications companies to process the data first, making it easier for investigators to use the information.

The Home Office expects such a plan would cost about £2bn over 10 years.

ISPs have already been forced to record basic communications data under the EU Data Retention Directive that came into force in the UK at the start of April.

But the government insists that better methods of retaining communications data are essential in fighting crime and terrorism.

“The capability to use communications data to protect the public is being eroded by new technology,” says the report.

Smith said Britain risks being among the first countries to lose existing access to communications data vital for the investigation of crime "for technological reasons".

Other countries have less of a problem, she said, because the UK industry is one of the most dynamic in the world due to de-regulation. Many other Western countries still have dominant national fixed-line telecoms companies compared w ith the UK's more open market, and "the UK's competitive communications market encourages companies to find new ways to cut costs and offer new services, many based in the complex world of the internet".

She said many new services will be offered by companies outside UK jurisdiction with no need to retain data or provide security agencies in the UK and the police with access to it.

"Consequently, it will become increasingly more difficult to obtain the communications data needed to support public safety,” she said.

"We need therefore to take action to maintain this crucial capability, ensuring that the necessary strict safeguards are retained."

She added: "Communications companies will continue to be at the heart of the proposed system. They would continue to store data as they do today. But we will need to find ways of collecting and storing data relating to communications services provided from overseas providers.

"Any reduction in communications data capabilities will seriously impair the effectiveness of our police and other services to protect the public."

The Internet Service Providers' Association said it welcomes the consultation, but expects the government to fund any extra data storage burden placed on ISPs.

Shadow home secretary Chris Grayling claimed the government had buckled under pressure against its original plans.

"The big problem is that the government has built a culture of surveillance which goes far beyond counter-terrorism and serious crime. Too many parts of government have too many powers to snoop on innocent people and that's really got to change,” he said.

"It is good that the home secretary appears to have listened to Conservative warnings about Big Brother databases. Now that she has admitted the public don't want their details held by the state in one place, perhaps she will look at other areas in which the government is trying to do precisely that."

SNP home affairs spokesman Pete Wishart welcomed the climb-down but demanded "absolute guarantees" about the security of the data companies are required to retain and that it will not be open to abuse.

He said ministers should be more cautious over opening the inbox to tracking.

"On social network sites alone, we are talking about sites used by tens of millions of people which contain personal information such as religious and political beliefs, and sexual orientation. The cost of this scheme is simply breathtaking," said Wishart.

The consultation seeks to find out the views of the public and affected parties as to the best way forward.