Earlier this month a rather unusual tech event took place in San Francisco.
The Decentralized Web Summit played host to a gathering of web luminaries such as Sir Tim Berners-Lee, Brewster Kahle and Vint Cerf. On top of that, activists and authors and screenwriters such as Jennifer Stisa Granick, Emili Jacobi, Mike Judge and Cory Doctorow put in an appearance, as did cryptocurrency pioneers like Zooko Wilcox, blockchain developers, and academics.
Then, there was what the Guardian's John Harris calls the Punk Rock Internet - companies like MaidSafe and Blockstack who play by their own decentralised rules.
Oh, and there was a sprinkling of techies from Microsoft, Google (Vint Cerf and others) and Mozilla in attendance too, along with a handful of venture capitalists looking for opportunities.
Uniting this diverse selection of delegates was the challenge of fixing the centralising tendencies of the internet and web.
Simply put, the internet's reliance on centralised hubs of servers and data centres means that the more servers you control the more power you have, with all the negative consequences that follow from the creation of data-haves and data-have-nots.
To redress the balance, data needs to be freed from silos with control handed back to users, but how to do that while retaining the convenience and ease-of-use of the current web?
Aside from the inevitable resistance by the powers that be, this turns out to be quite the technical challenge.
One task among a set of complex interlocking challenges is to separate data from the applications that use it. People could then store their personal data where they choose, granting or limiting access by applications as they please. For example, Berners-Lee's Solid platform enables everyone to have multiple 'pods' for their data allowing for fine-grained control.
Another element is authentication, ensuring that the data owner really is who they say they are, while ensuring real identities remain private by default.
Networking needs to be peer-to-peer rather than hub-and-spoke, with copies of files stored across multiple machines for redundancy and speed of throughput in a manner that users of torrent-based file-sharing services will be familiar with, but adding far more control and performance features.
And above all it will need to be easy to use, low latency and simple for developers to create decentralised applications for.
Computing contacted a number of contributors to the Summit before and after the event and asked about their take on progress towards a viable decentralised web.
No project is an island
The web touches almost every aspect of modern life. Re-architecting such a system will be a huge undertaking, one far too big for disparate bunches of developers working alone. MaidSafe COO Nick Lambert was among many urging more collaboration.
"Certainly, there are some efforts to work together on problem solving, but this is not happening universally," he said. "Everyone at the event was clearly united in a common purpose to make the internet more private and secure, but the key takeaway for me is how we foster greater cohesion among the different projects."
Money: no longer too tight to mention
Concerns about attracting VC funding haunted 2016, but those worries have largely evaporated as a result of the crypto goldrush which has given a huge boost to the value of the tokens that support many projects. Booms can turn to busts, of course, and sudden wealth can bring challenges of its own, but for now the gloom has lifted.
While some fear an inevitable clampdown on cryptocurrencies by the authorities, OmiseGO's Althea Allen, who chaired a debate on the issue, said the worst may not happen.
"What I took away from talking with those excellent thinkers was actually quite a hopeful picture for the future of decentralised finance," she said. "By all their accounts, they have found regulators to be more open to the possibilities of crypto than we tend to assume, with less default bias toward corporate interests, and largely concerned with the same things that we are: security, privacy, consumer protections; generally speaking, making honest people's lives easier and not harder."
Awareness of the bigger picture
Mindful of the developing relationship with the authorities, governance was front and centre of many discussions, a sign of growing maturity in decentralised thinking. For Miriam Avery, director of strategic foresight at Mozilla's Emerging Technologies department, valuable lessons can be learned from those working "in countries where corruption is blatant, regulation is ineffective, and centralised control points cause palpable harm."
"The threat model is changing such that these harms are relevant to people who are less acutely aware of their causes. For instance, the things Colombian Ethereum hackers are worried about are things that we should all be a little worried about."
Avery continued: "At the same time, digging into these projects we can already see pitfalls in the 'governance' of the software projects themselves, from the prevalence of benevolent dictators to disagreements on the limits of moral relativism. There's room to grow these technologies through healthy, inclusive open source communities, and I'm excited to see that growth."
The door needs to be wedged open, or it will be slammed shut again
Another Mozillan, software engineer Irakli Gozalishvili, said: "It was reassuring to see that the community is actively thinking and talking about not only making decentralised web a place that serves people, but also how to create technology that can't be turned into corporate silos or tools for empowering hate groups."
Any decentralised web worthy of that name needs to be quick and responsive at scale, said MaidSafe's Lambert. "There is a long way to go to create a user experience that will encourage everyone to adopt the decentralised approach. For example, none of the demonstrations at the summit were able to show scalability to millions of users."
The decentralised web, with a few notable exceptions, is still very 'engineering-y' with most of the effort going into the back-end rather than the user interface. The networking may be futuristic but the front end is (with a few honourable exceptions) still Web 1.0. Which is fine at the development stage but projects will soon need to move on from demonstrating capabilities to making apps that people actually want to use.
Creating an easy onramp is an essential step. Mozilla is piloting decentralised web browsing via WebExtension APIs, the first of the 'major' vendors to do so, although others have been working in this area for a while, notably the Beaker browser for navigating DAT sites and ZeroNet.
A long list of necessary developments includes a human-readable decentralised replacement for the DNS system, search engines, and proof that crypto-based incentive systems for the supply and demand of resources can make for a scalable economy.
And the next Decentralized Web Summit? Hanamura wouldn't be drawn on a date. "We're still recovering from organising this one," she said.
06/08/2018 - Maintaining the momentum
If the 2016 Decentralized Web Summit was a call to action, in 2018 it's all about working code. That's according to Wendy Hanamura, director of partnerships at the Internet Archive, the organisation that hosted both events. However, there's still a fair way to go before it goes anything like mainstream.
The Internet Archive's mission is to preserve the outputs of culture, turning analogue books, files and recordings into digital, storing digital materials for posterity and preserving web pages going back to 1996 in the Wayback Machine.
Unsurprisingly given its aims, the organisation is sitting on a mountain of data - more than 40 petabytes and rising fast. It has recently started experimenting with decentralised technologies as a way of spreading the load and ensuring persistence, including file sharing and storage protocols WebTorrent, DAT and IPFS, the database GUN and P2P collaborative editor YJS.
And it's open to looking at more in the future. "We're glad to be in at the ground floor," said Hanamura. "We have no horse in the race. We're looking for all of them to succeed so we're looking at different protocols for different functions."
Despite some substantial progress, few decentralised projects could yet be described as 'enterprise ready'. More work is required in many different areas, one of which is providing more straightforward ways for non-technical users to become involved.
Hanumara pointed to developments among big-name browsers including Firefox, Chrome and Brave as among the most promising for improved user experience. Mozilla demonstrated a Firefox API for decentralised systems at the event.
"Participants were able to talk to each other directly browser to browser without a server involved, and they thought that was tremendously exciting," she said.
For Ruben Verborgh of the Solid project, the cross-pollination required to overcome some of the challenges is hampered by the diversity of approaches.
"Ironically, the decentralised community itself is also very decentralised, with several smaller groups doing highly similar things," he said. "Finding common ground and interoperability will be a major challenge for the future since we can only each do our thing if we are sufficiently compatible with what others do."
While it's still too early for projects to merge or consolidate around standards, Hanamura said she witnessed "lots of meetings in corridors and deals being struck about how you could tweak things to work together."
"That's another way you can make it scale," she added.
The summit had strong ideological underpinnings. Hanamura described it as "an event for the heart. People came to share," she said.
The strength of small open-source projects with big ideas is that they can easily sustain shared ideals, but this can be hard to maintain as they evolve, she went on.
"Many founders said governance was their biggest worry. You need a team of developers who believe in you and are willing to work with you - if not they can fork the code and create something very different."
In 2016 the main concern was very different: it was funding. The success of cryptocurrency token sales (ICOs) have removed many of these worries, at least for some. A lot of money has flowed into decentralised technologies, for example Filecoin recently raised $230m in an ICO and Blockstack made $50m. But this can be a double-edged sword as rapid expansion and bags of cash make team cohesion more challenging to maintain, Hanamura believes.
"It makes it a dangerous time. We came to this with a purpose, to make a web that's better for everyone. So we need to keep our eye on the North Star."
Once the technologies hit the mainstream, there will be other challenges too, including legal ones.
"As this ecosystem grows it has to be aware of the regulations on the books around the world but also those pending," said Hanamura. "We have to have a strong voice for keeping areas where we can sandbox these technologies. We need a governance system to keep it decentralised otherwise it can get centralised again."
01/08/2018 - Why is decentralising the web so hard to achieve?
Tim Berners-Lee and his colleagues faced a number of tough challenges when inventing the web, including having to build early browsers and protocols from scratch and overcoming initial scepticism (his original idea was labelled ‘vague but exciting' by his boss at CERN). The nascent web also needed to be brought into being under the radar, and the terms for the release of its code carefully formulated to guarantee its free availability for all time. It took 18 months to persuade CERN that this was the right course.
"Had the technology been proprietary, and in my total control, it would probably not have taken off. The decision to make the web an open system was necessary for it to be universal. You can't propose that something be a universal space and at the same time keep control of it," said Berners-Lee in 1998.
The original web was designed to be decentralised, but over the course of time it has been largely fenced off by a small number of quasi-monopolistic powers we know as 'the tech giants'. This makes designing a new decentralised internet - one that's ‘locked open' in the words of the Internet Archive's Brewster Kahle - a challenge even more daunting than those pioneers faced. The problem is the tech giants are very good at what they do, said Jamie Pitts, a member of the DevOps team with the Ethereum Foundation, speaking for himself rather than on behalf of his organisation.
"One of the key hurdles to decentralisation is the lock-in effect and current excellent user experience provided by the large, centralised web services," he said.
"Decentralised web technology must enable developers to produce high-quality systems enabling users to search, to connect with each other, and to conduct all forms of business. Until that happens, users will continue to be satisfied with the current set of options."
While a subset of users is worried about power imbalances, surveillance and lack of control and transparency, the fact is that most people don't care so long as there are bells and whistles aplenty. A tipping point must be achieved, as Althea Allen of OmiseGO put it.
"The only thing that will force those decentralised systems to change on a fundamental level is a mass shift by consumers toward decentralised systems."
Selling ads and services through the centralisation and mining of data (‘surveillance capitalism') has made the tech giants very powerful, and it can be hard to see beyond this model.
"The monopolisation that can occur in a rapidly-advancing technology space poses one of the greatest challenges to decentralisation," said Pitts.
"Aggregation of capital and talent results from the network effect of a successful commercially-run service, and developers and users can become locked-in. While many of their needs of users may be met by the dominant content provider, search engine, or social network, the monopolised network becomes a silo."
Moreover, the suck-up-all-the-data model has proven to be highly lucrative for the big boys, and while alternative economic methods for paying participants involving cryptocurrencies and micropayments are emerging, none has yet proved itself on the wider stage.
"There need to be viable business models for app developers that do not depend on advertisements or exploiting user behaviour and data," said Blockstack's Patrick Stanley.
On the systems side, there is a necessity to rethink the architecture to avoid central hubs. One of the toughest problems is achieving reliable consensus: with nodes seeing different versions of the ‘truth' (i.e. what events are happening and in what order), how can one ‘truth' be agreed upon without reference to a central arbiter? And how can this consensus be secured against faults and bad actors?
This longstanding conundrum was finally solved by the bitcoin blockchain a decade ago, and many efforts are ongoing to make it more efficient and a better fit for the decentralised web, the IoT and other applications. However, other projects, such as IPFS and MaidSafe's SAFE Network, don't use a blockchain, arriving at different methods for achieving consensus.
There are many ways to skin the decentralised cat - and that is another issue. What do people want, is it privacy, autonomy, security, an alternative economy, all of the above? Where are the tradeoffs and who decides the priorities? And how can the various strategies work together?
The problem is too big for one player to handle. MaidSafe's David Irvine sees collaboration as key to any solution, which was one reason why the firm open-sourced all its code.
"We want to collaborate with other companies in this space. We have the scars of developing specific functionality and are happy to work with companies to integrate that functionality where it makes sense."
31/07/2018 What might go wrong?
Technology is morally agnostic. Nuclear power provides the raw material for nuclear bombs. That new road can carry serial killers as well as saints. And while a decentralised web would redistribute power over personal data, it could also provide a convenient hiding place for the bad guys.
It's high time technologists started to see this issue in the round, said Danielle Robinson, co-executive director, of Code for Science & Society, a non-profit supporting collaboration in public interest technology.
"When technology is built, the biases of its creators are often embedded into the technology itself in ways that are very hard for the creators to see, until it's used for a purpose you didn't intend," she said during an interview with Internet Archive. "So I think it's really important that we talk about this stuff."
The increased privacy and security built into decentralised web technologies makes it easier for anyone to collaborate in a secure fashion. And that includes hate groups.
"They're on the current existing web, and they're also on the decentralised web, and I think it's important for our community to talk about that," she said. "We need a deeper exploration that's not just 'oh you know, we can't control that'."
In a separate interview, Matt Zumwalt, program manager at Protocol Labs, creator of Inter-Plantetary File System (IPFS), argued that proponents of decentralised web need to think about how it might be gamed.
"We should be thinking, really proactively, about what are the ways in which these systems can be co-opted, or distorted, or gamed or hijacked, because people are going to try all of those things," he said.
The decentralised web is still an early stage project, and many involved in its creation are motivated by idealism, he went on, drawing parallels with the early days of the World Wide Web. Lessons should be learned from that experience about how reality is likely to encroach on the early vision, he said.
"I think we need to be really careful, and really proactive about trying to understand, what are these ideals? What are the things we dream about seeing happen well here, and how can we protect those dreams?"
Mitra Ardron, technical lead for decentralisation at the Internet Archive, believes that one likely crunch point will be when large firms try to take control.
"I think that we may see tensions in the future, as companies try and own those APIs and what's behind them," he said. "Single, unified companies will try and own it."
However, he does not think this will succeed because he believes people will not accept a monolith. Code can be forked and "other people will come up with their own approaches."
Researchers used earlier observations from NASA's MESSENGER mission to probe Mercury's inner core
Espionage campaign has compromised the websites of more than 40 organisations over the past two years
Wedlake Bell's James Castro-Edwards explains why Bounty received such a big fine under the old Data Protection Act - which would almost certainly have been much larger under GDPR
Big changes are under way at SAP, and HANA's weaknesses may be the root cause