11 ms·

Locking the Web Open: A Call for a Distributed Web

221 points by hachiya 11y ago

wwwtyro 11y ago
I'm all for this, and consider it to be inevitable in the long run. In the short term, however, it seems like the major hurdle will be getting one of these projects into the mainstream: for the most part, the web already does what most people want it to do, and those people aren't going to be bothered to install a new web browser so that they can do things they're already doing. Especially if it lacks the features, performance, or ease-of-use of their current browser. So, how do we address this? Is there a "killer app" for the distributed web that will motivate people to move to it? Can we use existing web tech like Web-RTC to bootstrap the system? Maybe a workable avenue is mobile, where people are pretty comfortable installing new applications - what if we built the next social network into an app based on the distributed web? I don't know the answer, but I'd love to hear any ideas/brainstorming you clever people have to offer.
- lsjroberts 11y ago
  You may be interested in https://ind.ie/about/vision/ https://ind.ie/about/vision/
  - molsongolden 11y ago
    https://indiewebcamp.com/ https://indiewebcamp.com/ is a bit more interesting and open.
- gagege 11y ago
  I love the idea of doing a distributed social network via a mobile app. Being a native app, it could be kind of seamless as well, for the user.
- mtgx 11y ago
  Going mainstream would probably require something like Firefox or at least Ubuntu integrating IPFS.
- simon_vetter 11y ago
  Implementing other protocol handlers into existing web browsers and operating systems would be a good start IMO. Taking ipfs/ipns [1] as an example, having handlers inside web browsers would allow people to link from http[s]:// to ipfs:// and vice versa in a seamless way, lowering the barrier to migrate. From there on, there's nothing preventing you from distributing your application code (html/css/javascript/whathaveyou) over ipfs, and make use of WebRTC for user-to-user interactions. Obviously http[s] is going to stick around for a while as it has its use cases (basically anything that deals with a centralized service, from online banking to search engines to apis), but having a secondary, peer to peer means of distributing content and applications would be a major plus. [1] http://ipfs.io http://ipfs.io (they have a working implementation in Go with an HTTP gateway as well as a FUSE filesystem)
- petrosh 11y ago
  WebRTC seems the weakest point to me: WebRTC still needs servers for clients to exchange metadata to coordinate communication http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/ http://www.html5rocks.com/en/tutorials/webrtc/infrastructure...
basicplus2 11y ago
If every one had a wireless node on their house you might get an open web.. Just leaving the problem of connecting to the backbone
- slxh 11y ago
  too bad Google blocks adhoc WiFi on Android... that might have helped too.
deleted 11y ago
[deleted]
EGreg 11y ago
I think it's obvious that the current web is decentralized, but is heavily server-based. At the same time, there is something about propagating applications across these servers... Russia can ban Reddit but they can't ban Wordpress. For the moment, that is what we are working on at http://platform.qbix.com http://platform.qbix.com (and have been for the past 4 years). Making it easy to have a distributed social network the same way bitcoin makes money distributed. Now, how would you take it further and make the web entirely peer to peer, so you wouldn't have to trust servers with your security and politics? You can have additional schemes like http and https, for various methods of delivery and storage. I wrote this FIVE years ago but nothing seems to have been done about it since then: https://news.ycombinator.com/item?id=2023475 https://news.ycombinator.com/item?id=2023475 That would be an easy first step, that would do a lot. It's 2015 and we can't even have XAuth (http://techcrunch.com/2010/04/18/spearheaded-by-meebo-xauth-looks-to-make-social-sites-smarter/ http://techcrunch.com/2010/04/18/spearheaded-by-meebo-xauth-...) in the browser! (We would need a space for storing preferences where websites from any domain could read what was written.)
- jeena 11y ago
  > Russia can ban Reddit but they can't ban Wordpress. That's why I'm part of the IndieWeb https://jeena.net/indieweb https://jeena.net/indieweb or http://indiewebcamp.com/ http://indiewebcamp.com/ The nicest thing about that all is that I don't need to wait until someone else writes a whole new WWW, my own website already is a small part of the whole big thing, I just make my HTML more machine readable and implement something like pingback (but easier, it is called webmentions). With this small building blocks I am, together with others, building a social network which we don't even need to call that.
  - EGreg 11y ago
    IndieWeb looks great. I am going to try to get involved with it. Could you please get in touch with me by email? You can find it at http://qbix.com/about http://qbix.com/about -- just click on "contact". I would like to find out more about this movement ... I'm beginning to participate more in the Offline First, Distributed Web, Mesh Networking and other such movements. Our company's spent 4 years building a platform that would decentralize social networking, because we see it as the catalyst to giving users control of their own data. Most people in the world are just using centralized services these days, and it's directly related to how difficult it is to make a seamless social layer for the web. So I think that we're solving a solution parallel to what bitcoin did with money. A good solution unleashes new possibilities, like the Web itself did, like Email did. Anyway, reach out if you can! - Greg
vezzy-fnord 11y ago
The author's proposals are strikingly similar to Xanadu, right down to suggesting embedded micropayments: https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rules https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rul... Ted Nelson will rise. Sort of, not really.
dweinus 11y ago
I want this for all the reasons they list, but it seems there are huge unanswered questions for anything beyond a permission-less static page. Imagine you are developing a modern web app in the locked open paradigm. Is all system data distributed, including private user data and passwords? The only solution I can come up with is homomorphic encryption, which is not performant enough and still probably leaves a huge timing/structure analysis attack area if anyone can download the database. If I make any mistakes on the database security, the entire DB is already pre-leaked to the world? The final dencryption/encryption happens in client javascript, which is a whole other hornets' nest. Besides that, the implication is that I write my entire system stack in client javascript that is exposed to everyone, including any proprietary algorithms or credentials? Even if that was ok, and the system can live in the user cloud, where does system processing that is independent of user activity (scheduled tasks, etc) happen? Again, I want all of these problems to be solved, but they are nontrivial.
- colordrops 11y ago
  A system like this that uses passwords would be poorly designed. You would use private keys that are never stored online.
- kodablah 11y ago
  Point by point...(sorry for the long post) "homomorphic encryption, which is not performant enough" It is fast enough on a per viewer basis, and in a DHT downloading the database doesn't mean it was all encrypted w/ one key. Each user encrypts his data as needed, or common groups of users encrypt data for each other with each others keys. "If I make any mistakes on the database security" This is why encryption is the underpinning. Sure you can still leak your private key like you can leak an SSH key today. "in client javascript" Nobody would use a distributed network where this was the case. In many cases (i.e. MaidSafe) they are developing a browser plugin for client side to communicate with the backend. "where does system processing that is independent of user activity (scheduled tasks, etc) happen?" Many of these now-being-designed systems have a pay-for-computing concept. Granted several (not all, unless you want to be limited by a single-file-line blockchain forever) have to agree on the results. Give some computing for other computes and get some. As for "scheduled task" timing issues are inherently difficult for these systems and I don't expect the "system" to trigger a job but rather a user to trigger it. Introducing timing into these distributed networks can be hairy. The real problem that needs to be tackled is a way for the common human to hold his private key in his memory or some other non-digitally-retrievable way.
williamcotton 11y ago
It's really exciting to see the these various technologies coming together! We're working on the micropayments for authors and rights-holders aspect of this: https://github.com/blockai/openpublish https://github.com/blockai/openpublish https://github.com/blockai/bitstore-client https://github.com/blockai/bitstore-client
dikaiosune 11y ago
I would love to live in this future -- but where's the incentive for businesses? How do they make more money developing in this way? How do users get more value accessing sites developed in a purely decentralized fashion? How do we avoid JavaScript being the basis for all of this? Interesting (almost exciting) vision, but I don't see why the majority of existing users would move. They just don't get much value out of privacy, versioning, reliability, etc. They get enough of those things out of Gmail, Facebook, et al for their purposes already.
- vezzy-fnord 11y ago
  The interests of society do not revolve around making life as convenient as possible for business. It is the onus on business to adapt itself to new conditions. A distributed web scheme should orient itself around giving its users freedom. All other concerns are secondary.
  - zaphar 11y ago
    This may be philosophically correct. But in a non platonic realm you must somehow fund these sorts of things. If it's not possible to make money on them then it may not be feasible to implement for the masses. The keyword being masses there. There all kinds of distributed p2p web projects out there. None of them appeal to a majority though they are all niche products. Most of these proposals forget the need fund marketing, promotion, and scale.
    - astazangasta 11y ago
      Wikipedia is one of the best sites on the web and didn't think in this way. The whole point of such a system is to remove capital requirements (hardware) from the process of building massive websites. It inherently means less funding is required.
      - api 11y ago
        Hardware is only a tiny, tiny part of the cost of developing a system for general use by huge numbers of people. If you don't believe me, do a price check on cloud hosting and then keep in mind that cloud hosting must be profitable so the prices you're seeing have some non-trivial margin attached. Software development isn't free. Sure, OSS developers can donate their time, but there's a selection process that comes into play. OSS developers do what they do partly because they enjoy it, and so they tend to gravitate to the types of development that is fun. Fun stuff includes deep systems stuff, algorithms, distributed systems, the flashier aspects of UI/UX, machine learning, etc. Fixing stupid bugs that stem from stupid compatibility issues and adding stupid features for stupid use cases is not fun. Debugging edge cases that afflict 1% of your users occasionally is not fun. Porting to popular but crap platforms is not fun. Supporting legacy platforms and APIs is not fun, nor is maintaining backward compatibility. Accessibility features and translations are not fun. Supporting right-to-left language is not fun. Rewriting your entire already-working app to support the next web fad (e.g. "responsive mobile design") is not fun. I could go on forever. There is a really really really long tail of these horrors. This is why OSS rules in the systems/algorithms/etc. space but drools when you get close to the user. This is why every major end user OS, site, or platform is commercial. People have to be paid to torture themselves with that stuff. Yet these sorts of problems are precisely the ones that make the difference between something only geeks (with time on their hands) would want to use and something regular people who aren't computer experts would want to use (or computer experts without time on their hands). For a distributed, decentralized system to challenge the silos of the web, it would absolutely need funding. Eliminating hardware and bandwidth costs is easy; eliminating HR costs is not. There will never be a volunteer-developed mainstream platform for the same reason there are no volunteer-developed mainstream anything elses. To make a truly polished product of any kind requires pain. Disclosure: I currently run a distributed networking / SDN startup. The core technology has been up and running for years with few modifications. It started as a side project so I haven't kept careful account, but I'd easily estimate that upwards of 90% of the development time spent on this project has been on trying to get it to the point that it can install and run trouble-free on multiple platforms and is easy enough for mere mortals to use. Getting the core plumbing, crypto, etc. working was the most fun and the most intellectually challenging, but it was a huge minority of time spent. Had there been no chance of a commercial application I would have stopped there and it'd be yet another piece of interesting GitHub networking orphanware, because for the love of God who would voluntarily try to port such a thing to Windows?!? let alone Android (It's in C, so enjoy your JNI pain).
charbz 11y ago
Data synchronization and Memory management is a major flaw in the concept of a distributed web as described by this article. Is the author suggesting taking all application data that exists on all web servers today, and hosting it on each device connected to the network (billions of devices) ?
saintx 11y ago
A good roadmap for a new distributed web should be broken down by OSI model layer, showing what protocols and technologies exist that need to be replaced, what levels of the OSI model they span, and identifies single points of failure lower in the stack that must be accommodated. Too few people understand how brittle the web is by its reliance on the "magical" underpinnings of the Internet continuing to "just work". For example, let's say we want privacy, anonymity and high availability for something fundamental like name lookups. It's not enough to simply replace DNS with namecoin (L7), if there's a critical vulnerability in openssl on linux that could force a fork in the network, possibly leading to existing blocks getting orphaned (L6), if every single session that goes through AT&T gets captured, and the corresponding netflow stored in perpetuity for later analysis and deanonymization (L5), if this application's traffic could be used for reflection amplification attacks (L4) due to host address spoofing (L3). One might try to get around those issues by direct transmission of traffic between network endpoints (asynchronous peer-to-peer ad hoc wireless networks via smartphones or home radio beacons, for example), but then you not only need to deal with MAC address spoofing and VLAN circumvention, (L2) but with radio signal interference from all the noisy radios turned up to max broadcast volume, shouting over one another, trying to be heard (L1) and accomplishing little more than forcing TCP retransmissions higher up in the stack. And really what's the point, when you can't even trust that the physical radios in your phone or modem aren't themselves vulnerable to their fundamentally insecure baseband processor and its proprietary OS? Turns out, what you were relying on to be "just a radio" has its own CPU and operating system with their own vulnerabilities. Solving this from the top down with a "killer app" is impossible without addressing each layer of the protocol stack. Each layer in the network ecosystem is under constant attack. Every component is itself vulnerable to weaknesses in all the layers above and below it. Vulnerabilities in the top layers can be used to saturate and overwhelm the bottom layers (like when Wordpress sites are used to commit HTTP reflection and amplification attacks), and vulnerabilities in the lower layers can be used to subvert, expose, and undermine the workings of the layers above them. The stuff in the middle (switches) are under constant threat of misuse from weaknesses both above AND below. It might be tempting for an app developer to read this blog post and think "Oh wow, what a novel idea! Why is nobody doing this?" But in reality, legions of security and network researchers, as well as system, network, and software engineers around the world toil daily to uncover and address the core vulnerabilities that hinder these sorts of efforts.
sebastianconcpt 11y ago
Well in this regard, I'm pretty mindblown by the possibilities of https://ethereum.org/ https://ethereum.org/
Titanous 11y ago
I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. It would allow for a lot of interesting experimentation and research.
- nekopa 11y ago
  Is that even possible? I don't know the latest size of the IA, but it must be ridiculously huge by now, (1 billion pages a week added) bandwidth cost would be massive. Maybe they could offer a mail-us-a-multi-petabyte-hdd service... Returned a few weeks later full of data :)
  - Titanous 11y ago
    It's totally possible, they already have the infrastructure in place and 14PB of data available for download. Unfortunately the Wayback Machine data is not currently exposed publicly.
    - nekopa 11y ago
      Why do you think that is? It seems like they are really open with most of their stuff, so why haven't they exposed the wayback with an api? Then again, wouldn't it be pretty trivial to scrape? (I say this as I'm working on a hellish scraping project, and the wayback machine seems like it would be a walk in the park to scrape)
- justin66 11y ago
  > I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. Have you asked them? Did they refuse outright?
  - Titanous 11y ago
    I have not personally asked, but I think the Archive Team has. I don't know the reasons behind the policy.
omouse 11y ago
That's been the goal of the Freenet project for a while, to build a distributed encrypted network protocol. It distributes storage and processing which is why full encryption is necessary; you don't want 10 people reading your email when it's distributed across their machines. The challenge for Freenet has been speed and fun. To have something like Facebook you have to download a JAR plugin for Freenet that adds that capability. That's not fun. The speed is slow because of the encryption and constant syncing. It might be better to look at the MediaGoblin and Pump.io (and StatusNet to some extent) for ideas on federated platforms. The challenge there again is fun; it isn't fun to set things up.
- heywire 11y ago
  Am I the only one who is scared to even try things like Freenet and Tor, for the risk that someone will somehow transmit illegal content across my connection? I'm not talking about content piracy, but the types of things that I don't even want to type for fear of having the terms associated with my username.
  - omouse 11y ago
    First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for the cops to know your machine was used to store the drug listings, the cops would have to spy on your machine and crack the encryption of the Freenet protocol and essentially monitor it. This is why undercover work is important to the police. If no one reports you for the crime of buying drugs and no one discovers the drugs in transit, then the police don't know what's happening. The only way to catch mobsters was through some undercover work and hoping that someone in the criminal network would squeal. If one criminal says the other 10 criminals actually had a hand in committing a crime, the police have more to investigate and can build a case. If you're not buying drugs or selling drugs and the data related to the drug listings is encrypted when stored on your machine and encrypted when served from your machine, you may be unknowingly helping a criminal to buy or sell drugs. But I'm not sure how that's discoverable by the police and I'm not sure how it would be turned into a criminal investigation. By the argument that you're allowing this, then all ISPs and cell providers are in big trouble because they also enable drug dealing. There are some horror stories for Tor node operators though.
nopcode 11y ago
Just replacing DNS with a decentralised alternative would be a big step, yet appeared to be an impossible one (zookos triangle). Something this big requires everyone using the internet to switch to the new system or it will never work, and that will never happen. It's the dancing pig problem. We can't go back on the decisions that have been made, only go forward.
LukeB42 11y ago
I've been thinking of how to decentralise the web as-is since 2011, the current development branch for this perspective on it is here: https://github.com/LukeB42/Uroko/tree/development https://github.com/LukeB42/Uroko/tree/development It's basically a collaborative caching proxy. One process is a proxy that can also coordinate multiple users editing the same page and a subprocess acts as a DHT node. You can use a raft-like log of hashes of pubkey,content and the previous hash to keep a history of edits in the network. the hard part is this: How do you trust the validity of a singular node having a url you're requesting? It entails a rating system, and then it becomes the byzantine generals problem where the overlay network should be able to tolerate up to a third of its malicious nodes saying they're all trustworthy. Feedback/any help would be much appreciated.
- notduncansmith 11y ago
  So the "log of hashes of pubkey,content and the previous hash" is conceptually similar to a blockchain, and I think reading into how that works (consensus, trust, etc) would give you some insight into the issues you're describing. You may also find the IPFS project of interest: http://ipfs.io/ http://ipfs.io/
  - LukeB42 11y ago
    I've found IPFS very interesting and reccommended it to peers, but it lacks the collaborative editing aspect. Trusting the initial public offering of a resource is still an interesting issue. IPFS is content-addressable by hash, addresses map to their content in a computable way. The idea for the distributed hash table in Uroko is that the keys are existing URLs. Imagine thousands of peers all saying they have a new page on the domain "google.com" and you can see what makes this a fun problem to solve.
    - notduncansmith 11y ago
      > Trusting the initial public offering of a resource is still an interesting issue. As you mention, resources on IPFS are addressed by the hash, so I'm curious what you mean by "trust" here - do you mean that you can't trust the accuracy/validity of the content? I would assume that content on this kind of network is signed by the publishing party, so if the signature checks out against your PKI, you can trust the content. I'm also curious as to why Git (or a spiritually similar adaptation) doesn't fit the needs of what you have in mind. Come to think of it, I don't think I see the use-case for Uroko - would you mind explaining?
dubwubz 11y ago
It's too late for that. What was once the internet is now basically glorified cable TV. At this point, it's pretty much inevitable that it's going full Disney or bust. Hopefully bust.
bobajeff 11y ago
I agree with one of the comments on that page that decentralizing the Internet is fundamental to decentralizing the Web. So in my mind the problems that need to be solved are: Information-Centric Networking > Unstructured Mesh Networking > Distributed Data Storage > P2P Information Retrieval
acd 11y ago
To enable p2p web Use ipfs or morph.is
Animats 11y ago
Kahle's approach works only for static content. It's not hard to distribute static content; BitTorrent does it just fine. The Internet Archive stores static content. Kahle thinks in terms of static content, because that's what the Internet Archive does. But it's less of the Web today. Despite that, it's good to have a way to distribute static content. Academic publishing, after all, is almost all static content. That should be widely distributed. It's not like academic journals pay their authors. There's the problem that distributing content means someone else pays for storing and serving it. This is part of what killed USENET, once the binary groups (mostly pirated stuff and porn) became huge. There's a scaling problem with replication. Federated networks are interesting, and there are several federated social networks. A few even have a number of servers in two digits. You could have a federated Facebook replacement that costs each user under a dollar a month at current hosting prices. No ads. The concept is not getting any traction. Kahle wants a system with "easy mechanisms for readers to pay writers." That's either micropayments or an app store, both of which are worse than the current Web.
- gglitch 11y ago
  Why are micropayments worse than the current web? People have differing opinions on the advertising-pays-for-content-so-don't-block-it issue, but are you referring to something technical?
  - Animats 11y ago
    No, just the general failure of the pay-to-read model. Other than the New York Times, the Wall Street Journal, and the Economist, few general publications with a paywall make money. They all have large, worldwide reporting staffs. Nobody is going to pay to read your blog. Pando Daily is trying pay-to-read. It's too soon to tell how that will work out.
    - scholia 11y ago
      Text is a rather small part of web traffic. Most bandwidth nowadays is used to deliver multimedia content (music and movies), much of it in real time. Several people have written articles about how the Internet has been and is being changed to work as a more efficient video distribution network, which is a long way from the original idea..... Some of that is paid for by subscriptions (eg Netflix, Spotify) though I assume the vast majority is -- like most radio and television -- paid for by advertising.
schoen 11y ago
I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community). I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers in the P2P networks and record detailed information about the IP addresses of peers that uploaded and downloaded particular files. There are often paradoxes where decentralization helps privacy and anonymity in some ways but harms it in others -- for example, if you run your own mail server instead of using Gmail, then you've prevented Google from knowing who communicates with whom, but allowed a network adversary to learn that information directly, where the network adversary might not know the messaging relationships if everyone on the network used Gmail. I guess a related point is that information about who is doing what online exists somewhere by default, unless careful privacy engineering reduces the amount of information that's out there. Making the simplest kinds of architectural changes could just shift the location where the information exists, for example from Google or Yahoo or Amazon to dozens of random strangers, some of whom might be working for an adversary.
- api 11y ago
  The only mechanism I'm aware of that truly allows anonymity over your own connection (or a connection that can be tied to you) is onion routing. On top of that, you must do it from a separate device or isolated VM to prevent hardware fingerprinting. Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.
  - phkahler 11y ago
    >> The only mechanism I'm aware of that truly allows anonymity... We have a need for both solid anonymity and zero anonymity. I think the first step is to be able to authenticate whom you are communicating with, and to reach them without a central authority. After that, you can choose to strip identifying information, or build a web of trust, or anything else. I think privacy can be built on top of an authenticated net, but the reverse is probably not possible. Today we have neither.
Sleaker 11y ago
I was liking what the article was suggesting but then it shamelessly plugged BitTorrent inc. which is one of those 'big companies' you don't want touching anything related to privacy or freedom.
jokoon 11y ago
Cool, that's a nice way of promoting those technologies since many don't understand them. I wish those things would land on the IETF board. I wonder what snowden think about those. I would surely make things much harder for the NSA to do massive surveillance.

221 points by hachiya 11y ago

I'm all for this, and consider it to be inevitable in the long run. In the short term, however, it seems like the major hurdle will be getting one of these projects into the mainstream: for the most part, the web already does what most people want it to do, and those people aren't going to be bothered to install a new web browser so that they can do things they're already doing. Especially if it lacks the features, performance, or ease-of-use of their current browser. So, how do we address this? Is there a "killer app" for the distributed web that will motivate people to move to it? Can we use existing web tech like Web-RTC to bootstrap the system? Maybe a workable avenue is mobile, where people are pretty comfortable installing new applications - what if we built the next social network into an app based on the distributed web? I don't know the answer, but I'd love to hear any ideas/brainstorming you clever people have to offer.

lsjroberts 11y ago
You may be interested in https://ind.ie/about/vision/ https://ind.ie/about/vision/
- molsongolden 11y ago
  https://indiewebcamp.com/ https://indiewebcamp.com/ is a bit more interesting and open.
gagege 11y ago
I love the idea of doing a distributed social network via a mobile app. Being a native app, it could be kind of seamless as well, for the user.
mtgx 11y ago
Going mainstream would probably require something like Firefox or at least Ubuntu integrating IPFS.
simon_vetter 11y ago
Implementing other protocol handlers into existing web browsers and operating systems would be a good start IMO. Taking ipfs/ipns [1] as an example, having handlers inside web browsers would allow people to link from http[s]:// to ipfs:// and vice versa in a seamless way, lowering the barrier to migrate. From there on, there's nothing preventing you from distributing your application code (html/css/javascript/whathaveyou) over ipfs, and make use of WebRTC for user-to-user interactions. Obviously http[s] is going to stick around for a while as it has its use cases (basically anything that deals with a centralized service, from online banking to search engines to apis), but having a secondary, peer to peer means of distributing content and applications would be a major plus. [1] http://ipfs.io http://ipfs.io (they have a working implementation in Go with an HTTP gateway as well as a FUSE filesystem)
petrosh 11y ago
WebRTC seems the weakest point to me: WebRTC still needs servers for clients to exchange metadata to coordinate communication http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/ http://www.html5rocks.com/en/tutorials/webrtc/infrastructure...

basicplus2 11y ago

If every one had a wireless node on their house you might get an open web.. Just leaving the problem of connecting to the backbone

slxh 11y ago
too bad Google blocks adhoc WiFi on Android... that might have helped too.

deleted 11y ago

[deleted]

EGreg 11y ago

I think it's obvious that the current web is decentralized, but is heavily server-based. At the same time, there is something about propagating applications across these servers... Russia can ban Reddit but they can't ban Wordpress. For the moment, that is what we are working on at http://platform.qbix.com http://platform.qbix.com (and have been for the past 4 years). Making it easy to have a distributed social network the same way bitcoin makes money distributed. Now, how would you take it further and make the web entirely peer to peer, so you wouldn't have to trust servers with your security and politics? You can have additional schemes like http and https, for various methods of delivery and storage. I wrote this FIVE years ago but nothing seems to have been done about it since then: https://news.ycombinator.com/item?id=2023475 https://news.ycombinator.com/item?id=2023475 That would be an easy first step, that would do a lot. It's 2015 and we can't even have XAuth (http://techcrunch.com/2010/04/18/spearheaded-by-meebo-xauth-looks-to-make-social-sites-smarter/ http://techcrunch.com/2010/04/18/spearheaded-by-meebo-xauth-...) in the browser! (We would need a space for storing preferences where websites from any domain could read what was written.)

jeena 11y ago
> Russia can ban Reddit but they can't ban Wordpress. That's why I'm part of the IndieWeb https://jeena.net/indieweb https://jeena.net/indieweb or http://indiewebcamp.com/ http://indiewebcamp.com/ The nicest thing about that all is that I don't need to wait until someone else writes a whole new WWW, my own website already is a small part of the whole big thing, I just make my HTML more machine readable and implement something like pingback (but easier, it is called webmentions). With this small building blocks I am, together with others, building a social network which we don't even need to call that.
- EGreg 11y ago
  IndieWeb looks great. I am going to try to get involved with it. Could you please get in touch with me by email? You can find it at http://qbix.com/about http://qbix.com/about -- just click on "contact". I would like to find out more about this movement ... I'm beginning to participate more in the Offline First, Distributed Web, Mesh Networking and other such movements. Our company's spent 4 years building a platform that would decentralize social networking, because we see it as the catalyst to giving users control of their own data. Most people in the world are just using centralized services these days, and it's directly related to how difficult it is to make a seamless social layer for the web. So I think that we're solving a solution parallel to what bitcoin did with money. A good solution unleashes new possibilities, like the Web itself did, like Email did. Anyway, reach out if you can! - Greg

vezzy-fnord 11y ago

The author's proposals are strikingly similar to Xanadu, right down to suggesting embedded micropayments: https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rules https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rul... Ted Nelson will rise. Sort of, not really.

dweinus 11y ago

I want this for all the reasons they list, but it seems there are huge unanswered questions for anything beyond a permission-less static page. Imagine you are developing a modern web app in the locked open paradigm. Is all system data distributed, including private user data and passwords? The only solution I can come up with is homomorphic encryption, which is not performant enough and still probably leaves a huge timing/structure analysis attack area if anyone can download the database. If I make any mistakes on the database security, the entire DB is already pre-leaked to the world? The final dencryption/encryption happens in client javascript, which is a whole other hornets' nest. Besides that, the implication is that I write my entire system stack in client javascript that is exposed to everyone, including any proprietary algorithms or credentials? Even if that was ok, and the system can live in the user cloud, where does system processing that is independent of user activity (scheduled tasks, etc) happen? Again, I want all of these problems to be solved, but they are nontrivial.

colordrops 11y ago
A system like this that uses passwords would be poorly designed. You would use private keys that are never stored online.
kodablah 11y ago
Point by point...(sorry for the long post) "homomorphic encryption, which is not performant enough" It is fast enough on a per viewer basis, and in a DHT downloading the database doesn't mean it was all encrypted w/ one key. Each user encrypts his data as needed, or common groups of users encrypt data for each other with each others keys. "If I make any mistakes on the database security" This is why encryption is the underpinning. Sure you can still leak your private key like you can leak an SSH key today. "in client javascript" Nobody would use a distributed network where this was the case. In many cases (i.e. MaidSafe) they are developing a browser plugin for client side to communicate with the backend. "where does system processing that is independent of user activity (scheduled tasks, etc) happen?" Many of these now-being-designed systems have a pay-for-computing concept. Granted several (not all, unless you want to be limited by a single-file-line blockchain forever) have to agree on the results. Give some computing for other computes and get some. As for "scheduled task" timing issues are inherently difficult for these systems and I don't expect the "system" to trigger a job but rather a user to trigger it. Introducing timing into these distributed networks can be hairy. The real problem that needs to be tackled is a way for the common human to hold his private key in his memory or some other non-digitally-retrievable way.

williamcotton 11y ago

It's really exciting to see the these various technologies coming together! We're working on the micropayments for authors and rights-holders aspect of this: https://github.com/blockai/openpublish https://github.com/blockai/openpublish https://github.com/blockai/bitstore-client https://github.com/blockai/bitstore-client

dikaiosune 11y ago

I would love to live in this future -- but where's the incentive for businesses? How do they make more money developing in this way? How do users get more value accessing sites developed in a purely decentralized fashion? How do we avoid JavaScript being the basis for all of this? Interesting (almost exciting) vision, but I don't see why the majority of existing users would move. They just don't get much value out of privacy, versioning, reliability, etc. They get enough of those things out of Gmail, Facebook, et al for their purposes already.

vezzy-fnord 11y ago
The interests of society do not revolve around making life as convenient as possible for business. It is the onus on business to adapt itself to new conditions. A distributed web scheme should orient itself around giving its users freedom. All other concerns are secondary.
- zaphar 11y ago
  This may be philosophically correct. But in a non platonic realm you must somehow fund these sorts of things. If it's not possible to make money on them then it may not be feasible to implement for the masses. The keyword being masses there. There all kinds of distributed p2p web projects out there. None of them appeal to a majority though they are all niche products. Most of these proposals forget the need fund marketing, promotion, and scale.
  - astazangasta 11y ago
    Wikipedia is one of the best sites on the web and didn't think in this way. The whole point of such a system is to remove capital requirements (hardware) from the process of building massive websites. It inherently means less funding is required.
    - api 11y ago
      Hardware is only a tiny, tiny part of the cost of developing a system for general use by huge numbers of people. If you don't believe me, do a price check on cloud hosting and then keep in mind that cloud hosting must be profitable so the prices you're seeing have some non-trivial margin attached. Software development isn't free. Sure, OSS developers can donate their time, but there's a selection process that comes into play. OSS developers do what they do partly because they enjoy it, and so they tend to gravitate to the types of development that is fun. Fun stuff includes deep systems stuff, algorithms, distributed systems, the flashier aspects of UI/UX, machine learning, etc. Fixing stupid bugs that stem from stupid compatibility issues and adding stupid features for stupid use cases is not fun. Debugging edge cases that afflict 1% of your users occasionally is not fun. Porting to popular but crap platforms is not fun. Supporting legacy platforms and APIs is not fun, nor is maintaining backward compatibility. Accessibility features and translations are not fun. Supporting right-to-left language is not fun. Rewriting your entire already-working app to support the next web fad (e.g. "responsive mobile design") is not fun. I could go on forever. There is a really really really long tail of these horrors. This is why OSS rules in the systems/algorithms/etc. space but drools when you get close to the user. This is why every major end user OS, site, or platform is commercial. People have to be paid to torture themselves with that stuff. Yet these sorts of problems are precisely the ones that make the difference between something only geeks (with time on their hands) would want to use and something regular people who aren't computer experts would want to use (or computer experts without time on their hands). For a distributed, decentralized system to challenge the silos of the web, it would absolutely need funding. Eliminating hardware and bandwidth costs is easy; eliminating HR costs is not. There will never be a volunteer-developed mainstream platform for the same reason there are no volunteer-developed mainstream anything elses. To make a truly polished product of any kind requires pain. Disclosure: I currently run a distributed networking / SDN startup. The core technology has been up and running for years with few modifications. It started as a side project so I haven't kept careful account, but I'd easily estimate that upwards of 90% of the development time spent on this project has been on trying to get it to the point that it can install and run trouble-free on multiple platforms and is easy enough for mere mortals to use. Getting the core plumbing, crypto, etc. working was the most fun and the most intellectually challenging, but it was a huge minority of time spent. Had there been no chance of a commercial application I would have stopped there and it'd be yet another piece of interesting GitHub networking orphanware, because for the love of God who would voluntarily try to port such a thing to Windows?!? let alone Android (It's in C, so enjoy your JNI pain).

charbz 11y ago

Data synchronization and Memory management is a major flaw in the concept of a distributed web as described by this article. Is the author suggesting taking all application data that exists on all web servers today, and hosting it on each device connected to the network (billions of devices) ?

saintx 11y ago

A good roadmap for a new distributed web should be broken down by OSI model layer, showing what protocols and technologies exist that need to be replaced, what levels of the OSI model they span, and identifies single points of failure lower in the stack that must be accommodated. Too few people understand how brittle the web is by its reliance on the "magical" underpinnings of the Internet continuing to "just work". For example, let's say we want privacy, anonymity and high availability for something fundamental like name lookups. It's not enough to simply replace DNS with namecoin (L7), if there's a critical vulnerability in openssl on linux that could force a fork in the network, possibly leading to existing blocks getting orphaned (L6), if every single session that goes through AT&T gets captured, and the corresponding netflow stored in perpetuity for later analysis and deanonymization (L5), if this application's traffic could be used for reflection amplification attacks (L4) due to host address spoofing (L3). One might try to get around those issues by direct transmission of traffic between network endpoints (asynchronous peer-to-peer ad hoc wireless networks via smartphones or home radio beacons, for example), but then you not only need to deal with MAC address spoofing and VLAN circumvention, (L2) but with radio signal interference from all the noisy radios turned up to max broadcast volume, shouting over one another, trying to be heard (L1) and accomplishing little more than forcing TCP retransmissions higher up in the stack. And really what's the point, when you can't even trust that the physical radios in your phone or modem aren't themselves vulnerable to their fundamentally insecure baseband processor and its proprietary OS? Turns out, what you were relying on to be "just a radio" has its own CPU and operating system with their own vulnerabilities. Solving this from the top down with a "killer app" is impossible without addressing each layer of the protocol stack. Each layer in the network ecosystem is under constant attack. Every component is itself vulnerable to weaknesses in all the layers above and below it. Vulnerabilities in the top layers can be used to saturate and overwhelm the bottom layers (like when Wordpress sites are used to commit HTTP reflection and amplification attacks), and vulnerabilities in the lower layers can be used to subvert, expose, and undermine the workings of the layers above them. The stuff in the middle (switches) are under constant threat of misuse from weaknesses both above AND below. It might be tempting for an app developer to read this blog post and think "Oh wow, what a novel idea! Why is nobody doing this?" But in reality, legions of security and network researchers, as well as system, network, and software engineers around the world toil daily to uncover and address the core vulnerabilities that hinder these sorts of efforts.

sebastianconcpt 11y ago

Well in this regard, I'm pretty mindblown by the possibilities of https://ethereum.org/ https://ethereum.org/

Titanous 11y ago

I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. It would allow for a lot of interesting experimentation and research.

nekopa 11y ago
Is that even possible? I don't know the latest size of the IA, but it must be ridiculously huge by now, (1 billion pages a week added) bandwidth cost would be massive. Maybe they could offer a mail-us-a-multi-petabyte-hdd service... Returned a few weeks later full of data :)
- Titanous 11y ago
  It's totally possible, they already have the infrastructure in place and 14PB of data available for download. Unfortunately the Wayback Machine data is not currently exposed publicly.
  - nekopa 11y ago
    Why do you think that is? It seems like they are really open with most of their stuff, so why haven't they exposed the wayback with an api? Then again, wouldn't it be pretty trivial to scrape? (I say this as I'm working on a hellish scraping project, and the wayback machine seems like it would be a walk in the park to scrape)
justin66 11y ago
> I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. Have you asked them? Did they refuse outright?
- Titanous 11y ago
  I have not personally asked, but I think the Archive Team has. I don't know the reasons behind the policy.

omouse 11y ago

That's been the goal of the Freenet project for a while, to build a distributed encrypted network protocol. It distributes storage and processing which is why full encryption is necessary; you don't want 10 people reading your email when it's distributed across their machines. The challenge for Freenet has been speed and fun. To have something like Facebook you have to download a JAR plugin for Freenet that adds that capability. That's not fun. The speed is slow because of the encryption and constant syncing. It might be better to look at the MediaGoblin and Pump.io (and StatusNet to some extent) for ideas on federated platforms. The challenge there again is fun; it isn't fun to set things up.

heywire 11y ago
Am I the only one who is scared to even try things like Freenet and Tor, for the risk that someone will somehow transmit illegal content across my connection? I'm not talking about content piracy, but the types of things that I don't even want to type for fear of having the terms associated with my username.
- omouse 11y ago
  First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for the cops to know your machine was used to store the drug listings, the cops would have to spy on your machine and crack the encryption of the Freenet protocol and essentially monitor it. This is why undercover work is important to the police. If no one reports you for the crime of buying drugs and no one discovers the drugs in transit, then the police don't know what's happening. The only way to catch mobsters was through some undercover work and hoping that someone in the criminal network would squeal. If one criminal says the other 10 criminals actually had a hand in committing a crime, the police have more to investigate and can build a case. If you're not buying drugs or selling drugs and the data related to the drug listings is encrypted when stored on your machine and encrypted when served from your machine, you may be unknowingly helping a criminal to buy or sell drugs. But I'm not sure how that's discoverable by the police and I'm not sure how it would be turned into a criminal investigation. By the argument that you're allowing this, then all ISPs and cell providers are in big trouble because they also enable drug dealing. There are some horror stories for Tor node operators though.

nopcode 11y ago

Just replacing DNS with a decentralised alternative would be a big step, yet appeared to be an impossible one (zookos triangle). Something this big requires everyone using the internet to switch to the new system or it will never work, and that will never happen. It's the dancing pig problem. We can't go back on the decisions that have been made, only go forward.

LukeB42 11y ago

I've been thinking of how to decentralise the web as-is since 2011, the current development branch for this perspective on it is here: https://github.com/LukeB42/Uroko/tree/development https://github.com/LukeB42/Uroko/tree/development It's basically a collaborative caching proxy. One process is a proxy that can also coordinate multiple users editing the same page and a subprocess acts as a DHT node. You can use a raft-like log of hashes of pubkey,content and the previous hash to keep a history of edits in the network. the hard part is this: How do you trust the validity of a singular node having a url you're requesting? It entails a rating system, and then it becomes the byzantine generals problem where the overlay network should be able to tolerate up to a third of its malicious nodes saying they're all trustworthy. Feedback/any help would be much appreciated.

notduncansmith 11y ago
So the "log of hashes of pubkey,content and the previous hash" is conceptually similar to a blockchain, and I think reading into how that works (consensus, trust, etc) would give you some insight into the issues you're describing. You may also find the IPFS project of interest: http://ipfs.io/ http://ipfs.io/
- LukeB42 11y ago
  I've found IPFS very interesting and reccommended it to peers, but it lacks the collaborative editing aspect. Trusting the initial public offering of a resource is still an interesting issue. IPFS is content-addressable by hash, addresses map to their content in a computable way. The idea for the distributed hash table in Uroko is that the keys are existing URLs. Imagine thousands of peers all saying they have a new page on the domain "google.com" and you can see what makes this a fun problem to solve.
  - notduncansmith 11y ago
    > Trusting the initial public offering of a resource is still an interesting issue. As you mention, resources on IPFS are addressed by the hash, so I'm curious what you mean by "trust" here - do you mean that you can't trust the accuracy/validity of the content? I would assume that content on this kind of network is signed by the publishing party, so if the signature checks out against your PKI, you can trust the content. I'm also curious as to why Git (or a spiritually similar adaptation) doesn't fit the needs of what you have in mind. Come to think of it, I don't think I see the use-case for Uroko - would you mind explaining?

dubwubz 11y ago

It's too late for that. What was once the internet is now basically glorified cable TV. At this point, it's pretty much inevitable that it's going full Disney or bust. Hopefully bust.

bobajeff 11y ago

I agree with one of the comments on that page that decentralizing the Internet is fundamental to decentralizing the Web. So in my mind the problems that need to be solved are: Information-Centric Networking > Unstructured Mesh Networking > Distributed Data Storage > P2P Information Retrieval

acd 11y ago

To enable p2p web Use ipfs or morph.is

Animats 11y ago

Kahle's approach works only for static content. It's not hard to distribute static content; BitTorrent does it just fine. The Internet Archive stores static content. Kahle thinks in terms of static content, because that's what the Internet Archive does. But it's less of the Web today. Despite that, it's good to have a way to distribute static content. Academic publishing, after all, is almost all static content. That should be widely distributed. It's not like academic journals pay their authors. There's the problem that distributing content means someone else pays for storing and serving it. This is part of what killed USENET, once the binary groups (mostly pirated stuff and porn) became huge. There's a scaling problem with replication. Federated networks are interesting, and there are several federated social networks. A few even have a number of servers in two digits. You could have a federated Facebook replacement that costs each user under a dollar a month at current hosting prices. No ads. The concept is not getting any traction. Kahle wants a system with "easy mechanisms for readers to pay writers." That's either micropayments or an app store, both of which are worse than the current Web.

gglitch 11y ago
Why are micropayments worse than the current web? People have differing opinions on the advertising-pays-for-content-so-don't-block-it issue, but are you referring to something technical?
- Animats 11y ago
  No, just the general failure of the pay-to-read model. Other than the New York Times, the Wall Street Journal, and the Economist, few general publications with a paywall make money. They all have large, worldwide reporting staffs. Nobody is going to pay to read your blog. Pando Daily is trying pay-to-read. It's too soon to tell how that will work out.
  - scholia 11y ago
    Text is a rather small part of web traffic. Most bandwidth nowadays is used to deliver multimedia content (music and movies), much of it in real time. Several people have written articles about how the Internet has been and is being changed to work as a more efficient video distribution network, which is a long way from the original idea..... Some of that is paid for by subscriptions (eg Netflix, Spotify) though I assume the vast majority is -- like most radio and television -- paid for by advertising.

schoen 11y ago

I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community). I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers in the P2P networks and record detailed information about the IP addresses of peers that uploaded and downloaded particular files. There are often paradoxes where decentralization helps privacy and anonymity in some ways but harms it in others -- for example, if you run your own mail server instead of using Gmail, then you've prevented Google from knowing who communicates with whom, but allowed a network adversary to learn that information directly, where the network adversary might not know the messaging relationships if everyone on the network used Gmail. I guess a related point is that information about who is doing what online exists somewhere by default, unless careful privacy engineering reduces the amount of information that's out there. Making the simplest kinds of architectural changes could just shift the location where the information exists, for example from Google or Yahoo or Amazon to dozens of random strangers, some of whom might be working for an adversary.

api 11y ago
The only mechanism I'm aware of that truly allows anonymity over your own connection (or a connection that can be tied to you) is onion routing. On top of that, you must do it from a separate device or isolated VM to prevent hardware fingerprinting. Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.
- phkahler 11y ago
  >> The only mechanism I'm aware of that truly allows anonymity... We have a need for both solid anonymity and zero anonymity. I think the first step is to be able to authenticate whom you are communicating with, and to reach them without a central authority. After that, you can choose to strip identifying information, or build a web of trust, or anything else. I think privacy can be built on top of an authenticated net, but the reverse is probably not possible. Today we have neither.

Sleaker 11y ago

I was liking what the article was suggesting but then it shamelessly plugged BitTorrent inc. which is one of those 'big companies' you don't want touching anything related to privacy or freedom.

jokoon 11y ago

Cool, that's a nice way of promoting those technologies since many don't understand them. I wish those things would land on the IETF board. I wonder what snowden think about those. I would surely make things much harder for the NSA to do massive surveillance.