6 ms·

I had a rant about this on Twitter yesterday[0]. Google makes money precisely because the web is centralised. If we moved to P2P systems they could still provi

by benkant 11y ago

I had a rant about this on Twitter yesterday[0]. Google makes money precisely because the web is centralised. If we moved to P2P systems they could still provide an index, but I'd wager far less data would ultimately pass through them. Not to mention that if people were more in control of their data rather than LinkedIn, Facebook, Flickr, YouTube, I bet they'd be less inclined to having it indexed publically simply because they have a choice. There's all sorts of network effects and shitty incentives at play, and it's a shame. My twitter rant (not the whole thing even) > though they have their place, centralised systems reinforce the role of the middle man, which is prime for rent-seeking and lopsided value > in reality networks exist on a continuum between centralised/decentralised. The web makes it difficult to choose the correct degree per case > both centralised and decentralised systems have trust issues, but they are different in kind not magnitude > the incentives are wrong for innovation. Google requires centralised, so HTTP is fine. Ubiquity, so HTML is fine > web developers have spent years becoming skilled in their corner and are incentivised to defend and perpetuate the platform > if you think discoverability, zero-install and sandboxes are only possible on the web, I invite you to consult the literature > we could have decentralised, secure, simple, efficient primitives, but network effects and incentives steer us away > tech solutions are moot unless they incentivise behaviour that leads to better returns for everyone. layers on HTML/HTTP will never do that [0] https://twitter.com/benkant https://twitter.com/benkant

kbenson 11y ago
In what way do you think the web is centralized?
- markbnj 11y ago
  You could probably argue that Google's near-monopoly on search is a form of centralized control.
  - kbenson 11y ago
    As much as people like to bandy that term about, I don't think of (less than) 68% of all searches as a monopoly. Two out of three people is a lot, but it's not nearly enough to force some sort of information control (whether that information is result, or other people exclaiming how much better their search engine is working).
- jacquesm 11y ago
  That's a good question which deserves an honest answer but this comment box is really too small for that and essay sized comments are frowned upon. For starters: navigating the web in the beginning consisted of clicking links which caused you to go from one website to another. This all worked well when (a) the web was small and (b) there were (hardly) no trash pages. Search engines changed that, and once they got 'good enough' the link graph became a mere starting point for crawling the web rather than the way we navigated from site to site. For a little while the link graph was used as a popularity measurement but this too changed (because of the huge number of low value links). Then we got silos. A 'silo' is a bunch of data locked up under a trade between users and large web properties. The trade is 'you give us your content and a bunch of information about yourself and we'll use that content to attract others and to sell ads'. Examples of such silos are Google, Yahoo and Facebook. Finally, if originally (and the internet itself) was strung together by a peer-to-peer approach it turned more and more into a division between producers and consumers, with the producers on the 'server' side and the consumers on the 'client' side. Mobile devices accessing the net further accelerated this trend, right now the only internet (not web) applications that are still peer-to-peer are torrent applications. For the most part the division on the web is complete and hosting a web server on your very powerful cable modem or DSL line would be grounds for termination of your access. Servers are hosted centrally and are operated by companies whereas clients are simply terminals that access the content stored on those servers. I hope that answers your question in enough detail, you could easily write a book about this.
  - kbenson 11y ago
    The internet is, by it's nature, peer to peer and decentralized. Cut a cable, or take out a large networks, the internet will route around it, either quickly (routes converging on a new peer) or slowly (a poorly connected network finding a new upstream to purchase connectivity through). That companies then build on top of this and implement services where they are the middle of both connections does not change this fundamentally, it just adds an optional layer. To assume our connections have upstream bandwidth that is never or rarely used is false. I would argue that we generate more content per-person than ever in history. The seer amount of pictures, videos, webcams, posts and comments is much higher than ever before. Are they hosting it directly from their connections? Usually not, but that's as much a case of being efficient and reaching an audience as it is in companies wanting control over the data. Even then, there are services which are decentralized from that, such as email. It's not efficient to host content yourself. Even the large networks use dedicated CDNs. For the end user, Facebook is a CDN. That said, I agree there is a clear move towards our data and services being handled by fewer, and larger entities, such as Google, Yahoo, Microsoft, Apple, Amazon. But they aren't a single entity, and I don't consider that centralized. Any one of those providers could implode today, and very little of their services could not be picked up by some competitor easily. I don't consider that centralized.
- benkant 11y ago
  It's distributed, but on the continuum of centralised - decentralised it is definitely centralised. How did my comment get from my computer to yours?
  - kbenson 11y ago
    Through a complex interrelationship of distinctly controlled networks that advertise routes and addresses and allow traffic based on complex business relationships (peering). The only case where that's not happening is where we both have the same ISP, and ycombinator happens to be hosted there as well. Running a traceroute from myself to news.ycombinator.com, I count two distinct networks not including my local one, and not including cloudfare. If those networks stopped talking to each other, my packets to hacker news would find another route, assuming my first hop had access to other networks (given time for the networks to determine a new route and my first hop had access to other peers).
    - benkant 11y ago
      We're talking about the web as an application layer protocol. By your definition everything that happens on the internet is decentralised. That's not untrue if you look at it from the point of view of TCP/IP, but that's tangential to the conversation we're having. You seem to be conflating the web with the internet.
      - kbenson 11y ago
        But even by that definition, the web isn't a single application, it's many applications, some of them compartmentalized (search, social), some of them not (email), and some in between (websites/blogs). If an application were centralized, I would expect a single provider you had to use, but instead, where it at least compartmentalized, you have a group or providers. Can you name a single service/application that you expect more than 5% of people use that has only a single provider? For search, you have Google, Yahoo, Bing, and other smaller players. Google is dominant here, but still has less than 68% of the market. For social, Facebook is the dominant player, but you yourself used a different social network to communicate on this subject, and there are many other providers with popularity that ebbs and flows. It's the same with anything I can think of. I'm not sure how this is considered centralized under any definition.
- thomasrossi 11y ago
  Also DNS servers are a pretty good example of centralized internet. Without 8.8.8.8 your browser turns clueless pretty quick:)
  - kbenson 11y ago
    No, my browser doesn't. Google's public DNS has little bearing on how I reach sites, unless I've specifically configured it that way. Either you really don't understand how DNS works, or you are simplifying to the point of just plain being wrong. You could argue that the root servers are too centralized, and that their control constitutes centralized DNS control, but since the only reason they have control is that all the different DNS servers use them as authorities, an argument could also be made that their control is more be convention than anything else, and all it would take is a competitor to ICANN that added some value, and eventually we could have multiple authorities. Whether that would be beneficial or detrimental is another discussion.
- jokoon 11y ago
  The nature of http and websites makes the web centralized: there's always a server, users don't really serve data, it's always stored somewhere. It's true that it's decentralized, that's it's easy to create websites, but in nature, if you shut down dns servers, you shut down 99% of the internet, which inclues HTML website. And I think that a decentralized web might be more easy to index (proof of work system, etc).
  - kbenson 11y ago
    That's not centralized, it's just less decentralized. Centralized and decentralized or on opposite ends of the spectrum. It's possible to be less decentralized and still be very far from centralized. There are many, many different entities providing all sorts of services, so I'm not sure how that portion can be seen as centralized at all. DNS, as you not, is probably the most centralized single point that everything relies on, but they simply have authority because we give them authority. If DNS server adminitrators decided to use different root servers, there's not a lot they can do about that. But I'll concede that authoritative DNS is fairly centralized, given it requires checking with a single authority, but even then, man entities(TLDs) have a say in what that authority says (but not the ultimate say).
    - jokoon 11y ago
      Well you're right, in nature and architecture the internet in decentralized, but the use most users make of it, is centralized. If you look at what internet.org attempted to do, that's actually how the internet is used most of the time. For consumers and most small businesses, internet is centralized. Technically, most of the internet is just http requests, meaning that there will always be this duality of servers and clients. Without web servers and their admins, there is nothing, and that's a form of control in my opinion: you can easily shut down a website.
      - kbenson 11y ago
        I still don't see that. A centralized internet, or event a centralized "web" as has been distinctly defined elsewhere here, implies a single authority. That doesn't exist, and I don't see it existing in the future. Which email provider do you want to use? Pick from hundreds. Which social network do you want to use? Pick from from the tens of candidates. Which blog platform do you want to use, pick from hundreds again. > Without web servers and their admins, there is nothing, and that's a form of control in my opinion: you can easily shut down a website. There are webservers, and admins. That hasn't changed. There's been a shift to larger sites, but there's still plenty of small ones. You sill have the options to put your site at many different locations, or use a platform such as Facebook, Blogger or Wordpress.
simonw 11y ago
"Google makes money precisely because the web is centralised" I see it exactly the other way: Google makes money precisely because the web is decentralised, and hence you can create an invaluable service by crawling it and creating a centralised search index.
- benkant 11y ago
  That's another way to look at it, but you're using decentralised to indicate there are many large nodes- I would say that's simply distributed. I'm using centralised to indicate that communications are via those large nodes, which mostly don't communicate with each other. It's client/server on a massive scale. Just because the servers are public, doesn't make it decentralised.

by benkant 11y ago

kbenson 11y ago

In what way do you think the web is centralized?

markbnj 11y ago
You could probably argue that Google's near-monopoly on search is a form of centralized control.
- kbenson 11y ago
  As much as people like to bandy that term about, I don't think of (less than) 68% of all searches as a monopoly. Two out of three people is a lot, but it's not nearly enough to force some sort of information control (whether that information is result, or other people exclaiming how much better their search engine is working).
jacquesm 11y ago
That's a good question which deserves an honest answer but this comment box is really too small for that and essay sized comments are frowned upon. For starters: navigating the web in the beginning consisted of clicking links which caused you to go from one website to another. This all worked well when (a) the web was small and (b) there were (hardly) no trash pages. Search engines changed that, and once they got 'good enough' the link graph became a mere starting point for crawling the web rather than the way we navigated from site to site. For a little while the link graph was used as a popularity measurement but this too changed (because of the huge number of low value links). Then we got silos. A 'silo' is a bunch of data locked up under a trade between users and large web properties. The trade is 'you give us your content and a bunch of information about yourself and we'll use that content to attract others and to sell ads'. Examples of such silos are Google, Yahoo and Facebook. Finally, if originally (and the internet itself) was strung together by a peer-to-peer approach it turned more and more into a division between producers and consumers, with the producers on the 'server' side and the consumers on the 'client' side. Mobile devices accessing the net further accelerated this trend, right now the only internet (not web) applications that are still peer-to-peer are torrent applications. For the most part the division on the web is complete and hosting a web server on your very powerful cable modem or DSL line would be grounds for termination of your access. Servers are hosted centrally and are operated by companies whereas clients are simply terminals that access the content stored on those servers. I hope that answers your question in enough detail, you could easily write a book about this.
- kbenson 11y ago
  The internet is, by it's nature, peer to peer and decentralized. Cut a cable, or take out a large networks, the internet will route around it, either quickly (routes converging on a new peer) or slowly (a poorly connected network finding a new upstream to purchase connectivity through). That companies then build on top of this and implement services where they are the middle of both connections does not change this fundamentally, it just adds an optional layer. To assume our connections have upstream bandwidth that is never or rarely used is false. I would argue that we generate more content per-person than ever in history. The seer amount of pictures, videos, webcams, posts and comments is much higher than ever before. Are they hosting it directly from their connections? Usually not, but that's as much a case of being efficient and reaching an audience as it is in companies wanting control over the data. Even then, there are services which are decentralized from that, such as email. It's not efficient to host content yourself. Even the large networks use dedicated CDNs. For the end user, Facebook is a CDN. That said, I agree there is a clear move towards our data and services being handled by fewer, and larger entities, such as Google, Yahoo, Microsoft, Apple, Amazon. But they aren't a single entity, and I don't consider that centralized. Any one of those providers could implode today, and very little of their services could not be picked up by some competitor easily. I don't consider that centralized.
benkant 11y ago
It's distributed, but on the continuum of centralised - decentralised it is definitely centralised. How did my comment get from my computer to yours?
- kbenson 11y ago
  Through a complex interrelationship of distinctly controlled networks that advertise routes and addresses and allow traffic based on complex business relationships (peering). The only case where that's not happening is where we both have the same ISP, and ycombinator happens to be hosted there as well. Running a traceroute from myself to news.ycombinator.com, I count two distinct networks not including my local one, and not including cloudfare. If those networks stopped talking to each other, my packets to hacker news would find another route, assuming my first hop had access to other networks (given time for the networks to determine a new route and my first hop had access to other peers).
  - benkant 11y ago
    We're talking about the web as an application layer protocol. By your definition everything that happens on the internet is decentralised. That's not untrue if you look at it from the point of view of TCP/IP, but that's tangential to the conversation we're having. You seem to be conflating the web with the internet.
    - kbenson 11y ago
      But even by that definition, the web isn't a single application, it's many applications, some of them compartmentalized (search, social), some of them not (email), and some in between (websites/blogs). If an application were centralized, I would expect a single provider you had to use, but instead, where it at least compartmentalized, you have a group or providers. Can you name a single service/application that you expect more than 5% of people use that has only a single provider? For search, you have Google, Yahoo, Bing, and other smaller players. Google is dominant here, but still has less than 68% of the market. For social, Facebook is the dominant player, but you yourself used a different social network to communicate on this subject, and there are many other providers with popularity that ebbs and flows. It's the same with anything I can think of. I'm not sure how this is considered centralized under any definition.
thomasrossi 11y ago
Also DNS servers are a pretty good example of centralized internet. Without 8.8.8.8 your browser turns clueless pretty quick:)
- kbenson 11y ago
  No, my browser doesn't. Google's public DNS has little bearing on how I reach sites, unless I've specifically configured it that way. Either you really don't understand how DNS works, or you are simplifying to the point of just plain being wrong. You could argue that the root servers are too centralized, and that their control constitutes centralized DNS control, but since the only reason they have control is that all the different DNS servers use them as authorities, an argument could also be made that their control is more be convention than anything else, and all it would take is a competitor to ICANN that added some value, and eventually we could have multiple authorities. Whether that would be beneficial or detrimental is another discussion.
jokoon 11y ago
The nature of http and websites makes the web centralized: there's always a server, users don't really serve data, it's always stored somewhere. It's true that it's decentralized, that's it's easy to create websites, but in nature, if you shut down dns servers, you shut down 99% of the internet, which inclues HTML website. And I think that a decentralized web might be more easy to index (proof of work system, etc).
- kbenson 11y ago
  That's not centralized, it's just less decentralized. Centralized and decentralized or on opposite ends of the spectrum. It's possible to be less decentralized and still be very far from centralized. There are many, many different entities providing all sorts of services, so I'm not sure how that portion can be seen as centralized at all. DNS, as you not, is probably the most centralized single point that everything relies on, but they simply have authority because we give them authority. If DNS server adminitrators decided to use different root servers, there's not a lot they can do about that. But I'll concede that authoritative DNS is fairly centralized, given it requires checking with a single authority, but even then, man entities(TLDs) have a say in what that authority says (but not the ultimate say).
  - jokoon 11y ago
    Well you're right, in nature and architecture the internet in decentralized, but the use most users make of it, is centralized. If you look at what internet.org attempted to do, that's actually how the internet is used most of the time. For consumers and most small businesses, internet is centralized. Technically, most of the internet is just http requests, meaning that there will always be this duality of servers and clients. Without web servers and their admins, there is nothing, and that's a form of control in my opinion: you can easily shut down a website.
    - kbenson 11y ago
      I still don't see that. A centralized internet, or event a centralized "web" as has been distinctly defined elsewhere here, implies a single authority. That doesn't exist, and I don't see it existing in the future. Which email provider do you want to use? Pick from hundreds. Which social network do you want to use? Pick from from the tens of candidates. Which blog platform do you want to use, pick from hundreds again. > Without web servers and their admins, there is nothing, and that's a form of control in my opinion: you can easily shut down a website. There are webservers, and admins. That hasn't changed. There's been a shift to larger sites, but there's still plenty of small ones. You sill have the options to put your site at many different locations, or use a platform such as Facebook, Blogger or Wordpress.

simonw 11y ago

"Google makes money precisely because the web is centralised" I see it exactly the other way: Google makes money precisely because the web is decentralised, and hence you can create an invaluable service by crawling it and creating a centralised search index.

benkant 11y ago
That's another way to look at it, but you're using decentralised to indicate there are many large nodes- I would say that's simply distributed. I'm using centralised to indicate that communications are via those large nodes, which mostly don't communicate with each other. It's client/server on a massive scale. Just because the servers are public, doesn't make it decentralised.