11 ms·

IPFS 0.5

410 points by willscott 6y ago

imglorp 6y ago
TL;DR to skip the glossy page. > The primary focus of this release was on improving content routing. That is, advertising and finding content. To that end, this release heavily focuses on improving the DHT. > A secondary focus in this release was improving content transfer, our data exchange protocols Other improvements to data store, libp2p, etc. source: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md
xur17 6y ago
> The Inter-Planetary Name System (IPNS), our system for creating mutable links for content addresses, now provides faster naming lookups and has a new experimental pubsub transport to speed up record distribution. Providing an IPNS record is now 30-40x faster in 1K node network simulations! IPNS performance was abysmal last time I tried to use ipfs (took > 30 seconds a fair amount of the time). Curious to see what it's like now.
- jarfil 6y ago
  The ping from Earth to Mars varies between 360 and 2600 seconds, so 30 might still be fast enough for an "interplanetary" thing.
  - quickthrower2 6y ago
    Surely they’d have an edge server eas-earth-1
  - Pick-A-Hill2019 6y ago
    I know your comment could be viewed as being flippant but none the less it made me chuckle.
  - espadrine 6y ago
    Doesn’t IPFS still rely on Kademlia? Since node IDs are random, node lookup may require multiple interplanetary hops, no? For instance, from N1 on Earth to the nearest XORwise node in its routing table which might be N2 on Mars, whose routing table finds the target node N3 on Earth.
    - momack2 6y ago
      We do use Kademlia - but note this release actually runs 2 DHTs - one for LAN connections and one for WAN connections. You could easily imagine a small Mars outpost as the LAN DHT where you can do fast retrieval for all content already available locally without hitting interplanetary lookup times. https://docs-beta.ipfs.io/recent-releases/go-ipfs-0-5/features/#improved-dht https://docs-beta.ipfs.io/recent-releases/go-ipfs-0-5/featur...
      - omginternets 6y ago
        What are your plans WRT Coral? I noticed there's a work-in-progress Go implementation. Is it going to supercede Kademlia eventually?
        whyrusleeping 6y ago
        Yeah, thats roughly the idea. We can evolve the current DHT into something coral-like, or if something better comes along, we will use that.
    - whyrusleeping 6y ago
      In an actual interplanetary setup, we would definitely not be using a single DHT across the entire space. The main idea once we get to that scale is either a different content routing system altogether, or sharded coral style DHTs (https://en.wikipedia.org/wiki/Coral_Content_Distribution_Network https://en.wikipedia.org/wiki/Coral_Content_Distribution_Net...)
    - hecturchi 6y ago
      Tbh, chances are you will find a faster path through earth peers. The query will explore multiple paths in parallel and timeout slow ones...
  - solarkraft 6y ago
    Isn't the point of IPFS exactly to get those pings far down due to the file being hosted on your planet?
mad44 6y ago
Here is a summary of IPFS white paper http://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-content-addressed.html http://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-c... The improvements announced are substantial.
- gunshai 6y ago
  You mean TRON /s Edit : makin jokes about plagiarism people.
LockAndLol 6y ago
Hold up... You can pull docker images from ipfs? How? And do IPNS entries finally last more than 30 seconds? It would be nice to not have to constantly keep a node up just to have an IPNS entry.
- momack2 6y ago
  IPNS records still have a default 24hr lifetime, but republishing your IPNS records is now much MUCH faster, so it should be much cheaper to run a process to republish these regularly. I assume you can also bump up your lifetime to >24hr if you want - but note that will mean nodes not using pubsub (aka getting updates pushed to them proactively) might be slow to get new updates. Here's what's new for IPNS in the release: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#ipns https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#ipn...
- luizfelberti 6y ago
  > You can pull docker images from ipfs? How? Like this https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain... Note that this is also similar to Kraken[1] from Uber, and Dragonfly[2] from Alibaba. Facebook also does container and artifact distribution using BitTorrent, but I can't find a good reference to it. [1] https://eng.uber.com/introducing-kraken/ https://eng.uber.com/introducing-kraken/ [2] https://d7y.io/en-us/ https://d7y.io/en-us/
- moritonal 6y ago
  Self-plug and an experiment only, but here's the blog of how I wrote a Storage Driver for Docker: https://blog.bonner.is/docker-registry-for-ipfs/ https://blog.bonner.is/docker-registry-for-ipfs/
omginternets 6y ago
I have a Go project that imports IPFS and libp2p libraries. What's the recommended way of upgrading those dependencies?
- willscott 6y ago
  The release notes contain a pretty extensive list of what APIs have changed. The path to upgrade the dependencies is probably to run `go get -u <dep>` for the direct dependencies you're including, and then fixing errors that pop up from doing so.
  - hecturchi 6y ago
    Depends what exactly you are importing and for what. I suggest you ask in the forums and post a link to your project. go get -u is dangerous as it updates all subdependencies used to latest. Usually take go-ipfs's go.mod as a guide on what versions to use.
    - omginternets 6y ago
      >go get -u is dangerous as it updates all subdependencies used to latest Yes, this is the problem I initially had; I would end up with a host of incompatible deps (leading me to wonder what the point of go.mod even was ... but I digress...). Sounds like this is going to be one of those super fun dependency hack-jobs :C Ah well, this is the price to pay for playing with beta software!
dzonga 6y ago
anyone know good python ipfs resources + decentralized sync-able embedded databases ?
- j88439h84 6y ago
  https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs is an alternative.
- momack2 6y ago
  orbitdb is a distributed database built on IPFS - looks like they have a python http client: https://github.com/orbitdb/py-orbit-db-http-client https://github.com/orbitdb/py-orbit-db-http-client the py-ipfs-http-client library is also actively maintained (but I think needs some small changes to work with IPFS 0.5): https://github https://github .com/ipfs-shipyard/py-ipfs-http-client/
lachlan-sneff 6y ago
I wish they hadn't picked go. Go isn't a bad language really, but it's a huge pain to integrate a go library into an app written in another language.
- echelon 6y ago
  C or Rust would be perfect. No GC runtime, no name mangling, fast, great ffi story. Rust needs to start edging out Go for new tools. Kubernetes, Envoy, IPFS, etc. would have benefitted from it. It might not have been time four years ago, but it's time now.
  - momack2 6y ago
    There's actually a new Rust-IPFS implementation looking for more contributors. Seems to be moving pretty fast - https://github.com/ipfs-rust/rust-ipfs https://github.com/ipfs-rust/rust-ipfs
  - spsoto 6y ago
    Envoy is written in C++, not Go.
    - echelon 6y ago
      I'm aware. The creator occasionally pokes fun at the Rust folks for arguing that Envoy should have been written in Rust. And rightly so--Rust wasn't mature at the time Envoy was written. If we revisit that now, C++ is the wrong language to start Envoy in today.
      - ncmncm 6y ago
        Citation needed.
        keeganpoppen 6y ago
        a citation is most definitely not needed. it's obviously a fucking opinion. and they are under absolutely no obligation to meet your standard of proof to demonstrate anything to you. clearly citations are not a necessary condition to posting-- this comment has no citations, but yet... hey look-- it's been posted! the horror! classic HN obnoxious "citation needed" commentary strikes again. i swear i'm just gonna keep doing this every time i see this... until i get bored and give up, i guess.
api 6y ago
This uses DHTs. How resilient is it against Sybil attacks? Also how does it work under global netsplit conditions, like if someone borks or attacks BGP in such a way that 1/3 of the world is not reachable? My impression is that DHTs fall down pretty hard under the latter scenario and are also pretty vulnerable to the Sybil scenario if the attacker has enough resources to mount a really serious attack. They're okay for low-value simple stuff that doesn't have much of an intrinsic bounty attached to it (like BitTorrent magnets), but trying to put a "decentralized web" on top of a DHT seems like a scenario where the instant it becomes popular it will get completely shredded for profit (spam, stealing Bitcoin, etc.). My rule of thumb is that anything designed for serious or large scale use (in other words that might get popular) needs to be built to withstand either a "nation state level attacker" threat model or a "Zerg rush of hundreds of thousands of profit motivated black hats" threat model. The Internet today is a war zone because today you can make money and gain power (e.g. by influencing elections) by messing with it.
- willscott 6y ago
  For BGP partitions / Significant network outage conditions: You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you. This is better than centralized solutions face at present - e.g. google not being available in China, and isn't too far off from what you might hope. For Sybils: You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence. For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT.
  - jude- 6y ago
    > You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you. This becomes hugely problematic the minute you start using IPNS. On one side of the split, the name `foo` can resolve to `bar`, but on the other side, it resolves to `baz`. If you're trying to impersonate someone, then a netsplit would make it easy for you to do so (barring an out-of-band content authentication protocol, of course). > You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence. On the contrary, a Sybil node operator can censor arbitrary content in a DHT by inserting their own nodes into the network that are all "closer" to the targeted key range in the key space than the honest nodes. This can be done by crawling the DHT, identifying the honest nodes that route to the targeted key range, and generating node IDs that correspond to key ranges closer than them. Honest nodes will (correctly) proceed to direct lookup requests to the attacker nodes, thereby leading to content censorship. Honest nodes can employ countermeasures to probe the network in order to try and see if/when this is happening, but an attacker node can be adapted to behave like an honest node when another honest node is talking to it. > For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT. Sure, and while this is a good thing, it's ultimately an arms race between the IPFS developers and network attackers who can fool the automated countermeasures. I'm not confident in its long-term ability to fend off attacks on the routing system.
carapace 6y ago
I really really want to use this, I have used it in the past. It works great as far as I could tell except for one thing: bandwidth limits[1]. Now I can do this myself, because I'm all "+337" and what not. (I used trickle as described in the comment[2]. Seemed to work fine. Nice and stable.) But I can't recommend Joe and Jane Consumer to install IPFS and some other thing with a straight face, because they'll say, "Well bittorrent can do it!?" and I don't have a good answer. Maybe there's an opportunity there to rent IPFS VMs to normal people? I dunno. https://github.com/ipfs/go-ipfs/issues/3065 https://github.com/ipfs/go-ipfs/issues/3065 https://github.com/ipfs/go-ipfs/issues/3065#issuecomment-415907797 https://github.com/ipfs/go-ipfs/issues/3065#issuecomment-415...
- nerdponx 6y ago
  Or at least a Docker image if there isn't already?
- tribler 6y ago
  plus IPFS wants to enforce copyrights worldwide: [1,2] Businesses principles plus monthly bandwidth usage matter for real people. [1] https://github.com/ipfs/community/blob/master/code-of-conduct.md#copyright-violations https://github.com/ipfs/community/blob/master/code-of-conduc... [2] https://discuss.ipfs.io/tos#8 https://discuss.ipfs.io/tos#8
  - johntash 6y ago
    I'm not sure if I understand the complaint around this.. If you put a file (that you do not have permission to distribute) on the internet in a public-accessible location, you should expect that someone will want it removed. IPFS is not trying to be an anonymous file-sharing service afaik.
    - kristopolous 6y ago
      The complaint is they shouldn't take a stance or at least one that is a little less firm. The permissions thing, we can all agree if it's a new movie that just came out, ok, yes, don't be spreading that. What if instead it's an academic journal article from 1930 in a publication that ceased operating in say 1940? You also don't have permission for this and it's still also under copyright. The strict interpretation would be "not that one either" while there's also some who say "it's ok, there's nobody to even ask, let historians do research". So some prefer to be grey about it like many are with obscenity and pornography. We don't for example, hide renaissance paintings with exposed beasts away from the public in basements for fear of getting shutdown by the police. There's a spirit of the law as well.
fg6hr 6y ago
IPFS is really just a mix of a torrent tracker with a torrent client, but once IPFS VMs start paying for themselves, people will sign-up in masses. I guess that's what Filecoin is about. Edit: The way I see filecoin working is anyone can post a reward for a file and once the file is provided, the reward is paid. In other words, it's bit like a brokerage that connects downloaders with uploaders. The difficultly is that this brokerage needs to be distributed and resilient.
- rglullis 6y ago
  I am yet to write more about this, but one thing that bothers me with Filecoin: the economics don't add up. The price per GB stored will tend to reach an equilibrium around the commodity price - i.e, costs of disks+computers+electricity+internet. Unless I am missing something, if the price gets higher, more people would buy disks and put them online, bringing the price down. If that is true, it means that a node can only be profitable if you are freeloading. And if you are freeloading, any price you get will be good which means that it tends to go even further down, perhaps even below the commodity cost. I may be missing something, but I really don't see this going beyond techies with spare disks playing around and definitely no way to run a Filecoin node on a VPS profitably.
  - anchpop 6y ago
    That is actually the intent of filecoin. In perfect competition, which is the ideal state for a commodity, there is no economic profit. The reason Amazon and other companies are able to make a profit selling cloud storage is because there is not yet perfect competition. Right now if I wanted to compete with Amazon, I could buy 2 or 3 petabytes of storage space and some bandwidth. But nobody would trust me not to lose their data. This is the differentiation that makes cloud storage less of a commodity than it could be. The goal with filecoin is to make it so you don't have to trust me, just some general guarantees about the filecoin network. If amazon for some reason was selling storage on this network, I could compete with them on equal footing (and whoever sold it more cheaply would win). Many commodities are in close-to-perfect competition. For instance, I don't care if grain comes from England or France, just that it's cheap. While it is very hard to make an economic profit selling grain, many people do farm grain and make enough money to support themselves (The money you pay your workers to survive is one of the costs of producing grain. That's true even if the only worker is you) If you could make a profit by running a filecoin node on a VPS, everyone would do that. By competing with each other you'd all bring the price down and down until it was no longer more profitable than selling any other commodity (and maybe even lower). So you're right to not expect to be able to do that.
aiyodev 6y ago
Can someone provide an example of a service that ipfs enables that can’t be provided on the regular web?
- StreamBright 6y ago
  No, and I was looking hard. IPFS is a slower, no-SLA version of S3 at this stage. People try to sell it with this planetary-scale marketing phrase just like they were trying to sell MongoDB with the web-scale phrase. Once you understand that the use cases for asymmetric uplink powered file sharing are latency and availability you also understand that the use cases are very limited for such storage. Mostly torrent really or cold backup maybe. If disagree please explain why.
- StavrosK 6y ago
  You can publish static content that doesn't go down when the server goes down and scales on its own as the content becomes more popular. It also has hashing built-in, so you can be sure the content you got is what you wanted, Things like distribution of apt packages becomes much more exciting when each computer can choose to redistribute the packages it got to others in the LAN or area, even offline. It's also very interesting to me how all the visitors to your website become servers as well, so content can never be "hugged to death" or links can never go stale, as long as at least one person has the content somewhere on their node. That, to me, is huge, as links now go stale with some regularity. Think of all the Geocities sites (and all versions of them) just existing for ever, regardless if Geocities decided to shut down. For example, here's my site on IPFS: https://ipfs.eternum.io/ipfs/QmVW6JejQkjLnBJacR8qcZi88WNTMwiBwNDMCFDzijtGAw/ https://ipfs.eternum.io/ipfs/QmVW6JejQkjLnBJacR8qcZi88WNTMwi... That can now "never" be lost, as long as someone cares enough about it to visit.
  - aiyodev 6y ago
    That's "how it works", not "what it can be used for". Regular websites can be designed to handle high traffic. Regular websites can make backups. What new thing does ipfs make possible that would make users install the software to use it?
truth_seeker 6y ago
>> Opera became the first major web browser to offer default IPFS support on Android, shortly after Brave started directly embedding the IPFS Companion extension (complete with a built-in js-ipfs node). This means millions of people around the world now have access to the decentralized web built directly into their browsers. Wow. Thank you IPFS community.
- StavrosK 6y ago
  I believe Opera only redirects to a gateway and doesn't implement IPFS itself, though, which is rather less exciting.
  - deleted 6y ago
    [deleted]
fit2rule 6y ago
I really wish there was a C/C++ implementation of IPFS that could be dropped in to a Linux, Windows, MacOS, or iOS cross platform app. I already ported the Go code to iOS once, and it wasn't that painful, but functionally it would be a lot more useful as a C++ code base. I don't build browser apps - I feel there is still a need for native apps especially in this particular space.
StavrosK 6y ago
I run https://www.eternum.io/ https://www.eternum.io/ (an IPFS pinning service) and created Hearth (https://hearth.eternum.io/ https://hearth.eternum.io/, a Dropbox-like way to publish files on IPFS), and this is a very welcome release. IPFS has been a real pain to work with in the past (the node would just consume all RAM and CPU and had to be restarted a lot), but it's been getting better, which is great to see. I really hope it gets good enough to run on everyone's desktop machine, since that's the way IPFS is meant to be deployed (rather than just on gateways). It seems that it doesn't take up too much RAM or CPU now, but it looks like it might be a problem bandwidth-wise, if you host some popular content. Still, great news overall.
jillesvangurp 6y ago
Good progress. I looked at IPFS a few months ago from the point of view of fitting a decentralized file store. The decentralized requirement came primarily from the point of view wanting to fit this in a broader product that is all about decentralized for various reasons. Key problems in this space: - decentralized is something techies obsess about but that has as of yet no business value whatsoever. Customers don't ask for it. Centralized alternatives are generally available and far more mature/easy to manage. The business ecosystem around IPFS is basically not there. - this space is dominated by hobbyists running stuff like this on their personal hardware doing this mostly for idealistic or other non incentivized (i.e. money) reasons. Nothing wrong with this but using it for something real brings a few requirements with it that are basically hard to address currently. - Filecoin has been 'coined' as the solution for this for years but seems nowhere near delivering on it's published roadmap. Last time I checked it had undelivered milestones in the past. As of yet this looks increasingly like something that is a bit dead in the water / a big distraction for coming up with better/alternate solutions. - Uptime guarantees for content are currently basically DYI. Nobody but you cares about your data. If you want your content to stay there, you basically need ... proper file hosting. As incentives and mechanisms for others to agree to host your content (aka pinning in ipfs) are not there, this is hard to fix. - integration with existing centralized solutions is kind of meh/barely supported. We actually looked at using s3 as store for ipfs just so we could onboard customers and give them some decent SLA (bandwidth would be another issue for that). There are some fringe projects on github implementing this but when we looked at it the smallish blocksizes in ipfs are kind of a non starter for using this at scale (think insane numbers of requests to s3). This stuff is not exactly mainstream. Obviously this wouldn't be needed if we could incentivize others to 'pin' content. But we can't currently.
- jonathanstrange 6y ago
  > decentralized is something techies obsess about but that has as of yet no business value whatsoever. I don't really understand that point. If it works, then it has the business value of saving you all running server and maintenance costs. For most larger businesses these costs may be insignificant and easily recovered, but for small businesses with a lot of customers they can make a huge difference. For example, I'm looking into P2P options for implementing a decentralized message forum in a game-like emulator and it would make no sense to even implement this feature with constant running costs for server space. Now getting the decentralized data management to work reliably out-of-the-box from behind various firewalls and different platforms, that's the big problem. So far, none of the libraries I've seen are very easy to use, some require a difficult installation and configuration or you need to your own STUN server or gateway, which kind of defeats the purpose.
kevincox 6y ago
I think IPFS is a great technology, and the protocol appears to be fairly well designed but to be frank, I tried to use go-ipfs and it is a very junky implementation. There are numerous issues that show an overall lack of good quality standards in the codebase. - Pin management is far to simple for any real use case. - GC is a complete GC on an arbitrary threshold. - API is based on the CLI and really weird (ex GET /api/v0/cp?arg={src}&arg={dest}, why don't the parameters have names? Why is it GET?) - They don't prove a way to upload a directory with references to existing IPFS objects, so you need to manually encode the data. - The manual encoding uses go-style names, they seem to have let their language choice leak into the protocol. - Lots of minor issues, such as the API implementing block size limits by simply cutting off the input data at the size limit, this is even before encoding it into the format where the size limit should be calculated and cause data corruption in some cases. - Their one abstract API for creating directories (MFS) has a number of issues. - It is incredibly slow for some reason. - It kindof but not really automatically pins all content recursively references. There are other weird choicess such as their directory sharding using hashing. It isn't clear to me which use case this improves over a btree but someone probably thought it sounded cooler. Additionally the sharding appears to be a single layer of sharding which means that it still has a size limit (just larger). I ended up sinking a ton of time into https://gitlab.com/kevincox/archlinux-ipfs-mirror/ https://gitlab.com/kevincox/archlinux-ipfs-mirror/ and in the end the number of small quirks were incredibly frustrating. I might go back and implement their hash-sharding logic to get the mirror working again but at this point I don't really want to interact with go-ipfs again. I thought it would be an interesting project to get involved in, and I have a lot of expertise that would be valuable however it appears that it is all a bit of a mess which is a real shame. Maybe with more success it can be re-written in cleaner fashion, as I said most of the issues are with the implementation not the protocol so there is definitely hope. I do honestly wish all of the success to the project.
skyde 6y ago
IPFS need a way to get paid for a block over a period of time. Does not need to be real money just token you can use later to pay for your file to be seeded. Basically I host your file and you host my file. With some algorithm to make sure number of seeder for each block never go to 0
- momack2 6y ago
  Check out Filecoin (https://filecoin.io https://filecoin.io) - token for buying/selling file storage for IPFS data. You can earn tokens for renting out your disk space and use them to purchase backup/hosting from others.
woodandsteel 6y ago
I was impressed to read that the IPFS public network scaled 30X in 2019. It sounds like they have gotten over a big hump for usability and utility.

410 points by willscott 6y ago

TL;DR to skip the glossy page. > The primary focus of this release was on improving content routing. That is, advertising and finding content. To that end, this release heavily focuses on improving the DHT. > A secondary focus in this release was improving content transfer, our data exchange protocols Other improvements to data store, libp2p, etc. source: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md

xur17 6y ago

> The Inter-Planetary Name System (IPNS), our system for creating mutable links for content addresses, now provides faster naming lookups and has a new experimental pubsub transport to speed up record distribution. Providing an IPNS record is now 30-40x faster in 1K node network simulations! IPNS performance was abysmal last time I tried to use ipfs (took > 30 seconds a fair amount of the time). Curious to see what it's like now.

jarfil 6y ago
The ping from Earth to Mars varies between 360 and 2600 seconds, so 30 might still be fast enough for an "interplanetary" thing.
- quickthrower2 6y ago
  Surely they’d have an edge server eas-earth-1
- Pick-A-Hill2019 6y ago
  I know your comment could be viewed as being flippant but none the less it made me chuckle.
- espadrine 6y ago
  Doesn’t IPFS still rely on Kademlia? Since node IDs are random, node lookup may require multiple interplanetary hops, no? For instance, from N1 on Earth to the nearest XORwise node in its routing table which might be N2 on Mars, whose routing table finds the target node N3 on Earth.
  - momack2 6y ago
    We do use Kademlia - but note this release actually runs 2 DHTs - one for LAN connections and one for WAN connections. You could easily imagine a small Mars outpost as the LAN DHT where you can do fast retrieval for all content already available locally without hitting interplanetary lookup times. https://docs-beta.ipfs.io/recent-releases/go-ipfs-0-5/features/#improved-dht https://docs-beta.ipfs.io/recent-releases/go-ipfs-0-5/featur...
    - omginternets 6y ago
      What are your plans WRT Coral? I noticed there's a work-in-progress Go implementation. Is it going to supercede Kademlia eventually?
      - whyrusleeping 6y ago
        Yeah, thats roughly the idea. We can evolve the current DHT into something coral-like, or if something better comes along, we will use that.
  - whyrusleeping 6y ago
    In an actual interplanetary setup, we would definitely not be using a single DHT across the entire space. The main idea once we get to that scale is either a different content routing system altogether, or sharded coral style DHTs (https://en.wikipedia.org/wiki/Coral_Content_Distribution_Network https://en.wikipedia.org/wiki/Coral_Content_Distribution_Net...)
  - hecturchi 6y ago
    Tbh, chances are you will find a faster path through earth peers. The query will explore multiple paths in parallel and timeout slow ones...
- solarkraft 6y ago
  Isn't the point of IPFS exactly to get those pings far down due to the file being hosted on your planet?

mad44 6y ago

Here is a summary of IPFS white paper http://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-content-addressed.html http://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-c... The improvements announced are substantial.

gunshai 6y ago
You mean TRON /s Edit : makin jokes about plagiarism people.

LockAndLol 6y ago

Hold up... You can pull docker images from ipfs? How? And do IPNS entries finally last more than 30 seconds? It would be nice to not have to constantly keep a node up just to have an IPNS entry.

momack2 6y ago
IPNS records still have a default 24hr lifetime, but republishing your IPNS records is now much MUCH faster, so it should be much cheaper to run a process to republish these regularly. I assume you can also bump up your lifetime to >24hr if you want - but note that will mean nodes not using pubsub (aka getting updates pushed to them proactively) might be slow to get new updates. Here's what's new for IPNS in the release: https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#ipns https://github.com/ipfs/go-ipfs/blob/master/CHANGELOG.md#ipn...
luizfelberti 6y ago
> You can pull docker images from ipfs? How? Like this https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ https://blog.ipfs.io/2020-02-14-improved-bitswap-for-contain... Note that this is also similar to Kraken[1] from Uber, and Dragonfly[2] from Alibaba. Facebook also does container and artifact distribution using BitTorrent, but I can't find a good reference to it. [1] https://eng.uber.com/introducing-kraken/ https://eng.uber.com/introducing-kraken/ [2] https://d7y.io/en-us/ https://d7y.io/en-us/
moritonal 6y ago
Self-plug and an experiment only, but here's the blog of how I wrote a Storage Driver for Docker: https://blog.bonner.is/docker-registry-for-ipfs/ https://blog.bonner.is/docker-registry-for-ipfs/

omginternets 6y ago

I have a Go project that imports IPFS and libp2p libraries. What's the recommended way of upgrading those dependencies?

willscott 6y ago
The release notes contain a pretty extensive list of what APIs have changed. The path to upgrade the dependencies is probably to run `go get -u <dep>` for the direct dependencies you're including, and then fixing errors that pop up from doing so.
- hecturchi 6y ago
  Depends what exactly you are importing and for what. I suggest you ask in the forums and post a link to your project. go get -u is dangerous as it updates all subdependencies used to latest. Usually take go-ipfs's go.mod as a guide on what versions to use.
  - omginternets 6y ago
    >go get -u is dangerous as it updates all subdependencies used to latest Yes, this is the problem I initially had; I would end up with a host of incompatible deps (leading me to wonder what the point of go.mod even was ... but I digress...). Sounds like this is going to be one of those super fun dependency hack-jobs :C Ah well, this is the price to pay for playing with beta software!

dzonga 6y ago

anyone know good python ipfs resources + decentralized sync-able embedded databases ?

j88439h84 6y ago
https://tahoe-lafs.org/trac/tahoe-lafs https://tahoe-lafs.org/trac/tahoe-lafs is an alternative.
momack2 6y ago
orbitdb is a distributed database built on IPFS - looks like they have a python http client: https://github.com/orbitdb/py-orbit-db-http-client https://github.com/orbitdb/py-orbit-db-http-client the py-ipfs-http-client library is also actively maintained (but I think needs some small changes to work with IPFS 0.5): https://github https://github .com/ipfs-shipyard/py-ipfs-http-client/

lachlan-sneff 6y ago

I wish they hadn't picked go. Go isn't a bad language really, but it's a huge pain to integrate a go library into an app written in another language.

echelon 6y ago
C or Rust would be perfect. No GC runtime, no name mangling, fast, great ffi story. Rust needs to start edging out Go for new tools. Kubernetes, Envoy, IPFS, etc. would have benefitted from it. It might not have been time four years ago, but it's time now.
- momack2 6y ago
  There's actually a new Rust-IPFS implementation looking for more contributors. Seems to be moving pretty fast - https://github.com/ipfs-rust/rust-ipfs https://github.com/ipfs-rust/rust-ipfs
- spsoto 6y ago
  Envoy is written in C++, not Go.
  - echelon 6y ago
    I'm aware. The creator occasionally pokes fun at the Rust folks for arguing that Envoy should have been written in Rust. And rightly so--Rust wasn't mature at the time Envoy was written. If we revisit that now, C++ is the wrong language to start Envoy in today.
    - ncmncm 6y ago
      Citation needed.
      - keeganpoppen 6y ago
        a citation is most definitely not needed. it's obviously a fucking opinion. and they are under absolutely no obligation to meet your standard of proof to demonstrate anything to you. clearly citations are not a necessary condition to posting-- this comment has no citations, but yet... hey look-- it's been posted! the horror! classic HN obnoxious "citation needed" commentary strikes again. i swear i'm just gonna keep doing this every time i see this... until i get bored and give up, i guess.

api 6y ago

This uses DHTs. How resilient is it against Sybil attacks? Also how does it work under global netsplit conditions, like if someone borks or attacks BGP in such a way that 1/3 of the world is not reachable? My impression is that DHTs fall down pretty hard under the latter scenario and are also pretty vulnerable to the Sybil scenario if the attacker has enough resources to mount a really serious attack. They're okay for low-value simple stuff that doesn't have much of an intrinsic bounty attached to it (like BitTorrent magnets), but trying to put a "decentralized web" on top of a DHT seems like a scenario where the instant it becomes popular it will get completely shredded for profit (spam, stealing Bitcoin, etc.). My rule of thumb is that anything designed for serious or large scale use (in other words that might get popular) needs to be built to withstand either a "nation state level attacker" threat model or a "Zerg rush of hundreds of thousands of profit motivated black hats" threat model. The Internet today is a war zone because today you can make money and gain power (e.g. by influencing elections) by messing with it.

willscott 6y ago
For BGP partitions / Significant network outage conditions: You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you. This is better than centralized solutions face at present - e.g. google not being available in China, and isn't too far off from what you might hope. For Sybils: You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence. For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT.
- jude- 6y ago
  > You may not find content partitioned on the other side of a netsplit from you, but the degraded condition is that you can still query and find content present in the same part of the network as you. This becomes hugely problematic the minute you start using IPNS. On one side of the split, the name `foo` can resolve to `bar`, but on the other side, it resolves to `baz`. If you're trying to impersonate someone, then a netsplit would make it easy for you to do so (barring an out-of-band content authentication protocol, of course). > You've left the attack you're worried about pretty vague. IPFS itself doesn't need to tackle many of the sybil-related issues by being content addressable (so only worrying about availability - not integrity) and not being a discovery platform - so not worrying about spam / influence. On the contrary, a Sybil node operator can censor arbitrary content in a DHT by inserting their own nodes into the network that are all "closer" to the targeted key range in the key space than the honest nodes. This can be done by crawling the DHT, identifying the honest nodes that route to the targeted key range, and generating node IDs that correspond to key ranges closer than them. Honest nodes will (correctly) proceed to direct lookup requests to the attacker nodes, thereby leading to content censorship. Honest nodes can employ countermeasures to probe the network in order to try and see if/when this is happening, but an attacker node can be adapted to behave like an honest node when another honest node is talking to it. > For the remaining degradation attacks - someone overwhelming the DHT with misbehaving nodes - there's been a bunch of work in this release looking at how to score peers and figure out which ones aren't worth keeping in the DHT. Sure, and while this is a good thing, it's ultimately an arms race between the IPFS developers and network attackers who can fool the automated countermeasures. I'm not confident in its long-term ability to fend off attacks on the routing system.

carapace 6y ago

I really really want to use this, I have used it in the past. It works great as far as I could tell except for one thing: bandwidth limits[1]. Now I can do this myself, because I'm all "+337" and what not. (I used trickle as described in the comment[2]. Seemed to work fine. Nice and stable.) But I can't recommend Joe and Jane Consumer to install IPFS and some other thing with a straight face, because they'll say, "Well bittorrent can do it!?" and I don't have a good answer. Maybe there's an opportunity there to rent IPFS VMs to normal people? I dunno. https://github.com/ipfs/go-ipfs/issues/3065 https://github.com/ipfs/go-ipfs/issues/3065 https://github.com/ipfs/go-ipfs/issues/3065#issuecomment-415907797 https://github.com/ipfs/go-ipfs/issues/3065#issuecomment-415...

nerdponx 6y ago
Or at least a Docker image if there isn't already?
tribler 6y ago
plus IPFS wants to enforce copyrights worldwide: [1,2] Businesses principles plus monthly bandwidth usage matter for real people. [1] https://github.com/ipfs/community/blob/master/code-of-conduct.md#copyright-violations https://github.com/ipfs/community/blob/master/code-of-conduc... [2] https://discuss.ipfs.io/tos#8 https://discuss.ipfs.io/tos#8
- johntash 6y ago
  I'm not sure if I understand the complaint around this.. If you put a file (that you do not have permission to distribute) on the internet in a public-accessible location, you should expect that someone will want it removed. IPFS is not trying to be an anonymous file-sharing service afaik.
  - kristopolous 6y ago
    The complaint is they shouldn't take a stance or at least one that is a little less firm. The permissions thing, we can all agree if it's a new movie that just came out, ok, yes, don't be spreading that. What if instead it's an academic journal article from 1930 in a publication that ceased operating in say 1940? You also don't have permission for this and it's still also under copyright. The strict interpretation would be "not that one either" while there's also some who say "it's ok, there's nobody to even ask, let historians do research". So some prefer to be grey about it like many are with obscenity and pornography. We don't for example, hide renaissance paintings with exposed beasts away from the public in basements for fear of getting shutdown by the police. There's a spirit of the law as well.

fg6hr 6y ago

IPFS is really just a mix of a torrent tracker with a torrent client, but once IPFS VMs start paying for themselves, people will sign-up in masses. I guess that's what Filecoin is about. Edit: The way I see filecoin working is anyone can post a reward for a file and once the file is provided, the reward is paid. In other words, it's bit like a brokerage that connects downloaders with uploaders. The difficultly is that this brokerage needs to be distributed and resilient.

rglullis 6y ago
I am yet to write more about this, but one thing that bothers me with Filecoin: the economics don't add up. The price per GB stored will tend to reach an equilibrium around the commodity price - i.e, costs of disks+computers+electricity+internet. Unless I am missing something, if the price gets higher, more people would buy disks and put them online, bringing the price down. If that is true, it means that a node can only be profitable if you are freeloading. And if you are freeloading, any price you get will be good which means that it tends to go even further down, perhaps even below the commodity cost. I may be missing something, but I really don't see this going beyond techies with spare disks playing around and definitely no way to run a Filecoin node on a VPS profitably.
- anchpop 6y ago
  That is actually the intent of filecoin. In perfect competition, which is the ideal state for a commodity, there is no economic profit. The reason Amazon and other companies are able to make a profit selling cloud storage is because there is not yet perfect competition. Right now if I wanted to compete with Amazon, I could buy 2 or 3 petabytes of storage space and some bandwidth. But nobody would trust me not to lose their data. This is the differentiation that makes cloud storage less of a commodity than it could be. The goal with filecoin is to make it so you don't have to trust me, just some general guarantees about the filecoin network. If amazon for some reason was selling storage on this network, I could compete with them on equal footing (and whoever sold it more cheaply would win). Many commodities are in close-to-perfect competition. For instance, I don't care if grain comes from England or France, just that it's cheap. While it is very hard to make an economic profit selling grain, many people do farm grain and make enough money to support themselves (The money you pay your workers to survive is one of the costs of producing grain. That's true even if the only worker is you) If you could make a profit by running a filecoin node on a VPS, everyone would do that. By competing with each other you'd all bring the price down and down until it was no longer more profitable than selling any other commodity (and maybe even lower). So you're right to not expect to be able to do that.

aiyodev 6y ago

Can someone provide an example of a service that ipfs enables that can’t be provided on the regular web?

StreamBright 6y ago
No, and I was looking hard. IPFS is a slower, no-SLA version of S3 at this stage. People try to sell it with this planetary-scale marketing phrase just like they were trying to sell MongoDB with the web-scale phrase. Once you understand that the use cases for asymmetric uplink powered file sharing are latency and availability you also understand that the use cases are very limited for such storage. Mostly torrent really or cold backup maybe. If disagree please explain why.
StavrosK 6y ago
You can publish static content that doesn't go down when the server goes down and scales on its own as the content becomes more popular. It also has hashing built-in, so you can be sure the content you got is what you wanted, Things like distribution of apt packages becomes much more exciting when each computer can choose to redistribute the packages it got to others in the LAN or area, even offline. It's also very interesting to me how all the visitors to your website become servers as well, so content can never be "hugged to death" or links can never go stale, as long as at least one person has the content somewhere on their node. That, to me, is huge, as links now go stale with some regularity. Think of all the Geocities sites (and all versions of them) just existing for ever, regardless if Geocities decided to shut down. For example, here's my site on IPFS: https://ipfs.eternum.io/ipfs/QmVW6JejQkjLnBJacR8qcZi88WNTMwiBwNDMCFDzijtGAw/ https://ipfs.eternum.io/ipfs/QmVW6JejQkjLnBJacR8qcZi88WNTMwi... That can now "never" be lost, as long as someone cares enough about it to visit.
- aiyodev 6y ago
  That's "how it works", not "what it can be used for". Regular websites can be designed to handle high traffic. Regular websites can make backups. What new thing does ipfs make possible that would make users install the software to use it?

truth_seeker 6y ago

>> Opera became the first major web browser to offer default IPFS support on Android, shortly after Brave started directly embedding the IPFS Companion extension (complete with a built-in js-ipfs node). This means millions of people around the world now have access to the decentralized web built directly into their browsers. Wow. Thank you IPFS community.

StavrosK 6y ago
I believe Opera only redirects to a gateway and doesn't implement IPFS itself, though, which is rather less exciting.
- deleted 6y ago
  [deleted]

fit2rule 6y ago

I really wish there was a C/C++ implementation of IPFS that could be dropped in to a Linux, Windows, MacOS, or iOS cross platform app. I already ported the Go code to iOS once, and it wasn't that painful, but functionally it would be a lot more useful as a C++ code base. I don't build browser apps - I feel there is still a need for native apps especially in this particular space.

StavrosK 6y ago

I run https://www.eternum.io/ https://www.eternum.io/ (an IPFS pinning service) and created Hearth (https://hearth.eternum.io/ https://hearth.eternum.io/, a Dropbox-like way to publish files on IPFS), and this is a very welcome release. IPFS has been a real pain to work with in the past (the node would just consume all RAM and CPU and had to be restarted a lot), but it's been getting better, which is great to see. I really hope it gets good enough to run on everyone's desktop machine, since that's the way IPFS is meant to be deployed (rather than just on gateways). It seems that it doesn't take up too much RAM or CPU now, but it looks like it might be a problem bandwidth-wise, if you host some popular content. Still, great news overall.

jillesvangurp 6y ago

Good progress. I looked at IPFS a few months ago from the point of view of fitting a decentralized file store. The decentralized requirement came primarily from the point of view wanting to fit this in a broader product that is all about decentralized for various reasons. Key problems in this space: - decentralized is something techies obsess about but that has as of yet no business value whatsoever. Customers don't ask for it. Centralized alternatives are generally available and far more mature/easy to manage. The business ecosystem around IPFS is basically not there. - this space is dominated by hobbyists running stuff like this on their personal hardware doing this mostly for idealistic or other non incentivized (i.e. money) reasons. Nothing wrong with this but using it for something real brings a few requirements with it that are basically hard to address currently. - Filecoin has been 'coined' as the solution for this for years but seems nowhere near delivering on it's published roadmap. Last time I checked it had undelivered milestones in the past. As of yet this looks increasingly like something that is a bit dead in the water / a big distraction for coming up with better/alternate solutions. - Uptime guarantees for content are currently basically DYI. Nobody but you cares about your data. If you want your content to stay there, you basically need ... proper file hosting. As incentives and mechanisms for others to agree to host your content (aka pinning in ipfs) are not there, this is hard to fix. - integration with existing centralized solutions is kind of meh/barely supported. We actually looked at using s3 as store for ipfs just so we could onboard customers and give them some decent SLA (bandwidth would be another issue for that). There are some fringe projects on github implementing this but when we looked at it the smallish blocksizes in ipfs are kind of a non starter for using this at scale (think insane numbers of requests to s3). This stuff is not exactly mainstream. Obviously this wouldn't be needed if we could incentivize others to 'pin' content. But we can't currently.

jonathanstrange 6y ago
> decentralized is something techies obsess about but that has as of yet no business value whatsoever. I don't really understand that point. If it works, then it has the business value of saving you all running server and maintenance costs. For most larger businesses these costs may be insignificant and easily recovered, but for small businesses with a lot of customers they can make a huge difference. For example, I'm looking into P2P options for implementing a decentralized message forum in a game-like emulator and it would make no sense to even implement this feature with constant running costs for server space. Now getting the decentralized data management to work reliably out-of-the-box from behind various firewalls and different platforms, that's the big problem. So far, none of the libraries I've seen are very easy to use, some require a difficult installation and configuration or you need to your own STUN server or gateway, which kind of defeats the purpose.

kevincox 6y ago

I think IPFS is a great technology, and the protocol appears to be fairly well designed but to be frank, I tried to use go-ipfs and it is a very junky implementation. There are numerous issues that show an overall lack of good quality standards in the codebase. - Pin management is far to simple for any real use case. - GC is a complete GC on an arbitrary threshold. - API is based on the CLI and really weird (ex GET /api/v0/cp?arg={src}&arg={dest}, why don't the parameters have names? Why is it GET?) - They don't prove a way to upload a directory with references to existing IPFS objects, so you need to manually encode the data. - The manual encoding uses go-style names, they seem to have let their language choice leak into the protocol. - Lots of minor issues, such as the API implementing block size limits by simply cutting off the input data at the size limit, this is even before encoding it into the format where the size limit should be calculated and cause data corruption in some cases. - Their one abstract API for creating directories (MFS) has a number of issues. - It is incredibly slow for some reason. - It kindof but not really automatically pins all content recursively references. There are other weird choicess such as their directory sharding using hashing. It isn't clear to me which use case this improves over a btree but someone probably thought it sounded cooler. Additionally the sharding appears to be a single layer of sharding which means that it still has a size limit (just larger). I ended up sinking a ton of time into https://gitlab.com/kevincox/archlinux-ipfs-mirror/ https://gitlab.com/kevincox/archlinux-ipfs-mirror/ and in the end the number of small quirks were incredibly frustrating. I might go back and implement their hash-sharding logic to get the mirror working again but at this point I don't really want to interact with go-ipfs again. I thought it would be an interesting project to get involved in, and I have a lot of expertise that would be valuable however it appears that it is all a bit of a mess which is a real shame. Maybe with more success it can be re-written in cleaner fashion, as I said most of the issues are with the implementation not the protocol so there is definitely hope. I do honestly wish all of the success to the project.

skyde 6y ago

IPFS need a way to get paid for a block over a period of time. Does not need to be real money just token you can use later to pay for your file to be seeded. Basically I host your file and you host my file. With some algorithm to make sure number of seeder for each block never go to 0

momack2 6y ago
Check out Filecoin (https://filecoin.io https://filecoin.io) - token for buying/selling file storage for IPFS data. You can earn tokens for renting out your disk space and use them to purchase backup/hosting from others.

woodandsteel 6y ago

I was impressed to read that the IPFS public network scaled 30X in 2019. It sounds like they have gotten over a big hump for usability and utility.