10 ms·
Database-less torrent website
- vindarel 5y agorelated: https://torrent-paradise.ml/ https://torrent-paradise.ml/ Also based on IPFS with this HTTP gateway. I find a lot of torrents in there.
- etagate 5y agoLove this website too. But in this case the database and search engine is hosted on the server, it only uses IPFS to distribute a copy of a folder ready out of the box (db included) to be hosted as another instance (a backup).
- mab122 5y agoYou actually should be able to access it using IPFS at/ipns/torrent-paradise.ml or /ipns/12D3KooWB3GY1u6zMLqnf3MJ8zhX3SS1oBj7VXk3xp6sJJiFGZXp but it seems that clicking search throws an JS error :/
- ThalesX 5y agoIf any dev is here, I'm on IPFS and can't seem to search, getting console error: > Uncaught ReferenceError: passQueryToResultpage is not defined
- jokoon 5y agoany similar link you know for DHT searches?
- danka 5y agoI wonder why the author ignored the option of compression in the post. Even with a simple gzip DEFLATE compression, those 10MB of plain text could get as small as a 1MB archive and possibly more, meaning that in a compressed 10MB payload you could fit much much more than 135K records.
- vermilingua 5y agoIt isn't 10MB of plain text though, it's 10MB of binary SQLite database. I agree that compression would be useful here, but I don't think a simple gzip DEFLATE would be. I was curious so I compressed that torrent db with a few different methods: 11.1MB 11116544B dump.sqlite 10.2MB 10155419B dump.csv 6.6MB 6573399B dump.sqlite.gz 6.6MB 6565771B dump.zip 5.6MB 5616842B dump.rar gzip is certainly suitable to be used in this situation, I stand corrected.
- rdubz 5y agoBack of envelope: The 10MB estimated size came from [100 bytes per row] * [100k rows]. 50 of the bytes per row were "description", which should compress well (2-3x, I'd guess). 40 bytes per row were the IPFS ID/hash, IIUC. I assumed this is like a Git hash, 40 hex chars, which is really just 20 bytes of entropy. He also estimated 14 bytes for the size (stored as a string representation of a decimal integer, up to 1e15 - 1, or 1PB?). That's about 50 bits or 6-7 bytes, as a binary integer. Sizes wouldn't be uniformly distributed though so it would compress to even fewer bytes. So if SQLite was smart (or one gzips the whole db file, like you did), it makes sense that a factor of 2 or so is reclaimable.
- lekevicius 5y agoOne unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?
- gerwim 5y agoWell there is Dolt: https://www.dolthub.com/ https://www.dolthub.com/
- rakoo 5y agoMutable torrents do exist (https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.
- ddtaylor 5y agoThis is actually really neat I wasn't aware of these BEPs. Do you know of any place where they are being used?
- rakoo 5y agolibtorrent has it, webtorrent has been working on it but development of this feature seems to have stalled. That's a real shame, because webtorrent can be used directly from the browser so it would have been very practical to use a single dependency and have pretty much the same functionalities as the article
- thinkloop 5y agoIPFS requires "pinning" content for it to exist, meaning you host it, whoever pins content becomes liable for its distribution, if no one pins it, it disappears. That's why you can't freely host things like child porn on IPFS.
- etagate 5y agoThis is true even for a torrent btw. And (un?)fortunately you can distribute anything on IPFS as long as it's encrypted.
- Gigachad 5y agoNo one is going to pin encrypted data they do not understand. And IPFS doesn’t get users to seed random files, you only seed stuff you downloaded. So really it’s not much different to hosting it on a personal http server except any downloaders can also host.
- mavhc 5y agoSounds very similar to bittorrent itself
- nybble41 5y ago> No one is going to pin encrypted data they do not understand. Select users who have the decryption key (communicated out-of-band) might be willing to pin the encrypted data for the benefit of others who also have the key. There are also for-pay pinning services who probably wouldn't care whether the data was encrypted. And whether it's pinned it or not, as long as it's downloaded the data remains available to be shared (at least until it's GC'd).
- Gigachad 5y agoOk but you could do that on literally any platform. I could encrypt data and stick it on google drive or aws and the hosts wouldn’t have any clue what it was.
- dfghdfhs 5y agoI love the internet.
- xialvjun 5y agoSince it has used IPFS, why not host the content of those torrents with IPFS?
- betwixthewires 5y agoYou can do that. And I like the idea. But for now, torrents are the go to method of sharing large pieces of data, and utilizing IPFS to move the centralized aspect of torrents to a decentralized space is a good idea.
- dfghdfhs 5y agoCould someone more savvy clarify.... On that page I do inspect, got console and do: loadDBAndExec('SELECT * FROM my_table LIMIT 10;') I get Uncaught ReferenceError: loadDBAndExec is not defined Why doesn't this work?
- manigandham 5y agoThat's just example code in the blog post. The live demo uses a webworker with different code: https://boredcaveman.xyz/demo/megacat/database-app.js https://boredcaveman.xyz/demo/megacat/database-app.js
- ronyfadel 5y agoI'm disappointed that I don't find result for "La vita e bella" but I do find results for "La vita è bella".
- yonixw 5y agoEnhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- Closi 5y agoExactly the thought I had reading this! Combining these projects would be awesome, proper semantic web! Plus it would mean near-zero loading times as the db size scaled (although ultimately some peers would still need the full database).
- lovasoa 5y agoWell, you could reimplement "absurd sql" over ipfs. Possible, but far from trivial.
- samwillis 5y agoAs the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent? (My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api) EDIT: After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfsfilesreadpath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI... EDIT2: In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfscatipfspath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...
- samwillis 5y agoContinuing my looking into this, the SQLite range requests trick was implemented in this [0] project. It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying. 0: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs 1: https://github.com/phiresky/sql.js-httpvfs#inspiration https://github.com/phiresky/sql.js-httpvfs#inspiration 2: https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net 3: https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
- politician 5y agoHow do immutable systems like IPFS manage the problem of updates to a blog?
- yonixw 5y agoYou have "publish ids" which you control using private keys.. like a dns... so you can have your main page address known and all the other posts linked from that page
- Gigachad 5y agoThey have another system called IPNS but unless it’s improved since I checked, it wasn’t very practical.
- lekevicius 5y agoWhat's wrong with IPNS? I'm using it for my decentralized website (via fleek), and it seems to work well.
- nybble41 5y agoIt has a reputation for being slow if you're not using the DNS-based version of the protocol (DNSLink). Though the experimental IPNS-over-PubSub interface helps if both sites are using it.
- tecleandor 5y agoI'm curious about why didn't they compress the SQLite file using gzip. Even in level 1, for superfast compression/decompression halved the file size.
- etagate 5y agoI knew I could do it, I just wanted to make a demo of the whole thing to see if it would have worked in practice. But yes there's space for a lot of improvements like compression and lazy load with sqlite (cited in a comment)
- tecleandor 5y agoGreat! I was honestly wondering because I didn't know if there were any technical limitations/problems, but it's quite interesting nonetheless.
- donkarma 5y agoThat's not going to stop takedowns, they don't care about if it's technically on your site or not.
- Chris2048 5y agoDo they also not care whether they technically have the authority of law or not?
- imtringued 5y agoThey can change the law to whatever is more profitable...
- jtbayly 5y agoExactly. I forget where I just saw this. It seemed like a particularly absurd example, but the examples are numerous. Torrent sites don’t have the copyrighted content either, but they get shut down all the time.
- roywiggins 5y agoBut the torrents themselves tend to stay seeded... if you stick the site itself on IPFS and anyone in any jurisdiction can "seed" it, then it's going to be hard to kill by court order.
- tablespoon 5y ago> That's not going to stop takedowns, they don't care about if it's technically on your site or not. Yep. "The underlying mistake is thinking about this like a game where you can make up rules for the government to follow." - https://news.ycombinator.com/item?id=29913036 https://news.ycombinator.com/item?id=29913036 Also the assumptions here kind of remind me of domain fronting: https://www.zdnet.com/article/amazons-aws-latest-to-give-up-domain-fronting-putting-censorship-apps-in-peril/ https://www.zdnet.com/article/amazons-aws-latest-to-give-up-.... Basically, it's assuming IPFS will protect it from the authorities, but what might actually happen is it makes IPFS a target of the authorities. Now that probably won't happen with this because the authorities don't actually care that much about torrenting/piracy, but the error is still there.
- ineedasername 5y agoThe headline doesn't make much sense. There's still a database, what changes is simply where it's stored when in use. "Torrent Website w/o DB Server" might work better. That aside, would this method scale? 135,000 torrents doesn't seem comprehensive, so I would expect real world use to have many more. Maybe a different SQLite db for different categories?
- moolcool 5y agoSince it's search is just based on prefixes, you could probably just alphabetically partition the databases.
- senjin 5y agoMakes me wonder if you could host a pouchdb in ipfs
- SuchAnonMuchWow 5y agoReading just the title, I was expecting storing/serving magnet files on DNS, as someone did a few years back if I remember correctly (just as a joke/poc, not as a serious solution). I'm pretty sure it was posted on hw, but I can't find the link anymore.
- kderbyma 5y agolove these kind of things. I am interested in distribution methods for smarter clients without needing a dedicated server to host from.
- MuchHustle 5y agoTheres already like 10 crypto projects implementing that. The most prominent is probably Bit Torrent Token (BTT). https://www.bittorrent.com/token/btt/ https://www.bittorrent.com/token/btt/
- Hakashiro 5y agoThese projects while technically impressive and interesting, I'm sure wouldn't make much of a difference to copyright holders. I can imagine something like this: - Please take down our IP - I can't because I am not hosting it - But you are displaying it on your website, take your website down or filter what your website displays to avoid showing copyrighted material And that's the end of it. It doesn't matter where the data is stored. It does matter, a lot, where the data is displayed. If your website displays illegal content, you're facilitating access so you're breaching the law. And, honestly, I'm not sure there's an easy solution to that other than using onion or I2P sites which are only marginally more safe against copyright takedown requests because, due to their very nature, they are obscure and difficult to access.