8 ms·

Academic Torrents

297 points by julianj 7y ago

yig 7y ago
2016 HN discussion: https://news.ycombinator.com/item?id=12381791 https://news.ycombinator.com/item?id=12381791 2014 HN discussion: https://news.ycombinator.com/item?id=7149006 https://news.ycombinator.com/item?id=7149006
- dang 7y ago
  2018 too: https://news.ycombinator.com/item?id=17744150 https://news.ycombinator.com/item?id=17744150
glofish 7y ago
Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
- derefr 7y ago
  One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.” Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere, ascribe metadata to the hash of the content it is meant to annotate. Then, later, you can reassemble them in either order—either by first finding a data set, hashing it, and then looking up metadata in some metadata-hosting service; or by first browsing a catalogue of indexed metadata, finding out about a dataset that meets your needs, and then retrieving the data set by its hash. Which is to say: with digital data, library science (creating metadata and chains-of-custody and indexing them for search) and archiving (ensuring access to pristine artifacts over time) don’t need to happen at the same time, in the same place. There can be separate “artifact hosting” and “metadata library” services. (Which is especially helpful in contexts where private IP is involved—you can still keep in your metadata library, the metadata for a data-set you don’t have the rights to; and those with the rights can go get the data-set themselves.)
  - bordercases 7y ago
    Aaaand someone has to do the work for computing the index and annotating the hashes.
    - robbya 7y ago
      I think it's worth recognizing that this is a good first step in a hard problem. Hosting many TB of data for free isn't easy. Building an index on top of that data isn't easy either, and it looks like no such index exists today, but if someone decided to build that index they wouldn't need to worry about the hosting portion of the problem. That's a great starting point.
  - metasj 7y ago
    This flexibility in time, specialization, and order of operation is surely one of the joys of modern digital collections. Library scientists might say archiving and structuring and curation are all facets of that science. And you'll also want a hash search engine that finds related hashes, as there can be many revisions + versions, only some of which have some metadata.
  - ska 7y ago
    > s that you don’t need to keep digital data’s metadata attached to the data “at the hip.” You don't have to, but it's still mostly a good idea. But this stuff isn't either-or. We can have both. This is especially true for research oriented files, where consumers are often unable or unwilling to maintain a functional metadata store, and do a lot of manual file handling. Saying "well, somebody could have set up a super-awesome metadata system that track this" doesn't magically make those resources exist.
- ieee8023 7y ago
  There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e58964c89b7da99e0edf7 https://academictorrents.com/details/d52ccc21455c7a82fd6e589...
  - glofish 7y ago
    Ok, I see that there is code provided there. Better than nothing but geez, it is not really what metadata should be like def get_labels(rightside): met = {} met['brain'] = ( 1. * (rightside != 0).sum() / (rightside == 0).sum()) met['tumor'] = ( 1. * (rightside > 2).sum() / ((rightside != 0).sum() + 1e-10)) met['has_enough_brain'] = met['brain'] > 0.30 met['has_tumor'] = met['tumor'] > 0.01 return met I will say that it is very handy to know exactly how the labels were computed. What I really meant is a way to search and select data based on metadata. For example has_tumor. Also note how everything is still one single blob, to get one line of any of the files, one would need to download everything.
    - Mathnerd314 7y ago
      Bittorrent does support partial downloads that request only some files or byte ranges out of a torrent. Some of the torrents are just compressed zip's but for the others you could look at the code / documentation to see which files were relevant before downloading 10GB of data. I think the abstract is sufficient for searching data; expecting some kind of smart database that can handle all the weird formats science uses is a bit much.
      - rakoo 7y ago
        There are even torrent clients that export a FUSE VFS so you can use your standard tools.
    - mtone 7y ago
- neffy 7y ago
  To err is human, to forgive divine, to fix immortal.
aldoushuxley001 7y ago
This is amazing, really a great source of data.
robbya 7y ago
https://academictorrents.com/about.php#mirroring https://academictorrents.com/about.php#mirroring Using RSS to allow mirrors to host different subjects is really clever, although some of the categories seem quite large (>5TB). It may be worth breaking up each category (sharding) to keep each to 100GB or less so a volunteer can pick a couple and not worry about running out of disk when a category grows. Then it would be good to track how many seeds each category-shard has so volunteers can help where it's most needed.
- DuskStar 7y ago
  Some individual items are multiple TB, which would make 100GB shards a little difficult.
krick 7y ago
Yeah, it could really benefit from some organizational work, like on more mature music torrent trackers or such. Categories, mandatory tags, unified names, reviewed by community-chosen category-wise moderators. In it's current state in's basically a file dump, either you have the direct link, or you can only hope to find something interesting. Not that much better than sharing magnet links via public pastebin records...
- colechristensen 7y ago
  One very interesting thing I wish would be studied in depth are the virtual economies of mature trackers. Limiting access to resources and granting increasing access for contributing and correcting quality has in places been extremely successful. It is interesting to see the varying quality and associated economic mechanics. Some environments, based just on prestige, have big problems with toxicity (StackOverflow, Wikipedia) which I didn't see at all in some music trackers.
  - ryacko 7y ago
    Wikipedia does cover that issue. Competing views are difficult to reconcile. https://en.wikipedia.org/w/index.php?title=Wikipedia:Systemic_bias&oldid=378210455 https://en.wikipedia.org/w/index.php?title=Wikipedia:Systemi... (using a version of the article from ten years ago because everything is unnecessarily verbose on wikipedia now)
    - ailideex 7y ago
      I'm not sure what the point of quoting that is really. I guess if you subscribe to the idea that reality is somehow modified by your age, sex, race, education or whatever the heck then it has some relevance but then the whole idea behind an encyclopedia seems pointless and we should just each maintain our own unique knowledge bases as they will have no relevance to someone other than us. That an article like that exists is patently absurd in my view and kind of makes me a bit ill. Things like that is what led to this: https://www.youtube.com/watch?v=C9SiRNibD14 https://www.youtube.com/watch?v=C9SiRNibD14 I really firmly believe that if you think there is a European (?) science and an African science and they are distinct and equally valid then either me or you do not belong on Wikipedia and I would actually like Wikipedia to clarify their mission in this light.
husainalshehhi 7y ago
Downloading some of this might be illegal. I see some entries that says "No license specified, the work may be protected by copyright."
DuskStar 7y ago
I wish I could add Gwern's Danbooru dataset [0] here - 2.7TB of labeled anime images. But they only support torrent files up to 10MB, and that's over 20MB for the full dataset or 12MB for the SFW low-rez set... Incidentally, when the torrent file for your anime image collection passes 20MB, something has obviously gone very w̵r̵o̵n̵g̵ right. 0: https://www.gwern.net/Danbooru2019 https://www.gwern.net/Danbooru2019
- DuskStar 7y ago
  I should probably point out that this dataset has been used for some machine learning tech demos in the past, for example This Waifu Does Not Exist [0], a StyleGAN-based automatic anime portrait generation tool. So it's not completely outside of what the site already hosts... 0: https://www.thiswaifudoesnotexist.net/ https://www.thiswaifudoesnotexist.net/
  - gwern 7y ago
    More than demos, papers too: https://www.gwern.net/Danbooru2019#applications https://www.gwern.net/Danbooru2019#applications

297 points by julianj 7y ago

yig 7y ago

2016 HN discussion: https://news.ycombinator.com/item?id=12381791 https://news.ycombinator.com/item?id=12381791 2014 HN discussion: https://news.ycombinator.com/item?id=7149006 https://news.ycombinator.com/item?id=7149006

dang 7y ago
2018 too: https://news.ycombinator.com/item?id=17744150 https://news.ycombinator.com/item?id=17744150

glofish 7y ago

Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.

derefr 7y ago
One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.” Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere, ascribe metadata to the hash of the content it is meant to annotate. Then, later, you can reassemble them in either order—either by first finding a data set, hashing it, and then looking up metadata in some metadata-hosting service; or by first browsing a catalogue of indexed metadata, finding out about a dataset that meets your needs, and then retrieving the data set by its hash. Which is to say: with digital data, library science (creating metadata and chains-of-custody and indexing them for search) and archiving (ensuring access to pristine artifacts over time) don’t need to happen at the same time, in the same place. There can be separate “artifact hosting” and “metadata library” services. (Which is especially helpful in contexts where private IP is involved—you can still keep in your metadata library, the metadata for a data-set you don’t have the rights to; and those with the rights can go get the data-set themselves.)
- bordercases 7y ago
  Aaaand someone has to do the work for computing the index and annotating the hashes.
  - robbya 7y ago
    I think it's worth recognizing that this is a good first step in a hard problem. Hosting many TB of data for free isn't easy. Building an index on top of that data isn't easy either, and it looks like no such index exists today, but if someone decided to build that index they wouldn't need to worry about the hosting portion of the problem. That's a great starting point.
- metasj 7y ago
  This flexibility in time, specialization, and order of operation is surely one of the joys of modern digital collections. Library scientists might say archiving and structuring and curation are all facets of that science. And you'll also want a hash search engine that finds related hashes, as there can be many revisions + versions, only some of which have some metadata.
- ska 7y ago
  > s that you don’t need to keep digital data’s metadata attached to the data “at the hip.” You don't have to, but it's still mostly a good idea. But this stuff isn't either-or. We can have both. This is especially true for research oriented files, where consumers are often unable or unwilling to maintain a functional metadata store, and do a lot of manual file handling. Saying "well, somebody could have set up a super-awesome metadata system that track this" doesn't magically make those resources exist.
ieee8023 7y ago
There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e58964c89b7da99e0edf7 https://academictorrents.com/details/d52ccc21455c7a82fd6e589...
- glofish 7y ago
  Ok, I see that there is code provided there. Better than nothing but geez, it is not really what metadata should be like def get_labels(rightside): met = {} met['brain'] = ( 1. * (rightside != 0).sum() / (rightside == 0).sum()) met['tumor'] = ( 1. * (rightside > 2).sum() / ((rightside != 0).sum() + 1e-10)) met['has_enough_brain'] = met['brain'] > 0.30 met['has_tumor'] = met['tumor'] > 0.01 return met I will say that it is very handy to know exactly how the labels were computed. What I really meant is a way to search and select data based on metadata. For example has_tumor. Also note how everything is still one single blob, to get one line of any of the files, one would need to download everything.
  - Mathnerd314 7y ago
    Bittorrent does support partial downloads that request only some files or byte ranges out of a torrent. Some of the torrents are just compressed zip's but for the others you could look at the code / documentation to see which files were relevant before downloading 10GB of data. I think the abstract is sufficient for searching data; expecting some kind of smart database that can handle all the weird formats science uses is a bit much.
    - rakoo 7y ago
      There are even torrent clients that export a FUSE VFS so you can use your standard tools.
  - mtone 7y ago
neffy 7y ago
To err is human, to forgive divine, to fix immortal.

aldoushuxley001 7y ago

This is amazing, really a great source of data.

robbya 7y ago

https://academictorrents.com/about.php#mirroring https://academictorrents.com/about.php#mirroring Using RSS to allow mirrors to host different subjects is really clever, although some of the categories seem quite large (>5TB). It may be worth breaking up each category (sharding) to keep each to 100GB or less so a volunteer can pick a couple and not worry about running out of disk when a category grows. Then it would be good to track how many seeds each category-shard has so volunteers can help where it's most needed.

DuskStar 7y ago
Some individual items are multiple TB, which would make 100GB shards a little difficult.

krick 7y ago

Yeah, it could really benefit from some organizational work, like on more mature music torrent trackers or such. Categories, mandatory tags, unified names, reviewed by community-chosen category-wise moderators. In it's current state in's basically a file dump, either you have the direct link, or you can only hope to find something interesting. Not that much better than sharing magnet links via public pastebin records...

colechristensen 7y ago
One very interesting thing I wish would be studied in depth are the virtual economies of mature trackers. Limiting access to resources and granting increasing access for contributing and correcting quality has in places been extremely successful. It is interesting to see the varying quality and associated economic mechanics. Some environments, based just on prestige, have big problems with toxicity (StackOverflow, Wikipedia) which I didn't see at all in some music trackers.
- ryacko 7y ago
  Wikipedia does cover that issue. Competing views are difficult to reconcile. https://en.wikipedia.org/w/index.php?title=Wikipedia:Systemic_bias&oldid=378210455 https://en.wikipedia.org/w/index.php?title=Wikipedia:Systemi... (using a version of the article from ten years ago because everything is unnecessarily verbose on wikipedia now)
  - ailideex 7y ago
    I'm not sure what the point of quoting that is really. I guess if you subscribe to the idea that reality is somehow modified by your age, sex, race, education or whatever the heck then it has some relevance but then the whole idea behind an encyclopedia seems pointless and we should just each maintain our own unique knowledge bases as they will have no relevance to someone other than us. That an article like that exists is patently absurd in my view and kind of makes me a bit ill. Things like that is what led to this: https://www.youtube.com/watch?v=C9SiRNibD14 https://www.youtube.com/watch?v=C9SiRNibD14 I really firmly believe that if you think there is a European (?) science and an African science and they are distinct and equally valid then either me or you do not belong on Wikipedia and I would actually like Wikipedia to clarify their mission in this light.

husainalshehhi 7y ago

Downloading some of this might be illegal. I see some entries that says "No license specified, the work may be protected by copyright."

DuskStar 7y ago

I wish I could add Gwern's Danbooru dataset [0] here - 2.7TB of labeled anime images. But they only support torrent files up to 10MB, and that's over 20MB for the full dataset or 12MB for the SFW low-rez set... Incidentally, when the torrent file for your anime image collection passes 20MB, something has obviously gone very w̵r̵o̵n̵g̵ right. 0: https://www.gwern.net/Danbooru2019 https://www.gwern.net/Danbooru2019

DuskStar 7y ago
I should probably point out that this dataset has been used for some machine learning tech demos in the past, for example This Waifu Does Not Exist [0], a StyleGAN-based automatic anime portrait generation tool. So it's not completely outside of what the site already hosts... 0: https://www.thiswaifudoesnotexist.net/ https://www.thiswaifudoesnotexist.net/
- gwern 7y ago
  More than demos, papers too: https://www.gwern.net/Danbooru2019#applications https://www.gwern.net/Danbooru2019#applications