9 ms·

Announcing the data.gov archive

382 points by govideo 2y ago

govideo 2y ago
From the post: Today we released our archive of data.gov on Source Cooperative. The 16TB collection includes over 311,000 datasets harvested during 2024 and 2025, a complete archive of federal public datasets linked by data.gov. It will be updated daily as new datasets are added to data.gov. This is the first release in our new data vault project to preserve and authenticate vital public datasets for academic research, policymaking, and public use.
deleted 2y ago
[deleted]
fredoliveira 2y ago
Honestly a shame it has to come to this. Sure, people elected this administration and I guess with that comes with a bunch things I disagree with. But the removal of years of scientific research and data from the web (paid for by citizens with their taxes) is absolutely unacceptable. Ravaging CDC data, climate data, etc is horrendous and unforgivable.
- nxm 2y ago
  [flagged]
  - fredoliveira 2y ago
    At face value I agree with everything USAID stands for, but it sounds like you have some examples in mind you want to provide.
    - matwood 2y ago
      Good luck getting any responses that have been corroborated outside of Musk's ramblings. So much for 'transparency'. If someone was truly looking to audit places to save money, it's always been odd to me that USAID was first. Now this story comes out... https://www.newsweek.com/usaid-elon-musk-starlink-probe-ukraine-2027054 https://www.newsweek.com/usaid-elon-musk-starlink-probe-ukra...
  - d3nj4l 2y ago
    Getting upset that USAID spent the government equivalent of pocket change on programs that nominally helped other countries and practically bound them to US soft power is (1) cutting off your nose to spite your face and (2) not equivalent to deleting data that money was already spent on.
  - MeetingsBrowser 2y ago
    We cancelled $100,000 in life saving donations because half a penny was spent on something considered “woke”.
- zombot 2y ago
  It's today's equivalent of book burnings. "where they burn books, they will ultimately burn people as well." Those who delete research will ultimately delete people as well. https://en.wikiquote.org/wiki/Heinrich_Heine https://en.wikiquote.org/wiki/Heinrich_Heine
black_puppydog 2y ago
Great to see there's some resistance. What I'm missing from this announcement though is any mention of how they intend to secure this "vault" against the current government. I'm assuming good intentions on the part of Harvard, but keeping this data online against the express will of the government is gonna cost (political) capital. And from what I can see, the archive is hosted by US entities on US-controlled servers on US soil? This is the same thing that's been bothering me with archive.org lately, by the way. I haven't found a good way to simply (for some reasonable definition definition of "simple") contribute 10 TiB or so of redundant storage on my (european) home server either. That kind of thing might (have to) serve to ensure tamper-resistance for that data, given the current political climate on both sides of the pond. Any pointers welcome.
- EnnEmmEss 2y ago
  If I remember correctly, Harvard has immunity to eminent domain under the Massachusetts constitution. Maybe it has a similar right which would make it immune to such attacks?
  - black_puppydog 2y ago
    That may be so, but given what Trump and Musk have been up to, the situation of the courts, and how they blatantly don't give a f*k about what's constitutional or not, I wouldn't rely on this so-called "immunity".
    - 0xEF 2y ago
      I really hope I am wrong, but I'm planning on seeing some headlines about Musk shutting down Harvard next week over this. For anyone who still thinks the existing laws, constitutions and policies mean anything to this current regime, prepare to get some whiplash. They are proving that none of that matters if they simply ignore it and do what they want to do anyway.
      - shakna 2y ago
        It wouldn't be the first time that Musk has called for Harvard to be defunded. [0] [0] https://www.businessinsider.com/elon-musk-calling-for-harvard-defunding-2023-12 https://www.businessinsider.com/elon-musk-calling-for-harvar...
fnands 2y ago
Will does this include all USGS data? This is a topic that came up at work today as we rely on this data and are considering backing up most of the Lidar data from there ourselves (100s of TB probably) EDIT: no, looks like it is only the footprints
cyberlimerence 2y ago
Is anyone out there archiving USGS/NOAA datasets ? It sounds ridiculous, but this appears to be where we are now. There is a submission about NOAA on the frontpage now: "Scientists on alert as NOAA restricts contact with foreign nationals" [1] [1] https://news.ycombinator.com/item?id=42970814 https://news.ycombinator.com/item?id=42970814
- fs111 2y ago
  ArchiveTeam is working on everything US-Government and you can help https://tracker.archiveteam.org/ https://tracker.archiveteam.org/ https://wiki.archiveteam.org/index.php/US_Government https://wiki.archiveteam.org/index.php/US_Government
deleted 2y ago
[deleted]
Rebuff5007 2y ago
I find it assuming that the might of the American government -- in trying to take a bunch of data offline -- is being resisted by a digital "militia" of hobbyist archivers and non profits. Theres something that about this that just rings second amendment. Personally I think the concept of civilians having weapons to be a check on a nation state is absurd, but in this case it feels pretty empowering.
- p3rls 2y ago
  Larping the larpers.
- jppope 2y ago
  Well I wouldn't really call it the "American Government" per say... Its a Geriatric former reality TV show host elected to the presidency by offering to do for America what he did for steak or private education. That guy and his cronies really aren't the American Government. They were just elected to be in charge of the American Government.
deleted 2y ago
[deleted]
frontalier 2y ago
this archive is going to disappear before the summer comes https://youtu.be/5RpPTRcz1no?t=1511 https://youtu.be/5RpPTRcz1no?t=1511
mindcrime 2y ago
Very happy this is happening. There's a ridiculous amount of incredibly valuable data, scientific documents, etc. "out there" that are at risk. I haven't had much time to look at this yet and see what all is there, but whether currently included or not, a couple of things I really hope get archived are the contents of the DTIC (Defense Technical Information Center) document repository (lots of really interesting older scientific publications) and the NASA TRS (Technical Report Server). I'm working on my own archive of at least some portion of the DTIC stuff just to be on the safe side. So far everything I've tried to access is still there, but who knows how long that will last.

382 points by govideo 2y ago

From the post: Today we released our archive of data.gov on Source Cooperative. The 16TB collection includes over 311,000 datasets harvested during 2024 and 2025, a complete archive of federal public datasets linked by data.gov. It will be updated daily as new datasets are added to data.gov. This is the first release in our new data vault project to preserve and authenticate vital public datasets for academic research, policymaking, and public use.

deleted 2y ago

[deleted]

fredoliveira 2y ago

Honestly a shame it has to come to this. Sure, people elected this administration and I guess with that comes with a bunch things I disagree with. But the removal of years of scientific research and data from the web (paid for by citizens with their taxes) is absolutely unacceptable. Ravaging CDC data, climate data, etc is horrendous and unforgivable.

nxm 2y ago
[flagged]
- fredoliveira 2y ago
  At face value I agree with everything USAID stands for, but it sounds like you have some examples in mind you want to provide.
  - matwood 2y ago
    Good luck getting any responses that have been corroborated outside of Musk's ramblings. So much for 'transparency'. If someone was truly looking to audit places to save money, it's always been odd to me that USAID was first. Now this story comes out... https://www.newsweek.com/usaid-elon-musk-starlink-probe-ukraine-2027054 https://www.newsweek.com/usaid-elon-musk-starlink-probe-ukra...
- d3nj4l 2y ago
  Getting upset that USAID spent the government equivalent of pocket change on programs that nominally helped other countries and practically bound them to US soft power is (1) cutting off your nose to spite your face and (2) not equivalent to deleting data that money was already spent on.
- MeetingsBrowser 2y ago
  We cancelled $100,000 in life saving donations because half a penny was spent on something considered “woke”.
zombot 2y ago
It's today's equivalent of book burnings. "where they burn books, they will ultimately burn people as well." Those who delete research will ultimately delete people as well. https://en.wikiquote.org/wiki/Heinrich_Heine https://en.wikiquote.org/wiki/Heinrich_Heine

black_puppydog 2y ago

Great to see there's some resistance. What I'm missing from this announcement though is any mention of how they intend to secure this "vault" against the current government. I'm assuming good intentions on the part of Harvard, but keeping this data online against the express will of the government is gonna cost (political) capital. And from what I can see, the archive is hosted by US entities on US-controlled servers on US soil? This is the same thing that's been bothering me with archive.org lately, by the way. I haven't found a good way to simply (for some reasonable definition definition of "simple") contribute 10 TiB or so of redundant storage on my (european) home server either. That kind of thing might (have to) serve to ensure tamper-resistance for that data, given the current political climate on both sides of the pond. Any pointers welcome.

EnnEmmEss 2y ago
If I remember correctly, Harvard has immunity to eminent domain under the Massachusetts constitution. Maybe it has a similar right which would make it immune to such attacks?
- black_puppydog 2y ago
  That may be so, but given what Trump and Musk have been up to, the situation of the courts, and how they blatantly don't give a f*k about what's constitutional or not, I wouldn't rely on this so-called "immunity".
  - 0xEF 2y ago
    I really hope I am wrong, but I'm planning on seeing some headlines about Musk shutting down Harvard next week over this. For anyone who still thinks the existing laws, constitutions and policies mean anything to this current regime, prepare to get some whiplash. They are proving that none of that matters if they simply ignore it and do what they want to do anyway.
    - shakna 2y ago
      It wouldn't be the first time that Musk has called for Harvard to be defunded. [0] [0] https://www.businessinsider.com/elon-musk-calling-for-harvard-defunding-2023-12 https://www.businessinsider.com/elon-musk-calling-for-harvar...

fnands 2y ago

Will does this include all USGS data? This is a topic that came up at work today as we rely on this data and are considering backing up most of the Lidar data from there ourselves (100s of TB probably) EDIT: no, looks like it is only the footprints

cyberlimerence 2y ago

Is anyone out there archiving USGS/NOAA datasets ? It sounds ridiculous, but this appears to be where we are now. There is a submission about NOAA on the frontpage now: "Scientists on alert as NOAA restricts contact with foreign nationals" [1] [1] https://news.ycombinator.com/item?id=42970814 https://news.ycombinator.com/item?id=42970814

fs111 2y ago
ArchiveTeam is working on everything US-Government and you can help https://tracker.archiveteam.org/ https://tracker.archiveteam.org/ https://wiki.archiveteam.org/index.php/US_Government https://wiki.archiveteam.org/index.php/US_Government

deleted 2y ago

[deleted]

Rebuff5007 2y ago

I find it assuming that the might of the American government -- in trying to take a bunch of data offline -- is being resisted by a digital "militia" of hobbyist archivers and non profits. Theres something that about this that just rings second amendment. Personally I think the concept of civilians having weapons to be a check on a nation state is absurd, but in this case it feels pretty empowering.

p3rls 2y ago
Larping the larpers.
jppope 2y ago
Well I wouldn't really call it the "American Government" per say... Its a Geriatric former reality TV show host elected to the presidency by offering to do for America what he did for steak or private education. That guy and his cronies really aren't the American Government. They were just elected to be in charge of the American Government.

deleted 2y ago

[deleted]

frontalier 2y ago

this archive is going to disappear before the summer comes https://youtu.be/5RpPTRcz1no?t=1511 https://youtu.be/5RpPTRcz1no?t=1511

mindcrime 2y ago

Very happy this is happening. There's a ridiculous amount of incredibly valuable data, scientific documents, etc. "out there" that are at risk. I haven't had much time to look at this yet and see what all is there, but whether currently included or not, a couple of things I really hope get archived are the contents of the DTIC (Defense Technical Information Center) document repository (lots of really interesting older scientific publications) and the NASA TRS (Technical Report Server). I'm working on my own archive of at least some portion of the DTIC stuff just to be on the safe side. So far everything I've tried to access is still there, but who knows how long that will last.