5 ms·

Were you not around for the Big Data heyday a decade ago?

by CincinnatiMan 5mo ago

varispeed 5mo ago
Until thumb drives became large enough to fit most datasets it stopped becoming Big Data. Just normal data.
- ffsm8 5mo ago
  We have thumb drives that can store petabytes of data? Or did you mean the "big data" crowd which thought 500GB was noteworthy? I don't think anyone took those serious, neither in 2010s nor now. That was always "small" data
  - varispeed 5mo ago
    Most companies using term "big data" had datasets in TB region. One company I had a gig at had full Hadoop cluster setup and their whole dataset was 40GB. Their marketing had all the big data adjacent keywords over the brochures for clients.
    - gf000 5mo ago
      That's a decent quality 3 hours movie :D
  - butlike 5mo ago
    > We have thumb drives that can store petabytes of data We do?
    - ffsm8 5mo ago
      Please provide a link.
      - BizarroLand 5mo ago
        You would need 4 and change of these 245tb Kioxias to hold 1 petabyte, and an entire server grade computer to run them. https://www.tomshardware.com/pc-components/ssds/kioxia-unveils-245tb-ssd-the-worlds-highest-capacity-storage-device-could-store-12-500-4k-movies https://www.tomshardware.com/pc-components/ssds/kioxia-unvei... Or 250 of these ~$400 4tb flash drives and an insane number of dongles to connect them all: https://www.slashgear.com/1847725/largest-usb-thumb-drive-highest-capacity-can-buy/ https://www.slashgear.com/1847725/largest-usb-thumb-drive-hi...
        vunuxodo 5mo ago
        Plus one more for your parity drive.
    - dylan604 5mo ago
      It was a question that you've edited out the punctuation. You're asking the exact same thing as the person you've replied
  - 0x457 5mo ago
    My rule of thumb was "can it fit in RAM on a server?" If it can, then it's not big data. 500GB is in the "fits" category.
    - gf000 5mo ago
      You can quadruple that and could still fit in server RAM
      - 0x457 5mo ago
        Correct, and until then it won't be "big data"
- jmalicki 5mo ago
  To some degree IMO big data is still a mindset when it might take a day to process your data in a normal SQL query. Some tech doesn't scale to the data size for all use cases, and you need different solutions.
ToucanLoucan 5mo ago
Hell you mean a decade ago? I still see businesses running losses left right and center saying that they're gonna monetize user data, any day now. Related "monetizing user data" seems to just mean ads. Ads on everything, forever, until the userbase gets fed up and moves to a new service that definitely won't do that, and the cycle repeats about every 3 years.

Were you not around for the Big Data heyday a decade ago?

varispeed 5mo ago

Until thumb drives became large enough to fit most datasets it stopped becoming Big Data. Just normal data.

ffsm8 5mo ago
We have thumb drives that can store petabytes of data? Or did you mean the "big data" crowd which thought 500GB was noteworthy? I don't think anyone took those serious, neither in 2010s nor now. That was always "small" data
- varispeed 5mo ago
  Most companies using term "big data" had datasets in TB region. One company I had a gig at had full Hadoop cluster setup and their whole dataset was 40GB. Their marketing had all the big data adjacent keywords over the brochures for clients.
  - gf000 5mo ago
    That's a decent quality 3 hours movie :D
- butlike 5mo ago
  > We have thumb drives that can store petabytes of data We do?
  - ffsm8 5mo ago
    Please provide a link.
    - BizarroLand 5mo ago
      You would need 4 and change of these 245tb Kioxias to hold 1 petabyte, and an entire server grade computer to run them. https://www.tomshardware.com/pc-components/ssds/kioxia-unveils-245tb-ssd-the-worlds-highest-capacity-storage-device-could-store-12-500-4k-movies https://www.tomshardware.com/pc-components/ssds/kioxia-unvei... Or 250 of these ~$400 4tb flash drives and an insane number of dongles to connect them all: https://www.slashgear.com/1847725/largest-usb-thumb-drive-highest-capacity-can-buy/ https://www.slashgear.com/1847725/largest-usb-thumb-drive-hi...
      - vunuxodo 5mo ago
        Plus one more for your parity drive.
  - dylan604 5mo ago
    It was a question that you've edited out the punctuation. You're asking the exact same thing as the person you've replied
- 0x457 5mo ago
  My rule of thumb was "can it fit in RAM on a server?" If it can, then it's not big data. 500GB is in the "fits" category.
  - gf000 5mo ago
    You can quadruple that and could still fit in server RAM
    - 0x457 5mo ago
      Correct, and until then it won't be "big data"
jmalicki 5mo ago
To some degree IMO big data is still a mindset when it might take a day to process your data in a normal SQL query. Some tech doesn't scale to the data size for all use cases, and you need different solutions.

ToucanLoucan 5mo ago

Hell you mean a decade ago? I still see businesses running losses left right and center saying that they're gonna monetize user data, any day now. Related "monetizing user data" seems to just mean ads. Ads on everything, forever, until the userbase gets fed up and moves to a new service that definitely won't do that, and the cycle repeats about every 3 years.