10 ms·

i mean, they also have one of the largest ceph deployments. anything is scalable with no budget.

by maybeben 2y ago

hi-v-rocknroll 2y ago
They probably consume Panasas, IBM, DDN, and BeeGFS gear and licensing too.
- adev_ 2y ago
  Nop. Most internal data is spread between Ceph and home-made distributed storage system named EOS (https://indico.cern.ch/event/138478/contributions/149912/attachments/116849/165874/EOS-ALICE.pdf https://indico.cern.ch/event/138478/contributions/149912/att...) running over commodity hardware. The only commerical-backed storage system is the long term storage tape system. Still it has an home-made overlay API over it to interface with the rest of the systems.
- rob_c 2y ago
  Good god no. Nowhere near anything so crass. CEPH and EOS all the way
- dark-star 2y ago
  All their important "administrative" stuff (think Active Directory/LDAP user database, mail and other essential services) run on proprietary storage systems from a commercial vendor (not IBM though), with enterprise support and all. At least that was the case a few years ago when we last talked to one of the heads of IT at CERN, but I guess it hasn't drastically changed
pas 2y ago
slide 22 states that the cost is 1 CHF/TB/month (on 10+2 erasure coded disks), though it would be interesting to do a breakdown of costs (development, hardware, maintenance, datacenter, servicing, management, etc..)
- pclmulqdq 2y ago
  1 CHF/TB/month is a bit expensive for storage at that scale, so it would definitely be interesting to see what they're spending the money on and what they are (and aren't) counting in that price.
  - hackernewds 2y ago
    No budget often tags along with no accountability
  - rob_c 2y ago
    Tape backup, accessibility, networking, availability... At 1CHF/TB that's a lot better than my local university still charging >100x that for such services internally
    - pclmulqdq 2y ago
      Economies of scale in storage are significant. Also, I don't know why you put up with your university charging 100x that when you can store things on AWS for $5-10/TB/month (or less). That comes with all the guarantees (or more) of durability and availability you get from the university.
      - lnauta 2y ago
        Don't forget bandwidth costs at this scale are significant to say the least.
        pclmulqdq 2y ago
        I assume most of that exabyte is stored in 2-3 datacenters, in which case the bandwidth cost is actually relatively small. Downloading it would cost a fortune (or take an eternity), but if it stays in the datacenter (or stays in AWS), bandwidth is cheap.

i mean, they also have one of the largest ceph deployments. anything is scalable with no budget.

hi-v-rocknroll 2y ago

They probably consume Panasas, IBM, DDN, and BeeGFS gear and licensing too.

adev_ 2y ago
Nop. Most internal data is spread between Ceph and home-made distributed storage system named EOS (https://indico.cern.ch/event/138478/contributions/149912/attachments/116849/165874/EOS-ALICE.pdf https://indico.cern.ch/event/138478/contributions/149912/att...) running over commodity hardware. The only commerical-backed storage system is the long term storage tape system. Still it has an home-made overlay API over it to interface with the rest of the systems.
rob_c 2y ago
Good god no. Nowhere near anything so crass. CEPH and EOS all the way
dark-star 2y ago
All their important "administrative" stuff (think Active Directory/LDAP user database, mail and other essential services) run on proprietary storage systems from a commercial vendor (not IBM though), with enterprise support and all. At least that was the case a few years ago when we last talked to one of the heads of IT at CERN, but I guess it hasn't drastically changed

pas 2y ago

slide 22 states that the cost is 1 CHF/TB/month (on 10+2 erasure coded disks), though it would be interesting to do a breakdown of costs (development, hardware, maintenance, datacenter, servicing, management, etc..)

pclmulqdq 2y ago
1 CHF/TB/month is a bit expensive for storage at that scale, so it would definitely be interesting to see what they're spending the money on and what they are (and aren't) counting in that price.
- hackernewds 2y ago
  No budget often tags along with no accountability
- rob_c 2y ago
  Tape backup, accessibility, networking, availability... At 1CHF/TB that's a lot better than my local university still charging >100x that for such services internally
  - pclmulqdq 2y ago
    Economies of scale in storage are significant. Also, I don't know why you put up with your university charging 100x that when you can store things on AWS for $5-10/TB/month (or less). That comes with all the guarantees (or more) of durability and availability you get from the university.
    - lnauta 2y ago
      Don't forget bandwidth costs at this scale are significant to say the least.
      - pclmulqdq 2y ago
        I assume most of that exabyte is stored in 2-3 datacenters, in which case the bandwidth cost is actually relatively small. Downloading it would cost a fortune (or take an eternity), but if it stays in the datacenter (or stays in AWS), bandwidth is cheap.