6 ms·

Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. The amortization periods on that spending is ridiculously long. Effec

Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. The amortization periods on that spending is ridiculously long. Effectively every single H100 in existence now will be e-waste in 5 years or less. Not exactly railroad infrastructure here, or even dark fiber.

fooker 1y ago
> Effectively every single H100 in existence now will be e-waste in 5 years or less. This remains to be seen. H100 is 3 years old now, and is still the workhorse of all the major AI shops. When there's something that is obviously better for training, these are still going to be used for inference. If what you say is true, you could find a A100 for cheap/free right now. But check out the prices.
- fxtentacle 1y ago
  Yeah, I can rent an A100 server for roughly the same price as what the electricity would cost me.
  - fooker 1y ago
    That is true for almost any cloud hardware.
  - fennecbutt 1y ago
    Because they buy the electricity in bulk so these things are not the same.
  - typpilol 1y ago
    Where?
    - Cheer2171 1y ago
      ~$1.25-1.75/hr at Runpod or vast.ai for an A100 Edit: https://getdeploying.com/reference/cloud-gpu/nvidia-a100 https://getdeploying.com/reference/cloud-gpu/nvidia-a100
      - diziet 1y ago
        The A100 SXM4 has a TDP of 400 watts, let's say about 800 with cooling etc overhead. Bulk pricing per KWH is about 8-9 cents industrial. We're over an order of magnitude off here. At 20k per card all in price (MSRSP + datacenter costs) for the 80GB version, with a 4 year payoff schedule the card costs 57 cents per hour (20,000/24/365/4) assuming 100% utilization.
9rx 1y ago
> Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. That which survived, at least. A whole lot of rail infrastructure was not viable and soon became waste of its own. There was, at one time, ten rail lines around my parts, operated by six different railway companies. Only one of them remains fully intact to this day. One other line retained a short section that is still standing, which is now being used for car storage, but was mostly dismantled. The rest are completely gone. When we look back in 100 years, the total amortization cost for the "winner" won't look so bad. The “picks and axes” (i.e. H100s) that soon wore down, but were needed to build the grander vision won't even be a second thought in hindsight.
- lesuorac 1y ago
  If 1/10 investment lasts 100 years that seems pretty good to me. Plus I'd bet a lot of the 9/10 of that investment had a lot of the material cost re-coup'd when scrapping the steel. I don't think you're going to recoup a lot of money from the H100s.
  - 9rx 1y ago
    Much like LLMs. There are approximately 10 reasonable players giving it a go, and, unless this whole AI thing goes away, never to be seen again, it is likely that one of them will still be around in 100 years. H100s are effectively consumables used in the construction of the metaphorical rail. The actual rail lines had their own fare share of necessary tools that retained little to no residual value after use as well. This isn't anything unique.
    - munk-a 1y ago
      H100s being thought of as consumables is keen - it much better to analogize the H100s to coal and chip manufacturer the mine owner - than to think of them as rails. They are impermanent and need constant upkeep and replacement - they are not one time costs that you build as infra and forget about.
- palmotea 1y ago
SJC_Hacker 1y ago
> Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. The amortization periods on that spending is ridiculously long. Are we? I was under the impression that the tracks degraded due to stresses like heat/rain/etc. and had to be replaced periodically.
- ralph84 1y ago
  The track bed, rails, and ties will have been replaced many times by now. But the really expensive work was clearing the right of way and the associated bridges, tunnels, etc.
  - intrasight 1y ago
    I am really digging the railroad analogies in this discussion! There are some striking similarities if you do the right mappings and timeframe transformations. I am an avid rail-to-trail cycler and more recently a student of the history of the rail industry. The result was my realization that the ultimate benefit to society and to me personally is the existence of these amazing outdoor recreation venues. Here in Western PA we have many hundreds of miles of rail-to-trail. My recent realization is that it would be totally impossible for our modern society to create these trails today. They were built with blood, sweat, tears and much dynamite - and not a single thought towards environmental impact studies. I estimate that only ten percent of the rail lines built around here are still used for rail. Another ten percent have become recreational trails. That percent continues to rise as more abandoned rail lines transition to recreational use. Here in Western PA we add a couple dozen miles every year. After reading this very interesting discussion, I've come to believe that the AI arms race is mainly just transferring capital into the pockets of the tool vendors - just as was the case with the railroads. The NVidia chips will be amortized over 10 years and the models over perhaps 2 years. Neither has any lasting value. So the analogy to rail is things like dynamite and rolling stock. What in AI will maintain value? I think the data center physical plants, power plants and transmission networks will maintain their value longer. I think the exabytes of training data will maintain their value even longer. What will become the equivalent of rail-to-trail? I doubt that any of the laborers or capitalists building rail lines had foreseen that their ultimate value to society would be that people like me could enjoy a bike ride. What are the now unforeseen long-term benefit to society of this AI investment boom? Rail consolidated over 100 years into just a handful of firms in North America, and my understanding is that these firms are well-run and fairly profitable. I expect a much more rapid shakeout and consolidation to happen in AI. And I'm putting my money on the winners being Apple first and Google second. Another analogy I just thought of - the question of will the AI models eventually run on big-iron or in ballpoint pens. It is similar to the dichotomy of large-scale vs miniaturized nuclear power sources in Asimov's Foundation series (a core and memorable theme of the book that I haven't seen in the TV series).
Spooky23 1y ago
How was your trip down the third Avenue El? Did your goods arrive via boxcar to 111 8th Ave?
- selimthegrim 1y ago
  At the rate they are throwing obstacles at the promised subway which they got rid of the 3rd Ave El for maybe his/her grandkids will finish the trip.
hyperbovine 1y ago
> Effectively every single H100 in existence now will be e-waste in 5 years or less. This is definitely not true, the A100 came out just over 5 years ago and still goes for low five figures used on eBay.

fooker 1y ago

> Effectively every single H100 in existence now will be e-waste in 5 years or less. This remains to be seen. H100 is 3 years old now, and is still the workhorse of all the major AI shops. When there's something that is obviously better for training, these are still going to be used for inference. If what you say is true, you could find a A100 for cheap/free right now. But check out the prices.

fxtentacle 1y ago
Yeah, I can rent an A100 server for roughly the same price as what the electricity would cost me.
- fooker 1y ago
  That is true for almost any cloud hardware.
- fennecbutt 1y ago
  Because they buy the electricity in bulk so these things are not the same.
- typpilol 1y ago
  Where?
  - Cheer2171 1y ago
    ~$1.25-1.75/hr at Runpod or vast.ai for an A100 Edit: https://getdeploying.com/reference/cloud-gpu/nvidia-a100 https://getdeploying.com/reference/cloud-gpu/nvidia-a100
    - diziet 1y ago
      The A100 SXM4 has a TDP of 400 watts, let's say about 800 with cooling etc overhead. Bulk pricing per KWH is about 8-9 cents industrial. We're over an order of magnitude off here. At 20k per card all in price (MSRSP + datacenter costs) for the 80GB version, with a 4 year payoff schedule the card costs 57 cents per hour (20,000/24/365/4) assuming 100% utilization.

9rx 1y ago

> Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. That which survived, at least. A whole lot of rail infrastructure was not viable and soon became waste of its own. There was, at one time, ten rail lines around my parts, operated by six different railway companies. Only one of them remains fully intact to this day. One other line retained a short section that is still standing, which is now being used for car storage, but was mostly dismantled. The rest are completely gone. When we look back in 100 years, the total amortization cost for the "winner" won't look so bad. The “picks and axes” (i.e. H100s) that soon wore down, but were needed to build the grander vision won't even be a second thought in hindsight.

lesuorac 1y ago
If 1/10 investment lasts 100 years that seems pretty good to me. Plus I'd bet a lot of the 9/10 of that investment had a lot of the material cost re-coup'd when scrapping the steel. I don't think you're going to recoup a lot of money from the H100s.
- 9rx 1y ago
  Much like LLMs. There are approximately 10 reasonable players giving it a go, and, unless this whole AI thing goes away, never to be seen again, it is likely that one of them will still be around in 100 years. H100s are effectively consumables used in the construction of the metaphorical rail. The actual rail lines had their own fare share of necessary tools that retained little to no residual value after use as well. This isn't anything unique.
  - munk-a 1y ago
    H100s being thought of as consumables is keen - it much better to analogize the H100s to coal and chip manufacturer the mine owner - than to think of them as rails. They are impermanent and need constant upkeep and replacement - they are not one time costs that you build as infra and forget about.
palmotea 1y ago

SJC_Hacker 1y ago

> Yep, we are (unfortunately) still running on railroad infrastructure built a century ago. The amortization periods on that spending is ridiculously long. Are we? I was under the impression that the tracks degraded due to stresses like heat/rain/etc. and had to be replaced periodically.

ralph84 1y ago
The track bed, rails, and ties will have been replaced many times by now. But the really expensive work was clearing the right of way and the associated bridges, tunnels, etc.
- intrasight 1y ago
  I am really digging the railroad analogies in this discussion! There are some striking similarities if you do the right mappings and timeframe transformations. I am an avid rail-to-trail cycler and more recently a student of the history of the rail industry. The result was my realization that the ultimate benefit to society and to me personally is the existence of these amazing outdoor recreation venues. Here in Western PA we have many hundreds of miles of rail-to-trail. My recent realization is that it would be totally impossible for our modern society to create these trails today. They were built with blood, sweat, tears and much dynamite - and not a single thought towards environmental impact studies. I estimate that only ten percent of the rail lines built around here are still used for rail. Another ten percent have become recreational trails. That percent continues to rise as more abandoned rail lines transition to recreational use. Here in Western PA we add a couple dozen miles every year. After reading this very interesting discussion, I've come to believe that the AI arms race is mainly just transferring capital into the pockets of the tool vendors - just as was the case with the railroads. The NVidia chips will be amortized over 10 years and the models over perhaps 2 years. Neither has any lasting value. So the analogy to rail is things like dynamite and rolling stock. What in AI will maintain value? I think the data center physical plants, power plants and transmission networks will maintain their value longer. I think the exabytes of training data will maintain their value even longer. What will become the equivalent of rail-to-trail? I doubt that any of the laborers or capitalists building rail lines had foreseen that their ultimate value to society would be that people like me could enjoy a bike ride. What are the now unforeseen long-term benefit to society of this AI investment boom? Rail consolidated over 100 years into just a handful of firms in North America, and my understanding is that these firms are well-run and fairly profitable. I expect a much more rapid shakeout and consolidation to happen in AI. And I'm putting my money on the winners being Apple first and Google second. Another analogy I just thought of - the question of will the AI models eventually run on big-iron or in ballpoint pens. It is similar to the dichotomy of large-scale vs miniaturized nuclear power sources in Asimov's Foundation series (a core and memorable theme of the book that I haven't seen in the TV series).

Spooky23 1y ago

How was your trip down the third Avenue El? Did your goods arrive via boxcar to 111 8th Ave?

selimthegrim 1y ago
At the rate they are throwing obstacles at the promised subway which they got rid of the 3rd Ave El for maybe his/her grandkids will finish the trip.

hyperbovine 1y ago

> Effectively every single H100 in existence now will be e-waste in 5 years or less. This is definitely not true, the A100 came out just over 5 years ago and still goes for low five figures used on eBay.