8 ms·
A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Go
by arcade79 2y ago
A lot of misunderstandings among the commenters here.
From the link:
"the total compute cost it would take to replicate the paper"
It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper.
For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priority jobs. It's effectively free resources (except electricity consumption). If any "more important" jobs with a higher priority comes in and asks for the resources, the paper-writers jobs will just be preempted.
- mrazomor 2y agoThis assumes the common resources (CPU, RAM, etc.), not the ones required for the LLM training (GPU, TPU, etc.). It's different economy. TL; DR: It's not ~free.
- akutlay 2y agoWhy does GPU matter? Do you think GCP keeps GPU utilization at 100% at all times?
- mrazomor 2y agoWhat the OP is referring to requires overprovisioning of the high priority traffic and the sine-like utilization (without it, the benefits of the "batch" tier is close to zero -- the preemption is too high for any meaningful work when you are close to the top of the utilization hill). You get that organically when you are serving lots of users. And, there's not much GPUs etc. used for that. Training LLMs gives you a different utilization pattern. The "best effort" resources aren't as useful in that setup.
- bbminner 2y agoBecause accelerators (tpus, gpus) unlike ram/cpu are notoriously hard to timeshare and vitrualize. So if you get evicted in an environment like that, you have to reload your entire experiment state from a model checkpoint. With giant models like that, it might take dozens of minutes. As a result, I doubt that these experiments are done using "spare" resources - in that case, constant interruptions and reloading would result in these experiments finishing sometime around the heat death of the universe :)
- bombcar 2y agoThis is the side effect of underutilized capital and it’s present in many cases. For example, if YOU want to rent a backhoe to do some yard rearrangement it’s going to cost you. But Bob who owns BackHoesInc has them sitting around all the time when they’re not being rented or used; he can rearrange his yard wholesale or almost free.
- deleted 2y ago[deleted]
- thaumasiotes 2y ago> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low-value work. Option value is real value; being prepared for unforeseen but urgent circumstances is a real use.
- nathancahill 2y agoSame effect when leasing companies let office space sit unoccupied for years on end. The future value is higher than the marginal value of reducing the price to fill it with a tenant.
- unyttigfjelltol 2y agoReal estate is a playground for irrationally hopeful or stubborn participants.
- Bjartr 2y agoThat may be part of it for spaces properties left unleased for years, but I believe it's not the only part. I believe the larger factor, and someone correct me if they have a better understanding of this, is that for commercially rented properties the valuation used to determine the mortgage terms you get takes into account what you claim to be able to get from rent. Renting for less than that reduces the valuation and can put you upside down on the mortgage. But the bank will let you defer mortgage payments, effectively taking each month of mortgage duration and moving it from now to after the last month of the mortgage duration, extending the time they earn interest for. So if no one want to lease the space at that price after a prior lessee leaves for whatever reason, it's better for the property owner financially to leave the space vacant, sometimes for years, until someone willing to pay that price comes along, than to lower the rent and get a tenant.
- punnerud 2y agoCan others also buy the “best effort” tier? If the job could easily run for weeks, even when you could buy your way for doing it in a day. Then have a bidding on this “best effort” resource, where they factor in electricity at any given time
- v3ss0n 2y agoSure,.land a job there, work the way all up against the cooperate bs and toxicity and you can get best effort tier. Those effort needs to be added in the cost calculation too.
- v3ss0n 2y agoSure,.land a job there, work the way all up against the cooperate bs and toxicity and you can get best effort tier. Those effort needs to be added in the cost calculation too
- curt15 2y agoIs the "best effort" tier similar to AWS spot instances?
- deleted 2y ago[deleted]
- WJW 2y agoAt every cloud provider there's probably a tier below "spot" (or whatever the equivalent is called at AWS's competitors) that is used for the low-priority jobs of the cloud provider itself.
- jeffbee 2y agoYou can speculate about this or you can look at how Google's internal workloads actually run, because they have released a large and detailed set of traces from Borg. They're really open about this. https://github.com/google/cluster-data https://github.com/google/cluster-data
- dweekly 2y agoPossible corollary: it may be difficult to regularly turn out highly compute-dependent research if you're paying full retail rack rates for your hardware (i.e. using someone else's cloud).
- huijzer 2y agoStill, don’t get high on your own supply.
- imtringued 2y agoAccording to neoclassical economists this is impossible since you can easily and instantaneously scale infrastructure up and down continuously at no cost and the future is known so demand can be predicted reliably. The problem with neoclassical economics is that it doesn't concern itself with the physical counterpart of liquidity. It is assumed that the physical world is just as liquid as the monetary world. The "liquidity mismatch" between money and physical capital must be bridged through overprovisioning on the physical side. If you want the option to choose among n different products, but only choose m products, then the n - m unsold products must be priced into the m bought products. If you can repurpose the unsold products, then you make a profit or you can lower costs for the buyer of the m products. I would even go as far as to say that the production of liquidity is probably the driving force of the economy, because it means we don't have to do complicated central planning and instead use simple regression models.
- jopsen 2y ago> I would even go as far as to say that the production of liquidity is probably the driving force of the economy. Isn't that all what high frequency traders would say? :) Perhaps there is some limit at which additional liquidity doesn't offer much value?
- marcosdumay 2y agoI think you completely misunderstood the GP. There isn't much there about stocks markets.
- 152334H 2y agoIs it free-priority based? I was told by an employee that GDM internally has a credits system for TPU allocation, with which researchers have to budget out their compute usage. I may have completely misunderstood what they were describing, though.
- deleted 2y ago[deleted]
- rldjbpin 2y agoif this is the way they pull it off consistently, it might be a good business models for those working on research like stability to also moonlight a gpu cloud service. it is a hustle only for the near future while this bubble lasts, but can help reduce costs.