mota7

10 ms·

1.
by mota7 3mo ago
Heat is mostly driven by leakage current and gate capacitance. The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to control with smaller device
2.
by mota7 4mo ago
If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual implementation of this is
3.
by mota7 1y ago
Is there really that big a different in TFLOPS between the GB100 and GB202 chips? The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?
4.
by mota7 1y ago
Land the western australia wheat belt sells for less than $1000/acre. Is that very expensive?
5.
by mota7 1y ago
Yes that bugged me too. If you replace 'precisely' with 'approximately' everywhere in the article it becomes much improved ;)
6.
by mota7 1y ago
There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core. So GPUs have ~120 small systolic arrays, one per SM (aka, a tensorcor
7.
by mota7 2y ago
The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow. So to make a drastic change in direction
8.
by mota7 2y ago
Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab. And where a and b are both small-ish, ab is really small and can just be ignored. So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns ou
9.
by mota7 2y ago
I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature. They're effectively doing softmax with a fixed temperature, but it's unclear that this work is g
10.
by mota7 4y ago
The paper says "... optimized on next-word prediction only". Which is absolutely correct in 2023. ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.
11.
by mota7 4y ago
Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'. It's a delicate balance based on the locations that rows are being read and writte
12.
by mota7 4y ago
> Gradient accumulation doesn't work with batch norms so you really need that memory. Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated? (precisely be
13.
by mota7 4y ago
It's hard to convey just how ridiculously complex and expensive 5nm is compared with 90nm. 5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers. As a data point: 5nm lithography equipment is > $1
14.
by mota7 4y ago
Excellent example: "reflective cistern tank with a reflection of the back of a truck transporting stop signs" People tend to forget just how hard the edge cases in vision are!
15.
by mota7 4y ago
Do you know why the blood thinners helped? (I'm assuming there was some underlying condition that this treated?)
16.
by mota7 4y ago
The problem is that predicting a pixel requires knowing what the pixels around it looks like. But if we start with lots of noise, then the neighboring pixels are all just noise and have no signal. You could also think of this as: We start w
17.
by mota7 4y ago
Minimum is around 1 kWh/m^3 for sea-water levels of salt concentration. (It varies a fair bit depending on salinity, the actual salts involved, the temperature etc etc).
18.
by mota7 4y ago
This is surprisingly poor production? Peak insolation varies widely, but 1000W/m^2 is a typical value. 5.8L/hr/m^2 means that it's using something like 180kWh/m^3 on raw solar insolation. For comparison, reverse osm
19.
by mota7 4y ago
Nuts, you are correct that it isn't optimal. It's not quite as simple as just 2 bits per digit, because the last digit can only be large if the earlier digits are small. This makes the worst case is (1,2,6) which encodes as 48, fo
20.
by mota7 4y ago
For the first case, you can't pick a number lower than any existing number. So if 4 is already picked, then only 5 or 6 can be picked, but if we pick 6, then there's no third number available to pick, so there only 1 possibility (
21.
by mota7 4y ago
Yes. The formulation in the blog post is a bit messy, but possibly more general way to think about it is something like: 0. set y = 1. set x = 0. 1. How many possible choices are there _given what we've picked so far_? Call it N. 2. Pi
22.
by mota7 6y ago
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in comparison to t
23.
by mota7 6y ago
The difficultly here is that there's an implicit assumption: 'Noise' is implicitly defined to be anything that isn't learned by the network. Now in same cases that may indeed be actual process noise! But there are many f
24.
by mota7 6y ago
Can't up-vote this hard enough!
25.
by mota7 6y ago
'half' is ambiguous here: They lose 1 bit of expressiveness for the mantissa, and 1 bit in the exponent, giving an efficiency loss of 2 bits from 32 == ~3%. So there's very little loss in using single-precision floating point
26.
by mota7 8y ago
Looks like this is a re-hash of the data from https://www.oag.com/blog/billion-dollar-route