spi

8 ms·

1.
by spi 9mo ago
Yes exactly, I fear that shortening the training time would skew the results. In the very short term, smaller batch size is typically better just because you need a certain amount of gradient updates to move away from the original random, h
2.
by spi 9mo ago
Sorry came a bit late to this reply. Interesting, well, nobody says it's a monotonic function :-) in the limit of _very_ large batches you of course are worse off, because you take a very large amount of computation before taking a sin
3.
by spi 9mo ago
Sorry I just opened that file now, and browsed through it very quickly, but my eye fell on the excerpt: ``` However, we did not observe any speedup by increasing the batch size from 65536 to 131072 for the first stage, thus, we restrict the
4.
by spi 9mo ago
Mmh not really. As OP shows, speed increases with larger batch size, but only initially, until the GPU has high enough utilization; then speed improvements flatten out (although you might get OOM before that and not "really" see t
5.
by spi 9mo ago
Thanks, very nice to see these results! Certainly using GPUs with more RAM makes things simpler to scale. Gradient accumulation is as easy as adding a counter for number of steps and an "if counter % gradient_accumulation_steps:` aroun
6.
by spi 9mo ago
A separate comment about conclusions about why they are worse than OpenAI GPT2 - which to me feel to be missing the point. One main point is batch size - I'd agree with Gemini here. Batch size <= 5 with 1024 seq len is really tiny.
7.
by spi 9mo ago
This is a very nice, detailed post! I have a few minor comments though (maybe a few are discussed somewhere, it's a _long_ article and I can't claim 100% coverage :-) ): Calling it "training LLM" is a bit misleading. Thi
8.
by spi 2y ago
Aside from the weirdness of calling "good old" something that was released 17 months ago :-D I mean, deep learning is evolving at crazy rhythm, but you just can't assume a good paper gets written in days. That said, as others
9.
by spi 2y ago
I know nothing about what makes an industry succeed or fail, and also nothing about web tech, but working in the field I can comment on: > tensorflow looks like currently loosing to pytorch - seems like google got bored and more developm
10.
by spi 2y ago
I know this is HN and here it's not a popular opinion, but maximum security is _not_ always a good idea. Even setting aside the problem of many different actors having to access these details mentioned below, there's value in a si
11.
by spi 2y ago
Do you have sources for "The MFU can be above 40% and certainly well above the 35 % in the estimate"? Looking at [1], the authors there claim that their improvements were needed to push BERT training beyond 30% MFU, and that the &
12.
by spi 2y ago
I'm into AI but not into sound, so I might be saying something stupid here, but I think using something like this for very high volume like concerts would be possibly outright impossible, but, even if not, certainly quite dangerous and
13.
by spi 2y ago
Yep that also sounded weird to me. I had, IIRC, three of my wisdom teeth removed as a teenager, I was living in Italy back then. I think two of them in a single session. General anaesthesia wasn't even an option, the whole thing happen
14.
by spi 2y ago
Variety matters a lot. If you pay 1000 trained labellers, you get 1000 POVs for a good amount of money, and likely can't even think of 1000 good questions to have them ask. If you let 1000000 people give you feedback on random topics f
15.
by spi 2y ago
If climate change were visible at that scale (tiny resolution between 0 and 40 degrees) we'd be all boiled since a while. Still, you can see signs: the maximum temperature until 1990 or so seems to be around 35 degrees, since then ther
16.
by spi 2y ago
IIRC, GPT-4 would actually be a bit _smaller_ to visualize than GPT3. Details are not public, but from the leaks GPT-4 (at least, some by-now old version of it) was a mixture of expert, with every model having around 110B parameters [1]. So
17.
by spi 2y ago
Nothing to pardon, asking questions is always the right thing to do :-) I also didn't look into the paper in great details, although I'm quite sure I am not fooling myself, but still take this with a grain of salt. My understandin
18.
by spi 2y ago
The weights are different, because the model is different. As jzbontar below mentions, the crucial point is that the random noise mask is the same. The diffusion models are trained to turn random noise to an image, and they are deterministi
19.
by spi 3y ago
I suppose varying the neural net size wouldn't be the best way of doing that; very small nets can have very "unhuman-like" behaviour. I'm not an expert on reinforcement learning, but for other fields in deep learning tha
20.
by spi 3y ago
Sharing my experience here. My background is in math (Ph.D. and a couple of postdoc years) before switching to practitioner in deep learning. This year I taught a class at university (as invited prof) in deep learning for students doing a m
21.
by spi 3y ago
I guess I should be wiser than contradicting LeCun on a public forum, but his math doesn't really work out. It only works if there is a unique correct answer to any question, in which case e=1/dict_size which is clearly false - ev
22.
by spi 3y ago
(I'm also replying myself concerning the problem itself). Unless I'm getting myself completely wrong, this also seems to be a very unusually simple problem for IMO's standards. I don't think I ever got myself solving one
23.
by spi 3y ago
Good to know it can do that, in the pasted chat above it didn't. To be honest, it surprised me it couldn't, this isn't exactly a very hard guess given the computation results. It doesn't convince me GPT4 is anywhere clos
24.
by spi 3y ago
A decent start? It says absolutely nothing about how to solve it, except repeating the question. The part where it tries out the few first numbers is entirely wrong, given that 6 does _not_ satisfy the condition (2 does not divide 3+6=9) an
25.
by spi 3y ago
Well, nuclear power is literally the reason why in this map: https://app.electricitymaps.com/map France is almost constantly green, while Germany, Poland, and most of the US, are just different shades of brown...
26.
by spi 3y ago
Are you implying that if there was no Nuclear plant in Fukushima, the Tsunami would not have happened, or the people living there wouldn't have been needed to be evacuated?
27.
by spi 3y ago
I've worked in two companies in Germany and, unless I'm wrong, both of them had 6 months periods for Senior position. At least the second one has no problem retaining good staff. As several others mention here, this is mostly on p
28.
by spi 3y ago
Yes of course, sorry my write-up was confusing: I meant that "adding a ReLU between the two linear layers" (the second option) would result in more parameters than "directly removing the second linear layer" (the first o
29.
by spi 3y ago
Well it depends what you mean by “best” :-) removing the linear layer is the easiest solution (indeed you can’t remove the embedding one; in theory you could replace embedding + linear by one hot encoding + linear, adapting the input dimens
30.
by spi 3y ago
Kudos for the work! Stupid comment (not really on the main topic of the blogpost, but might be useful anyway for future "toy example" models): in the initial SimpleBrokenModel class [EDIT: and also in SimpleModel), there is actual

More ›