eden-u4

7 ms·

1.
by eden-u4 5mo ago
I don't understand the RMS table, shouldn't it be non commutative? Eg "example 0 vs 1"'s RMS != "example 1 vs 0"'s RMS? Which doesn't seem the case for the checkpoints I checked.
2.
by eden-u4 5mo ago
There are not enough samples in that book to generate new "infinite" data.
3.
by eden-u4 5mo ago
Guardrailing is usually done with a smaller model (< 1b) to filter out simple "not aligned prompt" and not waste compute.
4.
by eden-u4 6mo ago
But what if your rubber duck is actually steering your thought process (since you may not have a consolidated one)? In this way I think the AI as editor is far better than a rubber duck AI. While in the former, it might point out your mista
5.
by eden-u4 7mo ago
This is basically a diffusion model: start from a random seed, use a generative process to transform it into something
6.
Neural Scaling and the Quanta Hypothesis(ericjmichaud.com)
3 points by eden-u4 8mo ago|1 comments
7.
by eden-u4 9mo ago
mine too, but none was such a dick. also, anything related to school (particularly at a young age), is not viewed as something to boast of (at least in my experience in italy, serbia and portugal).
8.
by eden-u4 10mo ago
Also, they have all the infra to actually use all that tpus advantage (as well as actual researchers, contrariwise to OpenAI)
9.
by eden-u4 10mo ago
what type of experiments did you run in less than a week to be so dismissing? (seriously curious)
10.
A Philosophy of Software Design vs. Clean Code (2024)(github.com)
1 points by eden-u4 10mo ago|0 comments
11.
by eden-u4 10mo ago
Not OP, but I guess based on your comment: > But the extent and the way in which Zig specifically puts it to use -- which includes, but is not limited to, how it is used to replace other features that can then be avoided (and all without
12.
by eden-u4 11mo ago
wow, thanks for this long explanation.
13.
by eden-u4 1y ago
so they are basically using a similar idea to that of a stirling engine in thermoelectric generator or they use a different mechanism to produce energy?
14.
by eden-u4 1y ago
No open model/weights?
15.
by eden-u4 1y ago
I don't have much experience with ROCm for large trainings, but NVIDIA is still shit with driver+cuda version+other things. The only simplification is due to ubuntu and other distros that already do the heavy lift by installing all req
16.
by eden-u4 1y ago
why don't you ask the model about the shrinked system prompt and the original system prompt? in this way you can infer whether the same relevant informations are "stored" in the hidden state of the model. Or better yet, check
17.
by eden-u4 1y ago
I dunno, these reasoning models seems kinda "dumb" because they try to bootstrap itself via reasoning, even though a simple direct answer might not exist (for example key information are missing for a proper answer). Ask something
18.
Chemdle(chemdle.com)
2 points by eden-u4 2y ago|0 comments
19.
by eden-u4 2y ago
When they say "in 20-50 years" it is implied that it might not exist ever. See nuclear fusion or AGI.
20.
by eden-u4 2y ago
I think the issue with RL is that, in order for a model to perform well in a task, you have to make it stubborn. In the same way a student that thinks outside the scope of the task might not perform well in a graded exam, but that does not
21.
by eden-u4 2y ago
because non-tech friends don't have coding problems, which is the only problem LLM are ok at.
22.
by eden-u4 2y ago
anyone knows which program they used to record the demo?
23.
by eden-u4 2y ago
meta is large enough (and diversified enough), when OpenAI valuation goes to pennies (because LLM won't solve any real problem, nor they'll remain relevant in the long term and open source alternative are more than enough for most
24.
by eden-u4 2y ago
this project only uses kaggle metadata and abstract from arxiv. Moreover it is "focused" on only 5-6 categories in the arxiv. Therefore, the costs are marginal. Plus you could use a mixed system: first you index the abstract of th
25.
by eden-u4 2y ago
if you try hard enough you can always ~latinize~ ASCIIfy a language
26.
by eden-u4 2y ago
ah, numerical instability in the warmup stage might be the issue then?
27.
by eden-u4 2y ago
that's a wrong simile given that you would get the same end result in both cases. I'm not using a transformer, just a plain Feedforward, Relu and dropout for a simple classifier. I don't know, I can be wrong. I hope and some
28.
by eden-u4 2y ago
Tried the source code on a toy model: adam took 2 epochs to train a 10k parameters model, this didn't achieve anything useful in 20. Tweaked a bit the hyper parameters and such, but nothing. Probably a bogus implementation?