SEGyges

7 ms·

1.
by SEGyges 1y ago
i hate "stochastic parrot" because it's not even really meaningful I think it's true that models are statistical, inasmuch as P(A|B) where B is the prior sequence is what the loss is computing, and that's statistica
2.
by SEGyges 1y ago
This is true of the system as a whole, but the core neural network is still a next-token predictor.
3.
by SEGyges 1y ago
fortunately i wrote an entire post about what the difference is between the parts of this that it is easy to make sense of and the parts of it that it is prohibitively difficult to make sense of and it was posted on hackernews
4.
by SEGyges 1y ago
I agree, what we do is much closer to growing them than to engineering them. We basically engineer the conditions for growth, and then check the results and try again. My best argument that insights from neuroscience will transfer to neural
5.
by SEGyges 1y ago
My short explanation would be that even for RL, you are training on a next token objective; but the next token is something that has been selected very very carefully for solving the problem, and was generated by the model itself. So you&#x
6.
by SEGyges 1y ago
Every LLM provider caches their KV-cache, it's a publicly documented technique (go stuff that KV in redis after each request, basically) and a good engineering team could set it up in a month.
7.
The Biggest Statistic About AI Water Use Is a Lie(verysane.ai)
3 points by SEGyges 1y ago|1 comments
8.
by SEGyges 1y ago
Because this specific number comes up constantly and is incredibly frustrating it seemed like it really needed to be addressed directly. If we can get past this specific thing we can perhaps have a fact-based conversation about what's
9.
by SEGyges 1y ago
I am not deep into the Linus weeds but my impression is that he doesn't especially care if he's on the receiving end of this. It only started to feel different from "well, the Linux list is the PvP zone" when Linus was s
10.
by SEGyges 2y ago
it is not necessarily 16x if you, e.g., decrease model width by a factor of 4 or so also, but yeah naively the RAM and FLOPs scale up by n^2
11.
by SEGyges 2y ago
you might have better luck giving the LM the original document and having it generate its own OCR independently, then asking the llm to tiebreak between its own generation and the OCR output while the image is still in the context window un
12.
by SEGyges 2y ago
tokens are on average four characters and the number of residual streams (and therefore RAM) the LLM allocates to a given sequence is proportionate to the number of units of input. the flops is proportionate to their square in the attenti
13.
by SEGyges 3y ago
In a world where the options were to 1) pay the author, 2) implement guaranteed citation of the author any time the model gave an answer that was directly derivative, with an option to not do so if the summary was sufficiently vague, or 3)
14.
by SEGyges 3y ago
These are fancy Markov chains in the sense that humans are just chemicals and computers just do math. Technically true, but not even "overly reductive"; it is just wrong if it is used to imply that, e.g., humans just swirl around
15.
by SEGyges 3y ago
This is the correct one.
16.
by SEGyges 3y ago
The counterparties on related legal action are sufficiently litigious that it is probably smarter to DM the magnet link.
17.
by SEGyges 3y ago
You are uploading 5 billion examples of <something>. You cannot filter it manually, of course, because there are five billion of it. Given that it is the year 2024, how hard is it to be positive that a well-resourced team at Stanford
18.
by SEGyges 3y ago
I am pretty sure if the authors were trying to license their works for this purpose we would just not use them at all; it is difficult to see under what circumstances they would stand to profit from this other than by suing people after the
19.
by SEGyges 3y ago
By "these algorithms", do you mean the ones that currently exist, or the ones that will exist next month, next year, or in 2034?
20.
by SEGyges 3y ago
Huckabee v Bloomberg, Meta, et al
21.
by SEGyges 3y ago
See my other comment replying to that.
22.
by SEGyges 3y ago
You and the guy below you in this thread should probably tag me on twitter, same tag as here, I can point you. I do not especially want to leave the discord link in a frontpage hn thread.
23.
by SEGyges 3y ago
I have good news about who wrote this paper
24.
by SEGyges 3y ago
I picked the wrong day to refactor an npm build.