11 ms·

Do Machine Learning Models Memorize or Generalize?

454 points by 1wheel 3y ago

davidguetta 3y ago
hierarchize would be a better term than generalize
- 3cats-in-a-coat 3y ago
  Generalize is seeing common principles, patterns, between disparate instances of a phenomena. It's a proper word for this.
  - davidguetta 3y ago
    Generalize has a tendency to imply you can extrapolate. And in most case it's actually the opposite that happens: neural nets tend to COMPRESS the data. (which in turn is a good thing in many case because the data is noisy)
    - 3cats-in-a-coat 3y ago
      The point of compression is to decompress after. That's what happens during inference, and when the extrapolation occurs. Let's say I tell GPT "write 8 times foobar". Will it? Well then it understands me and can extrapolate from the request to the proper response, without having specifically "write 8 times foobar" in its model. Most decompression algorithms focus on predicting the next token (byte, term, etc.), believe it or not. The more accurately they predict the next token, the less information you need to store to correct misprediction.
  - Chabsff 3y ago
    That's a common mechanism to achieve generalization, but the term is a little more general (heh) than that. It specifically refers to correctly handling data that lives outside the distribution presented by the training data. It's a description of a behavior, not a mechanism. Which may or may not be appropriate depending on whether you are talking about *what* the model does or *how* it achieves it.
    - 3cats-in-a-coat 3y ago
      Kinda fuzzy what's "in the distribution", because it depends on how deeply the model interprets it. If it understands examples outside the distribution... that kinda puts them in the distribution. General understanding makes the information in the distribution very wide. Shallow understanding makes it very narrow. Like say recognizing only specific combinations of pixels verbatim.
      - Chabsff 3y ago
        I think you are misinterpreting. The distribution present in the training set in isolation (the one I'm referring to, and is not fuzzy in the slightest) is not the same thing as the distribution understood by the trained model (the one you are referring to, and is definitely more conceptual and hard to characterize in non-trivial cases). "Generalization" is simply the theoretical measure of how much the later extends beyond the former, regardless of how that's achieved.
        3cats-in-a-coat 3y ago
        I'm saying how you determine the distribution in the training set depends on what the model understands and what the people who selected the dataset understand. There's no distribution of meaning in the training set that's independent of interpretation and understanding. Aside from maybe the literal series of bits (and words and pixels) in it, as encoded. In statistics that is not as severe a problem because you can plot how the data distribution lies in a commonly agreed upon position in one or more clearly defined and agreed upon dimensions. And you can look at the chart and talk about this shared interpretation objectively, and its distribution. Although as a matter of fact just as often it matters what questions you asked, and how and when and whom you asked, for the distribution of answers you got. Lying with statistics is easy as it's full of hidden variables. This is why statistics is great when the data is simple and the analysis is simple, mathematical, objective, but social studies tend to yield, whatever you want them to yield. So. What dimensions are we talking about with a self-evolved model? You have some understanding of what the data is, subjective to you. Maybe your team has some shared understanding of what the data covers, you have overlap. But the model has its own understanding, evolved independently. How much does it overlap with you? Not as much as you think. It's a problem decades old, that people give to the model data that contains things they didn't realize it contains. They themselves didn't see that. And then get surprised by the results. Say when an apple falls on your head, did you realize this contains the data required to describe classic mechanics? For centuries, billions of people didn't realize. To Newton it was there as clear as daylight. In the apple's fall. I know, the example is a myth, but the principle stands. Another example, a video of the change of light patterns reflected on the floor around the corner of room where a person, out of frame, is writing on a computer. What does this data contain? You think nothing much. Maybe it contains how a floor looks. To a model, it can easily also contain what the person who is not in frame, wrote on their keyboard. So given all this... what IS in the distribution? Depends with whose eyes you're looking. Your eyes are not the most objective eyes, nor the most intelligent eyes. You have no anchor to point to as the ultimate arbiter of what complex data contains or does not.
- version_five 3y ago
  Anything would be better than "grokking". From what I gather they're talking about double descent which afaik is the consequence of overparameterization leading to a smooth interpolation between the training data as opposed to what happens in traditional overfitting. Imagine a polynomial fit with the same degree as the number of data points (swinging up and down wildly away from the data) compared with a much higher degree fit that could smoothly interpolate between the points while still landing right on them. None of this is what I would call generalization, it's good interpolation, which is what deep learning does in a very high dimensional space. It's notoriously awful at extrapolating, ie generalizing to anything without support in the training data.
  - Jack000 3y ago
    double descent is a different phenomenon from grokking
    - tysam_and 3y ago
      Nope, they are the same, just that grokking is when the KL between the representable information of the implicit biases and the data is extremely high (i.e. the network is poorly-designed or oriented for the task). It's an informal term that not everyone accepts. Double-descent is acceptable as it describes a general phenomenon that is a natural consequence of a phase transition during neural network training. Grokking is like, to me, the 'fetch' of neural network terms. It's not new, it adds a seeming layer of separation from double-descent (which is is -- just very delayed), and it's not really accepted by everyone. I personally do not like it at all. Especially because language affects _our_ implicit biases about what neural networks can and cannot do. We've already seen that their capacities and performance can be pushed way beyond what we traditionally expect of them. But to summarize, they are the same. And this is why we need good terminology, as well, because poor adoption and boosting of improper terminology induces excess regret in the information exchange surface between agents in a game-theoretic sense in this lovely landscape of the ML world.
  - visarga 3y ago
    > It's notoriously awful at extrapolating, ie generalizing to anything without support in the training data. Scientists are also pretty lousy at making new discoveries without labs. They just need training data.
- ot 3y ago
  "hierarchize" only describes your own mental model of how knowledge organization and reasoning may work in the model, not the actual phenomenon being observed here. "generalize" means going from specific examples to general cases not seen before, which is a perfectly good description of the phenomenon. Why try to invent a new word?
  - davidguetta 3y ago
    > hierarchize" only describes your own mental model of how knowledge organization and reasoning may work in the model, not the actual phenomenon being observed here It's not true, if you look at deep CNN the lower layers show lines, the higher complex stuff like eyes or football players etc.. Herarchisation of information actually emerges naturally in NNs. Generalization often implies extrapolation on new data, which is just not the case most of the time with NNs and why i didn't like the word
MagicMoonlight 3y ago
Memorise because there is no decision component. It attempts to just brute force a pattern rather than thinking through the information and making a conclusion.
wwarner 3y ago
This is such a good explainer
deleted 3y ago
[deleted]
greenflag 3y ago
It seems the take home is weight decay induces sparsity which helps learn the "true" representation rather than an overfit one. It's interesting the human brain has a comparable mechanism prevalent in development [1]. I would love to know from someone in the field if this was the inspiration for weight decay (or presumably just the more equivalent nn pruning [2]). [1] https://en.wikipedia.org/wiki/Synaptic_pruning https://en.wikipedia.org/wiki/Synaptic_pruning [2] https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network) https://en.wikipedia.org/wiki/Pruning_(artificial_neural_net...
- deleted 3y ago
  [deleted]
- visarga 3y ago
  The inspiration for weight decay was to reduce the capacity to memorize of the model until it perfectly fits the complexity of the task, not more not less. A model more complex than the task is over-fitting, the other one is under-fitting. Got to balance them out. But the best cure for over-fitting is to make the dataset larger and ensure data diversity. LLMs have datasets so large they usually train one epoch.
  - nightski 3y ago
    It sounds nice in theory, but the data itself could be problematic. There is no temporal nature to it. You can have duplicate data points, many data points that are closely related but describe the same thing/event/etc.. So while only showing the model each data point once ensures you do not introduce any extra weight on a data point, if the dataset itself is skewed it doesn't help you at all. Just by trying to make the dataset diverse you could skew things to not reflect reality. I just don't think enough attention has been paid to the data, and too much the model. But I could be very wrong. There is a natural temporality to the data humans receive. You can't relive the same moment twice. That said, human intelligence is on a scale too and may be affected in the same way.
    - visarga 3y ago
_ache_ 3y ago
Does anyone know how that charts are created ? I bet that it's half generated by some sort of library and them manually improved but the generated animated SVG are beautiful.
- 1wheel 3y ago
  Basically just a bunch of d3 — could be cleaned up significantly, but that's hard to do while iterating and polishing the charts. I also have a couple of little libraries for things like annotations, interleaving svg/canvas and making d3 a bit less verbose. - https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking https://github.com/PAIR-code/ai-explorables/tree/master/sour... - https://1wheel.github.io/swoopy-drag/ https://1wheel.github.io/swoopy-drag/ - https://github.com/gka/d3-jetpack https://github.com/gka/d3-jetpack - https://roadtolarissa.com/hot-reload/ https://roadtolarissa.com/hot-reload/
  - iaw 3y ago
    I was going to ask the same question. Those are some great visualizations
mjburgess 3y ago
Statistical learning can typically be phrased in terms of k nearest neighbours In the case of NNs we have a "modal knn" (memorising) going to a "mean knn" ('generalising') under the right sort of training. I'd call both of these memorising, but the latter is a kind of weighted recall. Generalisation as a property of statistical models (ie., models of conditional freqs) is not the same property as generalisation in the case of scientific models. In the latter a scientific model is general because it models causally necessary effects from causes -- so, necessarily if X then Y. Whereas generalisation in associative stats is just about whether you're drawing data from the empirical freq. distribution or whether you've modelled first. In all automated stats the only diff between the "model" and "the data" is some sort of weighted averaging operation. So in automated stats (ie., ML,AI) it's really just whether the model uses a mean.
- bippihippi1 3y ago
  it's been proven that all models learned by gradient descent are equivalent to kernel machines. interpolation isn't generalization. if theres a new input sufficiently different from the training data the behaviour is unknown
  - xapata 3y ago
    One weird trick ... There's some fox and hedgehog analogy I've never understood.
  - visarga 3y ago
    but when the model trains on 13T tokens it is hard to be OOD
  - drdeca 3y ago
    Can you say what that says about the behavior described with the modular arithmetic in the article? And, in particular, how to interpret the fact that different hyperparameters determined whether runs, obtaining equally high accuracy on the training data, got good or bad scores on the test data, in terms of the "view it as a kernel machine/interpolation" lens? My understanding is that the behavior in at least one of those "models learned by gradient descent are equivalent to [some other model]" papers, works by constructing something which is based on the entire training history of the network. Is that the kernel machines one, or some other one?
taeric 3y ago
I'm curious how representative the target function is? I get that it is common for you to want a model to learn the important pieces of an input, but a string of bits, and only caring about the first three, feels particularly contrived. Literally a truth table on relevant parameters of size 8? And trained with 4.8 million samples? Or am I misunderstanding something there? (I fully expect I'm misunderstanding something.)
- jaggirs 3y ago
  I have observed this pattern before in computer vision tasks (train accuracy flatlining for a while before test acc starts to go up). The point of the simple tasks is to be able to interpret what could be going on behind the scenes when this happens.
  - taeric 3y ago
    No doubt. But I have also seen what people thought were generalized models failing on outlier, but valid, data. Quite often. Put another way, it isn't just how simple this task seems to be in the number of terms that are important, but isn't it also a rather dense function? Probably better question to ask is how sensitive are models that are looking at less dense functions to this? (Or more dense.). I'm not trying to disavow the ideas.
    - visarga 3y ago
      Maybe humans are also failing a lot in out of distribution settings. It might be inherent.
      - taeric 3y ago
        We have names for that. :D. Stereotypes being a large one. Racism being motivated interpretation on the same ideas. Right?
        lucubratory 3y ago
deleted 3y ago
[deleted]
gorjusborg 3y ago
Grr, the AI folks are ruining the term 'grok'. It means roughly 'to understand completely, fully'. To use the same term to describe generalization... just shows you didn't grok grokking.
- erwald 3y ago
  "Grok" in AI doesn't quite describe generalization, it's more specific that that. It's more like "delayed and fairly sudden generalization" or something like that. There was some discussion of this in the comments of this post[1], which proposes calling the phenomenon "eventual recovery from overfitting" instead. [1] https://www.lesswrong.com/posts/GpSzShaaf8po4rcmA/qapr-5-grokking-is-maybe-not-that-big-a-deal https://www.lesswrong.com/posts/GpSzShaaf8po4rcmA/qapr-5-gro...
  - gorjusborg 3y ago
    Whoever suggested 'eventual recovery from overfitting' is a kindred spirit. Why throw away the context and nuance? That decision only further leans into the 'AI is magic' attitude.
    - jeremyjh 3y ago
      No, actually this is just how language evolves. I'm glad we have the word "car" instead of "carriage powered by internal combustion engine" even if it confused some people 100 years ago when the term became used exclusively to mean something a bit more specfic. Of course the jargon used in a specific sub-field evolves much more quickly than common usage because the intended audience of paper like this is expected to be well-read and current in the field already.
      - smolder 3y ago
        Language devolves just as it evolves. We (the grand we) regularly introduce ambiguity --words and meanings with no useful purpose, or that are worse than useless. I'm not really weighing in on the appropriateness of the use "grok" in this case. It's just a pet peeve of mine that people bring out "language evolves" as an excuse for why any arbitrary change is natural and therefore acceptable and we should go with the flow. Some changes are strictly bad ones. A go-to example is when "literally" no longer means "literally", but its opposite, or nothing at all. We don't have a replacement word, so now in some contexts people have to explain that they "literally mean literally".
blueyes 3y ago
If your data set is too small, they memorize. If you train them well on a large dataset, they learn to generalize.
- visarga 3y ago
  they only generalise with big datasets, that is the rule
  - blueyes 3y ago
    That's what I said.
mostertoaster 3y ago
Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done.
- pillefitz 3y ago
  That is essentially what embeddings do
  - nightski 3y ago
    Maybe, except from my understanding an embedding vector tends to be much larger than the source token (due to the high dimensionality of the embedding space). So it's almost like a reverse compression in a way. That said I know vector DBs have much more efficient ways of storing those vector embedding.
    - jncfhnb 3y ago
      Tokens are not 1:1 with vectors.
- ComputerGuru 3y ago
  That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-boosts-the-human-brain-s-memory-capacity-10-fold/ https://www.scientificamerican.com/article/new-estimate-boos...
  - downboots 3y ago
    Can "distill knowledge" be made precise ?
ComputerGuru 3y ago
PSA: if you’re interested in the details of this topic, it’s probably best to view TFA on a computer as there is data in the visualizations that you can’t explore on mobile.
esafak 3y ago
I haven't read the latest literature but my understanding is that "grokking" is the phase transition that occurs during the coalescing of islands of understanding (increasingly abstract features) that eventually form a pathway to generalization. And that this is something associated with over-parameterized models, which have the potential to learn multiple paths (explanations). https://en.wikipedia.org/wiki/Percolation_theory https://en.wikipedia.org/wiki/Percolation_theory A relevant, recent paper I found from a quick search: The semantic landscape paradigm for neural networks (https://arxiv.org/abs/2307.09550 https://arxiv.org/abs/2307.09550)
ajuc 3y ago
I was trying to make an AI for my 2d sidescrolling game with asteroid-like steering learn from recorded player input + surroundings. It generalized splendidly - it's conclusion was that you always need to press "forward" and do nothing else, no matter what happens :)
huijzer 3y ago
A bit of both, but it does certainly generalize. Just look into the sentiment neuron from OpenAI in 2017 or come up with an unique question to ChatGPT.
SimplyUnknown 3y ago
First of all, great blog post with great examples. Reminds me of distill.pub used to be. Second, the article correctly states that typically L2 weight decay is used, leading to a lot of weights with small magnitudes. For models that generalize better, would it then be better to always use L1 weight decay to promote sparsity in combination with longer training? I wonder whether deep learning models that only use sparse fourier features rather than dense linear layers would work better...
- qumpis 3y ago
  Slightly related but sparsity-inducing activation function Relu is often used in neural networks
- medium_spicy 3y ago
  Short answer: if the inputs can be represented well on the Fourier basis, yes. I have a patent in process on this, fingers crossed. Longer answer: deep learning models are usually trying to find the best nonlinear basis in which to represent inputs; if the inputs are well-represented (read that as: can be sparsely represented) in some basis known a-priori, it usually helps to just put them in that basis, e.g., by FFT’ing RF signals. The challenge is that the overall-optimal basis might not be the same as those of any local minima, so you’ve got to do some tricks to nudge the network closer.
superkuh 3y ago
There were no auto-discovery RSS/Atom feeds in the HTML, no links to the RSS feed anywhere, but by guessing at possible feed names and locations I was able to find the "Explorables" RSS feed at: https://pair.withgoogle.com/explorables/rss.xml https://pair.withgoogle.com/explorables/rss.xml
lachlan_gray 3y ago
It looks like grid cells! https://en.wikipedia.org/wiki/Grid_cell https://en.wikipedia.org/wiki/Grid_cell If you plot a head map of a neuron in the hidden layer on a 2D chart where one axis is $a$ and the other is $b$, I think you might get a triangular lattice. If it's doing what I think it is, then looking at another hidden neuron would give a different lattice with another orientation + scale. Also you could make a base 67 adding machine by chaining these together. I also can't help the gut feeling that the relationship between W_in-proj's neurons compared to the relationship between W_out-proj's neurons looks like the same mapping as the one between the semitone circle and the circle of fifths https://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Pitch_class_space_star.svg/220px-Pitch_class_space_star.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Pi...
tehjoker 3y ago
Well they memorize points and lines (or tanh) between different parts of the space right? So it depends on whether a useful generalization can be extracted from the line estimation and how dense the points on the landscape are no?
xaellison 3y ago
what's the TLDR: memorize, or generalize?
djha-skin 3y ago
How is this even a shock. Anyone who so much as taken a class on this knows that even the simplest of perceptron networks, decision trees, or any form of machine learning model generalizes. That's why we use them. If they don't, it's called overfit[1], where the model is so accurate on the training data that its inferential ability on new data suffers. I know that the article might be talking about a higher form of generalization with LLMs or whatever, but I don't see why the same principle of "don't overfit the data" wouldn't apply to that situation. No, really: what part of their base argument is novel? 1: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting
- rosenjcb 3y ago
  There's so many idiots in the AI space that are completely ignorant of how Machine Learning works. The worst are the grifters that fearmonger about AI safety by regurgitating singularity memes.
- halflings 3y ago
  The interesting part is the sudden generalization. Simple models predicting simple things will generally slowly overfit, and regularization keeps that overfitting in check. This "grokking" phenomenon is when a model first starts by aggressively overfitting, then gradually prunes unnecessary weights until it suddenly converges on the one generalizable combination of weights (as it's the only one that both solves the training data and minimizes weights). Why is this interesting? Because you could argue that this justifies using overparametrized models with high levels of regularization; e.g. models that will tend to aggressively overfit, but over time might converge to a better solution by gradual pruning of weights. The traditional approach is not to do this, but rather to use a simpler model (which would initially generalize better, but due to its simplicity might not be able to learn the underlying mechanism and reach higher accuracy).
  - timy2shoes 3y ago
    It's interesting that the researchers chose example problems where the minimum norm solution is the best at generalization. What if that's not the case?
lewhoo 3y ago
So, the TLDR could be: they memorize at first and then generalize ?
- drdeca 3y ago
  depends on the hyperparameters, and the architecture (and probably the task)
flyer_go 3y ago
I don't think I have seen an answer here that actually challenges this question - from my experience, I have yet to see a neural network actually learn representations outside the range in which it was trained. Some papers have tried to use things like sinusoidal activation functions that can force a neural network to fit a repeating function, but on its own I would call it pure coincidence. On generalization - its still memorization. I think there has been some proof that chatgpt does 'try' to perform some higher level thinking but still has problems due to the dictionary type lookup table it uses. The higher level thinking or agi that people are excited about is a form of generalization that is so impressive we don't really think of it as memorization. But I actually question if our wantingness to generate original thought isn't as actually separate from what we currently are seeing.
- jhaenchen 3y ago
  The issue is that we are prone to inflate the complexity of our own processing logic. Ultimately we are pattern recognition machines in combination with abstract representation. This allows us to connect the dots between events in the world and apply principles in one domain to another. But, like all complexity, it is reduceable to component parts. (In fact, we know this because we evolved to have this ability. )
  - agalunar 3y ago
    Calling us "pattern recognition machines capable of abstract representation" I think is correct, but is (rather) broad description of what we can do and not really a comment on how our minds work. Sure, from personal observation, it seems like we sometimes overcomplicate self-analysis ("I'm feeling bad – why? oh, there are these other things that happened and related problems I have and maybe they're all manifestations of one or two deeper problems, &c" when in reality I'm just tired or hungry), but that seems like evidence we're both simpler than we think and also more complex than you'd expect (so much mental machinery for such straightforward problems!). I read Language in Our Brain [1] recently and I was amazed by what we've learned about the neurologicial basis of language, but I was even more astounded at how profoundly little we know. > But, like all complexity, it is reduceable to component parts. This is just false, no? Sometimes horrendously complicated systems are made of simple parts that interact in ways that are intractable to predict or that defy reduction. [1] https://mitpress.mit.edu/9780262036924/language-in-our-brain https://mitpress.mit.edu/9780262036924/language-in-our-brain
tipsytoad 3y ago
Seriously, are they only talking about weight decay? Why so complicated?
jimwhite42 3y ago
I'm not sure if I'm remembering it right, but I think it was on a Raphaël Millière interview on Mindscape, where Raphaël said something along the lines of when there are many dimensions in a machine learning model, the distinction between interpolation and extrapolation is not clear like it is in our usual areas of reasoning. I can't work out if this could be something similar to what the article is talking about.
agumonkey 3y ago
They ponderize.
aappleby 3y ago
They digest.
westurner 3y ago
If you omit the training data points where the baseball hits the ground, what will a machine learning model predict? You can train a classical ML model on the known orbits of the planets in the past, but it can presumably never predict orbits given unseen n-body gravity events like another dense mass moving through the solar system because of classical insufficiency to model quantum problems, for example. Church-Turing-Deutsch doesn't say there could not exist a Classical / Quantum correspondence; but a classical model on a classical computer cannot be sufficient for quantum-hard problems. (e.g. Quantum Discord says that there are entanglement and non-entanglement nonlocal relations in the data.) Regardless of whether they sufficiently generalize, [LLMs, ML Models, and AutoMLs] don't yet Critically Think and it's dangerous to take action without critical thought. Critical Thinking; Logic, Rationality: https://en.wikipedia.org/wiki/Critical_thinking#Logic_and_rationality https://en.wikipedia.org/wiki/Critical_thinking#Logic_and_ra...
lsh123 3y ago
Current ML models neither memorize or generalize, but instead approximate.

454 points by 1wheel 3y ago

davidguetta 3y ago

hierarchize would be a better term than generalize

3cats-in-a-coat 3y ago
Generalize is seeing common principles, patterns, between disparate instances of a phenomena. It's a proper word for this.
- davidguetta 3y ago
  Generalize has a tendency to imply you can extrapolate. And in most case it's actually the opposite that happens: neural nets tend to COMPRESS the data. (which in turn is a good thing in many case because the data is noisy)
  - 3cats-in-a-coat 3y ago
    The point of compression is to decompress after. That's what happens during inference, and when the extrapolation occurs. Let's say I tell GPT "write 8 times foobar". Will it? Well then it understands me and can extrapolate from the request to the proper response, without having specifically "write 8 times foobar" in its model. Most decompression algorithms focus on predicting the next token (byte, term, etc.), believe it or not. The more accurately they predict the next token, the less information you need to store to correct misprediction.
- Chabsff 3y ago
  That's a common mechanism to achieve generalization, but the term is a little more general (heh) than that. It specifically refers to correctly handling data that lives outside the distribution presented by the training data. It's a description of a behavior, not a mechanism. Which may or may not be appropriate depending on whether you are talking about *what* the model does or *how* it achieves it.
  - 3cats-in-a-coat 3y ago
    Kinda fuzzy what's "in the distribution", because it depends on how deeply the model interprets it. If it understands examples outside the distribution... that kinda puts them in the distribution. General understanding makes the information in the distribution very wide. Shallow understanding makes it very narrow. Like say recognizing only specific combinations of pixels verbatim.
    - Chabsff 3y ago
      I think you are misinterpreting. The distribution present in the training set in isolation (the one I'm referring to, and is not fuzzy in the slightest) is not the same thing as the distribution understood by the trained model (the one you are referring to, and is definitely more conceptual and hard to characterize in non-trivial cases). "Generalization" is simply the theoretical measure of how much the later extends beyond the former, regardless of how that's achieved.
      - 3cats-in-a-coat 3y ago
        I'm saying how you determine the distribution in the training set depends on what the model understands and what the people who selected the dataset understand. There's no distribution of meaning in the training set that's independent of interpretation and understanding. Aside from maybe the literal series of bits (and words and pixels) in it, as encoded. In statistics that is not as severe a problem because you can plot how the data distribution lies in a commonly agreed upon position in one or more clearly defined and agreed upon dimensions. And you can look at the chart and talk about this shared interpretation objectively, and its distribution. Although as a matter of fact just as often it matters what questions you asked, and how and when and whom you asked, for the distribution of answers you got. Lying with statistics is easy as it's full of hidden variables. This is why statistics is great when the data is simple and the analysis is simple, mathematical, objective, but social studies tend to yield, whatever you want them to yield. So. What dimensions are we talking about with a self-evolved model? You have some understanding of what the data is, subjective to you. Maybe your team has some shared understanding of what the data covers, you have overlap. But the model has its own understanding, evolved independently. How much does it overlap with you? Not as much as you think. It's a problem decades old, that people give to the model data that contains things they didn't realize it contains. They themselves didn't see that. And then get surprised by the results. Say when an apple falls on your head, did you realize this contains the data required to describe classic mechanics? For centuries, billions of people didn't realize. To Newton it was there as clear as daylight. In the apple's fall. I know, the example is a myth, but the principle stands. Another example, a video of the change of light patterns reflected on the floor around the corner of room where a person, out of frame, is writing on a computer. What does this data contain? You think nothing much. Maybe it contains how a floor looks. To a model, it can easily also contain what the person who is not in frame, wrote on their keyboard. So given all this... what IS in the distribution? Depends with whose eyes you're looking. Your eyes are not the most objective eyes, nor the most intelligent eyes. You have no anchor to point to as the ultimate arbiter of what complex data contains or does not.
version_five 3y ago
Anything would be better than "grokking". From what I gather they're talking about double descent which afaik is the consequence of overparameterization leading to a smooth interpolation between the training data as opposed to what happens in traditional overfitting. Imagine a polynomial fit with the same degree as the number of data points (swinging up and down wildly away from the data) compared with a much higher degree fit that could smoothly interpolate between the points while still landing right on them. None of this is what I would call generalization, it's good interpolation, which is what deep learning does in a very high dimensional space. It's notoriously awful at extrapolating, ie generalizing to anything without support in the training data.
- Jack000 3y ago
  double descent is a different phenomenon from grokking
  - tysam_and 3y ago
    Nope, they are the same, just that grokking is when the KL between the representable information of the implicit biases and the data is extremely high (i.e. the network is poorly-designed or oriented for the task). It's an informal term that not everyone accepts. Double-descent is acceptable as it describes a general phenomenon that is a natural consequence of a phase transition during neural network training. Grokking is like, to me, the 'fetch' of neural network terms. It's not new, it adds a seeming layer of separation from double-descent (which is is -- just very delayed), and it's not really accepted by everyone. I personally do not like it at all. Especially because language affects _our_ implicit biases about what neural networks can and cannot do. We've already seen that their capacities and performance can be pushed way beyond what we traditionally expect of them. But to summarize, they are the same. And this is why we need good terminology, as well, because poor adoption and boosting of improper terminology induces excess regret in the information exchange surface between agents in a game-theoretic sense in this lovely landscape of the ML world.
- visarga 3y ago
  > It's notoriously awful at extrapolating, ie generalizing to anything without support in the training data. Scientists are also pretty lousy at making new discoveries without labs. They just need training data.
ot 3y ago
"hierarchize" only describes your own mental model of how knowledge organization and reasoning may work in the model, not the actual phenomenon being observed here. "generalize" means going from specific examples to general cases not seen before, which is a perfectly good description of the phenomenon. Why try to invent a new word?
- davidguetta 3y ago
  > hierarchize" only describes your own mental model of how knowledge organization and reasoning may work in the model, not the actual phenomenon being observed here It's not true, if you look at deep CNN the lower layers show lines, the higher complex stuff like eyes or football players etc.. Herarchisation of information actually emerges naturally in NNs. Generalization often implies extrapolation on new data, which is just not the case most of the time with NNs and why i didn't like the word

MagicMoonlight 3y ago

Memorise because there is no decision component. It attempts to just brute force a pattern rather than thinking through the information and making a conclusion.

wwarner 3y ago

This is such a good explainer

deleted 3y ago

[deleted]

greenflag 3y ago

It seems the take home is weight decay induces sparsity which helps learn the "true" representation rather than an overfit one. It's interesting the human brain has a comparable mechanism prevalent in development [1]. I would love to know from someone in the field if this was the inspiration for weight decay (or presumably just the more equivalent nn pruning [2]). [1] https://en.wikipedia.org/wiki/Synaptic_pruning https://en.wikipedia.org/wiki/Synaptic_pruning [2] https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network) https://en.wikipedia.org/wiki/Pruning_(artificial_neural_net...

deleted 3y ago
[deleted]
visarga 3y ago
The inspiration for weight decay was to reduce the capacity to memorize of the model until it perfectly fits the complexity of the task, not more not less. A model more complex than the task is over-fitting, the other one is under-fitting. Got to balance them out. But the best cure for over-fitting is to make the dataset larger and ensure data diversity. LLMs have datasets so large they usually train one epoch.
- nightski 3y ago
  It sounds nice in theory, but the data itself could be problematic. There is no temporal nature to it. You can have duplicate data points, many data points that are closely related but describe the same thing/event/etc.. So while only showing the model each data point once ensures you do not introduce any extra weight on a data point, if the dataset itself is skewed it doesn't help you at all. Just by trying to make the dataset diverse you could skew things to not reflect reality. I just don't think enough attention has been paid to the data, and too much the model. But I could be very wrong. There is a natural temporality to the data humans receive. You can't relive the same moment twice. That said, human intelligence is on a scale too and may be affected in the same way.
  - visarga 3y ago

_ache_ 3y ago

Does anyone know how that charts are created ? I bet that it's half generated by some sort of library and them manually improved but the generated animated SVG are beautiful.

1wheel 3y ago
Basically just a bunch of d3 — could be cleaned up significantly, but that's hard to do while iterating and polishing the charts. I also have a couple of little libraries for things like annotations, interleaving svg/canvas and making d3 a bit less verbose. - https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking https://github.com/PAIR-code/ai-explorables/tree/master/sour... - https://1wheel.github.io/swoopy-drag/ https://1wheel.github.io/swoopy-drag/ - https://github.com/gka/d3-jetpack https://github.com/gka/d3-jetpack - https://roadtolarissa.com/hot-reload/ https://roadtolarissa.com/hot-reload/
- iaw 3y ago
  I was going to ask the same question. Those are some great visualizations

mjburgess 3y ago

Statistical learning can typically be phrased in terms of k nearest neighbours In the case of NNs we have a "modal knn" (memorising) going to a "mean knn" ('generalising') under the right sort of training. I'd call both of these memorising, but the latter is a kind of weighted recall. Generalisation as a property of statistical models (ie., models of conditional freqs) is not the same property as generalisation in the case of scientific models. In the latter a scientific model is general because it models causally necessary effects from causes -- so, necessarily if X then Y. Whereas generalisation in associative stats is just about whether you're drawing data from the empirical freq. distribution or whether you've modelled first. In all automated stats the only diff between the "model" and "the data" is some sort of weighted averaging operation. So in automated stats (ie., ML,AI) it's really just whether the model uses a mean.

bippihippi1 3y ago
it's been proven that all models learned by gradient descent are equivalent to kernel machines. interpolation isn't generalization. if theres a new input sufficiently different from the training data the behaviour is unknown
- xapata 3y ago
  One weird trick ... There's some fox and hedgehog analogy I've never understood.
- visarga 3y ago
  but when the model trains on 13T tokens it is hard to be OOD
- drdeca 3y ago
  Can you say what that says about the behavior described with the modular arithmetic in the article? And, in particular, how to interpret the fact that different hyperparameters determined whether runs, obtaining equally high accuracy on the training data, got good or bad scores on the test data, in terms of the "view it as a kernel machine/interpolation" lens? My understanding is that the behavior in at least one of those "models learned by gradient descent are equivalent to [some other model]" papers, works by constructing something which is based on the entire training history of the network. Is that the kernel machines one, or some other one?

taeric 3y ago

I'm curious how representative the target function is? I get that it is common for you to want a model to learn the important pieces of an input, but a string of bits, and only caring about the first three, feels particularly contrived. Literally a truth table on relevant parameters of size 8? And trained with 4.8 million samples? Or am I misunderstanding something there? (I fully expect I'm misunderstanding something.)

jaggirs 3y ago
I have observed this pattern before in computer vision tasks (train accuracy flatlining for a while before test acc starts to go up). The point of the simple tasks is to be able to interpret what could be going on behind the scenes when this happens.
- taeric 3y ago
  No doubt. But I have also seen what people thought were generalized models failing on outlier, but valid, data. Quite often. Put another way, it isn't just how simple this task seems to be in the number of terms that are important, but isn't it also a rather dense function? Probably better question to ask is how sensitive are models that are looking at less dense functions to this? (Or more dense.). I'm not trying to disavow the ideas.
  - visarga 3y ago
    Maybe humans are also failing a lot in out of distribution settings. It might be inherent.
    - taeric 3y ago
      We have names for that. :D. Stereotypes being a large one. Racism being motivated interpretation on the same ideas. Right?
      - lucubratory 3y ago

deleted 3y ago

[deleted]

gorjusborg 3y ago

Grr, the AI folks are ruining the term 'grok'. It means roughly 'to understand completely, fully'. To use the same term to describe generalization... just shows you didn't grok grokking.

erwald 3y ago
"Grok" in AI doesn't quite describe generalization, it's more specific that that. It's more like "delayed and fairly sudden generalization" or something like that. There was some discussion of this in the comments of this post[1], which proposes calling the phenomenon "eventual recovery from overfitting" instead. [1] https://www.lesswrong.com/posts/GpSzShaaf8po4rcmA/qapr-5-grokking-is-maybe-not-that-big-a-deal https://www.lesswrong.com/posts/GpSzShaaf8po4rcmA/qapr-5-gro...
- gorjusborg 3y ago
  Whoever suggested 'eventual recovery from overfitting' is a kindred spirit. Why throw away the context and nuance? That decision only further leans into the 'AI is magic' attitude.
  - jeremyjh 3y ago
    No, actually this is just how language evolves. I'm glad we have the word "car" instead of "carriage powered by internal combustion engine" even if it confused some people 100 years ago when the term became used exclusively to mean something a bit more specfic. Of course the jargon used in a specific sub-field evolves much more quickly than common usage because the intended audience of paper like this is expected to be well-read and current in the field already.
    - smolder 3y ago
      Language devolves just as it evolves. We (the grand we) regularly introduce ambiguity --words and meanings with no useful purpose, or that are worse than useless. I'm not really weighing in on the appropriateness of the use "grok" in this case. It's just a pet peeve of mine that people bring out "language evolves" as an excuse for why any arbitrary change is natural and therefore acceptable and we should go with the flow. Some changes are strictly bad ones. A go-to example is when "literally" no longer means "literally", but its opposite, or nothing at all. We don't have a replacement word, so now in some contexts people have to explain that they "literally mean literally".

blueyes 3y ago

If your data set is too small, they memorize. If you train them well on a large dataset, they learn to generalize.

visarga 3y ago
they only generalise with big datasets, that is the rule
- blueyes 3y ago
  That's what I said.

mostertoaster 3y ago

Sometimes I think the reason human memory in some sense is so amazing, is what we lack in storage capacity that machines have, we makeup for in our ability to create patterns that compress the amount of information stored dramatically, and then it is like we compress those patterns together with other patterns and are able to extract things from it. Like it is an incredibly lossy compression, but it gets the job done.

pillefitz 3y ago
That is essentially what embeddings do
- nightski 3y ago
  Maybe, except from my understanding an embedding vector tends to be much larger than the source token (due to the high dimensionality of the embedding space). So it's almost like a reverse compression in a way. That said I know vector DBs have much more efficient ways of storing those vector embedding.
  - jncfhnb 3y ago
    Tokens are not 1:1 with vectors.
ComputerGuru 3y ago
That’s not exactly true, there doesn’t seem to be an upper bound (that we can reach) on storage capacity in the brain [0]. Instead, the brain actually works to actively distill knowledge that doesn’t need to be memorized verbatim into its essential components in order to achieve exactly this “generalized intuition and understanding” to avoid overfitting. [0]: https://www.scientificamerican.com/article/new-estimate-boosts-the-human-brain-s-memory-capacity-10-fold/ https://www.scientificamerican.com/article/new-estimate-boos...
- downboots 3y ago
  Can "distill knowledge" be made precise ?

ComputerGuru 3y ago

PSA: if you’re interested in the details of this topic, it’s probably best to view TFA on a computer as there is data in the visualizations that you can’t explore on mobile.

esafak 3y ago

I haven't read the latest literature but my understanding is that "grokking" is the phase transition that occurs during the coalescing of islands of understanding (increasingly abstract features) that eventually form a pathway to generalization. And that this is something associated with over-parameterized models, which have the potential to learn multiple paths (explanations). https://en.wikipedia.org/wiki/Percolation_theory https://en.wikipedia.org/wiki/Percolation_theory A relevant, recent paper I found from a quick search: The semantic landscape paradigm for neural networks (https://arxiv.org/abs/2307.09550 https://arxiv.org/abs/2307.09550)

ajuc 3y ago

I was trying to make an AI for my 2d sidescrolling game with asteroid-like steering learn from recorded player input + surroundings. It generalized splendidly - it's conclusion was that you always need to press "forward" and do nothing else, no matter what happens :)

huijzer 3y ago

A bit of both, but it does certainly generalize. Just look into the sentiment neuron from OpenAI in 2017 or come up with an unique question to ChatGPT.

SimplyUnknown 3y ago

First of all, great blog post with great examples. Reminds me of distill.pub used to be. Second, the article correctly states that typically L2 weight decay is used, leading to a lot of weights with small magnitudes. For models that generalize better, would it then be better to always use L1 weight decay to promote sparsity in combination with longer training? I wonder whether deep learning models that only use sparse fourier features rather than dense linear layers would work better...

qumpis 3y ago
Slightly related but sparsity-inducing activation function Relu is often used in neural networks
medium_spicy 3y ago
Short answer: if the inputs can be represented well on the Fourier basis, yes. I have a patent in process on this, fingers crossed. Longer answer: deep learning models are usually trying to find the best nonlinear basis in which to represent inputs; if the inputs are well-represented (read that as: can be sparsely represented) in some basis known a-priori, it usually helps to just put them in that basis, e.g., by FFT’ing RF signals. The challenge is that the overall-optimal basis might not be the same as those of any local minima, so you’ve got to do some tricks to nudge the network closer.

superkuh 3y ago

There were no auto-discovery RSS/Atom feeds in the HTML, no links to the RSS feed anywhere, but by guessing at possible feed names and locations I was able to find the "Explorables" RSS feed at: https://pair.withgoogle.com/explorables/rss.xml https://pair.withgoogle.com/explorables/rss.xml

lachlan_gray 3y ago

It looks like grid cells! https://en.wikipedia.org/wiki/Grid_cell https://en.wikipedia.org/wiki/Grid_cell If you plot a head map of a neuron in the hidden layer on a 2D chart where one axis is $a$ and the other is $b$, I think you might get a triangular lattice. If it's doing what I think it is, then looking at another hidden neuron would give a different lattice with another orientation + scale. Also you could make a base 67 adding machine by chaining these together. I also can't help the gut feeling that the relationship between W_in-proj's neurons compared to the relationship between W_out-proj's neurons looks like the same mapping as the one between the semitone circle and the circle of fifths https://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Pitch_class_space_star.svg/220px-Pitch_class_space_star.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Pi...

tehjoker 3y ago

Well they memorize points and lines (or tanh) between different parts of the space right? So it depends on whether a useful generalization can be extracted from the line estimation and how dense the points on the landscape are no?

xaellison 3y ago

what's the TLDR: memorize, or generalize?

djha-skin 3y ago

How is this even a shock. Anyone who so much as taken a class on this knows that even the simplest of perceptron networks, decision trees, or any form of machine learning model generalizes. That's why we use them. If they don't, it's called overfit[1], where the model is so accurate on the training data that its inferential ability on new data suffers. I know that the article might be talking about a higher form of generalization with LLMs or whatever, but I don't see why the same principle of "don't overfit the data" wouldn't apply to that situation. No, really: what part of their base argument is novel? 1: https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting

rosenjcb 3y ago
There's so many idiots in the AI space that are completely ignorant of how Machine Learning works. The worst are the grifters that fearmonger about AI safety by regurgitating singularity memes.
halflings 3y ago
The interesting part is the sudden generalization. Simple models predicting simple things will generally slowly overfit, and regularization keeps that overfitting in check. This "grokking" phenomenon is when a model first starts by aggressively overfitting, then gradually prunes unnecessary weights until it suddenly converges on the one generalizable combination of weights (as it's the only one that both solves the training data and minimizes weights). Why is this interesting? Because you could argue that this justifies using overparametrized models with high levels of regularization; e.g. models that will tend to aggressively overfit, but over time might converge to a better solution by gradual pruning of weights. The traditional approach is not to do this, but rather to use a simpler model (which would initially generalize better, but due to its simplicity might not be able to learn the underlying mechanism and reach higher accuracy).
- timy2shoes 3y ago
  It's interesting that the researchers chose example problems where the minimum norm solution is the best at generalization. What if that's not the case?

lewhoo 3y ago

So, the TLDR could be: they memorize at first and then generalize ?

drdeca 3y ago
depends on the hyperparameters, and the architecture (and probably the task)

flyer_go 3y ago

I don't think I have seen an answer here that actually challenges this question - from my experience, I have yet to see a neural network actually learn representations outside the range in which it was trained. Some papers have tried to use things like sinusoidal activation functions that can force a neural network to fit a repeating function, but on its own I would call it pure coincidence. On generalization - its still memorization. I think there has been some proof that chatgpt does 'try' to perform some higher level thinking but still has problems due to the dictionary type lookup table it uses. The higher level thinking or agi that people are excited about is a form of generalization that is so impressive we don't really think of it as memorization. But I actually question if our wantingness to generate original thought isn't as actually separate from what we currently are seeing.

jhaenchen 3y ago
The issue is that we are prone to inflate the complexity of our own processing logic. Ultimately we are pattern recognition machines in combination with abstract representation. This allows us to connect the dots between events in the world and apply principles in one domain to another. But, like all complexity, it is reduceable to component parts. (In fact, we know this because we evolved to have this ability. )
- agalunar 3y ago
  Calling us "pattern recognition machines capable of abstract representation" I think is correct, but is (rather) broad description of what we can do and not really a comment on how our minds work. Sure, from personal observation, it seems like we sometimes overcomplicate self-analysis ("I'm feeling bad – why? oh, there are these other things that happened and related problems I have and maybe they're all manifestations of one or two deeper problems, &c" when in reality I'm just tired or hungry), but that seems like evidence we're both simpler than we think and also more complex than you'd expect (so much mental machinery for such straightforward problems!). I read Language in Our Brain [1] recently and I was amazed by what we've learned about the neurologicial basis of language, but I was even more astounded at how profoundly little we know. > But, like all complexity, it is reduceable to component parts. This is just false, no? Sometimes horrendously complicated systems are made of simple parts that interact in ways that are intractable to predict or that defy reduction. [1] https://mitpress.mit.edu/9780262036924/language-in-our-brain https://mitpress.mit.edu/9780262036924/language-in-our-brain

tipsytoad 3y ago

Seriously, are they only talking about weight decay? Why so complicated?

jimwhite42 3y ago

I'm not sure if I'm remembering it right, but I think it was on a Raphaël Millière interview on Mindscape, where Raphaël said something along the lines of when there are many dimensions in a machine learning model, the distinction between interpolation and extrapolation is not clear like it is in our usual areas of reasoning. I can't work out if this could be something similar to what the article is talking about.

agumonkey 3y ago

They ponderize.

aappleby 3y ago

They digest.

westurner 3y ago

If you omit the training data points where the baseball hits the ground, what will a machine learning model predict? You can train a classical ML model on the known orbits of the planets in the past, but it can presumably never predict orbits given unseen n-body gravity events like another dense mass moving through the solar system because of classical insufficiency to model quantum problems, for example. Church-Turing-Deutsch doesn't say there could not exist a Classical / Quantum correspondence; but a classical model on a classical computer cannot be sufficient for quantum-hard problems. (e.g. Quantum Discord says that there are entanglement and non-entanglement nonlocal relations in the data.) Regardless of whether they sufficiently generalize, [LLMs, ML Models, and AutoMLs] don't yet Critically Think and it's dangerous to take action without critical thought. Critical Thinking; Logic, Rationality: https://en.wikipedia.org/wiki/Critical_thinking#Logic_and_rationality https://en.wikipedia.org/wiki/Critical_thinking#Logic_and_ra...

lsh123 3y ago

Current ML models neither memorize or generalize, but instead approximate.