jaschasd

6 ms·

1.
by jaschasd 3y ago
It does! See the effect strength plot near the bottom of the blog post, or Figures 2b and 3d in the paper. The effect strength on humans ranges from a few percent deviation of human judgements from chance for subtle adversarial perturbation
2.
by jaschasd 3y ago
This comment is incorrect. For experiments 1 through 4, N was 38, 389, 396, and 389. The subjects were not undergrad psych students. The article linked in the parent comment does not correspond to any experiment in the blog post or the Natu
3.
by jaschasd 4y ago
I can't reply directly -- is there a maximum thread depth, or a maximum conversation depth? Anyway -- I wanted to apologize for misreading -- I missed the parenthetical "interpolation" in your comment. I think we are both int
4.
by jaschasd 4y ago
(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)
5.
by jaschasd 4y ago
It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite be
6.
by jaschasd 4y ago
Thanks! I'm also adding this to my reading list now.
7.
by jaschasd 4y ago
Blog post author here. A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of the model being t
8.
by jaschasd 4y ago
See response to alexmolas -- I'm using GitHub Pages + Jekyll + Markdeep. You don't see the source repo because it's private, but I'm happy to share a code snapshot with you if you like -- email me for it.
9.
by jaschasd 4y ago
Yes! In the post I talk about both under- and over-parameterization being mitigations for overfitting, though I don't use the term double descent.
10.
by jaschasd 4y ago
+1. It doesn't require there to be a time axis -- but in practice, we almost always optimize incrementally, so it takes a while for the strong version of Goodhart's law to kick in.
11.
by jaschasd 4y ago
Yup, it's GitHub pages. It's just a private repo, so the world doesn't get to see my embarrassing edits and half written drafts. I would be happy to share a snapshot of the source code with you if you have a specific use for
12.
by jaschasd 4y ago
+1 to this comment! The barrier between (thinking we) know what changes should happen, and realizing those changes in the real world, is complex and frustrating and political.
13.
by jaschasd 4y ago
A gentle note that an incomplete piece of a goal (e.g. a loss function computed on a subset of the data) is a proxy for the full goal (e.g. the loss function on the full dataset). Similarly, concept drift can be a source of overfitting -- t
14.
by jaschasd 4y ago
A note that the datapoints you train on are part of the training objective. If you are using different data at test time than you use at training time, then you are measuring the wrong thing during training, the same as if you used a diff
15.
by jaschasd 4y ago
(blog post author here) Yes! The post actually talks about that a bit: how overfitting can result from treating "leaders that have the most support in the population" as a proxy for "leaders that act in the best interests of
16.
by jaschasd 4y ago
Blog post author here. Just for the record, I am very much in favor of metric-driven decision making. I suggest some ways we can make it more robust, and also that we should be aware that our metrics may not be measuring what we intend.
17.
by jaschasd 4y ago
Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For overfitting induced by d
18.
by jaschasd 5y ago
They show the source images next to the super-resolution output on the website for the iterative refinement paper: https://iterative-refinement.github.io/
19.
by jaschasd 6y ago
Don't forget Neural Tangents, a high level library for building and running experiments with infinite width neural networks: https://github.com/google/neural-tangents