6 ms·

If you're an explorer and you live in a small valley, then you will slowly begin to believe yourself a better explorer as time goes on. But one day you will ge

If you're an explorer and you live in a small valley, then you will slowly begin to believe yourself a better explorer as time goes on. But one day you will get to the edge of the valley and look out onto a vast landscape and believe that you have become a worse explorer. Checkout the volume of N-balls [1]. As the number of dimensions in a "sphere" become higher the "volume" of the sphere becomes smaller when you compare it to a hypercube of the same dimensional count. This is another way to saying that as the number of dimensions increases the closer the "halfway" point of the volume of the N-ball gets to the edge of the N-ball. Programming (and software engineering by extension) involves a lot of orthogonal issues. We can imagine this so be a really high dimensional space. My theory is that whenever you're in situation that can be represented by a high dimensional space you will learn a bit of knowledge and think you know something. Then you'll learn a bit more and discover there's more to learn. The more you learn the farther the horizon moves away. It will feel like you're getting worse. Just like finding out you live in a valley. What's really happening though is that your increase in skill is allowing you to realize how small your world was all along. Keep learning and trying and you'll get better. It will however, take a while. [1] - https://en.wikipedia.org/wiki/Volume_of_an_n-ball#Low_dimensions https://en.wikipedia.org/wiki/Volume_of_an_n-ball#Low_dimens...

Philomath 9y ago
Wise words! This certainly applies to programming but I've felt it too in my Aerospace career. I used to feel like the smartest guy, I always knew how to do things or learnt them really fast, and I couldn't imagine where the horizon was. Now, the more I see, the further I feel the horizon is, and it's pretty frustrating as it makes me feel less intelligent. Your comparison helped me to see things a bit clearer and to be a bit more motivated.
debaserab2 9y ago
Isn't this effectively the same as the Dunning Kruger effect (as it applies to this situation)? https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
- Verdex_2 9y ago
  More or less. I'm suggesting a mechanism for why something like the Dunning Kruger effect might occur and where we are likely to see it occurring.
  - debaserab2 9y ago
    That's a very interesting way of explaining it from a perspective I hadn't considered.
- anonymoushn 9y ago
  In the paper, ability is positively correlated with perceived ability. So we would need some other effect to explain people saying they got worse over time, or people's perceived ability decreasing over a time period in which their ability improved.
jasode 9y ago
>We can imagine this so be a really high dimensional space. My theory is that whenever you're in situation that can be represented by a high dimensional space you will learn a bit of knowledge and think you know something. Then you'll learn a bit more and discover there's more to learn. The more you learn the farther the horizon moves away. It will feel like you're getting worse. That is all true but I don't think the n-dimensions analogy applies to the particular essay the author wrote. He didn't look back on his accumulation of learning and suddenly notice that it was smaller part of a larger knowledge space. (E.g., programmer stumbles across "monads" and suddenly he realizes that's just the tip of the iceberg in a never-ending horizon of category theory.) Instead, his essay is lamenting the all-too-common "joy vs drudgery" in projects. It's a similar sentiment that aspiring writers who don't finish their "great novel" have. He then grades himself on his lack of motivation (words like "lazy") to stay on top of programming tasks that are not pure intellectual fun. On his idiosyncratic standard of "programming competence", he wonders if he is a "worse programmer". Of course, he can choose whatever "success metrics" he wishes to apply to his personal projects but some of us might think that staying on top of documentation (that nobody will read) or adding features to Lily (that a large population of programmers haven't adopted) can bring undue self-criticism. If we are to plow ahead into boring unpleasant tasks, there usually has to be a compelling motivation -- e.g. we think the personal project is the basis of a great business startup... or the code is part of a PhD paper due next Friday, etc.
- TheOtherHobbes 9y ago
  Maybe there's a simpler explanation: FOSS projects can easily turn into unpaid work, and devs tend to burn out. Projects that start as fun programming exercises eventually take on all the aspects of a full-time job with "customer" support, project planning, maintenance, bug fixing, and so on. It can become very draining, and if the project isn't unusually popular or high profile and/or there's no other dev support and/or the team isn't solid and mutually supportive it can be understandably difficult to stay motivated. One of the insidious things about FOSS is the way that devs can make themselves feel as if they ought to be doing all of these things, sometimes in addition to a full-time job. The problem isn't that they're not up to the task, it's that the expectation isn't very realistic.
hinkley 9y ago
It's not always that way though. We often suffer as if we have unique problems, but through house repairs (got a nice long chat with a couple GCs) and extracurriculars, I find that a lot of my problems look a good bit like other people's problems. I think the surface area thing is overblown. There are always stories about innovators who simply apply established solutions from problem space A to problem space B and everyone is amazed. The surface area problem certainly exists for the entire domain of human knowledge, but domains pretty much exist to wrangle problems that are manageable in the first place, even if it makes you myopic in the process. I get frustrated in interviews because everyone is just so convinced that the latest utility module is so new and unique and it's really just a retread of five other things. It might have some interesting semantics that change how you use it, but even that isn't always the case. Plus ça change, plus c'est la même chose. I've just started telling people that 'how' ages faster than 'why'.
deleted 9y ago
[deleted]
westoncb 9y ago
I think I get part of the N-ball analogy, but I'm trying to piece together the rest... So it seems like one problem/feature with high dimensional spaces is that you have more regions which are orthogonal to one other. So, it's possible to have quite a bit of motion, which to a human mind might give the impression that one is doing quite a lot—but then you might discover a new way of turning around and find that space extends in another direction, too, at which point you realize you covered a smaller fraction of the available space than you previously believed. What I don't get is how the volume of an N-ball with fixed radius going to zero as N goes to infinity plays into the analogy. When you say, "when you compare it to a hypercube of the same dimensional count," I don't get what in the programming analogy would link to the N-ball rather than the high dimensional hypercube. I also get the algebraic justification for this property of N-balls, but I still have no real geometric intuition for why it would be the case. Could you clarify what's meant by, "as the number of dimensions increases the closer the 'halfway' point of the volume of the N-ball gets to the edge of the N-ball"? Not sure what's meant by 'halfway' point here, or edge of the N-ball. My guess is that you're talking about distribution of volume, like if you have a tapered cup that gets small at the bottom, most of the volume will be at the top.
- Verdex_2 9y ago
  Checkout the generic formula for any number of dimensions [1]. The function in the denominator is a factorial (except extended to work on decimals instead of just integers). The functions in the numerator are exponents. Factorials grow much faster than exponents, so with an N-Ball we expect the "volume" to be small if we were to compare it to the corresponding hypercube when the dimension and when the diameter / length are all equal. After all we're comparing approximately diameter * R ^ D / D! and length * R ^ D. So if you put an N-Ball into a hypercube with length = diameter, what you'll discover is that the N-Ball is tiny when compared to the hypercube. Which means we have the weird and counter intuitive conclusion that all the "volume" of a hypercube is in the corners. Try calculating the volumes yourself using a high dimension like 100. Even better plug the whole thing into a script so you can generate a table and see the effect get progressively worse as you increase the dimension count. There is also a similar effect on the N-Ball itself. Imagine drawing a smaller circle inside of a circle where their centers are the same such that the inner circle contains exactly half of the area of the outer circle. Now imagine doing this in a sphere. Now a 4D sphere. Now a 5D sphere. What you'll find is that as you increase the dimension count, the radius of the inner circle will get longer and longer. Which means that half of the "volume" of an N-Ball slowly moves closer and closer to the outer edge of the N-Ball as you increase dimensions. Here's the punch line. Imagine that the N-Ball represents all there is to know about a topic. You begin in the center of the N-Ball and in order to learn more you have to walk to the outer edge of the N-Ball. Also imagine that you're only able to see what's right in front of your face. In the beginning at the center there is very little volume to look at. It will appear to be easy to understand what's going on. But as you walk to the edge of the N-Ball, the amount of volume that you're able to see increases. When you get right to the edge of a very high dimension N-Ball you will notice that you only know half of what is available to know. Simple topics are N-Balls with low dimensions. It's easy to "walk" your way to knowing half of everything there is to know. Complex topics are N-Balls with high dimensions. Programming (and by extension software engineering) is really high dimensional. The more you learn the more you notice there is to learn. When you walk to the edge of what seems possible you find yourself swimming in a sea that dwarfs you. But in the beginning it looks like there isn't really all that much going on. [1] - https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume
deleted 9y ago
[deleted]
lacampbell 9y ago
To use your analogy, when I start exploring outside the valley, I think to myself - "wow, a year ago I only knew this valley. Now I'm getting to know this new area! I've come a long way and I feel great I've made progress!". I tend to focus on the fact I am constantly expanding my knowledge, instead of worrying about all the stuff I don't know yet. I still consider myself a generalist, however. I can't say I'm an expert in any one area. Maybe that's why I feel the way I do.
- mpfundstein 9y ago
  great way of thinking. its all about the journey
disposablename 9y ago
TL;DR "The more you learn, the less you know"
edem 9y ago
It is more like traversing a tree. The more you learn (the deeper you go) the more branches you can explore which leads to the conclusion that the more you know the more you realize how much you still don't know.

Philomath 9y ago

Wise words! This certainly applies to programming but I've felt it too in my Aerospace career. I used to feel like the smartest guy, I always knew how to do things or learnt them really fast, and I couldn't imagine where the horizon was. Now, the more I see, the further I feel the horizon is, and it's pretty frustrating as it makes me feel less intelligent. Your comparison helped me to see things a bit clearer and to be a bit more motivated.

debaserab2 9y ago

Isn't this effectively the same as the Dunning Kruger effect (as it applies to this situation)? https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Verdex_2 9y ago
More or less. I'm suggesting a mechanism for why something like the Dunning Kruger effect might occur and where we are likely to see it occurring.
- debaserab2 9y ago
  That's a very interesting way of explaining it from a perspective I hadn't considered.
anonymoushn 9y ago
In the paper, ability is positively correlated with perceived ability. So we would need some other effect to explain people saying they got worse over time, or people's perceived ability decreasing over a time period in which their ability improved.

jasode 9y ago

>We can imagine this so be a really high dimensional space. My theory is that whenever you're in situation that can be represented by a high dimensional space you will learn a bit of knowledge and think you know something. Then you'll learn a bit more and discover there's more to learn. The more you learn the farther the horizon moves away. It will feel like you're getting worse. That is all true but I don't think the n-dimensions analogy applies to the particular essay the author wrote. He didn't look back on his accumulation of learning and suddenly notice that it was smaller part of a larger knowledge space. (E.g., programmer stumbles across "monads" and suddenly he realizes that's just the tip of the iceberg in a never-ending horizon of category theory.) Instead, his essay is lamenting the all-too-common "joy vs drudgery" in projects. It's a similar sentiment that aspiring writers who don't finish their "great novel" have. He then grades himself on his lack of motivation (words like "lazy") to stay on top of programming tasks that are not pure intellectual fun. On his idiosyncratic standard of "programming competence", he wonders if he is a "worse programmer". Of course, he can choose whatever "success metrics" he wishes to apply to his personal projects but some of us might think that staying on top of documentation (that nobody will read) or adding features to Lily (that a large population of programmers haven't adopted) can bring undue self-criticism. If we are to plow ahead into boring unpleasant tasks, there usually has to be a compelling motivation -- e.g. we think the personal project is the basis of a great business startup... or the code is part of a PhD paper due next Friday, etc.

TheOtherHobbes 9y ago
Maybe there's a simpler explanation: FOSS projects can easily turn into unpaid work, and devs tend to burn out. Projects that start as fun programming exercises eventually take on all the aspects of a full-time job with "customer" support, project planning, maintenance, bug fixing, and so on. It can become very draining, and if the project isn't unusually popular or high profile and/or there's no other dev support and/or the team isn't solid and mutually supportive it can be understandably difficult to stay motivated. One of the insidious things about FOSS is the way that devs can make themselves feel as if they ought to be doing all of these things, sometimes in addition to a full-time job. The problem isn't that they're not up to the task, it's that the expectation isn't very realistic.

hinkley 9y ago

It's not always that way though. We often suffer as if we have unique problems, but through house repairs (got a nice long chat with a couple GCs) and extracurriculars, I find that a lot of my problems look a good bit like other people's problems. I think the surface area thing is overblown. There are always stories about innovators who simply apply established solutions from problem space A to problem space B and everyone is amazed. The surface area problem certainly exists for the entire domain of human knowledge, but domains pretty much exist to wrangle problems that are manageable in the first place, even if it makes you myopic in the process. I get frustrated in interviews because everyone is just so convinced that the latest utility module is so new and unique and it's really just a retread of five other things. It might have some interesting semantics that change how you use it, but even that isn't always the case. Plus ça change, plus c'est la même chose. I've just started telling people that 'how' ages faster than 'why'.

deleted 9y ago

[deleted]

westoncb 9y ago

I think I get part of the N-ball analogy, but I'm trying to piece together the rest... So it seems like one problem/feature with high dimensional spaces is that you have more regions which are orthogonal to one other. So, it's possible to have quite a bit of motion, which to a human mind might give the impression that one is doing quite a lot—but then you might discover a new way of turning around and find that space extends in another direction, too, at which point you realize you covered a smaller fraction of the available space than you previously believed. What I don't get is how the volume of an N-ball with fixed radius going to zero as N goes to infinity plays into the analogy. When you say, "when you compare it to a hypercube of the same dimensional count," I don't get what in the programming analogy would link to the N-ball rather than the high dimensional hypercube. I also get the algebraic justification for this property of N-balls, but I still have no real geometric intuition for why it would be the case. Could you clarify what's meant by, "as the number of dimensions increases the closer the 'halfway' point of the volume of the N-ball gets to the edge of the N-ball"? Not sure what's meant by 'halfway' point here, or edge of the N-ball. My guess is that you're talking about distribution of volume, like if you have a tapered cup that gets small at the bottom, most of the volume will be at the top.

Verdex_2 9y ago
Checkout the generic formula for any number of dimensions [1]. The function in the denominator is a factorial (except extended to work on decimals instead of just integers). The functions in the numerator are exponents. Factorials grow much faster than exponents, so with an N-Ball we expect the "volume" to be small if we were to compare it to the corresponding hypercube when the dimension and when the diameter / length are all equal. After all we're comparing approximately diameter * R ^ D / D! and length * R ^ D. So if you put an N-Ball into a hypercube with length = diameter, what you'll discover is that the N-Ball is tiny when compared to the hypercube. Which means we have the weird and counter intuitive conclusion that all the "volume" of a hypercube is in the corners. Try calculating the volumes yourself using a high dimension like 100. Even better plug the whole thing into a script so you can generate a table and see the effect get progressively worse as you increase the dimension count. There is also a similar effect on the N-Ball itself. Imagine drawing a smaller circle inside of a circle where their centers are the same such that the inner circle contains exactly half of the area of the outer circle. Now imagine doing this in a sphere. Now a 4D sphere. Now a 5D sphere. What you'll find is that as you increase the dimension count, the radius of the inner circle will get longer and longer. Which means that half of the "volume" of an N-Ball slowly moves closer and closer to the outer edge of the N-Ball as you increase dimensions. Here's the punch line. Imagine that the N-Ball represents all there is to know about a topic. You begin in the center of the N-Ball and in order to learn more you have to walk to the outer edge of the N-Ball. Also imagine that you're only able to see what's right in front of your face. In the beginning at the center there is very little volume to look at. It will appear to be easy to understand what's going on. But as you walk to the edge of the N-Ball, the amount of volume that you're able to see increases. When you get right to the edge of a very high dimension N-Ball you will notice that you only know half of what is available to know. Simple topics are N-Balls with low dimensions. It's easy to "walk" your way to knowing half of everything there is to know. Complex topics are N-Balls with high dimensions. Programming (and by extension software engineering) is really high dimensional. The more you learn the more you notice there is to learn. When you walk to the edge of what seems possible you find yourself swimming in a sea that dwarfs you. But in the beginning it looks like there isn't really all that much going on. [1] - https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume

deleted 9y ago

[deleted]

lacampbell 9y ago

To use your analogy, when I start exploring outside the valley, I think to myself - "wow, a year ago I only knew this valley. Now I'm getting to know this new area! I've come a long way and I feel great I've made progress!". I tend to focus on the fact I am constantly expanding my knowledge, instead of worrying about all the stuff I don't know yet. I still consider myself a generalist, however. I can't say I'm an expert in any one area. Maybe that's why I feel the way I do.

mpfundstein 9y ago
great way of thinking. its all about the journey

disposablename 9y ago

TL;DR "The more you learn, the less you know"

edem 9y ago

It is more like traversing a tree. The more you learn (the deeper you go) the more branches you can explore which leads to the conclusion that the more you know the more you realize how much you still don't know.