7 ms·
Complexity theory puts limits on performance of gradient descent
- gerdesj 5y agoYou say minima and I say maxima; They say maybe over there ... no left a bit.
- miketery 5y ago> Brouwer’s fixed point theorem. The theorem says that for any continuous function, there is guaranteed to be one point that the function leaves unchanged — a fixed point, as it’s known. This is true in daily life. If you stir a glass of water, the theorem guarantees that there absolutely must be one particle of water that will end up in the same place it started from. This seems counter intuitive. Why would this be true? I would have thought that it’s almost impossible for a particle to end up in the same place as it started.
- im3w1l 5y agoIt only works for some idealized continous fluid. Not for actual water, which is made up of a large but finite amount of molecules.
- kevinventullo 5y agoEven then it doesn’t feel like a great example since it’s not obvious to me that stirring should be a continuous function. E.g. if I put half the glass into a container and then pour the container back into the glass, that is definitely a discontinuous operation!
- drdeca 5y agoBut stirring doesn’t break the connnectedness of the water like that?
- kevinventullo 5y agoI guess wherever you dip the spoon in will break it in a kind of discontinuous way?
- drdeca 5y agoIf the spoon was one of those spoons with holes in the, non-handle-part, then I think it would have to produce a discontinuity. But for a normal spoon shape, I don’t think dipping it in makes the map between points in the water fail to be continuous. I’m somewhat confident that there is a smooth path from an embedding of a cylinder to R^3 to an embedding of a cylinder which is a little taller and has a spoon shape removed from it, into R^3 .
- alanbernstein 5y agoI wonder if there is some idealized definition of "stirring" being used here. It seems that actual stirring could conceivably result in any possible rearrangement of particles. As a simple example, which is illustrative even though it seems unlikely as a result of stirring: leave the (x, y) coordinates of each particle the same, while changing the z coordinate as z -> z+k mod h. The result is effectively a discontinuous map (violating the theorem), even though the path of each individual particle must be continuous for physical reasons. Two different notions of continuity here.
- xarici_ishler 5y agohttps://www.youtube.com/watch?v=oX9aPNF6_h8 https://www.youtube.com/watch?v=oX9aPNF6_h8 This is a great video explaining Brouwer's theorem and its (theoretical) application to stirring a cup of water.
- joebob42 5y agoI'm with this guy. Obviously I'm missing something or not understanding what is being said here, because to my head f(x) = x+1 changes every single point and is continuous. Or in the glass of water point, what if I pour the water out on the table. It's certainly all been moved, yes? (To be clear I'm obviously wrong / not understanding the premise, but I would be curious to know what I'm missing)
- im3w1l 5y agoYour example function doesn't map any compact set back to itself.
- joebob42 5y agoYep. I notice mention of this additional constraint from another comment. As far as I can tell the article is just incorrect since it misses this constraint up to that point, unless I'm just a clown and somehow not seeing it. The upside here is that it seems perhaps I wasn't missing the premise, and it was just mis-stated :)
- voldacar 5y agoNo you are right, quanta sucks. Im sure there are thousands of other people thinking of x+1 and scratching their heads
- SilasX 5y agoSo add "mod p" to it.
- random314 5y agoTransformation is not continuous at the boundary p
- nonameiguess 5y agoIt only applies to functions on bounded sets. The reals are not bounded. There are other restrictions, but that is one of them.
- fshbbdssbbgdd 5y agoThe paragraph leaves out an important constraint: it’s only for functions that map some compact convex set to itself. So, the theory doesn’t apply to the sibling commenter’s example of f(x) = x + 1 because it doesn’t map a compact set to itself. One example of a compact convex set is a circle. Imagine some continuous function that maps a circle to a circle. An easy example is rotation. For rotation, the center stays in the same place. Now imagine some more complicated function that might do rotation, stretching, spiral distortions, whatever you can imagine. But it has to be continuous, meaning you can’t go off one edge and come back on the other side (among other things). Try to draw, or even just imagine a transformation like this and you’ll find that there’s always some point that stays in the same place. For a 3D object like a glass it’s hard to see the same thing intuitively, but Brouwer’s theorem says that it holds (and it also holds for higher-dimensional sets).
- kevinventullo 5y agoMinor wording nit: rather than the circle, I would say “consider the disc”. Usually in math “circle” means just the boundary whereas “disc” means the circle together with its interior.
- OrderlyTiamat 5y agothank you, that did clear it up for me.
- antman 5y agoWhat about a circle with 60cm diameter that I move one meter to the right? Something is missing in my understanding of the definition.
- legolas2412 5y agoIt has to map the same set onto itself.
- enriquto 5y ago> For a 3D object like a glass A glass is not convex so this theorem does not apply. You'd need other fixed-point theorems for that. If the glass has a handle the result is trivially false (consider rotating a doughnut, this does not have any fixed point).
- deleted 5y ago[deleted]
- techwizrd 5y agoThe devil lies in the details here. Brouwer's fixed point theorem says that for a continuous function f that maps a _compact, convex_ set to itself, there must exist a point x₀ such that f(x₀) = x₀. This theorem is only valid for sets that are compact (i.e., closed and bounded) and homeomorphic to convex. I think the 3-dimensional analogy (similar to the stirring of coffee and sugar Brouwer observed) is rather unintuitive. I think some of the proofs are quite beautiful, but I like this analogy from Wikipedia: > Take an ordinary map of a country, and suppose that that map is laid out on a table inside that country. There will always be a "You are Here" point on the map which represents that same point in the country.
- zitterbewegung 5y agoAlso, an example of Brouwer's fixed point theorem is the Y Combinator in the lambda calculus. See https://en.wikipedia.org/wiki/Fixed-point_combinator#Y_combinator https://en.wikipedia.org/wiki/Fixed-point_combinator#Y_combi...
- drdeca 5y agoWhat is the compact space in question there? I don’t see one. Or, I don’t even see an appropriate topology to apply? Are you sure?
- SquishyPanda23 5y agoAs others have mentioned, the article is missing a crucial constraint, which is essentially that it holds on the closed unit ball and spaces homeomorphic to it. As you mentioned, it's also not intuitive. It's by now well-digested, but it's a fairly surprising theorem, even to Brouwer. Finally, it's not constructive. Generally you can't figure out which point is fixed. If I recall correctly, frustration with this fact fueled Brouwer's interest in intuitionism, which gives us a type of logic where some propositions are neither true nor false but essentially undecidable. I'm not sure what you mean by a "particle" in your sentence. If you mean to go down to the quantum scale, then I'm sure all sorts of things go wrong in trying to apply the theorem. For one, even talking about the position of a particle at all is very different from talking about a point in Euclidean space. If you don't mean to go down to the quantum scale, then lots of maps have fixed points. Rotations an reflections are two large classes of examples.
- vlovich123 5y agoDoes intuitionism go beyond Gödel’s incompleteness theorem? Or is it an unrelated thing?
- deleted 5y ago[deleted]
- skulk 5y agoThis philosophy.se post answers that question: https://philosophy.stackexchange.com/questions/7194/how-is-g%C3%B6dels-incompleteness-theorem-interpreted-in-intuitionistic-logic https://philosophy.stackexchange.com/questions/7194/how-is-g...
- SquishyPanda23 5y agoThey are different things, although "unrelated" might be a bit strong. I'm sure there are interesting things to say about the incompleteness theorem and intuitionistic logic. Intuitionistic logic is a logic that is weaker than classical logic in the sense that there are some proofs in classical logic that don't work in intuitinistic logic. Specifically, the law of the excluded middle (either A is true or not A is true") does not hold. You can convince yourself that this is a reasonable type of logic by interpreting "A is true" to mean "I have a proof of A", and likewise "not A is true" means "I have a proof of not A". Then it's pretty reasonable that you neither have a proof of A or of not A. I haven't read Godel's proof, but my understanding is that he uses intuitionistic logic in some of the key steps. Since facts in intuitionistic logic are also facts in classical logic, then I believe the incompleteness theorem holds in both logics. But I'm not at all an expert here, so someone please correct me if I'm wrong. "Intuitionism" also refers to a specific philosophy of mathematics. But a lot of people just use the word to refer to the logic that came out of that philosophical school. Intuitionistic logic is also very common in type theory, so a lot of proof assistants use it.
- deleted 5y ago[deleted]
- arketyp 5y agoI stumbled upon this theorem after trying to come up with a mapping of color space, RGB to RGB, where each color maps to a new one. (It was for a mouse cursor effect in a painting app.) I couldn't come up with one, and indeed it is impossible!
- chompychop 5y agoAre there any constraints on the mapping? If no, then why not map each intensity value to the immediately next value, and then circle it around? For instance, 0 -> 1, 1 -> 2, 2 -> 3, ...., 255 -> 0.
- voldacar 5y agoThe function you are describing is not continuous, so the theorem does not apply. But yeah, it has no fixed point.
- cblconfederate 5y agoI think it's because we think of the drink points as molecules which can indeed move and rearrange in ways that continuous points cannot.
- zbendefy 5y agowatch this, it explains it well: https://www.youtube.com/watch?v=csInNn6pfT4 https://www.youtube.com/watch?v=csInNn6pfT4
- hyperbovine 5y agoIt's counter intuitive because it's obviously false as stated: f(x) = 1 + x for x in R. They left out the main assumption of the theorem, namely that the domain of the function is compact, and gets mapped (in)to itself.
- max_ 5y ago>I would have thought that it’s almost impossible for a particle to end up in the same place as it started. That's true on a short time scale. But on a longer time scale, you will eventually visit all possible states of the system. I think this is similar to the concept of ergodicity[0]. [0]: https://en.wikipedia.org/wiki/Ergodicity https://en.wikipedia.org/wiki/Ergodicity
- optimalsolver 5y agoDerivative-free optimization to the rescue!
- wenc 5y agoDFO is far slower to converge in general than gradient based methods even when it’s close to the optimum because it doesn’t use derivative information. Newton’s method on the other hand has local quadratic convergence. Not sure I would bet on DFOs to have better performance than gradient methods. DFOs have two advantages: they’re embarrassingly parallelizable and for non convex problems, they don’t succumb as easily to local optima. But they’re not better performing in general.
- Buttons840 5y ago> For example, gradient descent is often used in machine learning in ways that don’t require extreme precision. But a machine learning researcher might want to double the precision of an experiment. In that case, the new result implies that they might have to quadruple the running time of their gradient descent algorithm. That’s not ideal, but it is not a deal breaker. > But for other applications, like in numerical analysis, researchers might need to square their precision. To achieve such an improvement, they might have to square the running time of gradient descent, making the calculation completely intractable. I think a silver lining here is that a company having access to 10,000 times as much computing power as a common consumer can only achieve, say, a 10x better model. So the inequality isn't as extreme as it could be.
- scotth 5y agoThis is my favorite comment (please don't down vote)
- mjburgess 5y agoIs that how scientific modelling (ie., modelling of non-computable functions) works though? It's not necessarily that squaring precision leads to a "better model", its that it fails to be a model at all without that precision. Eg., consider a climate model with effects at the first-decimal place, and 10th decimal-place of some measurable quantity vs. time. The effects at the 10th dp. could aggregate to overwhelm the effects at the 1 dp. It isnt that the model is better if it can "compute at the 10th dp" -- its just wildly inaccurate otherwise.
- ben_w 5y ago10 decimal points reminds me of a conversation I had with my father. Last century my dad was working on radar/IFF simulations, and he had an anecdote (if you can call it that) about when they switched from normal to double precision, and needing to get more digits of π (I think from 10 to 20, the way he talked implied some very strange things about the programming language(s) he was using). The third time he told this story, late-teenage-me pointed out that the curvature of spacetime under GR had more impact than the extra digits. While the broader point is valid, IIRC climate (being about average and broad trends) is fine with the imperfections, and handles this by running multiple simulations with slightly different parameters and seeing what usually happens. Weather forecast likewise.
- dynm 5y ago> But a machine learning researcher might want to double the precision of an experiment. In that case, the new result implies that they might have to quadruple the running time of their gradient descent algorithm. > But for other applications, like in numerical analysis, researchers might need to square their precision. To achieve such an improvement, they might have to square the running time of gradient descent I don't quite understand this. Say f(p) is the complexity of gradient descent for precision p. The first paragraph seems to imply that f(p) = c×p^2, so that f(2p) = c×4×p^2 = f(p). However, the second paragraph then just seems to imply that f(p) = c×p. Am I missing something? Incidentally, I love that Quanta brings attention to work mostly just on the basis of how important it is. 99% of the other stuff you hear about new research is really just driven by PR departments at the researcher's home institutions.
- mmiyer 5y agoIf f(p) = cp^2 then f(p^2) = c(p^2)^2 = cp^4 which is square the running time, ignoring constant factors.
- FabHK 5y agoNow, that article was written by an academic, not a practitioner (my emphasis): > You can imagine a function as a landscape, where the elevation of the land is equal to the value of the function (the “profit”) at that particular spot. Gradient descent searches for the function’s local minimum [...]
- tzs 5y ago> But for other applications, like in numerical analysis, researchers might need to square their precision. To achieve such an improvement, they might have to square the running time of gradient descent, making the calculation completely intractable There is an easy way to counteract this. Let’s say your original runs in 1000 seconds. Squaring that would give 1000000 seconds. That’s a jump from under 17 minutes to over 11 days. The fix: measure everything in hours. Your original runs in 0.278 hours. Squaring that gives 0.077 hours. Squaring the precision now makes it run 3.6 times faster.
- jakeinspace 5y agoI assume this is a joke, but for those not aware: the correct calculation would be to take a measure of the original running time in terms of CPU cycles or some equivalent unitless value, and square that. You don't want to be squaring values with units attached, because squared hours (or squared seconds) are very much not simple units of time.
- mmiyer 5y agoCPU cycles isn't right either - this article is about theoretical CS not about a real machine (squared cycles is meaningless too). It seems the author is saying the algorithm running time is O(n^2) where n is precision. This is approximately cn^2 where c is a constant. Without knowing c we cannot say what squaring the running time precisely means.