6 ms·
But how do you classify a question as high vs low complexity? Some seemingly simple questions can turn out to be very very complex. For example, integer solutio
by bufferoverflow 1y ago
But how do you classify a question as high vs low complexity? Some seemingly simple questions can turn out to be very very complex. For example, integer solution to
x³ + y³ + z³ = 42
took over a hundred years of compute time to find.
Or another seemingly simple equation with positive integers x,y,z
x/(y+z)+y/(z+x)+z/(x+y) = 4
requires elliptic curve knowledge, and the solution is huge
x = 154476802108746166441951315019919837485664325669565431700026634898253202035277999
y = 36875131794129999827197811565225474825492979968971970996283137471637224634055579
z = 4373612677928697257861252602371390152816537558161613618621437993378423467772036
(Solution is discussed here: https://www.quora.com/How-do-you-find-the-positive-integer-solutions-to-frac-x-y+z-+-frac-y-z+x-+-frac-z-x+y-4 https://www.quora.com/How-do-you-find-the-positive-integer-s...)
- codelion 1y agoQuery complexity in this context is based on how many tokens it took for the model to respond to a query correctly based on a ground truth dataset like GSM8k. The adaptive classifier learns over this dataset and then we use it at inference for classification.
- bufferoverflow 1y agoSo it can be very very wrong. You're trading correctness for speed.
- codelion 1y agoYes, the goal here is to avoid overthinking and be as efficient as possible in terms of the minimal tokens required to solve a query. Often, queries that require too many tokens are unlikely to lead to correct answers anyways otherwise they would show up when we are learning the classifier.
- VagabundoP 1y agoIf you ask it to rethink the problem again because you've found a flaw, does it bump up the complexity and actually think about it. Like a person might give you a quick answer to something and then questioning the answer would cause them to think deeper about it.
- codelion 1y agoThe short answer is in general yes it helps improve the accuracy, there is a whole line of work on self consistency and critique that supports it. Many of those approaches are already implemented in optillm.
- baobabKoodaa 1y agoYes, if you only care about correctness, you always use the maximum possible inference compute. Everything that does not do that is trading correctness for speed.
- wat10000 1y agoIf compute is limited, then dedicating more resources to the questions that are more likely to need it will increase correctness overall, even if it may decrease correctness for some individual responses.
- xigency 1y ago> You're trading correctness for speed. That's AI in a nutshell.
- MrManatee 1y agoI think there exists a separate skill for classifying problems by difficulty, apart from being able to solve them. This skill can be developed from both directions by learning which problems have been solved and which haven't been. If someone asked me to find solutions to these example equations, there are three complications that I would immediately notice: 1. We are looking for solutions over integers. 2. There are three variables. 3. The degree of the equation is 3. Having all three is a deadly combination. If we were looking for solutions over reals or complex numbers? Solvable. Less than three variables? Solvable. Degree less than 3? Solvable. With all three complications, it's still not necessarily hard, but now it might be. We might even be looking at an unsolved problem. I haven't studied enough number theory to actually solve either of these problems, but I have studied enough to know where to look. And because I know where to look, it only takes me a few seconds to recognize the "this might be very difficult" vibe that both of these have. Maybe LLMs can learn to pick up on similar cues to classify problems as difficult or not so difficult without having needing to solve them. (Or, maybe they have already learned?)