10 ms·

For algorithms, a little memory outweighs a lot of time

343 points by makira 1y ago

kazinator 1y ago
Give algorithm designers a little more memory, and they will find a way to shave off the time. Give operating system development organizations a lot more memory and they will find a way to worsen the response time.
sieabahlpark 1y ago
[dead]
cperciva 1y ago
Minus the fuzz: A multitape Turing machine running in time t can be simulated using O(sqrt(t log t)) space (and typically more than t time). https://arxiv.org/abs/2502.17779 https://arxiv.org/abs/2502.17779
- diamondage 1y ago
  Should have come to the comments first!
LPisGood 1y ago
I think it is very intuitive that more space beats the pants off of more time. In time O(n) you can use O(n) cells on a tape, but there are O(2^n) possible configurations of symbols on a tape of length n (for an alphabet with 2 symbols), so you can do so much more with n space than with n time.
- thatguysaguy 1y ago
  Intuitive yes, but since P != PSPACE is still unproven it's clearly hard to demonstrate.
  - LPisGood 1y ago
    I think that since many people find it intuitive that P != NP, and PSPACE sits way on top of polynomial hierarchy that it is intuitive even if it’s unproven.
  - dragontamer 1y ago
    The article is about a new proof wherein P == PSPACE. Something we all intuitively expected but someone finally figured out an obscure way to prove it. -------- This is a really roundabout article that takes a meandering path to a total bombshell in the field of complexity theory. Sorry for spoiling but uhhh, you'd expect an article about P == PSPACE would get to the point faster....
    - LPisGood 1y ago
      This article is not about a proof that P = PSPACE. That would be way bigger news since it also directly implies P = NP.
  - porphyra 1y ago
    There's not even a proof that P != EXPTIME haha EDIT: I am a dumbass and misremembered.
    - doc_manhat 1y ago
      I think there is right? It's been a long time but I seem to remember it following from the time hierarchy theorem
    - LPisGood 1y ago
      I thought there was some simple proof of this, but all I can think of is time hierarchy theorem.
- deleted 1y ago
  [deleted]
- qbane 1y ago
  But you also spend time on updating cells, so it is not that intuitive.
  - LPisGood 1y ago
    I’m not sure what you mean here. If you’re in the realm of “more space” than you’re not thinking of the time it takes. More precisely, I think it is intuitive that the class of problems that can be solved in any time given O(n) space is far larger than the class of problems that can be solved in any space given O(n) time.
    - Almondsetat 1y ago
      If your program runs in O(n) time, it cannot use more than O(n) memory (upper bound on memory usage. If your program uses O(n) memory, it must run at least in O(n) time (lower bound on time).
      - deleted 1y ago
        [deleted]
      - bmacho 1y ago
        In other words M <= T.
      - withinboredom 1y ago
        This is pretty easy to refute: > If your program runs in O(n) time, it cannot use more than O(n) memory (upper bound on memory usage.[sic] This is clearly refuted by all software running today. Programs (especially games) clearly use more memory than there are instructions in the program. > If your program uses O(n) memory, it must run at least in O(n) time (lower bound on time). Memory bombs use an incredible amount of memory and do it incredibly quickly.
        Almondsetat 1y ago
        >Programs (especially games) clearly use more memory than there are instructions in the program. How can you access a piece of memory without issuing an instruction to the CPU? Also, "clearly" is not an argument. >Memory bombs use an incredible amount of memory and do it incredibly quickly. How can you access a piece of memory without issuing an instruction to the CPU? Also "incredibly quickly" is not an argument. Also also, O(n) is incredibly quick.
- hn_acc1 1y ago
  My intuition: the value of a cell can represent the result of multiple (many) time units used to compute something. If you cannot store enough intermediate results, you may end up needing to recalculate the same / similar results over and over - at least in some algorithms. So one cell can represent the results of hundreds of time units, and being able to store / load that one value to re-use it later can then replace those same hundreds of time units. In effect, space can be used for "time compression" (like a compressed file) when the time is used to compute similar values multiple times. If intermediate results are entirely uncorrelated, with no overlap in the work at all, that would not hold - space will not help you. Edit: This kind of problem is very rare. Think of a cache with 0 percent hit rate - almost never happens. And you can't really do it the other way around (at least not in current computing terms / concepts): you cannot use a single unit of time as a standin / proxy for hundreds of cells, since we don't quite have infinitely-broad SIMD architectures.
  - slt2021 1y ago
    I think this theorem applies well for modern LLMs: large language model with pre-computed weights can be used to compute very complex algorithms that approximate human knowledge, that otherwise were impossible or would have required many orders more compute to calculate
  - benchloftbrunch 1y ago
    As I understand it, this is the essential insight behind dynamic programming algorithms; the whole idea is to exploit the redundancies in a recursive task by memoizing the partial results of lower order stages.
  - RetroTechie 1y ago
    There's many algorithms with a space vs time tradeoff. But what works best, depends a lot on the time/energy cost of re-computing something, vs the storage/bandwidth cost of caching results. Expensive calculation, cheap storage → caching results helps. Limited bandwidth / 'expensive' storage, simple calculation (see: today's hyper-fast CPU+L1 cache combo's) → better to re-compute some things on the fly as needed. I suspect there's a lot of existing software (components) out there designed along the "save CPU cycles, burn storage" path, where in modern reality a "save storage, CPU cycles are cheap" would be more effective. CPU speeds have grown way way faster than main memory bandwidth (or even size?) over the last decades. For a datacenter, supercomputer, embedded system, PC or some end-user's phone, the metrics will be different. But same principle applies.
- frollogaston 1y ago
  Also, the O(1) random memory access assumption makes it easy to take memory for granted. Really it's something like O(n^(1/3)) when you're scaling the computer to the size of the problem, and you can see this in practice in datacenters. I forget the name of the O(1) access model. Not UMA, something else.
  - cperciva 1y ago
    O(n^(1/2)) really, since data centers are 2 dimensional, not 3 dimensional. (Quite aside from the practical "we build on the surface of the earth" consideration, heat dissipation considerations limit you to a 2 dimensional circuit in 3-space.)
    - frollogaston 1y ago
      If you have rows of racks of machines, isn't that 3 dimensions? A machine can be on top of, behind, or next to another that it's directly connected to. And the components inside have their own non-uniform memory access. Or if you're saying heat dissipation scales with surface area and is 2D, I don't know. Would think that water cooling makes it more about volume, but I'm not an expert on that.
      - jvanderbot 1y ago
        Spatial position has nothing (ok only a little) to do with topology of connections.
      - manwe150 1y ago
        That example would be two dimensions still in the limit computation, since you can keep building outwards (add buildings) but not scale upwards (add floors)
        frollogaston 1y ago
        You can add floors though. Some datacenters are 8 stories with cross-floor network fabrics.
- undefuser 1y ago
  I think it really depends on the task at hand, and not that intuitive. At some point accessing the memory might be slower than repeating the computation, especially when the storage is slow.
- IsTom 1y ago
  One one hand yes, on the other there might be some problems that are inherently difficult to parallelize (alternating machine PTIME is the same as deterministic PSPACE) where space doesn't buy you much. The jump from paper from t/log t to sqrt(t log t) is huge, but it still might be that unbounded parallelism doesn't buy you much more.
whatever1 1y ago
Lookup tables with precalculated things for the win! In fact I don’t think we would need processors anymore if we were centrally storing all of the operations ever done in our processors. Now fast retrieval is another problem for another thread.
- chowells 1y ago
  Oh, that's not a problem. Just cache the retrieval lookups too.
  - michaelhoney 1y ago
    it's pointers all the way down
    - drob518 1y ago
      Just add one more level of indirection, I always say.
      - EGreg 1y ago
        But seriously… the solution is often to cache / shard to a halfway point — the LLM model weights for instance — and then store that to give you a nice approximation of the real problem space! That’s basically what many AI algorithms do, including MCTS and LLMs etc.
- crmd 1y ago
  Reminds me of when I started working on storage systems as a young man and once suggested pre-computing every 4KB block once and just using pointers to the correct block as data is written, until someone pointed out that the number of unique 4KB blocks (2^32768) far exceeds the number of atoms in the universe.
  - ww520 1y ago
    The idea is not too far off. You could compute a hash on an existing data block. Store the hash and data block mapping. Now you can use the hash in anywhere that data block resides, i.e. any duplicate data blocks can use the same hash. That's how storage deduplication works in the nutshell.
lukeyoo 1y ago
[dead]
IvanK_net 1y ago
I am confused. If a single-tape turing machine receives a digit N in binary, and is supposed to write N ones on the tape, on the right side of the digit N, it performs N steps. If you expect N ones at the output, how can this machine be simulated in the space smaller than N? This machine must decrement the digit N at the beginning of the tape, and move to the end of the tape to write "1", so it runs in time O(N^2), not O(N)? (as it takes N "trips" to the end of the tape, and each "trip" takes 1, 2, 3 .. N steps) Since turing machines can not jump to any place on a tape in constant time (like computers can), does it have any impact on real computers?
- cperciva 1y ago
  Multitape Turing machines are far more powerful (in terms of how fast they can run, not computability) than single-tape machines. But to answer your question: "space" here refers to working space, excluding the input and output.
  - IvanK_net 1y ago
    A single tape machine is still a multi tape machine, only with one tape.
- iNic 1y ago
  This paper looks exclusively at decision problems, i.e. problems where the output is a single bit. EDIT: This makes sense because if you look at all problems with N outputs then that is just the same as "gluing together" N different decision problems (+ some epsilon of overhead)
  - IvanK_net 1y ago
    Oh okay, that was my second guess.
willmarquis 1y ago
This is just a reminder that memory isn’t just a constraint, it’s a resource.
- michaelsbradley 1y ago
  What resources are available (or not) and in what quantities are the most basic constraints for solving a problem/s with a computer.
felineflock 1y ago
Ryan Williams lecture (how he started): https://www.youtube.com/live/0DrFB2Cp7tg https://www.youtube.com/live/0DrFB2Cp7tg And paper: https://people.csail.mit.edu/rrw/time-vs-space.pdf https://people.csail.mit.edu/rrw/time-vs-space.pdf
vishnugupta 1y ago
Rainbow Tables FTW!
golly_ned 1y ago
“ If the problem is solved next week, Williams will be kicking himself. Before he wrote the paper, he spent months trying and failing to extend his result” What a strange, sad way to think about this. Academia is perverse.
- catoc 1y ago
  Nothing necessarily perverse here. I don’t know Williams but don’t image him disliking the other guy or being unhappy with progress in general, but just being someone who truly challenged himself only to find him being trumped a week later; and kicking himself for that. Either way, the week is not yet over, at least since the quanta article, so maybe no kicking will ensue.
  - golly_ned 1y ago
    I wonder how the researchers who supplied the tree evaluation algorithm felt when Williams supplied his proof. Dismay at having not kept their result under wraps for longer so they could claim credit for such an advance themselves? I hope not.
ziofill 1y ago
At the cost of sounding ridiculous: can there be a notion of "speed of light" in the theory of computation, determining the ultimate limit of memory (space) vs runtime?
- awanderingmind 1y ago
  You mean something like this https://en.wikipedia.org/wiki/Bremermann%27s_limit https://en.wikipedia.org/wiki/Bremermann%27s_limit or this https://en.wikipedia.org/wiki/Quantum_speed_limit https://en.wikipedia.org/wiki/Quantum_speed_limit?
  - ziofill 1y ago
    Oh wait, I just realized what I said was probably very stupid: I was thinking of some computational complexity theorem that links memory and runtime complexity classes in the same way that the "speed of light" sets an ultimate bound on the relation between actual space and actual time. But the speed of light is the maximum space in the smallest time, which computationally would correspond to filling the largest amount of memory in the shortest time :facepalm: (and thanks for the links!)
xlii 1y ago
From the „Camel Book”, one of my favorite programming books (not because it was enlightening, but because it was entertaining); on the Perl philosophy: “If you’re running out of memory, you can buy more. But if you’re running out of time, you’re screwed.”
- immibis 1y ago
  This can work both ways. If the program needs more memory than the computer has, it can't run until you buy more. But if it takes twice as long, at least it runs at all.
  - Dban1 1y ago
    brain brain brain
  - hinkley 1y ago
    The Camel book was written when Moore’s Law was trucking along. These days you can’t buy much more time but you used to be able to just fine. Now it’s horizontal scaling. Which is still more time.
  - const_cast 1y ago
    Modern computers have so much memory it feels like it doesn't matter. Spending that memory on arrays for algorithms or things like a Garbage Collector just make sense. And, extra memory is worthless. You WANT the summation of all your programs to use all your memory. The processor, on the other hand, can context switch and do everything in it's power to make sure it stays busy. The CPU is like an engine and memory is your gas tank. Idling the engine is bad, but leaving gas in the tank doesn't hurt, but it doesn't help either. I'm not gonna get to my destination faster because I have a full tank.
    - Mawr 1y ago
      Only if running one such memory-hungry program at a time, which usually cannot be afforded. Multi-program workloads are much more common and the strategy of using as much ram as possible can't work in that environment.
woolion 1y ago
It's unfortunate that Quanta links are so popular, when they include so much pseudo-poetic fluff around the mathematics. Below there's an entire thread to dismiss a misconception introduced by the quanta article. "I think it is very intuitive that more space beats the pants off of more time." (poster is absolutely right) The The article say "Until now, the only known algorithms for accomplishing certain tasks required an amount of space roughly proportional to their runtime, and researchers had long assumed there’s no way to do better.", which is interpreted as that there's a proportional relation between time and space. However, a quick look at the complexity hierarchy would never suggest such a thing. Reading more carefully, it says "known algorithms" for "certain tasks", but then where do you get a general intuition from such a particular statement?
- godelski 1y ago
  I think they used to be better but really have made a blatant turn. I really thought that wormhole fiasco would have killed them. To go 4 whole months before putting the editor's note is beyond egregious[0]. Mistakes happen, but 4 months kills all credibility. You have to act fast on those things! There were big names raising serious concerns on day 1 and it really shows they didn't do due diligence to get outside verification before running a piece that they knew would be really popular. All this accomplishes is discrediting science. Trading personal gains for eroding the very thing that they make their money off of. This is a major part of why Americans (and people) have such high distrust for science. News outlets, and in particular science focused news outlets, constantly spew inaccurate information. It really should be no wonder that so many people are confused about so many scientific topics, as unless they actually take the years it takes to become an expert in a field, they are going to have a difficult time distinguishing fact from fiction. And why shouldn't the average person expect to trust a source like Quanta? They're "full of experts", right? smh [0] This is the earliest archive I see with the note. Press back one day and it should not be there. Article was published on Nov 30 2022, along with a youtube video https://web.archive.org/web/20230329191417/https://www.quantamagazine.org/physicists-create-a-wormhole-using-a-quantum-computer-20221130/ https://web.archive.org/web/20230329191417/https://www.quant...
- bbrubaker 1y ago
  I'm the author of this article. If you ask a complexity theorist, they will tell you that they did in fact have a general intuition that certain problems require space close to to linear in time to solve (see e.g., Ryan's comment #22 on Scott Aaronson's blog post about the result: https://scottaaronson.blog/?p=8680 https://scottaaronson.blog/?p=8680, and the comments after that). The most intuitive way to see this is in a circuit/DAG picture, where the goal is to get from the input nodes of the graph to the output nodes. Some graphs are very "wide": cut the graph at some intermediate point, and there will be a lot of edges crossing the cut, each of which represents some information from an earlier stage in the computation that you'll need to remember to get to the output. Ryan's first result is a general-purpose method for doing any computation, even ones whose graph structure looks like this, in asymptotically far less space. That is precisely what makes the result so surprising! My article was quite explicit in multiple places that the universal/comprehensive character of the result was that counterintuitive part: - In the first paragraph: "memory was more powerful than computer scientists believed: A small amount would be as helpful as a lot of time in all conceivable computations." - Further down in the introduction, in the passage you quoted: "Until now, the only known algorithms for accomplishing certain tasks required an amount of space roughly proportional to their runtime, and researchers had long assumed there’s no way to do better. Williams’ proof established a mathematical procedure for transforming any algorithm — no matter what it does — into a form that uses much less space. - In the third section, I explicitly state that researchers do believe space is more powerful than time in the specific sense that you're criticizing my article for misrepresenting: "But complexity theorists suspect that PSPACE is a much larger class, containing many problems that aren’t in P. In other words, they believe that space is a far more powerful computational resource than time. This belief stems from the fact that algorithms can use the same small chunk of memory over and over, while time isn’t as forgiving — once it passes, you can’t get it back." - In the fourth section, I explain why researchers didn't think the HPV75 result could be improved further, despite their intuition that space is more powerful than time in the above sense: "While many problems can be solved with much less space than time, some intuitively seemed like they’d need nearly as much space as time." TCS (and complexity theory specifically) are complicated subjects. I spend a lot of time interviewing researchers and thinking about how to distill the results of my reporting into a form that is accessible to readers with widely varying levels of familiarity with the subject matter. You are of course well within your rights to critique my stylistic choices, the narrative aspects of the story, and the order in which I presented information, but I will push back against the claim that my article is spreading misinformation about complexity theory. You're referring to a misconception that arises, by your own admission, when you don't read carefully. If it's the headline you object to, you could lodge a similar complaint against the complexity theorist Lance Fortnow: https://blog.computationalcomplexity.org/2025/02/you-need-much-less-memory-than-time.html https://blog.computationalcomplexity.org/2025/02/you-need-mu....
ChrisMarshallNY 1y ago
Interesting. It's one of those things that I’ve always “just assumed,” without thinking about it. I did a lot of raster graphics programming, in my career, and graphics work makes heavy use of lookup tables. Yesterday, I posted a rather simple tool I wrote[0]: a server that “frontloads” a set of polygons into a database, and then uses them, at query time. It’s fairly fast (but I’m sure it could be a lot faster). I wrote it in a few hours, and got pretty good performance, right out of the starting gate. Pretty basic stuff. I doubt the pattern is unique, but it’s one that I’ve used for ages. It’s where I try to do as much “upfront” work as possible, and store “half-baked” results into memory. Like I said, I always “just worked that way,” and never really thought about it. There’s been a lot of “rule of thumb” stuff in my work. Didn’t have an MIT professor to teach it, but it’s sort of gratifying to see that it wasn’t just “old wives” stuff. There’s probably a ton of stuff that we do, every day, that we don’t think about. Some of it almost certainly originated from really smart folks like him, finding the best way (like the “winding number” algorithm, in that server[1]), and some of it also probably comes from “grug-brained programmers,” simply doing what makes sense. [0] https://news.ycombinator.com/item?id=44046227 https://news.ycombinator.com/item?id=44046227 [1] https://github.com/LittleGreenViper/LGV_TZ_Lookup/blob/e247f2f97fb83e5392d31f81c321814baa26dc1b/src/Sources/LGV_TZ_Lookup_Query.class.php#L101 https://github.com/LittleGreenViper/LGV_TZ_Lookup/blob/e247f...
- auto 1y ago
  I'm in the depths of optimization on a game right now, and it's interesting how the gains I'm making currently all seem to be a matter of scaling the concept of lookup tables, and using the right tool for the job. What I mean is that traditionally I think peoples' ideas of lookup tables are things like statically baked arrays setup at compile time, or even first thing at runtime, and they never change. But if you loosen your adherence to that last idea a bit, where a lookup table can change slightly over time, you can get a ton of mileage out of a comparatively small amount of memory compared to wasting cycles every frame. As for the right tool for the job, I've read tons of dev logs and research papers over the years about moving work to the GPU, but this last few months stint of ripping my game inside out has really made me see the light. It's not just lookup tables built at compile or early runtime, but lookup tables modified slightly over time, and sent to the GPU as textures and used there. Follow this train of thought long enough, and now we're just calling memory writes and reads "lookup tables" when they aren't really that anymore, but whos to say where the barrier really lies?
alkyon 1y ago
It's kind of insulting to the reader that they explain P complexity class without using the word polynomial ("all problems that can be solved in a reasonable amount of time")
- simpaticoder 1y ago
  Be generous - it saves a lot of time. Once you say "polynomial" readers will think, "like, ANY polynomial, even like n^100?!" and you'll have to explain, yes, but that's STILL better than exponential, etc. They avoided all of that
  - godelski 1y ago
    Quanta targets people who are above average. So I don't think it is too much for them to give a sentence or two stating that. Or even a little graphic could do wonders. I don't think it would take much time or effort to make a graphic like the one on wikipedia[0] and just throw in some equations within the ring. You can easily simplify too, by removing NL and merging EXP. Hell, look at the graphics here[1]. That's much more work. I don't think Quanta should be afraid of showing math to people. That's really their whole purpose. Even if I think they've made some egregious mistakes that make them untrustable...[2] [0] https://en.wikipedia.org/wiki/PSPACE#/media/File:Complexity_subsets_pspace.svg https://en.wikipedia.org/wiki/PSPACE#/media/File:Complexity_... [1] https://www.quantamagazine.org/june-huh-high-school-dropout-wins-the-fields-medal-20220705/ https://www.quantamagazine.org/june-huh-high-school-dropout-... [2] https://news.ycombinator.com/item?id=44067043 https://news.ycombinator.com/item?id=44067043
    - simpaticoder 1y ago
      I suppose my point is that the readers who will wonder about this are a) very likely to know about complexity classes already, or b)capable of learning about it themselves. Perhaps a simple link to something like https://complexityzoo.net/Petting_Zoo https://complexityzoo.net/Petting_Zoo would have been a nice middle-ground. Edit: Aaronson even mentions the n^100 problem in the section about P!
nottorp 1y ago
You can determine both the time and space complexity for any algorithm, should you need to. And in some cases you can adjust to have more of one or the other, depending on your particular constraints. More at eleven.
amelius 1y ago
> Williams’ proof established a mathematical procedure for transforming any algorithm — no matter what it does — into a form that uses much less space. Ok, but space is cheap, and we usually want to trade processing time for space. I.e., the opposite.
- JohnKemeny 1y ago
  Ryan made a dent (a tiny dent) in one of the most important open problems in mathematics. He's not trying to please programmers.
  - amelius 1y ago
    But what makes an open problem important? ;)
senfiaj 1y ago
I always thought that the "reverse" relationship between the time and the space requirements is explained by the simple fact that when you have a set of algorithms with certain time and space requirements and you constrain one of these parameters, the other parameter will not be necessarily (in practice oftentimes not) the most optimal. But it doesn't necessarily mean that the faster algorithm will require less memory or vice versa. This "reverse" relationship works with other parameters. Such as in sorting algorithms, where besides time and space requirements you have stability requirement. If you constrain the sorting algorithms by stability, you won't have any advantage (will likely to have some disadvantage) in performance over non constrained algorithms. And wee see that the known stable sorting algorithms are either slower or require more memory. Nobody has yet invented a stable sorting algorithm that has comparable performance (both time and space) to non stable sorting algorithms. That's it.
schmorptron 1y ago
As an aside, I really enjoy a lot of the quanta magazine articles! They manage to write articles that both appeal to compsci people as well as interested "outsiders". The birds-eye view and informal how and why they pack into their explanation style often gives me new perspectives and things to think about!
bgnn 1y ago
The poetic style of Quanta makes it impossible to understand what does this mean. Can someone familiar with the topic explain is this applicable to real world computers or just a theoretical scenario? Does this mean we need more memory in computers even if they need to run at a lower clock speed?

343 points by makira 1y ago

Give algorithm designers a little more memory, and they will find a way to shave off the time. Give operating system development organizations a lot more memory and they will find a way to worsen the response time.

sieabahlpark 1y ago

[dead]

cperciva 1y ago

Minus the fuzz: A multitape Turing machine running in time t can be simulated using O(sqrt(t log t)) space (and typically more than t time). https://arxiv.org/abs/2502.17779 https://arxiv.org/abs/2502.17779

diamondage 1y ago
Should have come to the comments first!

LPisGood 1y ago

I think it is very intuitive that more space beats the pants off of more time. In time O(n) you can use O(n) cells on a tape, but there are O(2^n) possible configurations of symbols on a tape of length n (for an alphabet with 2 symbols), so you can do so much more with n space than with n time.

thatguysaguy 1y ago
Intuitive yes, but since P != PSPACE is still unproven it's clearly hard to demonstrate.
- LPisGood 1y ago
  I think that since many people find it intuitive that P != NP, and PSPACE sits way on top of polynomial hierarchy that it is intuitive even if it’s unproven.
- dragontamer 1y ago
  The article is about a new proof wherein P == PSPACE. Something we all intuitively expected but someone finally figured out an obscure way to prove it. -------- This is a really roundabout article that takes a meandering path to a total bombshell in the field of complexity theory. Sorry for spoiling but uhhh, you'd expect an article about P == PSPACE would get to the point faster....
  - LPisGood 1y ago
    This article is not about a proof that P = PSPACE. That would be way bigger news since it also directly implies P = NP.
- porphyra 1y ago
  There's not even a proof that P != EXPTIME haha EDIT: I am a dumbass and misremembered.
  - doc_manhat 1y ago
    I think there is right? It's been a long time but I seem to remember it following from the time hierarchy theorem
  - LPisGood 1y ago
    I thought there was some simple proof of this, but all I can think of is time hierarchy theorem.
deleted 1y ago
[deleted]
qbane 1y ago
But you also spend time on updating cells, so it is not that intuitive.
- LPisGood 1y ago
  I’m not sure what you mean here. If you’re in the realm of “more space” than you’re not thinking of the time it takes. More precisely, I think it is intuitive that the class of problems that can be solved in any time given O(n) space is far larger than the class of problems that can be solved in any space given O(n) time.
  - Almondsetat 1y ago
    If your program runs in O(n) time, it cannot use more than O(n) memory (upper bound on memory usage. If your program uses O(n) memory, it must run at least in O(n) time (lower bound on time).
    - deleted 1y ago
      [deleted]
    - bmacho 1y ago
      In other words M <= T.
    - withinboredom 1y ago
      This is pretty easy to refute: > If your program runs in O(n) time, it cannot use more than O(n) memory (upper bound on memory usage.[sic] This is clearly refuted by all software running today. Programs (especially games) clearly use more memory than there are instructions in the program. > If your program uses O(n) memory, it must run at least in O(n) time (lower bound on time). Memory bombs use an incredible amount of memory and do it incredibly quickly.
      - Almondsetat 1y ago
        >Programs (especially games) clearly use more memory than there are instructions in the program. How can you access a piece of memory without issuing an instruction to the CPU? Also, "clearly" is not an argument. >Memory bombs use an incredible amount of memory and do it incredibly quickly. How can you access a piece of memory without issuing an instruction to the CPU? Also "incredibly quickly" is not an argument. Also also, O(n) is incredibly quick.
hn_acc1 1y ago
My intuition: the value of a cell can represent the result of multiple (many) time units used to compute something. If you cannot store enough intermediate results, you may end up needing to recalculate the same / similar results over and over - at least in some algorithms. So one cell can represent the results of hundreds of time units, and being able to store / load that one value to re-use it later can then replace those same hundreds of time units. In effect, space can be used for "time compression" (like a compressed file) when the time is used to compute similar values multiple times. If intermediate results are entirely uncorrelated, with no overlap in the work at all, that would not hold - space will not help you. Edit: This kind of problem is very rare. Think of a cache with 0 percent hit rate - almost never happens. And you can't really do it the other way around (at least not in current computing terms / concepts): you cannot use a single unit of time as a standin / proxy for hundreds of cells, since we don't quite have infinitely-broad SIMD architectures.
- slt2021 1y ago
  I think this theorem applies well for modern LLMs: large language model with pre-computed weights can be used to compute very complex algorithms that approximate human knowledge, that otherwise were impossible or would have required many orders more compute to calculate
- benchloftbrunch 1y ago
  As I understand it, this is the essential insight behind dynamic programming algorithms; the whole idea is to exploit the redundancies in a recursive task by memoizing the partial results of lower order stages.
- RetroTechie 1y ago
  There's many algorithms with a space vs time tradeoff. But what works best, depends a lot on the time/energy cost of re-computing something, vs the storage/bandwidth cost of caching results. Expensive calculation, cheap storage → caching results helps. Limited bandwidth / 'expensive' storage, simple calculation (see: today's hyper-fast CPU+L1 cache combo's) → better to re-compute some things on the fly as needed. I suspect there's a lot of existing software (components) out there designed along the "save CPU cycles, burn storage" path, where in modern reality a "save storage, CPU cycles are cheap" would be more effective. CPU speeds have grown way way faster than main memory bandwidth (or even size?) over the last decades. For a datacenter, supercomputer, embedded system, PC or some end-user's phone, the metrics will be different. But same principle applies.
frollogaston 1y ago
Also, the O(1) random memory access assumption makes it easy to take memory for granted. Really it's something like O(n^(1/3)) when you're scaling the computer to the size of the problem, and you can see this in practice in datacenters. I forget the name of the O(1) access model. Not UMA, something else.
- cperciva 1y ago
  O(n^(1/2)) really, since data centers are 2 dimensional, not 3 dimensional. (Quite aside from the practical "we build on the surface of the earth" consideration, heat dissipation considerations limit you to a 2 dimensional circuit in 3-space.)
  - frollogaston 1y ago
    If you have rows of racks of machines, isn't that 3 dimensions? A machine can be on top of, behind, or next to another that it's directly connected to. And the components inside have their own non-uniform memory access. Or if you're saying heat dissipation scales with surface area and is 2D, I don't know. Would think that water cooling makes it more about volume, but I'm not an expert on that.
    - jvanderbot 1y ago
      Spatial position has nothing (ok only a little) to do with topology of connections.
    - manwe150 1y ago
      That example would be two dimensions still in the limit computation, since you can keep building outwards (add buildings) but not scale upwards (add floors)
      - frollogaston 1y ago
        You can add floors though. Some datacenters are 8 stories with cross-floor network fabrics.
undefuser 1y ago
I think it really depends on the task at hand, and not that intuitive. At some point accessing the memory might be slower than repeating the computation, especially when the storage is slow.
IsTom 1y ago
One one hand yes, on the other there might be some problems that are inherently difficult to parallelize (alternating machine PTIME is the same as deterministic PSPACE) where space doesn't buy you much. The jump from paper from t/log t to sqrt(t log t) is huge, but it still might be that unbounded parallelism doesn't buy you much more.

whatever1 1y ago

Lookup tables with precalculated things for the win! In fact I don’t think we would need processors anymore if we were centrally storing all of the operations ever done in our processors. Now fast retrieval is another problem for another thread.

chowells 1y ago
Oh, that's not a problem. Just cache the retrieval lookups too.
- michaelhoney 1y ago
  it's pointers all the way down
  - drob518 1y ago
    Just add one more level of indirection, I always say.
    - EGreg 1y ago
      But seriously… the solution is often to cache / shard to a halfway point — the LLM model weights for instance — and then store that to give you a nice approximation of the real problem space! That’s basically what many AI algorithms do, including MCTS and LLMs etc.
crmd 1y ago
Reminds me of when I started working on storage systems as a young man and once suggested pre-computing every 4KB block once and just using pointers to the correct block as data is written, until someone pointed out that the number of unique 4KB blocks (2^32768) far exceeds the number of atoms in the universe.
- ww520 1y ago
  The idea is not too far off. You could compute a hash on an existing data block. Store the hash and data block mapping. Now you can use the hash in anywhere that data block resides, i.e. any duplicate data blocks can use the same hash. That's how storage deduplication works in the nutshell.

lukeyoo 1y ago

[dead]

IvanK_net 1y ago

I am confused. If a single-tape turing machine receives a digit N in binary, and is supposed to write N ones on the tape, on the right side of the digit N, it performs N steps. If you expect N ones at the output, how can this machine be simulated in the space smaller than N? This machine must decrement the digit N at the beginning of the tape, and move to the end of the tape to write "1", so it runs in time O(N^2), not O(N)? (as it takes N "trips" to the end of the tape, and each "trip" takes 1, 2, 3 .. N steps) Since turing machines can not jump to any place on a tape in constant time (like computers can), does it have any impact on real computers?

cperciva 1y ago
Multitape Turing machines are far more powerful (in terms of how fast they can run, not computability) than single-tape machines. But to answer your question: "space" here refers to working space, excluding the input and output.
- IvanK_net 1y ago
  A single tape machine is still a multi tape machine, only with one tape.
iNic 1y ago
This paper looks exclusively at decision problems, i.e. problems where the output is a single bit. EDIT: This makes sense because if you look at all problems with N outputs then that is just the same as "gluing together" N different decision problems (+ some epsilon of overhead)
- IvanK_net 1y ago
  Oh okay, that was my second guess.

willmarquis 1y ago

This is just a reminder that memory isn’t just a constraint, it’s a resource.

michaelsbradley 1y ago
What resources are available (or not) and in what quantities are the most basic constraints for solving a problem/s with a computer.

felineflock 1y ago

Ryan Williams lecture (how he started): https://www.youtube.com/live/0DrFB2Cp7tg https://www.youtube.com/live/0DrFB2Cp7tg And paper: https://people.csail.mit.edu/rrw/time-vs-space.pdf https://people.csail.mit.edu/rrw/time-vs-space.pdf

vishnugupta 1y ago

Rainbow Tables FTW!

golly_ned 1y ago

“ If the problem is solved next week, Williams will be kicking himself. Before he wrote the paper, he spent months trying and failing to extend his result” What a strange, sad way to think about this. Academia is perverse.

catoc 1y ago
Nothing necessarily perverse here. I don’t know Williams but don’t image him disliking the other guy or being unhappy with progress in general, but just being someone who truly challenged himself only to find him being trumped a week later; and kicking himself for that. Either way, the week is not yet over, at least since the quanta article, so maybe no kicking will ensue.
- golly_ned 1y ago
  I wonder how the researchers who supplied the tree evaluation algorithm felt when Williams supplied his proof. Dismay at having not kept their result under wraps for longer so they could claim credit for such an advance themselves? I hope not.

ziofill 1y ago

At the cost of sounding ridiculous: can there be a notion of "speed of light" in the theory of computation, determining the ultimate limit of memory (space) vs runtime?

awanderingmind 1y ago
You mean something like this https://en.wikipedia.org/wiki/Bremermann%27s_limit https://en.wikipedia.org/wiki/Bremermann%27s_limit or this https://en.wikipedia.org/wiki/Quantum_speed_limit https://en.wikipedia.org/wiki/Quantum_speed_limit?
- ziofill 1y ago
  Oh wait, I just realized what I said was probably very stupid: I was thinking of some computational complexity theorem that links memory and runtime complexity classes in the same way that the "speed of light" sets an ultimate bound on the relation between actual space and actual time. But the speed of light is the maximum space in the smallest time, which computationally would correspond to filling the largest amount of memory in the shortest time :facepalm: (and thanks for the links!)

xlii 1y ago

From the „Camel Book”, one of my favorite programming books (not because it was enlightening, but because it was entertaining); on the Perl philosophy: “If you’re running out of memory, you can buy more. But if you’re running out of time, you’re screwed.”

immibis 1y ago
This can work both ways. If the program needs more memory than the computer has, it can't run until you buy more. But if it takes twice as long, at least it runs at all.
- Dban1 1y ago
  brain brain brain
- hinkley 1y ago
  The Camel book was written when Moore’s Law was trucking along. These days you can’t buy much more time but you used to be able to just fine. Now it’s horizontal scaling. Which is still more time.
- const_cast 1y ago
  Modern computers have so much memory it feels like it doesn't matter. Spending that memory on arrays for algorithms or things like a Garbage Collector just make sense. And, extra memory is worthless. You WANT the summation of all your programs to use all your memory. The processor, on the other hand, can context switch and do everything in it's power to make sure it stays busy. The CPU is like an engine and memory is your gas tank. Idling the engine is bad, but leaving gas in the tank doesn't hurt, but it doesn't help either. I'm not gonna get to my destination faster because I have a full tank.
  - Mawr 1y ago
    Only if running one such memory-hungry program at a time, which usually cannot be afforded. Multi-program workloads are much more common and the strategy of using as much ram as possible can't work in that environment.

woolion 1y ago

It's unfortunate that Quanta links are so popular, when they include so much pseudo-poetic fluff around the mathematics. Below there's an entire thread to dismiss a misconception introduced by the quanta article. "I think it is very intuitive that more space beats the pants off of more time." (poster is absolutely right) The The article say "Until now, the only known algorithms for accomplishing certain tasks required an amount of space roughly proportional to their runtime, and researchers had long assumed there’s no way to do better.", which is interpreted as that there's a proportional relation between time and space. However, a quick look at the complexity hierarchy would never suggest such a thing. Reading more carefully, it says "known algorithms" for "certain tasks", but then where do you get a general intuition from such a particular statement?

godelski 1y ago
I think they used to be better but really have made a blatant turn. I really thought that wormhole fiasco would have killed them. To go 4 whole months before putting the editor's note is beyond egregious[0]. Mistakes happen, but 4 months kills all credibility. You have to act fast on those things! There were big names raising serious concerns on day 1 and it really shows they didn't do due diligence to get outside verification before running a piece that they knew would be really popular. All this accomplishes is discrediting science. Trading personal gains for eroding the very thing that they make their money off of. This is a major part of why Americans (and people) have such high distrust for science. News outlets, and in particular science focused news outlets, constantly spew inaccurate information. It really should be no wonder that so many people are confused about so many scientific topics, as unless they actually take the years it takes to become an expert in a field, they are going to have a difficult time distinguishing fact from fiction. And why shouldn't the average person expect to trust a source like Quanta? They're "full of experts", right? smh [0] This is the earliest archive I see with the note. Press back one day and it should not be there. Article was published on Nov 30 2022, along with a youtube video https://web.archive.org/web/20230329191417/https://www.quantamagazine.org/physicists-create-a-wormhole-using-a-quantum-computer-20221130/ https://web.archive.org/web/20230329191417/https://www.quant...
bbrubaker 1y ago
I'm the author of this article. If you ask a complexity theorist, they will tell you that they did in fact have a general intuition that certain problems require space close to to linear in time to solve (see e.g., Ryan's comment #22 on Scott Aaronson's blog post about the result: https://scottaaronson.blog/?p=8680 https://scottaaronson.blog/?p=8680, and the comments after that). The most intuitive way to see this is in a circuit/DAG picture, where the goal is to get from the input nodes of the graph to the output nodes. Some graphs are very "wide": cut the graph at some intermediate point, and there will be a lot of edges crossing the cut, each of which represents some information from an earlier stage in the computation that you'll need to remember to get to the output. Ryan's first result is a general-purpose method for doing any computation, even ones whose graph structure looks like this, in asymptotically far less space. That is precisely what makes the result so surprising! My article was quite explicit in multiple places that the universal/comprehensive character of the result was that counterintuitive part: - In the first paragraph: "memory was more powerful than computer scientists believed: A small amount would be as helpful as a lot of time in all conceivable computations." - Further down in the introduction, in the passage you quoted: "Until now, the only known algorithms for accomplishing certain tasks required an amount of space roughly proportional to their runtime, and researchers had long assumed there’s no way to do better. Williams’ proof established a mathematical procedure for transforming any algorithm — no matter what it does — into a form that uses much less space. - In the third section, I explicitly state that researchers do believe space is more powerful than time in the specific sense that you're criticizing my article for misrepresenting: "But complexity theorists suspect that PSPACE is a much larger class, containing many problems that aren’t in P. In other words, they believe that space is a far more powerful computational resource than time. This belief stems from the fact that algorithms can use the same small chunk of memory over and over, while time isn’t as forgiving — once it passes, you can’t get it back." - In the fourth section, I explain why researchers didn't think the HPV75 result could be improved further, despite their intuition that space is more powerful than time in the above sense: "While many problems can be solved with much less space than time, some intuitively seemed like they’d need nearly as much space as time." TCS (and complexity theory specifically) are complicated subjects. I spend a lot of time interviewing researchers and thinking about how to distill the results of my reporting into a form that is accessible to readers with widely varying levels of familiarity with the subject matter. You are of course well within your rights to critique my stylistic choices, the narrative aspects of the story, and the order in which I presented information, but I will push back against the claim that my article is spreading misinformation about complexity theory. You're referring to a misconception that arises, by your own admission, when you don't read carefully. If it's the headline you object to, you could lodge a similar complaint against the complexity theorist Lance Fortnow: https://blog.computationalcomplexity.org/2025/02/you-need-much-less-memory-than-time.html https://blog.computationalcomplexity.org/2025/02/you-need-mu....

ChrisMarshallNY 1y ago

Interesting. It's one of those things that I’ve always “just assumed,” without thinking about it. I did a lot of raster graphics programming, in my career, and graphics work makes heavy use of lookup tables. Yesterday, I posted a rather simple tool I wrote[0]: a server that “frontloads” a set of polygons into a database, and then uses them, at query time. It’s fairly fast (but I’m sure it could be a lot faster). I wrote it in a few hours, and got pretty good performance, right out of the starting gate. Pretty basic stuff. I doubt the pattern is unique, but it’s one that I’ve used for ages. It’s where I try to do as much “upfront” work as possible, and store “half-baked” results into memory. Like I said, I always “just worked that way,” and never really thought about it. There’s been a lot of “rule of thumb” stuff in my work. Didn’t have an MIT professor to teach it, but it’s sort of gratifying to see that it wasn’t just “old wives” stuff. There’s probably a ton of stuff that we do, every day, that we don’t think about. Some of it almost certainly originated from really smart folks like him, finding the best way (like the “winding number” algorithm, in that server[1]), and some of it also probably comes from “grug-brained programmers,” simply doing what makes sense. [0] https://news.ycombinator.com/item?id=44046227 https://news.ycombinator.com/item?id=44046227 [1] https://github.com/LittleGreenViper/LGV_TZ_Lookup/blob/e247f2f97fb83e5392d31f81c321814baa26dc1b/src/Sources/LGV_TZ_Lookup_Query.class.php#L101 https://github.com/LittleGreenViper/LGV_TZ_Lookup/blob/e247f...

auto 1y ago
I'm in the depths of optimization on a game right now, and it's interesting how the gains I'm making currently all seem to be a matter of scaling the concept of lookup tables, and using the right tool for the job. What I mean is that traditionally I think peoples' ideas of lookup tables are things like statically baked arrays setup at compile time, or even first thing at runtime, and they never change. But if you loosen your adherence to that last idea a bit, where a lookup table can change slightly over time, you can get a ton of mileage out of a comparatively small amount of memory compared to wasting cycles every frame. As for the right tool for the job, I've read tons of dev logs and research papers over the years about moving work to the GPU, but this last few months stint of ripping my game inside out has really made me see the light. It's not just lookup tables built at compile or early runtime, but lookup tables modified slightly over time, and sent to the GPU as textures and used there. Follow this train of thought long enough, and now we're just calling memory writes and reads "lookup tables" when they aren't really that anymore, but whos to say where the barrier really lies?

alkyon 1y ago

It's kind of insulting to the reader that they explain P complexity class without using the word polynomial ("all problems that can be solved in a reasonable amount of time")

simpaticoder 1y ago
Be generous - it saves a lot of time. Once you say "polynomial" readers will think, "like, ANY polynomial, even like n^100?!" and you'll have to explain, yes, but that's STILL better than exponential, etc. They avoided all of that
- godelski 1y ago
  Quanta targets people who are above average. So I don't think it is too much for them to give a sentence or two stating that. Or even a little graphic could do wonders. I don't think it would take much time or effort to make a graphic like the one on wikipedia[0] and just throw in some equations within the ring. You can easily simplify too, by removing NL and merging EXP. Hell, look at the graphics here[1]. That's much more work. I don't think Quanta should be afraid of showing math to people. That's really their whole purpose. Even if I think they've made some egregious mistakes that make them untrustable...[2] [0] https://en.wikipedia.org/wiki/PSPACE#/media/File:Complexity_subsets_pspace.svg https://en.wikipedia.org/wiki/PSPACE#/media/File:Complexity_... [1] https://www.quantamagazine.org/june-huh-high-school-dropout-wins-the-fields-medal-20220705/ https://www.quantamagazine.org/june-huh-high-school-dropout-... [2] https://news.ycombinator.com/item?id=44067043 https://news.ycombinator.com/item?id=44067043
  - simpaticoder 1y ago
    I suppose my point is that the readers who will wonder about this are a) very likely to know about complexity classes already, or b)capable of learning about it themselves. Perhaps a simple link to something like https://complexityzoo.net/Petting_Zoo https://complexityzoo.net/Petting_Zoo would have been a nice middle-ground. Edit: Aaronson even mentions the n^100 problem in the section about P!

nottorp 1y ago

You can determine both the time and space complexity for any algorithm, should you need to. And in some cases you can adjust to have more of one or the other, depending on your particular constraints. More at eleven.

amelius 1y ago

> Williams’ proof established a mathematical procedure for transforming any algorithm — no matter what it does — into a form that uses much less space. Ok, but space is cheap, and we usually want to trade processing time for space. I.e., the opposite.

JohnKemeny 1y ago
Ryan made a dent (a tiny dent) in one of the most important open problems in mathematics. He's not trying to please programmers.
- amelius 1y ago
  But what makes an open problem important? ;)

senfiaj 1y ago

I always thought that the "reverse" relationship between the time and the space requirements is explained by the simple fact that when you have a set of algorithms with certain time and space requirements and you constrain one of these parameters, the other parameter will not be necessarily (in practice oftentimes not) the most optimal. But it doesn't necessarily mean that the faster algorithm will require less memory or vice versa. This "reverse" relationship works with other parameters. Such as in sorting algorithms, where besides time and space requirements you have stability requirement. If you constrain the sorting algorithms by stability, you won't have any advantage (will likely to have some disadvantage) in performance over non constrained algorithms. And wee see that the known stable sorting algorithms are either slower or require more memory. Nobody has yet invented a stable sorting algorithm that has comparable performance (both time and space) to non stable sorting algorithms. That's it.

schmorptron 1y ago

As an aside, I really enjoy a lot of the quanta magazine articles! They manage to write articles that both appeal to compsci people as well as interested "outsiders". The birds-eye view and informal how and why they pack into their explanation style often gives me new perspectives and things to think about!

bgnn 1y ago

The poetic style of Quanta makes it impossible to understand what does this mean. Can someone familiar with the topic explain is this applicable to real world computers or just a theoretical scenario? Does this mean we need more memory in computers even if they need to run at a lower clock speed?