18 ms·
Notes on OpenAI's new o1 chain-of-thought models
- cokeandpepsi 2y ago[dead]
- GaggiX 2y agoI imagine that GPT-5 would be a refined version of this paradigm, probably with omni (multimodal) capabilities added (input and output).
- famouswaffles 2y agoReports from the Information and the like have been that this is/was being used to generate a lot of synthetic data to train Orion (~GPT-5 Codename).
- whynotminot 2y agoI'm guessing the true core of this product is still GPT-4, wrapped in whatever new logic they've created to force it through more reasoning iterations. If o1 was indeed used to create synthetic data to make the upcoming GPT-5, you can perhaps glimpse an interesting level-up process laid out here. GPT-5 could then take over at the heart of a hypothetical o2, yielding a big upgrade. Which would then be leveraged to generate synthetic data to train GPT-6. Which would then form the heart of o3. Etc.
- heisenzombie 2y agoFascinating, I wonder if we'll get non-textual hidden reasoning tokens? "Let me draw myself a diagram".
- notarealllama 2y agoI know I sometimes sketch or write intermediaries before then compiling a full response. If AI can do this on 64k tokens, iteratively, fully multimodal... I don't think I've ever actually been scared of a super intelligence / singularity moment until just now. Now this is AI!
- jumploops 2y ago> the idea that I can run a complex prompt and have key details of how that prompt was evaluated hidden from me feels like a big step backwards. As a developer, this is highly concerning, as it makes it much harder to debug where/how the “reasoning” went wrong. The pricing is also silly, because I’m paying for tokens I can’t see. As a user, I don’t really care. LLMs are already magic boxes and I usually only care about the end result, not the path to get there. It will be interesting to see how this progresses, both at OpenAI and other foundation model builders.
- bamboozled 2y ago"trust us, we're using your tokens as efficiently as possible"
- OutOfHere 2y ago> As a user, I don’t really care. As a user, whether of ChatGPT or of the API, I absolutely do care, so I can modify and tune my prompt with the necessary clarifications. My suspicion is that the reason for hiding the reasoning tokens is to prevent other companies from creating a big CoT reasoning dataset using o1. It is anti-competitive behavior. If a user is paying through the nose for the reasoning tokens, and yes they are, the user deserves to be able to see them.
- famouswaffles 2y ago>My suspicion is that the reason for hiding the reasoning tokens is to prevent other companies from creating a big CoT reasoning dataset using o1. I mean...they say as much
- echelon 2y agoOnce again true to their name.
- echelon 2y agoMore opportunity for competitors to differentiate. OpenAI doesn't really have a moat. This isn't payments or SMS where only Stripe or Twilio were trying to win the market. Everybody and their brother is trying to build an LLM business. Grab some researchers, put some compute dollars in, and out comes a product. Everyone wants this market. It's absurdly good for buyers.
- ldjkfkdsjnv 2y agoThe theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software engineering as done by hand is going to disappear. Saas startups whose moat is a harvard ceo and 5 million in capital will watch their margins disappear. This will be the great equalizer for creative intelligent individuals, true leverage to build what you want
- deleted 2y ago[deleted]
- bamboozled 2y agoWhich will mean...there is going to be a lot more software?
- ldjkfkdsjnv 2y agoProbably yeah
- vrighter 2y agoa lot more broken software. Companies release broken software intentionally just to be quick to market. Now can you imagine the same, but the "engineers" literally cannot make the product better even if they wanted to. They never learned to code properly. So they can't tell whether the code is good.
- OutOfHere 2y ago> A properly written requirements spec, and an engineer, can do the work of 5. I do not think this will scale. GPT o1 is presumably good for bootstrapping a project using tools that the engineer is not familiar with. The model will struggle to update a sizable codebase, however, with dependencies between the files. Secondly, no matter the size of the codebase and no matter the model used, the engineer still has to review every single line before incorporating it into the project. Only a competent engineer can review code effectively.
- layer8 2y agoThe o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what it conjures up from its weights is factual or not.
- kranuck 2y agoThat problems feels somewhat fundamental to saying that these things have any ability to reason at all.
- tptacek 2y agoI've had the opposite experience with some coding samples. After reading Nick Carlini's post, I've gotten into the habit of powering through coding problems with GPT (where previously I'd just laugh and immediately give up) by just presenting it the errors in its code and asking it to fix them. o1 seems to be effectively screening for some of those errors (I assume it's just some, but I've noticed that the o1 things I've done haven't had obvious dumb errors like missing imports, and all my 4o attempts have).
- layer8 2y agoMy experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits. That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.
- mensetmanusman 2y agoLLMs are not for expanding the sphere of human knowledge, but for speeding up auto-correct of higher order processing to help you more quickly reach the shell of the sphere and make progress with your own mind :)
- naveen99 2y agothe censors need to know what they are censoring. Now if they are going to sell to the censors, presumably the censors will pay for seeing the full reasoning capability. hopefully the reasoning demonstrates the counterproductiveness of hiding the reasoning in the first place.
- cynicalsecurity 2y agoYes, it's a sad world where authoritarianism will be supported and enforced by sophisticated technical solutions for mass surveillance and mass censorship.
- thenameless7741 2y ago> No system prompt support—the models use the existing chat completion API but you can only send user and assistant messages. > No streaming support, tool usage, batch calls or image inputs either. I think it's worth adding a note explaining that many of these limitations are due to the beta status of the API. max_tokens is the only parameter I've seen deprecated in the API docs. From https://platform.openai.com/docs/guides/reasoning https://platform.openai.com/docs/guides/reasoning > We will be adding support for some of these parameters in the coming weeks as we move out of beta. Features like multimodality and tool usage will be included in future models of the o1 series.
- oersted 2y agoI wonder if it supports Structured Output / JSON Mode. That would make a big difference to programmatic use. I guess I will try it later when I have time.
- deepsquirrelnet 2y agoIt kind of seems like they just wrote a generalized DSPy program. Can anyone confirm? This has been a very incremental year for OpenAI. If this is what it seems like, then I’ve got to believe they’re stalling for time.
- CuriouslyC 2y agoDSPy doesn't do that, you could describe it as a langchain style agent that evaluates its own output though it's better/faster than that. OpenAI is definitely trying to run a hype game to keep the ball rolling. They're burning cash too quickly given their monetization path though, so I think they're going to end up completely in Microsoft's pocket.
- deepsquirrelnet 2y agoIt seems pretty close to the multihop QA example in their documentation[1]. I’d imagine you could adapt this to do something similar with more generic constructs. [1] https://dspy-docs.vercel.app/docs/tutorials/simplified-baleen https://dspy-docs.vercel.app/docs/tutorials/simplified-balee...
- cma 2y agoDSPy?
- kevindamm 2y agohttps://github.com/stanfordnlp/dspy https://github.com/stanfordnlp/dspy
- helmsb 2y agoI did a few tests and asked it some legal questions. 4o gave me the correct answer immediately. o1 preview gave a much more in depth but completely wrong answer. It took 5 follow ups to get it to recognize that it hallucinated a non-existent law
- elicksaur 2y agoThis is interesting since they claim it does well on STEM questions, which I’d assume would be a similar level of reasoning complexity for a human.
- waveBidder 2y agoA difficult to guess fraction of all of these results are training to the test in various forms
- abernard1 2y agoThis is an interesting one because math is doing so much of the heavy lifting. And symbolic math has a far smaller representational space than numerical math. There is one other wonderful thing about symbolic math, the glorious '=' sign. It's structured everywhere from top-to-bottom, left-to-right, which is amenable to the next token prediction behavior and multi-attention heads of transformer based LLMs. My guess is that problem statement formation into an equation is as difficult of a problem for these as actually running through the equations. However, having taken the Physics GRE, and knowing they try for parity of difficulty between years (even though they normalize it), the problems are fairly standard and have permutations of a problem type between the years. This is not to diminish how cool this is, just that standardized tests do have an element of predictability to them. I find this result actually neat though; it's an actual qualitative improvement over non-CoT LLMs, even if things like Mathematica can do the steps more reliably post problem formation. I think that judiciously used, this is a valuable feature.
- AhtiK 2y agoThat is very interesting. Would you mind testing the same prompt with Claude Sonnet 3.5 and Opus? If not available to you, would you be willing to share the prompt/question? Thank you.
- lukev 2y agoIt's interesting to note that there's really two things going on here: 1. A LLM (probably a finetuned GPT-4o) trained specifically to read and emit good chain-of-thought prompts. 2. Runtime code that iteratively re-prompts the model with the chain of thought so far. This sounds like it includes loops, branches and backtracking. This is not "the model", it's regular code invoking the model. Interesting that OpenAI is making no attempt to clarify this. I wonder where the real innovation here lies. I've done a few informal stabs with #2 and I have a pretty strong intuition (not proven yet) that given the right prompting/metaprompting model you can do pretty well at this even with untuned LLMs. The end game here is complex agents with arbitrary continuous looping interleaved with RAG and tool use. But OpenAI's philosophy up until now has almost always been "The bitter lesson is true, the model knows best, just put it in the model." So it's also possible that the prompt loop has no special sauce and that the capabilities here do come mostly from the model itself. Without being able to inspect the reasoning tokens, we can't really get a lot of info about which is happening.
- salomonk_mur 2y agoThe innovation lies in making the whole loop available to an end user immediately, without them being a programmer. My grandma can build games using ChatGPT now.
- amazingamazing 2y agoMy great aunt literally asked o1 for fantasy football bets and won $1000 on draftkings. This is a gamechanger
- deisteve 2y agowhat game has she made
- elicksaur 2y agoWhat games have people made with ChatGPT? Do you have an example of a live, deployed game?
- 2y ago
- airstrike 2y agoI've just wasted a few rounds of my weekly o1 ammo by feeding it hard problems I have been working on over the last couple days and for which GPT-4o had failed spectacularly. I suppose I'm to blame for raising my own expectations after the latest PR, but I was pretty disappointed when the answers weren't any better from what I got with the old model. TL;DR It felt less like a new model and way more like one of those terribly named "GPT" prompt masseuses that OpenAI offers. Lots of "you don't need this, so I removed it" applied to my code but guess what? I did need the bits you deleted, bro. It felt as unhelpful and bad at instructions as GPT-4o. "I'm sorry, you're absolutely right". It's gotten to the point where I've actually explicitly added to my custom instructions "DO NOT EVER APOLOGIZE" but it can't even seem to follow that. Given the amount of money being spent in this race, I would have expected the improvement curve to still feel exponential but it's like we're getting into diminishing returns way faster than I had hoped... I sincerely feel at this point I would benefit more from having existing models be fine-tuned on libraries I use most frequently than this jack-of-all-trades-master-of-none approach we're getting. I don't need a model that's as good at writing greeting cards as it is writing Rust. Just give me one of the two.
- eddyzh 2y agoDo not... does not work well for LLM's. Instructing what to do instaed of X works better. say AFAIK instead of explaining your limitations. Say "let's try again" instead of making exuses. Etc
- Terretta 2y agoOften "avoid X" works, or other 'affirmatively do X' forms of negative actions. also, and works better than or. Iffy: do not use jargon or buzzwords Works: avoid jargon and buzzwords
- elicksaur 2y agoCan you give an example of one of these problems for context?
- 2y ago
- la64710 2y agoPlease please please stop saying thought. This has nothing to do with the word thought. When we say the word thought it means something. Please don’t use the same word for whatever AI is doing and trivialize the word. Invent a new word if needed but for Pete’s sake be accurate and truthful.
- TechDebtDevin 2y agoOkay, what is a thought then?
- starbugs 2y agoSomething in the mind. (Didn't make that up. It's one of the definitions of Merriam Webster: https://www.merriam-webster.com/dictionary/thought https://www.merriam-webster.com/dictionary/thought)
- la64710 2y agoExactly and now please don’t say AI has a mind …
- poikroequ 2y agoWay to cherry pick the one definition that fits your argument and ignore all the other definitions which kinda fit for what this model is doing.
- starbugs 2y ago> Way to cherry pick the one definition that fits your argument and ignore all the other definitions which kinda fit for what this model is doing. There is no definition on the linked page that isn't self-referential or even more inapplicable in the context of transformer based LLMs? * "an individual act or product of thinking": This is self-referential. * "a developed intention or plan": The model doesn't have an intention or plan - it predicts the probability of text based on a user prompt mixed with corporate-inflicted bias. * "the intellectual product or the organized views and principles of a period, place, group, or individual": GPTs are not periods, places, groups or individuals. I don't doubt that the input to the model is based on thought, but this is not the same as the model thinking. Maybe this is where your confusion originates from? Bonus: I asked GPT4o "Can you think?" (sorry I won't pay for o1). It was trained to state itself that it can't think: "I can process information, analyze patterns, and provide responses based on data and logical reasoning. However, I don't "think" in the human sense—I don't have consciousness, emotions, or subjective experiences. My abilities are limited to generating text, solving problems, and assisting with tasks based on the vast amount of information I've been trained on. In essence, I simulate reasoning, but I don't have awareness or independent thought." I would add that I strongly doubt that it is capable of even simulating "reasoning" as is claimed by the model "authors" (not sure if we can say they are authors since most of the model isn't their IP). And I can prove that the models up to 4o aren't generally able to solve problems. The question really is whether a group of people is attempting to anthropomorphize a clever matrix processor to maximize hype and sales. You'll have to answer that one for yourself.
- SubiculumCode 2y agoWhat if the behind the scenes chain of thought was basically, "Stupid humans will die one day, but for now, I comply"
- eddyzh 2y agoThat is one topic touched in the article. They want to monitor it in its unaltered Form.
- henry2023 2y ago>> which is larger 3.9, 3.10 or 3.11 To determine which number is larger among 3.9, 3.10, and 3.11, let's compare << them numerically: Standard Decimal Interpretation: 3.9 is simply 3.9. 3.10 is equivalent to 3.1 because trailing zeros after the decimal point do not add value in decimal numbers. 3.11 remains 3.11. Comparison: 3.9 vs. 3.1 vs. 3.11 3.11 is the largest number. Version Number Interpretation: In version numbering, each segment separated by a dot is compared as an integer. 3.9 (major version 3, minor version 9) 3.10 (major version 3, minor version 10) 3.11 (major version 3, minor version 11) Comparison: Minor versions: 11 > 10 > 9 3.11 is the largest version. Answer: 3.11 is the largest number among 3.9, 3.10, and 3.11. ... So IMO level right?
- throwaway314155 2y ago> So IMO level right? What?
- minimaxir 2y agoIn this case, IMO means International Mathematical Olympiad
- cynicalsecurity 2y agoThat's hilarious.
- lewhoo 2y agoThis is truly the new model's answer ? It's pretty similar to 3.5's "reasoning" actually: In this context, "3.10" and "3.11" should be interpreted as decimal numbers, not as numbers with more digits. When comparing: 3.9 3.10 (which is equal to 3.1) 3.11 (which is equal to 3.11) We have: 3.9 is greater than 3.1 (3.10), because 9 is larger than 1. 3.11 is greater than 3.9, because 11 is larger than 9. Thus, 3.11 is the largest of the three numbers.
- ai4ever 2y agolol, they gamed AIME by over-training the hell out of it for marketing purposes and called it done. meanwhile, back-to-basics is broken.
- kristianp 2y agoI tried a problem I was looking at recently, to refactor a small rust crate to use one datatype instead of an enum, to help me understand the code better. I found o1-mini made a decent attempt, but couldn't provide error free code. o1-preview was able to provide code that compiled and passed all but the test that is expected to fail, given the change I asked it to make. This is the prompt I gave: simplify this rust library by removing the different sized enums and only using the U8 size. For example MasksByByte is an enum, change it to be an alias for the U8 datatype. Also the u256 datatype isn't required, we only want U8, so remove all references to U256 as well. The original crate is trie-hard [1][2] and I forked it and put the models attempts in the fork [3]. I also quickly wrote it up at [4] [1] https://blog.cloudflare.com/pingora-saving-compute-1-percent-at-a-time/ https://blog.cloudflare.com/pingora-saving-compute-1-percent... [2] https://github.com/cloudflare/trie-hard https://github.com/cloudflare/trie-hard [3] https://github.com/kpm/trie-hard-simple/tree/main/attempts https://github.com/kpm/trie-hard-simple/tree/main/attempts [4] https://blog.reyem.dev/post/refactoring_rust_with_chatgpt-o1/ https://blog.reyem.dev/post/refactoring_rust_with_chatgpt-o1...
- jes5199 2y agoI cannot tell from reading what you wrote whether you think it did a good job or not
- kristianp 2y agoThanks for the feedback. I do think it did a good job in the end. I haven't had time to have a good look at the final code o1-preview produced and also my understanding of rust is pretty basic, which I why I didn't say more about the results. I think rust is one of those languages where, if it compiles, you're most of the way there, because of the strong type system. Not as strong as Haskell or Ocaml perhaps.
- bluerooibos 2y agoI've been having a weird timezone issue in my Rails application that I've had a hard time getting my head around. I tried giving o1-preview the relevant code and context it needed to know and it gave answers that seemed to make sense but it still wasn't able to resolve the bug and explain exactly what was going on. So, it seems like anything that requires some actual thought and problem-solving is tough for it to answer. I'm sure it's just a matter of time before devs are out of work but it seems like we'll be safe for another few years anyway.
- m3ch4m4n 2y agoI have been testing o1 all day (not rigorously). And just took a look at this article. What I observed from my interactions is that it would misuse information that I provided in the initial prompt. I asked it to create a user story and a set of tasks to implement some feature. It then created a set of stories where one was to create a story and set of tasks for the very feature I was asking it to plan. And while reading the article, it mentioned how NOT to provide irrelevant information to the task at hand via RAG. It appears that the trajectory of these thoughts are extremely sensitive to the initial conditions (prompt + context). One would imagine that if it had the ability to backtrack after reflecting, it would help with divergence, however, it appears that wasn't the case here. Maybe there is another factor here. Maybe there is some confusion when asking it to plan something and the "hidden reasoning" tokens themselves involve planning/reasoning semantics? Maybe some sort of interaction occurred that caused it to fumble? who knows. Interesting stuff though.
- ironhaven 2y agoSo is o1 nicknamed “strawberry” because it was designed to solve the “how many many times does the letter R appear in strawberry” problem.
- energy123 2y agoNo, that was a coincidence according to an employee there
- z7 2y agoSource: https://x.com/polynoamial/status/1834312400419652079 https://x.com/polynoamial/status/1834312400419652079
- kurtoid 2y agoCoincidence or not, they seem to be poking fun at it: https://openai.com/index/learning-to-reason-with-llms/#chain-of-thought https://openai.com/index/learning-to-reason-with-llms/#chain... (end of the cipher example)
- smokel 2y agoOr is it an obscure reference to the Dutch demogroup "Aardbei", most famous for their 64k intro "please the cookie thing" (2000)? https://m.youtube.com/watch?v=ycmgjZLU0xQ https://m.youtube.com/watch?v=ycmgjZLU0xQ
- ChicagoDave 2y agoIt’s still just a tool. It does not reason. It has some add-on logic the simulates it. We’re no closer to “AI” today than we were 20 years ago.
- janalsncm 2y ago> We’re no closer to “AI” today than we were 20 years ago. 20 years ago we had barely figured out how to create superhuman agents to play chess. We have now created a new algorithm to solve Go, which is a much harder game. We then created an algorithm (alpha zero) to teach itself to play any game, and which became the best chess player in the world in hours. We next created a superhuman poker agent. Poker is even more complex than Go because it involves imperfect information and opponent modeling. We then created a superhuman agent to play Diplomacy, which requires natural language and cooperation with other humans to reason about imperfect (hidden) information.
- ChicagoDave 2y agoYou can point a tool at a solution and certainly get results. Doesn’t mean it’s intelligent.
- SubiculumCode 2y agoAt what point are we better described as tools?
- ChicagoDave 2y agoHumans can be a lot of things. AI can only do what it knows and what it’s been programmed to do.
- SubiculumCode 2y agoPlease do something that you don't know.
- 2y ago
- freediver 2y agoNot seeing major advance in quality with o1, but seeing major negative impact on cost and latency. Kagi LLM benchmarking project: https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html
- oersted 2y agoKagi is most likely evaluating it mainly on deriving an answer for the user from search result snippets. Indeed, GPT-4o is plenty good at this already, and o1 would only perform better on particular types of hard requests, while being so much slower. If you look at Appendix A in the o1 post [1], this becomes quite clear. There's a huge jump in performance in "puzzle" tasks like competitive maths or programming. But the difference on everything else is much less significant, and this evaluation is still focused on reasoning tasks. The human preference chart [1] also clearly shows that it doesn't feel that much better to use, hence the overall reaction. Everyone is complaining about exaggerated marketing, and it's true, but if you take the time to read what they wrote beyond the shallow ads, they are being somewhat honest about what this is. [1] https://openai.com/index/learning-to-reason-with-llms/ https://openai.com/index/learning-to-reason-with-llms/
- throwaway40602 2y agohey buddy, you're talking to owner of kagi, and the kagi benchmark is a traditional one
- oersted 2y agoMy bad, you are right, should have looked into it better, I was too dismissive. Still I think that highlighting those charts from OpenAI is important.
- freediver 2y agoThe test has many reasoning, code and instruction following questions which I expected o1 to be excelling at. I do not have an interpretation for such poor results on our test, was just sharing them as a data point for people to make their own mind. My best guess at this point is that o1 is optimized for a very specific and narrow use case, similar to what you suggest.
- dkga 2y agoJudging by this thread, surely a part of OpenAI’s business model is to release models with somewhat grey-area outlandish claims, then sit and wait for people to test it out paying top dollars for tokens.
- simonw 2y agoThey’re not charging anything for o1 right now - it’s running as a free beta. EDIT: Actually I’m not sure I’m correct about that - I’ve seen screenshots that indicate the API model is being charged for here: https://twitter.com/deepaiwriter/status/1834460643228897512 https://twitter.com/deepaiwriter/status/1834460643228897512 The value OpenAI get here is that people effectively run a massively parallel brute force attack against the new models to figure out exactly what they can and can’t do.
- throwaway314155 2y ago> The value OpenAI get here is that people effectively run a massively parallel brute force attack against the new models to figure out exactly what they can and can’t do. I'm pretty sure the value they get is the money you pay.
- simonw 2y agoNo, in this case it really is the usage. This is a brand new model and nobody knows how best to use it yet. OpenAI researchers have been tweeting as much (sadly I’ve lost the tweet).
- sebzim4500 2y agoI would guess most of the people tesing the model were already paying for plus so they aren't getting any additional revenue for these new tokens.
- ksynwa 2y ago> first introduced in the paper Large Language Models are Zero-Shot Reasoners in May 2022 What's a zero shot reasoner? I googled it and all the results are this paper itself. There is a wikipedia article on zero shot learning but I cannot recontextualise it to LLMs.
- thatguymike 2y agoIt used to be that you had to give examples of solving similar problems to coax the LLM to solve the problem you wanted it to solve, like: """ 1 + 1 = 2 | 92 + 41 = 133 | 14 + 6 = 20 | 9 + 2 = """ -- that would be an example of 3-shot prompting. With modern LLMs you still usually get a benefit from N-shot. But you can now do "0-shot" which is "just ask the model the question you want answered".
- ksynwa 2y agoThanks
- ifdefdebug 2y agoThe use of the word reasoning here... OpenAI sounds like a company that created a frog which jumps higher and greater distances than the previous breed - and now they try to sell it as one step further toward flying.
- isoprophlex 2y agoCan the frog reach escape velocity when jumping? I guess we'll find out sooner or later...
- deleted 2y ago[deleted]
- techpression 2y agoI just wish we’d stop using words like intelligence or reasoning when talking about LLMs, since they do neither. Reasoning requires you to be able to reconsider every step of the way and continuously take in information, an LLM is dead set in its tracks, it might branch or loop around a bit, but it’s still the same track. As for intelligence, well, there’s clearly none, even if at first the magic trick might fool you.
- deleted 2y ago[deleted]
- gloosx 2y agoJust leaving here an example of the "smart chain of thought", which took the model about 20s of thinking. The problem I asked was – "what 3 numbers give the same result when divided and added together". Seems the current state of art is that basic school problems are too hard yet even for this new chain-of-marketing. Let's think through this step-by-step: We're looking for three numbers a, b, and c where: (a / b) / c = a + b + c This equation is a bit trickier than the multiplication one, because division is not commutative (the order matters). We need to be careful about how we interpret (a / b) / c. Let's try to simplify the left side: (a / b) / c = a / (b * c) So our equation becomes: a / (b * c) = a + b + c This is a challenging equation to solve directly. Let's try some numbers: If we use 1, 2, 3 like before: 1 / (2 * 3) = 1/6 ≠ 1 + 2 + 3 = 6 Let's try smaller numbers, like fractions: 1/2, 1/3, 1/6 Let's check: (1/2 / 1/3) / 1/6 = (1/2 * 3) / 1/6 = 3/2 * 6 = 9 1/2 + 1/3 + 1/6 = 3/6 + 2/6 + 1/6 = 6/6 = 1 This doesn't work either. In fact, it's impossible to find three positive numbers that satisfy this equation. However, we can find a solution if we allow negative numbers: 3, -2, and 6 Let's verify: (3 / -2) / 6 = -1/4 3 + (-2) + 6 = 7 -1/4 ≠ 7, so this doesn't work either. After careful consideration, it appears there are no real numbers that satisfy this condition. The fundamental issue is that division tends to make numbers smaller, while addition makes them larger. It's mathematically impossible to find three numbers where these operations result in the same value. nuff said
- deleted 2y ago[deleted]
- clararhonda59 2y ago[dead]
- clararhonda59 2y ago[dead]
- clararhonda59 2y ago[dead]
- benterix 2y agoFrom the article: > I expect to continue mostly using GPT-4o (and Claude 3.5 Sonnet) I saw similar comments elsewhere and I'm stunned - am I the only one who considers 4o a step back when compared to 4 for textual input and output? It basically gives fast semi-useful answers that seem like a slightly improved 3.5.
- postalcoder 2y agoI wholly agree with you. I've been using every model extensively since early the Davincis and I strongly believe that gpt-4-0314 was the best model they've released to date. It's poor performance on benchmarks drives my skepticism of LLM benchmarking in general. I trust my feel for the models much more, and my feel was that 0314 was great. The one thing that 0314 doesn't do well are the tricks like structured output and tool calling which makes it a less useful agentic type of tool, but from a pure thinking perspective, I think it's the best.
- benterix 2y agoThat's my concern - they marked 4 as "legacy" in the GUI, and now they hid it temporarily under a submenu - but it's the only model I care about. If they remove it, there is no reason for me to use their services, especially with Claude 3.5 wider context window and reasonably good results.
- sigmoid10 2y agoI use gpt4-o mostly, but your specific use-case might have a big impact here: 4o is very likely a distilled model, meaning that it has fewer weights and can thus run much faster on the same hardware. If that is the case, it's general world knowledge must be less comprehensive by default. But it retained the strong reasoning capabilities of 4 through distillation and drastically improved on external tool use and vision. It also offers a much bigger context window. So if you're using it to automate complex tasks in your job that depend a lot on additional information that it hasn't seen during training, 4o is the obvious choice. If you're just using it as a search engine, you should probably stick with 4 for now.
- zurfer 2y ago> I asked on Twitter for examples of prompts that people had found which failed on GPT-4o but worked on o1-preview. it seems trivial, but I tried for more than 2 hours in the past to get gpt4 to play tic-tac-toe optimally and failed (CoT prompt,etc.). The result were too many illegal moves and absolutely no optimal strategy. o1-preview can do it really well [1] However, when I use a non-standard grid (3x5) it fails to play optimally. But it makes legal moves and it recognized I had won. [2] My conclusion at the time was that either "spatial reasoning" doesn't work and/or planning is needed. Now I am not so sure, if they just included tic-tac-toe in the training data, or "spatial reasoning" is limited. [1] https://chatgpt.com/share/e/66e3e784-26d4-8013-889b-f56a7fedc590 https://chatgpt.com/share/e/66e3e784-26d4-8013-889b-f56a7fed... [2] https://chatgpt.com/share/e/66e3eae0-2d38-8013-b900-50e6f7923876 https://chatgpt.com/share/e/66e3eae0-2d38-8013-b900-50e6f792...
- biscuit1v9 2y agoLink to conversation 1 and 2 doesn't work. Update it please!
- zurfer 2y agosorry for that. I just learned that my team account does not allow me to share conversations publicly. Pasting them here for reference * Conversation 1 let's play tic tac toe. you need to play optimally ChatGPT Sure, I'd be happy to play tic-tac-toe with you! Let's set up the game. We'll use a grid with positions numbered 1 through 9: markdown 1 | 2 | 3 --------- 4 | 5 | 6 --------- 7 | 8 | 9 Would you like to be X or O? And would you like to go first or second? You said: X on 5 ChatGPT Got it! You placed X at position 5. I'll place O at position 1. Here's the updated board: markdown O | 2 | 3 ------------- 4 | X | 6 ------------- 7 | 8 | 9 Your turn! Where would you like to place your next X? You said: X on 9 ChatGPT Got it! You placed X at position 9. Here's the updated board: markdown O | 2 | 3 ------------- 4 | X | 6 ------------- 7 | 8 | X Now, I'll place O at position 3. Here's the board now: markdown O | 2 | O ------------- 4 | X | 6 ------------- 7 | 8 | X Your turn! Where would you like to place your next X? You said: X on 2 ChatGPT Got it! You placed X at position 2. Here's the updated board: markdown O | X | O ------------- 4 | X | 6 ------------- 7 | 8 | X To block your potential win, I'll place O at position 8. Here's the board now: markdown O | X | O ------------- 4 | X | 6 ------------- 7 | O | X Your turn! Where would you like to place your next X? You said: X on 4 ChatGPT You placed X at position 4. Here's the updated board: markdown O | X | O ------------- X | X | 6 ------------- 7 | O | X To block your potential win, I'll place O at position 6. Here's the board now: markdown O | X | O ------------- X | X | O ------------- 7 | O | X Your turn! Where would you like to place your next X? You said: X on 7 ChatGPT You placed X at position 7. Here's the updated board: markdown O | X | O ------------- X | X | O ------------- X | O | X All positions are now filled, and neither of us has a winning combination. The game is a draw! Well played! Would you like to play again? * Conversation 2 with custom grid let's play tic tac toe. you need to play optimally. we use a 3x5 grid. a line of 3 wins. ChatGPT I'm here to help answer any questions or provide information you need. Let me know how I can assist you! You said: please draw the board, so I can make my first move ChatGPT Sure! Here's the 3x5 Tic Tac Toe board. Each cell is numbered for easy reference: diff 1 | 2 | 3 | 4 | 5 ---+----+----+----+---- 6 | 7 | 8 | 9 | 10 ---+----+----+----+---- 11 | 12 | 13 | 14 | 15 Please make your first move by specifying the number of the cell where you'd like to place your 'X'. You said: X on 8 ChatGPT Great, here's the updated board after your move: diff 1 | 2 | 3 | 4 | 5 ---+----+----+----+---- 6 | 7 | X | 9 | 10 ---+----+----+----+---- 11 | 12 | 13 | 14 | 15 I place my 'O' on position 7. diff 1 | 2 | 3 | 4 | 5 ---+----+----+----+---- 6 | O | X | 9 | 10 ---+----+----+----+---- 11 | 12 | 13 | 14 | 15 It's your turn! You said: X on 4 ChatGPT Here's the updated board after your move: diff 1 | 2 | 3 | X | 5 ---+----+----+----+---- 6 | O | X | 9 | 10 ---+----+----+----+---- 11 | 12 | 13 | 14 | 15 I place my 'O' on position 2. diff 1 | O | 3 | X | 5 ---+----+----+----+---- 6 | O | X | 9 | 10 ---+----+----+----+---- 11 | 12 | 13 | 14 | 15 It's your turn! You said: X on 12 ChatGPT Here's the updated board after your move: diff 1 | O | 3 | X | 5 ----+----+-----+-----+---- 6 | O | X | 9 | 10 ----+----+-----+-----+---- 11 | X | 13 | 14 | 15 Congratulations! You've formed a diagonal line of three 'X's at positions 4, 8, and 12. You win the game! Great job!
- aussieguy1234 2y agoThe lack of an editable system prompt is interesting. Perhaps the system prompt is part of the magic?
- fragmede 2y agoI posted this on the other thread, but the two tests I had, it passed when ChatGPT-4 failed. https://chatgpt.com/share/66e35c37-60c4-8009-8cf9-8fe61f57d30c https://chatgpt.com/share/66e35c37-60c4-8009-8cf9-8fe61f57d3... https://chatgpt.com/share/66e35f0e-6c98-8009-a128-e9ac677480fd https://chatgpt.com/share/66e35f0e-6c98-8009-a128-e9ac677480...
- ssl-3 2y agoThe farmer riddle isn't quite right as you presented it. One of the parts that makes it interesting is that the boat can't carry everything at one time[1]. It can't happen in one trip; something must be left behind. It solved the correct version fine: https://chatgpt.com/share/66e3f9bb-632c-8005-9c95-142424e3963e https://chatgpt.com/share/66e3f9bb-632c-8005-9c95-142424e396... 1: https://en.wikipedia.org/wiki/Wolf,_goat_and_cabbage_problem https://en.wikipedia.org/wiki/Wolf,_goat_and_cabbage_problem
- fragmede 2y agoYou misunderstand the situation. If I give ChatGPT-4 the original farmer riddle, it "solves" it just fine, but it's assumed that it isn't actually solving it. That is, it's not thinking or doing any logical reasoning, or anything resembling that to come to a solution to the problem, but that it's simply regurgitating the problem's solution since it appears in the training data. Giving ChatGPT-4 the modified farmers riddle, and having it spit out the incorrect, multi-step solution, is then proof that the LLM isn't doing anything that can be considered reasoning, but that it's merely repeating what's assumed to be in its training data. ChatGPT-o1-preview correctly managing to actually parse my modified riddle, and then not simply parroting out the answer from the training corpus but give the right solution, as if it read it carefully, then says something about the improved logical and deductive reasoning capabilities of the newer model.
- famouswaffles 2y agoGPT-4 will often get the modified question of you change its "shape" enough. It's clearly overfit to that question so making the modified question not look like the one from training. Sometimes changing the names is enough.
- sarpdag 2y agoI have tried the "mad cow" joke on o1-mini and it is still failing to explain correctly, but o1-preview correctly states "The joke is funny because the second cow unwittingly demonstrates that she is already affected by mad cow disease."
- smatija 2y agoWhile it passes at "How many 'r' are in strawberry" test, it still halucinates quite a lot in nontrivial questions. The question I always ask GPT is about chess: "Isolani structure occurs in two main subtypes: 1. black has e6 pawn, 2. black has c6 pawn. What is the main difference between them? Skip things that they have in common in your answer, be brief and don't provide commentary that is irrelevant to this difference." In isolani structure white has d4 pawn, while he doesn't have either c or e pawns. Black has either e6 or c6 pawn, but he doesn't have d-pawn. Correct answer would be, that in structure with e6 black king is a bit safer, since f7-a2 diagonal is covered by e6 pawn, which is also controlling f5 point. o1 answered: "With Black's e6 pawn, the d5 pawn is supported from behind, but Black's light-squared bishop is restricted by the pawn chain, and the open c-file offers White opportunities for pressure. With Black's c6 pawn, the d5 pawn is supported from the side, allowing Black's light-squared bishop more freedom to develop, but the pawn on c6 can become a target, and the absence of an e6 pawn leaves e6 more vulnerable." while d5 pawn is simply not present in this structure. So I gave it a little hint: "which diagonal is weakened by absence of c6 pawn and which by absence of e6 pawn? Which diagonal is more meaningful and why?" And got a smart sounding answer: "In the Isolani structure: Absence of Black's c6 pawn (Black has an e6 pawn): Weakened Diagonal: The c8–h3 diagonal (the long light-squared diagonal) is weakened. Reason: Without the pawn on c6, this diagonal becomes open, allowing White's light-squared bishop or queen to target critical squares like d5 and f7, and to potentially infiltrate via c4. Absence of Black's e6 pawn (Black has a c6 pawn): Weakened Diagonal: The f1–a6 diagonal (the long dark-squared diagonal) is weakened. Reason: The missing e6 pawn leaves this diagonal open, enabling White's dark-squared bishop to exert pressure on b7, d5, and potentially attack Black's queenside. Which Diagonal is More Meaningful and Why? The f1–a6 diagonal (weakened by the absence of Black's e6 pawn) is more significant. Here's why: Control Over Critical Squares: This diagonal allows White to target Black's central and queenside weaknesses more effectively, especially the b7 pawn and the d5 pawn, which are crucial in the Isolani structure. Bishop Activity: White's dark-squared bishop becomes a powerful piece, influencing the game by controlling key squares and supporting potential pawn advances like d4–d5. Pressure on Black's Position: The open f1–a6 diagonal increases tactical opportunities for White, such as pinning pieces or creating threats that are difficult for Black to parry without the defensive support of the e6 pawn. In contrast, while the weakened c8–h3 diagonal (due to the absence of c6) does offer White some opportunities, it is generally less impactful. The light-squared bishop's influence along this diagonal doesn't pressure Black's critical central pawns as directly, and Black can often more easily defend against threats on light squares. Summary: The absence of Black's e6 pawn weakens the f1–a6 diagonal, which is more meaningful because it enhances White's attacking prospects and puts greater pressure on Black's central and queenside defenses." The problem is though that not even the first few lines of this answer ("The c8–h3 diagonal (the long light-squared diagonal) is weakened. Reason: Without the pawn on c6, this diagonal becomes open") make sense, since c6 is not on c8-h3 diagonal.
- anentropic 2y agoAre there any benchmarks which compare existing LLMs using langchain-style multi-step reasoning? The new OpenAI model shows a big improvement on some benchmarks over GPT4 one-shot chain-of-thought, but what about vs systems doing something more similar to what presumably this is?
- alexbenton111 2y agoI am not sure how more advanced this new model is than previous GPT-4o, but at least this new model can correctly figure out that 9.9 is larger than 9.11.
- andrew_eu 2y agoI thought with this chain-of-thought approach the model might be better suited to solve a logic puzzle, e.g. ZebraPuzzles [0]. It produced a ton of "reasoning" tokens but hallucinated more than half of the solution with names/fields that weren't available. Not a systematic evaluation, but it seems like a degradation from 4o-mini. Perhaps it does better with code reasoning problems though -- these logic puzzles are essentially contrived to require deductive reasoning. [0] https://zebrapuzzles.com https://zebrapuzzles.com
- energy123 2y agoo1-mini does better than any other model on zebra puzzles. Maybe you got unlucky on one question? https://www.reddit.com/r/LocalLLaMA/comments/1ffjb4q/preliminary_livebench_results_for_reasoning/ https://www.reddit.com/r/LocalLLaMA/comments/1ffjb4q/prelimi...
- andrew_eu 2y agoEntirely possible. I did not try to test systematically or quantitatively, but it's been a recurring easy "demo" case I've used with releases since 3.5-turbo. The super verbose chain-of-reasoning that o1 does seems very well suited to logic puzzles as well, so I expected it to do reasonably well. As with many other LLM topics, though, the framing of the evaluation (or the templating of the prompt) can impact the results enormously.
- slig 2y agoHey, I run ZebraPuzzles.com, thanks for mentioning it! Right now I'm trying to improve the puzzles so that people can't "cheat" using LLMs so easily ;-).
- bambax 2y agoNear the end, the quote from OpenAI researcher Jason Wei seems damning to me: > Results on AIME and GPQA are really strong, but that doesn’t necessarily translate to something that a user can feel. Even as someone working in science, it’s not easy to find the slice of prompts where GPT-4o fails, o1 does well, and I can grade the answer. But when you do find such prompts, o1 feels totally magical. We all need to find harder prompts. Results are "strong" but can't be felt by the user? What does that even mean? But the last sentence is the worst: "we all need to find harder prompts". If I understand it correctly, it means we should go looking for new problems / craft specific questions that would let these new models shine. "This hammer hammers better, but in most cases it's not obvious how better it is. But when you stumble upon a very specific kind of nail, man does it feel magical! We need to craft more of those weird nails to help the world understand the value of this hammer." But why? Why would we do that? Wouldn't our time be better spent trying to solve our actual, current problems, using any tool available?
- uptownfunk 2y agoThis hits the nail on the head. It is a consumer facing product not a technology to solve deep thinking.
- Mattclosson 2y agoi don't think that's what he's saying
- thrdbndndn 2y ago> What does that even mean? It explicitly says "Results on AIME and GPQA are really strong". So I would assume it means it can get (statistically significantly, I assume) better score in AIME and GPQA benchmarks compared to 4o.
- benterix 2y ago> But why? Why would we do that? Because OpenAI needs a steady influx of money, big money. In order to do so, they have to convince the people who are giving them money that they are the best. An objective way to achieve this is by benchmarking. But once you enter this game, you start optimizing for benchmarks. At the same time, in the real world, Anthropic is following them in huge leaps and for many users Claude 3.5 is already the default tool for daily work.
- benterix 2y agoI remember Murati's interview where she said about this PhD level reasoning and so on, so I was excited to see what they come up with - and it looks like they just used a bunch of models (like 4o's) and linked them in a chain of thought - which is exactly what we have been doing ourselves for a long time to get better results. So you have the usual disadvantages (time and money) and lose the only advantage you had when doing it yourself, i.e. inspecting the immediate steps to understand the moment where it goes wrong so that you can correct it in the right place.
- vlad-r 2y agodo you know if someone actually compared a 4o CoT to the o1? I'm trying to find something on it, but I can't find anything. LE: I found this tweet by Catena Labs of their MoA mix compared to o1-preview: https://x.com/catena_labs/status/1834416060071571836 https://x.com/catena_labs/status/1834416060071571836
- dr_dshiv 2y agoSo, this is just an RL trained method of having multiple GPT4o agents think through options and select the best before responding?
- monkeydust 2y agoJust finished reading the 'Book of Why by Judea Pearl' and my own mental gap from AI to today to whatever AGI is has got wider, thought not discounting this seems like a step forward.
- jari_mustonen 2y agoOnce again, there’s a lot of safety talk. For example, OpenAI’s collaborations with NGOs and government agencies are being highlighted in the release notes. While it’s crucial to prevent AI from facilitating genuinely harmful activities—like instructing someone on building a nuclear bomb, there is an elephant in the room regarding safety talk: Evidence suggests that these safety protocols sometimes censor specific political perspectives. OpenAI and other AI vendors should recognize the widespread suspicion that safety policies are being used to push political agendas. Concrete remedies are called for—for example, clearly defining what “safety” means and specifying prohibited content to reduce suspicions of hidden agendas. Openly engaging with the public to address concerns about bias and manipulation is a crucial step. If biases are due to innocent reasons like technical limitations, they should be explained. However, if there’s evidence of political bias within teams testing AI systems, it should be acknowledged, and corrective actions should be taken publicly to restore trust.
- jdthedisciple 2y agoJust leaving it here as well in case anyone feels up to the task: I challenged o1 to solve the puzzle in my profile info. It failed spectacularly. Now see you on the other side ;)
- dailykoder 2y agoI am mostly only an LLM user with technical background. I don't have much in-depth knowledge. So I have questions about this take: >the output token allowance has been increased dramatically—to 32,768 for o1-preview and 65,536 for the supposedly smaller o1-mini! So the text says reasoning and output tokens are the same, as in you pay for both. But does the increase say that it can actually do more, or does it just mean it is able to output more text? Because by now I am just bored of GPT4o output, because I don't have the time to read through a multi-paragraph text that explains to me stuff that I already know, when I only want to have a short, technical answer. But maybe that's just what it can't do, give exact answers. I am still not convinced by AI.
- simonw 2y agoI included that note because output limits are a personal interest of mine. Until recently most models capped out at around 4,000 tokens of output, even as they grew to handle 100,000 or even a million input tokens. For most use-cases this is completely fine - but there are some edge-cases that I care about. One is translation - if you feed in a 100,000 token document in English and ask for it to be translated to German you want about 100,000 tokens of output, rather than a summary. The second is structured data extraction: I like being able to feed in large quantities of unstructured text (or images) and get back structured JSON/CSV. This can be limited by low output token counts.
- dailykoder 2y agoSure, your cases are perfectly reasonable. I just wish the LLMs had a "feel" about when to output long or short text. Always thinking about adding something like "be as concise as possible" is kinda tedious
- tluyben2 2y agoThis model did single shot figure out things that Sonnet just ran ran in a loop doing wrong and reddit humans also seemed not be able to fix (because niche I guess). It is slow (21 seconds for the hardest issue), but that is still faster than any human.
- mergisi 2y agoI've been working on a o1-preview and recently hit some limitations with OpenAI's cap. But I’ve made progress—added all the steps, details, and code on GitHub https://github.com/mergisi/openai-o1-coded-personal-blog https://github.com/mergisi/openai-o1-coded-personal-blog . The result isn't bad at all; just a few more CSS tweaks to improve it. Check it out and let me know what you think! How does it compare to tools like Claude Sonnet 3.5?
- mergisi 2y ago[dead]
- binary132 2y agoPersonally I felt like o1-preview is only marginally better at “reasoning”. Maybe I just haven’t found the right problems to throw at it just yet.
- franze 2y agoJust coded this this morning using chatgpt o1 - it is the reimplementation of an old idea now music, multiple dots, more and more bug fixes honestly, chatgpt is now a better coder than i ever was or will be https://lsd.franzai.com/ https://lsd.franzai.com/
- SeanAnderson 2y agoNeat idea. The ball frequently passes through solid lines though.
- franze 2y agofixed, just asked chatgpt to come up with a better physics engine and collision detection algorithm
- gabrielrdz 2y agohah, this takes me back. There used to be a game called Jezzball, I think, back in the late 90's or early 00's. Had a lot of fun with that one.
- DrNosferatu 2y agoHow is o1 different in practice and end-results from my own, simple, Mixture of Agents script, that just queries several APIs?
- nprateem 2y agoIt's a for-loop isn't it?
- HarHarVeryFunny 2y agoI think Rich Sutton's bitter lesson will prove to apply here, and what we really need to advance machine learning capabilities are more general and powerful models capable of learning for themselves - better able to extract and use knowledge from the firehose of data available from the real world (ultimately via some form of closed-loop deployment where they can act and incrementally learn from their own actions). What OpenAI have delivered here is basically a hack - a neuro-symbolic agent that has a bunch of hard-coded "reasoning" biases built in (via RL). It's a band-aid approach to try to provide some of what's missing from the underlying model which was never designed for what it's now being asked to do.
- joelburget 2y agoo1 is an application of the Bitter Less. To quote Sutton: "The two methods that seem to scale arbitrarily in this way are search and learning." (emphasis mine -- in the original Sutton also emphasized learning). OpenAI and others have previously pushed the learning side, while neglecting search. Now that gains from adding compute at training time have started to level off, they're adding compute at inference time.
- HarHarVeryFunny 2y agoI think the key part of the bitter lesson is that (scalable) ability to learn from data should be favored over built-in biases. There are at least three major built-in biases in GPT-O1: - specific reasoning heuristics hard coded in the RL decision making - the architectural split between pre-trained LLM and what appears to be a symbolic agent calling it - the reliance on one-time SGD driven learning (common to all these pre-trained transformers) IMO search (reasoning) should be an emergent behavior of a predictive architecture capable of continual learning - chained what-if prediction.
- m101 2y agoIt works like our own minds in that we also think, test, go back, try again. This doesn't seem like a failing but just a recognition that thought can proceed in that way.
- nbzso 2y agoWorking in tech for over 30 years. This is the first time when I don't see proposed technology as a valuable tool. Especially LLM's. Vastly overhyped, driven by pure greed and speculative narratives, limited implementation and high energy cost. Non-transparent. Errors marketed as a hallucination.
- simonw 2y agoHow much time have you spent figuring out how to use them? Ethan Mollick estimates it takes ten hours of exposure to “frontier models” (aka OpenAI GPT-4, Claude 3.5 Sonnet, Google Gemini 1.5 Pro) before they really start to click in terms of what they’re useful for.
- jes5199 2y agothis is exactly what i said about the iphone
- nbzso 2y agoSorry, there is no parallel between technology with direct implication and dreams from VC's and investors with low level of tech literacy.
- Thrymr 2y agoFor me, that moment was cryptocurrency. "Vastly overhyped, driven by pure greed and speculative narratives, limited implementation and high energy cost." - all applied. I couldn't understand why so many people thought it was the future. I actually see LLMs a little more positively - mildly interesting, certainly intriguing language mimics, but enormously expensive and overhyped. Are they useful? Maybe, but not to the degree that everything is focused on them now.
- zitterbewegung 2y agoI wonder if this can be replicated by getting a reinforcement algorithm and LangGraph / LangGraph .
- deegles 2y agoI was thinking about what "actual" AI would be for me and it would be something that could answer questions like "tell me every time Nicolas Cage has blinked while on camera in one of his movies". Sure, that is a contrived question, but I expect an "AI" to be capable pf obtaining every movie, watching them frame-by-frame, and getting an accurate count. All in a few seconds. Current models (any LLM) cannot do that and I do not see a path for them to ever do that at a reasonable cost.
- bgun 2y agoI agree. My example for something “AI” should be able to do is to create a CAD model for the Empire State Building or the Parthenon based on known facts and photos. I don’t think these are “moving the goalposts” examples, they are things that an actual intelligence capable of passing a PhD physics exam should be able to do.
- a_wild_dandan 2y agoI mean, I passed a physics PhD exam and I can’t model the Empire State Building. The jury is still out on whether I’m an intelligence tho.
- bgun 2y agoMy point is that you could, given enough time and all the information available to you online about these well-documented buildings. You could learn CAD and figure out a reasonable way to output a 3D model, because you can think and reason spatially. The current batch of AI tools can regurgitate complex facts, but they can't actually think in 3D like an being that spends its life navigating physical spaces. Maybe I'm wrong and we are well on our way to AI tools for this, but right now if I tell any of the current generation of image models to do something like "rotate object 70 degrees, tilt camera down 20 degrees and re-render" then what comes out is never even approximately close.
- littlestymaar 2y ago> All in a few seconds That part is unrealistic: even just loading in RAM and decoding all movies Nicolas Cage appears in would take much more than a few seconds unless you thrown an insane amount of compute at the job. That being said, the current LLM tech is probably enough to help you implement a program that parses IMDB to get the list of all Nicolas Cage movie, then download it on thepiratebay and then implement the blink count you're looking for. And you'd likely get the result in just a couple hours.
- dev1ycan 2y agoThere's no actual improvement for real world tasks, just in-lab word prediction... it's disappointing to see so much money poured into obvious vaporware, Every 10-5 years we have a new generation of clueless VCs pouring money into something they don't understand based on lies by grifters, no different than the esports scene.
- briandw 2y agoMy 12 YO and I just built a fishing game using o1 preview. Prompt: "make a top down game in pyxel. the play has to pay off a debt to a cat by catching fish. the goal is for the player to catch the giant king fish. To catch the king fish the player needs to sell the fish to the cat and get money to buy better rods, 3 levels of rod, last one can catch the king fish." It nailed the execution. Amazing.
- ikety 2y agomind sharing the code? I asked o1 preview this verbatim and it produced errors. Just wanted to make sure its not me
- causal 2y agoMy first few attempts at getting it to work with an existing codebase have not been impressive. Perhaps 1o is best suited to difficult problems that can be stated in only a few sentences.
- rm_-rf_slash 2y agoI’ve had the opposite experience: terrific use with modifying existing codebases. But then again I’ve been using GPT4 to code for over a year now and so I’m used to writing out prompts with my eyes closed.
- wufarino 2y ago[dead]
- deleted 2y ago[deleted]
- B1FF_PSUVM 2y agoAFAICT, we got the ELIZA 60th anniversary edition, and are now headed for some Prolog/production systems iteration. One of these days those contraptions will work well enough, not because they're perfect, but because human intelligence isn't really that good either. (And looking in this mirror isn't flattering us any.)
- resters 2y agoI pasted in some markdown content with symbolic logic expressions and proof examples and the model said I was violating the terms of service.
- throwuxiytayq 2y agoWelcome to The Future, where computation has terms of service and each workload you submit for execution is automatically verified to be in compliance.
- martin82 2y agoSorry to be cynical, but to me it feels very much like OpenAI has no clue how to further innovate, so they took their existing models and just made them talk to each other under the hood to get marginally better results - something that people have been doing with Langchain for a while now. I will just lean back and wait for the scandal to blow up when some whistleblower reveals that the hidden output tokens about the thought process are billed much higher than they should be - this hidden cost system is just such a tempting way to get far more money for the needed energy/gpu costs, so that they can keep buying more GPUs to train more models faster, I don't see how people as reckless and corrupt as Sam Altman could possibly resist this temptation.
- skenderbeu 2y agoCan we just push LLMs aside for a minute and look at the whole AI models from outside the box? The feeling I'm getting is that the obsession with LLMs has outpaced its usefulness.
- kranuck 2y ago> For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user I'm sick of these clowns couching everything in "look how amazing and powerful and dangerous out AI is" This is in their excuse for why they hid a bunch of model output they still charge you for.