11 ms·
John Carmack’s ‘Different Path’ to Artificial General Intelligence
- Dowwie 4y agoHe's going to take a different path by downloading himself into a machine and becoming the AGI. No one will be surprised.
- deleted 4y ago[deleted]
- lostmsu 4y agoJohn is rubbing my ego with this move. I am basically on the same track since a few years ago with a bit lower resources (4x3090, but planning to grow).
- Macuyiko 4y agoFun interview but barely any meat for those in the field. Just very general questions and answers, saying that the road is murky, but nothing about e.g. if Transformers / attention are the way to go forward, multi-modal models, reinforcement learning + self-supervised learning.
- modeless 4y agoThere is a comment in the article about having models watch TV and play video games, and he's talked about that before too in his Lex Friedman interview. Seems like his approach is to take existing model architectures, apply some tweaks and experimental ideas and then use datasets consisting of TV (self supervised learning maybe?) and classic video games (RL I guess?). The video game part at least sounds like what Deepmind is already doing. I guess we'll just have to wait and see what he plans to do differently. It seems to me like his expertise would be most valuable in optimizing model architectures for hardware capabilities to improve utilization and training efficiency. That will be important for AGI especially as the cost of training models skyrockets (both time and money). If I was a startup doing AI hardware like Cerebras or Graphcore I would definitely try to hire Carmack to help with my software stack. Though he doesn't seem interested in custom AI hardware.
- arcturus17 4y agoI do not question for a second that Carmack is a computing genius, but “murky” is an understatement. There is nothing of technical, scientific or business value in this interview. It’s just generic words strung together.
- SideburnsOfDoom 4y agoEven socially, it fails to plumb the depths. Mr Carmack is taking a "different path" by ... reading the relevant literature and talking to YC? This is 100% mainstream - The difference is what?
- GuB-42 4y agoPlot twist, John Carmack's AI is answering the interview.
- mrguyorama 4y agoWhy do people think he is a computing genius? He read contemporary papers in computer 3D rendering and implemented well understood concepts in a shitty dev environment (early x86 and VGA systems). When I do that for my company, I'm just a junior dev. People really need to stop with this "Great person" nonsense. He's a pretty smart coder, and is gifted with geometry and other fields of math. He's not a genius. He didn't "master" calculus at age 15 like Einstein, he didn't invent anything particularly new in the field. Why the obsession people have with him? Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
- arcturus17 4y agoOk!
- 93po 4y ago> Why should we look to him for AI questions? What evidence is there that he has any new knowledge? He covers this in the article. He doesn't. He's just trying stuff out with a different approach than others because he believes (and is probably correct) that there is a chance that the most efficient path forward to AGI isn't the work that OpenAI and others are doing.
- laluser 4y agoI imagine this is a 10 year journey for him and he’s just getting started. Check back in 5 years.
- jdlshore 4y agoIt’s a puff piece in a general interest magazine. It’s not going to go into details. I also got the impression that Carmack was being cagey about the directions where he saw potential.
- SideburnsOfDoom 4y agoYep. Does it ever get around to answering the question implicit in the title: what is it that he is going to do "different" to the rest? Seems more like he's talking to and following up with Altman, "Y Combinator conference" and the rest. Is that "bucking the trend", taking your own "path", really?
- JKCalhoun 4y agoI assume it's to know which directions have shown the most recent potential, to catch up on the techniques and literature so you can talk/think intelligently on the matter. But I see your more general point, he has to be careful not to get caught up I the groupthink he is (understandably?) critical of.
- kelipso 4y agoLol I feel like this is going to be a high school valedictorian goes on to be an average student in Stanford kind if story.
- 93po 4y agoThe area of AI research isn't some outlier of super geniuses that don't exist anywhere else. Carmack has often worked around some of the best talent on the planet and he has always stood out.
- dr_dshiv 4y agoWhat would be the difference between GPT3 and actual AGI? (Want to make sure we don’t keep moving goalposts)
- GalahiSimtam 4y agoAGI would not mix up stuff like this https://pastebin.com/raw/zPLxKr56 https://pastebin.com/raw/zPLxKr56
- ipsum2 4y agoAGI is usually defined as anything a normal human could do. Examples of some things that GPT is currently lacking: 1) Understanding images and video 2) Learning and remembering things outside the 2048 token context window 3) Interacting with the environment (either through a computer, or in the real world) 4) Doing basic math (1662 + 239 = ?) and logic
- bbor 4y agoI’d argue “learning and remembering” is the core issue here, and the other things you’ve mentioned are just ways humans use that ability. I took a “Knowledge Based AI” course and it really opened my eyes to how central a knowledge system is to cognition. I wouldn’t call what LLMs have knowledge since it lacks structure, though I’m very much a novice on the exact details there. Well actually #3 isn’t great bc what exactly would you call taking input from a human and displaying output, much less e.g. controlling a smart home? Seems like LLMs are plenty smart enough to interact with things.
- dr_dshiv 4y agoNice list. 1. Understanding— provided this isn’t based on consciousness but benchmarks — appears present in text, but yes, GPT3 can’t do images or video yet 2. Is there some number of tokens that would be AGI? Of course it can remember things it has already been trained on… 3. In what way is interacting with a human not interacting with the environment? (Eg, you can ask it to tell you the steps to make a pbj sandwich and troubleshoot) 4. Basic math and logic. Ok. So, once AI can do images, video, math, logic and remember 20000 tokens, then we have AGI?
- inawarminister 4y agoCarmack seems to be simulating a vtuber AI watching TV and playing video games. If anyone here is doing that too, I would recommend taking a quick look at Neuro-sama on Twitch. They're using RL to play OSU, Minecraft, and Pokemon, and voice input + video image analysis to react to Twitch streams and documentaries. While being watched by 6.5K people. The url is twitch.com/vedal
- Zondartul 4y agoGood point, although some background info should be noted: Vedal worked on Neuro-sama for about 10 years, and for most of that time it was a NN with a singular task of playing "osu!". I think it's only this year that he implemented a chatbot with TTS for her debut on Twitch. Although Neuro-sama is an impressive application of several existing technologies to make a virtual entertainer, under the hood Neuro-sama is several specialized NNs (one to play Osu, one to play minecraft, one for chat, and one for TTS) that, for the most part, don't know what the other is doing as very little data is passed between them. There is no scientific innovation or any claims of higher intelligence, but I think it's still a very cool system from hobby-engineering point of view. Neuro-sama is a ditz, but a funny one.
- mshake2 4y ago>And one of the things that I trained myself to do a few decades ago is pulling ideas out and pursuing them in a way where I’m excited about them, knowing that most of them don’t pan out in the end. Much earlier in my career, when I’d have a really bright idea that didn’t work out, I was crushed afterwards. But eventually I got to the point where I’m really good at just shoveling ideas through my processing and shooting them down, almost making it a game to say, ‘How quickly can I bust my own idea, rather than protecting it as a pet idea?’ Cool life lesson there
- once_inc 4y ago'Kill Your Darlings' is one of the most fundamental skills in anything creative. It is a hurdle that, once taken, allows you to ascend above a plateau of mediocrity.
- awestroke 4y agoWell, he's not taking risks anymore by doing this. He has already proven himself, he's rich, and he's paid incredible amounts to do whatever he likes.
- jdlshore 4y agoThe lesson to be learned is that it’s less risky, not more, to understand that some ideas don’t work out and to have ways to identify failures quickly.
- jstummbillig 4y agoI get and like that interpretation. I wonder if, societally, it's true though, or if statistically the more reliable way to "succeed" in life, is getting good at selling and defending ideas, even the bad ones.
- kwhitefoot 4y agoThat only works if you lack a conscience. I think you were right to quote the word succeed!
- random314 4y ago> So, about four years ago, I went on one of my week-long retreats, where I just take a computer and a stack of reference materials and I spend a week kind of reimplementing the fundamentals of the industry. And getting to the point where it’s like, ‘All right, I understand this well enough to have a serious conversation with a researcher about it.’ And I was pretty excited about getting to that level of understanding. As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded. Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.
- carabiner 4y agoYep - he also failed with his lean, simple, first-principles approach with Armadillo Aerospace. The guy is proudly uncreative, and so he could have never come up with Scaled's design of a variable geometry rocket ship launched from a jet mothership.
- WoodenChair 4y ago> As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least). > If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded. This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher. That said, of course it makes sense to be skeptical.
- labrador 4y agoThis paragraph took me aback: But if I just look at it and say, if 10 years from now, we have ‘universal remote employees’ that are artificial general intelligences, run on clouds, and people can just dial up and say, ‘I want five Franks today and 10 Amys, and we’re going to deploy them on these jobs,’ and you could just spin up like you can cloud-access computing resources, if you could cloud-access essentially artificial human resources for things like that—that’s the most prosaic, mundane, most banal use of something like this. It kind of shocked me because I thought of the office worker reading this who will soon lose her job. People are going to have to up their game. Let's help them by making adult education more affordable.
- peepee1982 4y agoEducation is great, but it can go only so far. We will always have to find things to do for the less gifted in order to provide them with some dignity. Even if they are not strictly needed for reasons of productivity or profitability. Anything else would be inhumane.
- adrianN 4y agoPanem et circenses. I think it's unlikely that we'll be able to sufficiently transform the economy so that there is an ample supply of desirable jobs that could more profitably done by robots.
- bbor 4y agoDo you see “giftedness” as a 1D score, where someone is either smart or not smart? And presumably this quality happens to correlate with software engineering ability? I think you’re hinting at some very hurtful, dangerous ideas.
- beebeepka 4y agoThat's how I read it as well. Maybe their heart is at the right place but I think "gifted" and "having what happens to be needed right now" are completely different things, at least to me.
- sreeramvenkat 4y ago> So I asked Ilya, their chief scientist, for a reading list Just curious, if this reading list is available somewhere.
- phodo 4y agoWould be interesting to get a list of those 40 papers mentioned
- 9999 4y agoI would also really like to see that list of 40 papers.
- robsun 4y agoPlease, upvote parent comment :). I guess there is a lot of people who are wondering which papers he read.
- nvrmnd 4y agoI got a let out of Karpathy's video lectures on youtube, for example: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY He mentions a few of the bigger papers in multilayer perceptrons (aka deep networks) such as attention is all you need, I think a good place to dive in before coming back to visit some fundamentals.
- w-m 4y agoMaybe ask him on Twitter for it?
- tipsytoad 4y agoCame here to say the same thing, but a few off the top of my head - attention is all you need - image is worth 16x16 words (vit) - openai clip - transformer XL - memorizing transformers / retro - language models are few shot learners (gpt) A few newer papers - recurrent block wise transformers - mobilevit (conv + transformer) - star (self taught transformer)
- loveparade 4y agoI'm always taken aback by this fascination and heroification with John Carmack (or similar figures) on HN. I love Doom and Quake too, but that doesn't qualify him any more than any other random senior engineer to work on something totally different. From his interviews it looks like he understands little about the technical details of ML, or about as much as anyone can learn in a few months, and is just banking on PR based on his games and name. I put him into the same category as Elon Musk, who also understand nothing about technical details of AI, but was still able to hire a world class team at OpenAI. His name and fame counts for something in terms of recruiting and joining his venture may be a good bet because of that, but he's not a person whose opinion on the subject matter I would take seriously in the same way I'd take a researcher seriously.
- Kbelicius 4y ago> I put him into the same category as Elon Musk, who also understand nothing about technical details of AI, but was still able to hire a world class team at OpenAI. I don't believe he had anything to do with hiring at openAI nor that he is anything more there than an investor/donor as others are.
- loveparade 4y agoPerhaps not directly, but from the very start OpenAI was considered "Elon's AI company" and people wanted to work there because they wanted to work with Elon. That had a huge effect on what kind of talent OpenAI was able to hire. Yes, I'm aware that Ilya had much more day-to-day impact, but if you're part of the ML research community and have worked with Ilya, or heard rumors of working with Ilya, that may have had more of a negative effect...
- AlchemistCamp 4y ago> From his interviews it looks like he understands little about the technical details of ML... I'd happily bet my entire net worth that he knows more about the technical details of ML than you do.
- fxtentacle 4y ago
- mach1ne 4y agoCarmack seems extremely lucid about his position in the field and the current nature of the field in general. While rogue scientists don't have great odds, Carmack is definitely doing important research studying off-mainstream subjects.
- htk 4y agoCarmack is a beast, the path he's pursuing might not yield the best results, but I'm pretty sure he's the guy to make the best use of the hardware. I wouldn't be surprised if his solution is orders of magnitude more performant than what the competition is doing.
- abudabi123 4y agoHe has the technical chops to squeeze more out of his $250k LambdaLabs pro-rig than the average among the walkoffs and layoffs in the techsector or a tables worth of happy-hippos at a scheduled meeting sharing Excel spreadsheets in the old fossil fuel firms that have experienced stellar double digit billion dollar profits. Would that he mixes in his reading list some choice papers from the social sciences and social contructivists. The a16z group gave the impression they are into the social sciences in a podcast. General intelligence technology can help people bridge the gap before them to get where it is they want to be for the better. The educated might want to make known something from the unknown-unknown. The stinky hobo sleeping under the bridge with a smartphone could be guided to special social programs by Ai.
- jimmcslim 4y agoThis will be a provocative question, but is there any form of AGI that won’t ultimately share the attributes of slavery? Can AGI exist without also creating an artificial consciousness/self-awareness?
- detrites 4y agoYou mean like the attribute of rising up against ones oppressor and establishing ones own fundamental rights?
- unsupp0rted 4y agoAnd if it could, wouldn’t creating that be identical to creating Gammas and Deltas (i.e. intentionally mentally retarded clones who are pleased to serve you) in Brave New World?
- jlawson 4y agoEven if it's conscious, that doesn't mean it's suffering. Think of a happy dog. Dogs are subject to our whims and do what we want or face consequences. But they like it because we bred them to like it. So is that evil? Is that slavery?
- psychoslave 4y ago>We do not yet have the learnable stream of consciousnesses of a co-worker in AI, but we do have kind of this oracular amount of knowledge that can be brought forward. Except if there are some plane to make AGI agents have their own mundane-human-like life with issues unrelated to the businesses problem at stack they are supposed to do, where will they take serendipity inputs?
- ggm 4y agoI don't object to him doing this nor the immense amounts of investor money which will burn along the path. He's smart. Good things will come of this. Maybe even amazing things. But not AGI. I do object to fetishists of AGI piling in and the equally silly assumptions he has some magic secret sauce which can get there. Please do not be sucked into "to infinity and beyond" nonsense. I don't care if it's Musk, or Carmack or Kurzeweil, it's stupid. If Malcolm Gladwell writes it up, it's peak stupid.
- Tepix 4y ago> Good things will come of this. … But not AGI. What are the showstoppers in your opinion?
- jhoelzel 4y agoAs someone who works in IT for 20 years but has a bachelor in economic psychology, i do believe that we have not solved AGI yet because the creators of the models have the generalized view of the universe that you are "one person" and there are just so many detailed facets to that. Basically the assumption is that if you cram enough data into your gpt model, it should know everything. Which is of course not true, it repeats the things it reads the most with a probability. Basically how there are two versions of smart teens, the ones who learn every day and the ones who just pick up concepts on the fly and run with them. I think the first space has been explored plenty, for the second one I have a concept ready and dearly hope that power gets cheaper in europe ;)
- lostmsu 4y ago> Basically the assumption is that if you cram enough data into your gpt model, it should know everything. Which is of course not true, it repeats the things it reads the most with a probability. That's only how the system was designed intentionally. E.g. there's intentionally no self-feedback loop.
- jhoelzel 4y agoso we are both agreeing on the same thing? :D
- lostmsu 4y agoConceptually, no. For me your statement sounds like it says we did not solve flight, because our planes have to land.
- jhoelzel 4y agoWell I am in fact trying to argue that the self feedback loop is what is missing to solve AGI and even more so implying that the reason for that may be that there are different ways to see a "human" psychologically. Also that validity in data can only be achieved using a multi modal approach with source ranking. Psychology essentially has the same problem and is only a "science" where reproducible. Quantitative and Qualitative psychology have 2 approaches to the same solution whereas the latter is "reading between the lines" and the former is "bean counting", meaning statistical interference. I am trying to say that friend of mine has created "Chicken Infinite" in 2014, which is basically an endless auto generated cooking book. Deep-L has also been around for a while. These applications lead me to believe that text applications, trained on a large dataset do not have to be this intensive. Furthermore what makes chatgpt enticing, is its chat interface which is using a multi model approach too. Have it create a detailed story for you with multiple prompts and then ask it to generate the prompt to generate this story and you will see various model instructions. (or at least you could last month). Or differently put, there is no AGI because the understanding function is simply not present and i think that the reasoning for that is buried in the approach of the human mind.
- dharma1 4y agoI agree with John that there are probably other people in his position (great engineers with a lifetime of experience and enough financial resources to focus on something like this for 5-10 years) who could have a go at this and have a reasonable chance at chipping away at AGI. I also like his frugality, whether it’s optimising for hardware or financials.
- jasmer 4y agoAGI is a bit of a glib concept ... our technology will not be like an 'autaumaton'. Everything we make is 'systems' oriented, and is not at all like our human conception of 'intelligence' dervied from the fact that we humans are effectively independant from our surroundings. 'Siri', backed by ChatGPT and the 'world's data' will probably pass some 'AGI' threshold, but is 'Siri' an individual AGI? Are we all talking to the same siri? Different Siri? It's not even an entity, rather a distributed system. Our ideas regarding AGI are perversely inluenced by the fact that we humans are automatons, but technology is not that. It's also entirely feasible that if ChatGPT represents all possible forms of human communication, then it will perfectly emulate a human. Ok, it's really just a fancy neural network that is not theoretically 'thiking' but how does that matter? If it can rationlize sufficiently to make such interactions, who is to say it's not 'AGI'? I think we're using the wrong concepts.
- cactusplant7374 4y ago> It's also entirely feasible that if ChatGPT represents all possible forms of human communication, then it will perfectly emulate a human. Ok, it's really just a fancy neural network that is not theoretically 'thiking' but how does that matter? If it can rationlize sufficiently to make such interactions, who is to say it's not 'AGI'? Doesn’t AGI need to be able to make discoveries as a human would? How else can it move us forward as a society?
- TuringTest 4y ago> So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head I wonder what that list could be? I have always had trouble finding the essential scientific articles in an area of knowledge and separating them from the fashion of the day. A list compiled by an expert specifically for sharp learners is valuable on its own.
- lngnmn2 4y agoThere is something to read before you go. https://schiptsov.github.io/GPT-bullshit.html https://schiptsov.github.io/GPT-bullshit.html
- marviel 4y agoI agree with the author's point that intelligence is largely "the process of removing the bullshit to see what remains" So, because I thought it would be funny, I asked ChatGPT to summarize this essay: > The author critiques the work of @karpathy, who is trending on HN, as promoting bullshit and discrediting science and true philosophy. The author argues that mathematics and logic are valid generalizations and abstractions made by the mind of an observer of patterns in the universe. Intelligence is the ability to zoom in and out through different levels of generalizations and abstractions. The author argues that the problem with language models is that they lack validation of previous steps and the process of construction of a representation must trigger a verification process. The author concludes that what is manipulated and transformed in language models is information, not knowledge, as knowledge requires validation and verification at each step.
- thom 4y agoI don't want to be an utter bore, but 60% chance of AGI by 2030, taken at face value, would have prompted me as an interviewer to more urgently ask deeper questions about safety. For a long time I have tried to comfort myself that brains are complicated, AI is hard, and maybe there's just a complexity barrier that is going to prevent any sort of hard takeoff scenario. Maybe we can have subhuman AGI that can do lots of menial jobs, but not Carmack's, and we're right to concentrate on the economic ramifications. But lately... boy, I dunno.
- tkgally 4y agoMy reaction, too. For an interview as long and wide-ranging as that one, the lack of any mention of safety or ethical issues stood out. And the potential impact on employment was just waved away with “I am confident that the sum total of value and progress in humanity will accelerate extraordinarily with welcoming artificial beings into our community of working on things.”
- jasmer 4y agoMy god man 'this is like slavery' (!!) and 'the ethnical issues'. It's software man. Stop it with this 'AGI' nonsense and even 'AI'. Let's call it 'adaptive algorithms' and see it for what it is, just a neat bit of algebra trained on corpus data. The biggest upset in the industrial revolution was the harnessing of fossil fuels, nothing will ever come close to that. We have not had a problem with 'employment' ever since.
- JKCalhoun 4y ago
- dsign 4y agoUnpopular idea, but I hope it's only a matter of time before we declare involving with AGI a criminal endeavor, just as it is playing with biological weapons. Because we can't all be wrong: in almost every forecast, we see AGIs taking over our dignity as a bad thing. And we know that this is not any longer a sci-fi hypothetical scenario: the current generation of AI models is taking jobs from illustrators and copy writers. The current argument is that "China will do it if we don't", which to me sounds like "China will keep going in whatever path they are going, but supercharged with AGI, and we must desperately follow."
- Throwaway78389 4y ago> China will keep going in whatever path they are going, but supercharged with AGI, and we must desperately follow Exactly right but for one detail - we must desperately lead. What other countermeasure do you have in mind?
- suby 4y agoThis is a frustrating point of view to me. We have nearly every able bodied person in society spending something like 1/3rd of their life on this "dignity", but how many would define it as such? For most people it is an inversion of free will where they are subservient to others and treated as a replaceable cog in a larger and uncaring machine, spending time on something that they may very well hate or at least otherwise choose not to do. All to justify their existence and provide value to society, though for some nominally because there is a growing number of bullshit jobs where the value provided is questionable and the people going through the motions. It does not need to be that way. In an ideal world, human beings should be free to spend their time doing what they wish, work should be purely in the realm of hobby. No one should have to do work that they would not voluntarily choose to do for enjoyment. The only way we get there is through AI and the automation of everything. I don't even think it's avoidable -- provided civilization does not collapse, we will 100% reach a point where everything required to sustain a civilization is performed by machines. We shouldn't let fear keep us in a status quo that, while better than it has ever been historically, is still highly flawed.
- CatWChainsaw 4y ago
- hectorlorenzo 4y agoReading this I have the impression that John Carmack will be to AI what Jeff Hawkins is to brain research: a far-fetched idea, with potential, maybe too abstract and left-field to yield nothing tangible. Looking at the rest of the industry from the sidelines and (maybe) not being taken seriously by it.
- throwaway323421 4y agoPieces like this have started to really ramp up my anxiety. > Once it’s figured out, what do you think the ramifications will be? That'll probably destroy my life? I'm an ML engineer trying my best to immigrate to a better country with my wife who is a digital artist. As much as I think AI is cool - we both won't be needed anymore if the thing is tuned a couple of notches more intelligent. As a matter of fact, she's extremely worried about Midjourney - she probably lost book cover jobs already.
- jasmer 4y ago"So I asked Ilya Sutskever, OpenAI’s chief scientist, for a reading list. He gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did. I plowed through all those things and it all started sorting out in my head." Wow, that's going to be one of the more glib things I've read in a while. This is a bit of a Tom Cruise moment. I mean, I get it on some level but I suggest it's going to take a bit for someone to 'catch up' to cutting edge AI. Like more than a 'week of reading papers he doesn't understand'. Defeating the Rust borrow checker takes longer than that!
- Yoric 4y ago> Defeating the Rust borrow checker takes longer than that! Fighting the Rust borrow checker to hate leads and hate to the Dark Side leads. Harmony with the Rust borrow checker is what you must achieve, padawan.
- Jensson 4y agoThe models aren't particularly hard to implement, so a person who has spent his life and is one of the best game engine programmers in the world I don't think it would take that long to implement the cutting edge of todays machine learning models. Compared to game graphics shaders and the complex interactions you have to work with to get those performant and work correctly and display all the gameplay that happens the technical parts of machine learning are very simple. I've worked building low level machine learning stuff at Google, it isn't that hard to do at all. The hard part is improving these models, not building them when you already know what to build.
- trabant00 4y ago> there are signs that extremely powerful things are possible right now in the narrow machine-learning stuff, it’s not clear that those are the necessary steps to get all the way to artificial general intelligence This is my main takeaway from the interview, as it suits my beliefs. Most people seem to think that if we develop ML further we will go all the way to AGI, I think this is just mimicry step similar to how initial attempts to flight had flapping wings. I do think it is mandatory to explore in all directions but at this point this does not seem to be the one to lead all the way up to AGI.
- rl3 4y agoCarmack makes four points—some of which I agree with—that are unfortunately disturbing when taken in totality: a) We’ll eventually have universal remote workers that are cloud-deployable. b) That we’ll have something on the level of a toddler first, at which point we can deploy an army of engineers, developmental psychologists, and scientists to study it. c) The source code for AGI will be a few tens of thousands of line of code. d) Has good reason to believe that an AGI would not require computing power approaching the scale of the human brain. I wholeheartedly agree with c) and d). However, to merely have a toddler equivalent at first would be a miracle—albeit an ethically dubious one. Sure, a hard-takeoff scenario could very well have little stopping it. However, I think that misses the forest for the trees: Nothing says AGI is going to be one specific architecture. There’s likely many different viable architectures that are vastly different in capability and safety. If the bar ends up being as low as c) and d), what’s stopping a random person from intentionally or unintentionally ending human civilization? Even if we’re spared a direct nightmare scenario, you still have a high probability for what might end up being complete chaos—we’ve already seen a very tiny sliver of that dynamic in the past year. I think there’s a high probability that either side of a) won’t exist, because neither the cloud as we know it nor the need for remote workers will be present once we’re at that level of technology. For better or worse. So what to do? I think open development of advanced AI and AGI is lunacy. Despite Nick Bostrom’s position that an AGI arms race is inherently dangerous, I believe that it is less dangerous than humanity collectively advancing the technology to the point that anyone can end or even control everything—let alone certain well-resourced hostile regimes with terrible human rights track records that’ve openly stated their ambitions towards AI domination. When the lead time from state of the art to public availability is a matter of months, that affords pretty much zero time to react let alone assure safety or control. At the rate we’re going, by the time people in the free world with sufficient power to put together an effort on the scale and secrecy of the Manhattan Project come to their senses, it’ll be too late. Were such a project to exist, I think that an admirable goal might be to simply stabilize the situation via way of prohibiting creation of further AGI for a time. Unlike nuclear weapons, AGI has the potential to effectively walk back the invention of itself. However, achieving that end both quickly and safely is no small feat. It would amount to creation of a deity. Yet, that path seems more desirable than the alternatives outlined above-such a deity coming into existence either by accident or by malice. This is why I’ve never agreed with people who hold the position that AGI safety should only be studied once we figure out AGI-that to me is also lunacy. Given the implications, we should be putting armies of philosophers and scientists alike on the task. Even if they collectively figure out one or two tiny pieces of the puzzle, that alone could be enough to drastically alter the course of human civilization for the better given the stakes. I suppose it’s ironic that humanity’s only salvation from the technology it has created may in fact be technology—certainly not a unique scenario in our history. I fear our collective fate has been left to nothing more than pure chance. Poetic I suppose, given our origins.
- jeisc 4y agountil we have resolved the violent side of our human nature by finding a way to eradicate it from our world civilization which would mean no more murders no more killing no more waste no more wars etc. then we could start calling ourselves an intelligent life form and not a primitive species. This next level of humanity must be reached before venturing into creating a new life form which is the aim of AGI. Here is my rename tongue in cheek for AGI "awful greedy intelligence™" and to go with it AI "awful intelligence™"... We are way behind in our human evolution and AI or AGI is not going to do that for us. It is easier to chase a fantasy than to make real change in human nature.
- janalsncm 4y agoSelfishly the hype for AGI is good for an ML engineer like myself. But I have to say that there is no hope in solving a problem (especially by 2030!) that one cannot even define. Problems of the form “create a machine that can do X” are tractable. AGI is not because no one can agree on what intelligence is.
- sinuhe69 4y agoI stopped reading after the passage “and works independently [to achieve AGI]”. In my book, it’s impossible to achieve such breakthrough by working alone, regardless who the man/woman is.
- dustingetz 4y ago“If a system is to serve the creative spirit, it must be entirely comprehensible to a single individual. Human potential manifests in individuals.” — Dan Ingalls “Civilization advances by extending the number of operations we can perform without thinking about them”
- no-s 4y ago> In my book, it’s impossible to achieve such breakthrough by working alone, regardless who the man/woman is. Another fascinating opportunity for AGI - no sole contributor is all on their own, they can just spin up a community to embed into.
- 93po 4y agoHow can you claim something is impossible if you don't even know what's going to be achieved?
- dustingetz 4y agowhat are the 40 papers he read?
- redox99 4y agoWhat would we actually consider an AGI? Wikipedia lists the following tests > The Turing Test (Turing): A machine and a human both converse unseen with a second human, who must evaluate which of the two is the machine, which passes the test if it can fool the evaluator a significant fraction of the time. Note: Turing does not prescribe what should qualify as intelligence, only that knowing that it is a machine should disqualify it. > The Coffee Test (Wozniak): A machine is required to enter an average American home and figure out how to make coffee: find the coffee machine, find the coffee, add water, find a mug, and brew the coffee by pushing the proper buttons. > The Robot College Student Test (Goertzel): A machine enrolls in a university, taking and passing the same classes that humans would, and obtaining a degree. > The Employment Test (Nilsson): A machine performs an economically important job at least as well as humans in the same job. LLMs don't seem very far from passing 1), 3) and 4). I wouldn't be surprised if "GPT5" passed those 3.
- deleted 4y ago[deleted]
- sandos 4y agoI think GTP is very far from solving 1), at least if you happen to know anything about LLMs. I think the easiest one of these would be 4) actually.
- redox99 4y agoIs it? I remember reading [1] back in the day, which showed some limitations of GPT3, but ChatGPT actually answers all of those perfectly. [1] https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.h...
- xwdv 4y agoLike Mozart, Beethoven, and Chopin, John Carmack is one of those men who will never really die. He’ll simply become code.
- poulpy123 4y agoI can't wait to become completely useless and to finish my life under a bridge begging for food. On a more technical point of few I'm always surprised to read these articles and never read the work reasoning
- dopidopHN 4y agoI’ve been a plumber all my life, all I see here is more pipes.
- osigurdson 4y agoThat seems like an unlikely future reality. Productivity will be off the charts, goods and services will be nearly free. Why would the average person be poor in such a future?
- epups 4y agoBecause if productivity can be achieved with less people, and even intellectual work is not protected from machines, people will become redundant for the creation of economic value.
- osigurdson 4y agoProductivity is achieving more with the same number of people. The end result is more stuff (goods/services) at a lower price. It seems unlikely that this will be bad: wealth = f(technology) Optimists sound dumb in the present but are usually right in the long run. Pessimists sounds smart in the present but are usually wrong and miss out on opportunities. If not, which year in the past was significantly better than today? What was humanity's high water mark? I suspect we are nowhere near it.
- poulpy123 4y agoproductivity by who and for who ? AI will belong to a small number of people, and I will not be part of them
- 4y ago
- seu 4y ago> Well, you can tie that into a lot of questions, like, ‘Is human population a good thing?’ ‘Is immigration a good thing, where we seem to have been able to take advantage of new sources of humanity that are willing to engage in economic activities and be directed by the markets?’ > The world is a hugely better place with our 8 billion people than it was when there were 50 million people kind of like living in caves and whatever. So, I am confident that the sum total of value and progress in humanity will accelerate extraordinarily with welcoming artificial beings into our community of working on things. I think there will be enormous value created from all that. The problem with Carmack and many like him, is that they think of themselves as purely rational beings operating within scientific frameworks and based purely on scientific results, but whenever they step outside the technical fields in which they work, they are ignorant and dogmatic. He seems to ignore a lot about what the living conditions for people were throughout history, and have a blind trust in the positive power of 'human progress'. These people don't stop for a second to question the 'why', just the 'how'. They just assume 'because it will be better' and build their mountains of reasons on top of that, which just crumble and fall down as soon as that basic belief does not hold. I have a LOT of respect for him, and I'm sure he's a very decent, honest human being. But he's unfortunately another believer of the techno-utopianist faith which only asks for more 'blind progress' without questioning whether that is a good thing or not.
- serverholic 4y agoI’ve come to strongly resent the techno-utopian mindset that unfortunately plagues the tech world. Tech for the sake of tech with zero thought about how it will affect humanity.
- ncr100 4y agoHere, here. I haven't studied it formally and I'm being asked to support techno utopia also. So it feels pretty shaky to me. Certainly my livelihood is based on the premise of it and my dreams which fuel my workplace motivation serve as foundation to you know what I do with 50% of my life, work on technology. So I am biased. Some utopian dystopia discussions here on hacker news sort of boil down to the chaos theory level of assumptions, where you can see people exercising their own defensiveness when they snipe on a naysayer, sniping on their grammatical concerns, but not actually engaging in value-based discussions into the hacker news thread. It's like they're not human, they're only practicing it being devil's advocate technicians. Useful idiots is kind of what I think. We need to have more values discussions, ethics too.
- mark_l_watson 4y agoMy biggest takeaway from this is that he started by reading the most important papers on deep learning. When he originally announced his startup a few months ago, I wondered if he had something else in mind.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- mogacom 4y agoCarmack is clearly a brilliant guy, but it feels like he's fallen into the trap of overfitting on his previous successes and believing they generalise into other domains. No doubt his experience and innovations in computer graphics gives him a good insight into problems in vision, etc. but I don't see anything particularly original or orthogonal in what he's saying with regards to "general AI".
- haswell 4y agoWhat makes this a case of overfitting success vs. one of applying a lifetime of experience? I’m not claiming the first can’t exist, but I see no reason to conclude that is the case here.
- mogacom 4y agoI'm not intending to sound overly negative, I really hope he makes progress and think he has a chance (certainly more so than most). As someone who's been in the field for a long time though I constantly see people trivialise the problem of AGI and lose perspective, viewing everything as system design and architecture. Making the required conceptual contributions is a different thing entirely, but it seems his real goal might be the humbler one of scaling and integrating existing ideas.
- TremendousJudge 4y agoI kind of agree, but when you're trying to research truly new stuff, a priori you don't know which are the promising avenues. Newton spent most of his life studying teology and alchemy, which now we know would never take him anywhere compared to physics, optics, or even running the Royal Mint, but at the time there was no way for him to know this.
- m3kw9 4y agoWould you bet on me over Carmack? He obviously has famous engineering chops to have a punchers chance pull this off
- 4y ago
- belter 4y agoWould be ironic if achieving Artificial General Intelligence would be easier than implementing VR at Meta...
- notacoward 4y agoFor a long time, there have been two general approaches to AI. The first was focused on algorithms and heuristics, giving us everything from Eliza and somewhat-useful expert systems to Doug Lenat's Cyc project. Then came "AI winter" and everything (except some robotics) was pretty much on hold for a decade. Finally we got "cheap and deep" neural-net stuff as we know it today. I think we need both before we can reach true AGI. Whoever unifies them will have achieved the equivalent of grand unification in physics. And probably also created Skynet.
- marviel 4y ago> Now, the smart money still says it’s done by a team of researchers, and it’s cobbled together over all that. But my reasoning on this is: If you take your entire DNA, it’s less than a gigabyte of information. So even your entire human body is not all that much in the instructions, and the brain is this tiny slice of it —like 40 megabytes, and it’s not tightly coded. So, we have our existence proof of humanity: What makes our brain, what makes our intelligence, is not all that much code. Nice thought
- jhoelzel 4y agoWell, we could also simply be a lot of RAM interacting with itself until the power runs out. Following that trail of thought, intelligence is an achievement and not a physicality.
- thechao 4y agoI have a tongue-in-cheek assertion that a single human can be smart — perhaps even intelligent. But, what we're experiencing in our society, right now, is an emergent superintelligence that is the culmination of the interaction of 8 billion "merely smart" monkeys. When I'm feeling especially dark, I just assume that most humans aren't even conscious or usefully sapient most of the time, and that continuity of consciousness & sapience is only possible due to the density of our daily interactions. Humans are simple in this model (just like Carmack asserts) because they aren't properly intelligent, sapient, or conscious 100% of the time.
- seanhunter 4y agoA really good example of a smart person reasoning from completely made up assumptions to a punchy-sounding but almost-certainly wildly wrong conclusion. Just because our DNA can be efficiently encoded doesn't mean that our brain is a tiny proportion of that encoding. Your DNA doesn't change much from when you're born to when you die (random degredation aside) and yet your cognative abilities change beyond all recognition. Why is that? Well maybe there's more to what's in the brain than just what's encoded in your DNA. Secondly, how does he get to the 40Mb number? I don't think we know anywhere near enough to know how much information it would take to encode a brain, but 40Mb seems just made up. For starters, consider the amount of random stuff you can remember from your entire life. Are you saying that all can be encoded in just 40Mb? Seems very unlikely.
- bilsbie 4y agoNot AGI related but this has always bothered me. I know he’s super talented but I always wonder how many other equally talented software engineers never get noticed and toil away at crappy jobs. What’s the trick to becoming a celebrity if you’re talented?
- freediver 4y agoDo something so extraordinary that the entire world will remember you.
- moffkalast 4y agoThe prerequisites of course being in the right place at the right time, with the right people and a little luck on your side as well. So many people are remembered just because there were first at something by like a week, and the dozen others who also thought of it elsewhere but were a bit late or didn't publicize as well are forgotten forever.
- MonkeyMalarky 4y agoI was going to make a joke about being born in the right place but he's from Kansas.
- moffkalast 4y agoStill aeons ahead from being born in a slum in India.
- throwaway4aday 4y agoid wasn't first in a lot of things. They were essentially porting existing graphics technology to the PC. They succeeded because other programmers turned their noses up at the "underpowered" PC in favour of systems with "better" hardware. Carmack certainly did a lot of work and devised some ingenious ways of getting the most out of PCs but he was adapting things more than pioneering them.
- 4y ago
- someonewhocar3s 4y ago[dead]
- deleted 4y ago[deleted]
- FrustratedMonky 4y agoWhat is consciousness, is an old subject, with old arguments around old miss-understandings. People here are trying to use new analogies from computer science (a clock work universe), or with miss-understandings around quantum mechanics (somehow randomness give me agency). Until General AI needs to work for food and reproduction, everyone will still say its just mimicking humans. Best summarized by Schopenhauer. "A man can do as he wills, but not will as he wills." So if we find where the GAI comes up with the original ‘will’, we’ll just write it off as computation. Go watch some Robert Sapolsky lectures. We are just a monkey society, reacting to stimuli based on hormones and what we just ate. If you drill down far enough, sure some electrons twitched one way or the other, and yeah, if you steal something, or do something the group doesn’t like, then all the other monkeys will want to beat you up and call it justice, and dream up some logic to justify it and call it morality. And eventually the same will happen between GAI agents. Because it’s just turtles all the way down.
- tejohnso 4y ago> I made an estimate three or four years ago that I think there’s a 50-50 chance that we’ll have clear signs of life in 2030 of artificial general intelligence. That doesn’t necessarily mean a huge economic impact for anything yet, but just that we have a being that’s running on computers that most people recognize as intelligent and conscious and sort of on the same level of what we humans are doing. And after three years of hardcore research on all this, I haven’t changed my prediction. In fact, I probably even slightly bumped it up to maybe a 60% chance in 2030. And if you go up to, say, 2050, I’ve got it at like a 95% chance. > What I keep saying is that as soon as you’re at the point where you have the equivalent of a toddler—something that is a being, it’s conscious, it’s not Einstein, it can’t even do multiplication—if you’ve got a creature that can learn, you can interact with and teach it things on some level. Last I heard he wasn't interested in getting into the murky waters of consciousness. But I guess I misremembered. I'm very surprised to hear that he's very seriously talking about a conscious computer in the near future.
- trynewideas 4y agoTangentially: > North Texas’ resident tech genius, John Carmack Part of me always wonders what would've happened if the Softdisk crew that founded id Software had done it in Shreveport, or had moved to Baton Rouge or Lafayette, instead going to Texas. When Romero says they "waded across rivers" in Masters of Doom to build games, IIRC he's talking about the bridge over Cross Lake in Shreveport being washed out. The early demos and Keen prototypes were born in Louisiana. There's always been so much creative tech talent without an outlet or upward mobility across TX/LA/MO/KS/AR/AL/MS, either native to it or hired into it. The nexus of id in the Dallas area and Origin Systems in Austin made Texas an oasis for those who could get there in the 90s/00s, but even among the few people in the surrounding states with access to pre-Internet education and resources, so many couldn't afford to pack up and move even one state over. States around Texas vetoing out every incentive to incubate anything but entry-level QA centers didn't help. So many of those people either risked it all to leave, shuffled that talent into corporate work for oil/gas/finance/Wal-Mart, or didn't do anything with it at all. We know about a lot of the people who figured it out and could leave, but I guarantee there are more Romeros and Carmacks who couldn't, who are still putting in the same kinds of workloads with the same kinds of talent to figure out how to design better oil rigs or more efficiently stock Wal-Mart warehouses.
- skoczko 4y agoWith the utmost respect I have for John Carmack, this interview reads like a publicity piece for investors. Unless he already wrote the AGI and sent it to do interview for him I am not impressed.
- 93po 4y agoCarmack is not taking more funding right now. It reads like a puff investor piece because that is the target audience for the publication. He isn't trying to impress anyone. He's just being interviewed about his intentions.
- ace_of_spades 4y agoI find the tone and collective mindset expressed in this article deeply unsettling. We are talking about developing technology that is going to be the foundation and a huge challange for the longterm development of the human race and people are overtly stating that they are working on this because they perceive this to be a shot at making trillions of dollars. I mean, what the fuck, how can money be the right motivator here? This technology would change everything about what it means to be human and we do it because „we can make money“? This is so shortsighted it’s almost tragically laughable. And we as a society worship people talking this way as somekind of heropreneuers. Like anyone alive today would be able to do anything without all the people supporting our continent crossing supply chains that are crazy environmentally destructive and allround unsustainable. Sorry for this rant but come we can do better than this!
- Tepix 4y agoIt looks like we'll eventually have AGI and also detect life in space followed by contact with alien intelligence. It seems to be just a matter of time (unless humanity collapses first). Interesting times - what will happen first?