6 ms·

Fun interview but barely any meat for those in the field. Just very general questions and answers, saying that the road is murky, but nothing about e.g. if Tran

Fun interview but barely any meat for those in the field. Just very general questions and answers, saying that the road is murky, but nothing about e.g. if Transformers / attention are the way to go forward, multi-modal models, reinforcement learning + self-supervised learning.

modeless 4y ago
There is a comment in the article about having models watch TV and play video games, and he's talked about that before too in his Lex Friedman interview. Seems like his approach is to take existing model architectures, apply some tweaks and experimental ideas and then use datasets consisting of TV (self supervised learning maybe?) and classic video games (RL I guess?). The video game part at least sounds like what Deepmind is already doing. I guess we'll just have to wait and see what he plans to do differently. It seems to me like his expertise would be most valuable in optimizing model architectures for hardware capabilities to improve utilization and training efficiency. That will be important for AGI especially as the cost of training models skyrockets (both time and money). If I was a startup doing AI hardware like Cerebras or Graphcore I would definitely try to hire Carmack to help with my software stack. Though he doesn't seem interested in custom AI hardware.
arcturus17 4y ago
I do not question for a second that Carmack is a computing genius, but “murky” is an understatement. There is nothing of technical, scientific or business value in this interview. It’s just generic words strung together.
- SideburnsOfDoom 4y ago
  Even socially, it fails to plumb the depths. Mr Carmack is taking a "different path" by ... reading the relevant literature and talking to YC? This is 100% mainstream - The difference is what?
- GuB-42 4y ago
  Plot twist, John Carmack's AI is answering the interview.
- mrguyorama 4y ago
  Why do people think he is a computing genius? He read contemporary papers in computer 3D rendering and implemented well understood concepts in a shitty dev environment (early x86 and VGA systems). When I do that for my company, I'm just a junior dev. People really need to stop with this "Great person" nonsense. He's a pretty smart coder, and is gifted with geometry and other fields of math. He's not a genius. He didn't "master" calculus at age 15 like Einstein, he didn't invent anything particularly new in the field. Why the obsession people have with him? Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
  - arcturus17 4y ago
    Ok!
  - 93po 4y ago
    > Why should we look to him for AI questions? What evidence is there that he has any new knowledge? He covers this in the article. He doesn't. He's just trying stuff out with a different approach than others because he believes (and is probably correct) that there is a chance that the most efficient path forward to AGI isn't the work that OpenAI and others are doing.
laluser 4y ago
I imagine this is a 10 year journey for him and he’s just getting started. Check back in 5 years.
jdlshore 4y ago
It’s a puff piece in a general interest magazine. It’s not going to go into details. I also got the impression that Carmack was being cagey about the directions where he saw potential.
SideburnsOfDoom 4y ago
Yep. Does it ever get around to answering the question implicit in the title: what is it that he is going to do "different" to the rest? Seems more like he's talking to and following up with Altman, "Y Combinator conference" and the rest. Is that "bucking the trend", taking your own "path", really?
- JKCalhoun 4y ago
  I assume it's to know which directions have shown the most recent potential, to catch up on the techniques and literature so you can talk/think intelligently on the matter. But I see your more general point, he has to be careful not to get caught up I the groupthink he is (understandably?) critical of.
- kelipso 4y ago
  Lol I feel like this is going to be a high school valedictorian goes on to be an average student in Stanford kind if story.
  - 93po 4y ago
    The area of AI research isn't some outlier of super geniuses that don't exist anywhere else. Carmack has often worked around some of the best talent on the planet and he has always stood out.

modeless 4y ago

There is a comment in the article about having models watch TV and play video games, and he's talked about that before too in his Lex Friedman interview. Seems like his approach is to take existing model architectures, apply some tweaks and experimental ideas and then use datasets consisting of TV (self supervised learning maybe?) and classic video games (RL I guess?). The video game part at least sounds like what Deepmind is already doing. I guess we'll just have to wait and see what he plans to do differently. It seems to me like his expertise would be most valuable in optimizing model architectures for hardware capabilities to improve utilization and training efficiency. That will be important for AGI especially as the cost of training models skyrockets (both time and money). If I was a startup doing AI hardware like Cerebras or Graphcore I would definitely try to hire Carmack to help with my software stack. Though he doesn't seem interested in custom AI hardware.

arcturus17 4y ago

I do not question for a second that Carmack is a computing genius, but “murky” is an understatement. There is nothing of technical, scientific or business value in this interview. It’s just generic words strung together.

SideburnsOfDoom 4y ago
Even socially, it fails to plumb the depths. Mr Carmack is taking a "different path" by ... reading the relevant literature and talking to YC? This is 100% mainstream - The difference is what?
GuB-42 4y ago
Plot twist, John Carmack's AI is answering the interview.
mrguyorama 4y ago
Why do people think he is a computing genius? He read contemporary papers in computer 3D rendering and implemented well understood concepts in a shitty dev environment (early x86 and VGA systems). When I do that for my company, I'm just a junior dev. People really need to stop with this "Great person" nonsense. He's a pretty smart coder, and is gifted with geometry and other fields of math. He's not a genius. He didn't "master" calculus at age 15 like Einstein, he didn't invent anything particularly new in the field. Why the obsession people have with him? Why should we look to him for AI questions? What evidence is there that he has any new knowledge?
- arcturus17 4y ago
  Ok!
- 93po 4y ago
  > Why should we look to him for AI questions? What evidence is there that he has any new knowledge? He covers this in the article. He doesn't. He's just trying stuff out with a different approach than others because he believes (and is probably correct) that there is a chance that the most efficient path forward to AGI isn't the work that OpenAI and others are doing.

laluser 4y ago

I imagine this is a 10 year journey for him and he’s just getting started. Check back in 5 years.

jdlshore 4y ago

It’s a puff piece in a general interest magazine. It’s not going to go into details. I also got the impression that Carmack was being cagey about the directions where he saw potential.

SideburnsOfDoom 4y ago

Yep. Does it ever get around to answering the question implicit in the title: what is it that he is going to do "different" to the rest? Seems more like he's talking to and following up with Altman, "Y Combinator conference" and the rest. Is that "bucking the trend", taking your own "path", really?

JKCalhoun 4y ago
I assume it's to know which directions have shown the most recent potential, to catch up on the techniques and literature so you can talk/think intelligently on the matter. But I see your more general point, he has to be careful not to get caught up I the groupthink he is (understandably?) critical of.
kelipso 4y ago
Lol I feel like this is going to be a high school valedictorian goes on to be an average student in Stanford kind if story.
- 93po 4y ago
  The area of AI research isn't some outlier of super geniuses that don't exist anywhere else. Carmack has often worked around some of the best talent on the planet and he has always stood out.