6 ms·

I built a low-latency AI companion that plays Skyrim with me

399 points by pantelisk 24d ago

Hilliard_Ohiooo 24d ago
Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.
- goda90 24d ago
  It's expensive, early attempts didn't impress, and many gamers are hostile to the tech because it threatens the livelihood of game creators and impacts gaming hardware prices.
- bbmatryoshka 24d ago
  at the current token prices, far too expensive, but in medium/long term I think we will see a revolution in this field, the concepts are already there (and the expensive proofs of concept already work)
- deleted 24d ago
  [deleted]
- numpad0 24d ago
  Because games are entertainment, but outputs generated by current AI are rage inducing and felt hostile, not entertaining. It just isn't past the "make it work" stage.
- pphysch 24d ago
  LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.
  - keyringlight 24d ago
    One challenge I see is that to work well and stand above 'easy' implementations the game would need to provide a rich context for the AI to exist in and interact with. Skyrim is a richly detailed world with an engine that allows that level of interaction, especially as it builds upon earlier elder scrolls games. Then there's the factor of whether commercial studios with the capacity to make that world in the first place (which has been a major draw for players) will invest further budget into AI to add more value, and if they have options to see a return on that investment via the main purchase price or additional costs.
    - igleria 24d ago
      just offer a separate subscription to host the companion model. I feel dirty typing this.
- snowram 24d ago
  It did, look at the game Suck Up or a variety of Skyrim mods. Problem is gamers are hostile to any obvious trace of IA in their games. Also, consumer hardware is overall too weak to run a convincing llm model.
- nottorp 24d ago
  It depends on if you want samey games or anything creative. You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but: One: do you really want that? There is such a thing as too much realism in a video game. Two: do you really want all NPCs in all games to speak nigerian business english? Reminder for 2: https://www.theguardian.com/technology/2024/apr/16/techscape-ai-gadgest-humane-ai-pin-chatgpt https://www.theguardian.com/technology/2024/apr/16/techscape...
  - flaunf221 24d ago
    It's a simple thing: Who wants to read AI-generated articles and respond to AI-generated comments? Probably not many. For same reason gamers mostly don't want to chat with AI-backed NPCs. More is only better when it has meaning.
    - nottorp 23d ago
      > More is only better when it has meaning. Well there are a lot of AAA gaming companies that should learn that. But instead they feel they have to ad MOAR CONTENT and then complain of balooning costs...
- brightstep 23d ago
  I'm especially interested in how this can be applied to the world of the game. I think that creates much more interesting results than NPCs who can hold dynamic conversation. Imagine an MMO where quests you complete actually shape the world (its landscape, structures and enemies). Then dynamic quests can be created based on the new world, creating a living loop.
- utopiah 23d ago
  IMHO at least 2 reasons, the jagged frontier (you cant easily evaluate the capabilities of your companion, I expanded a bit on https://news.ycombinator.com/item?id=49419952 https://news.ycombinator.com/item?id=49419952 ) and probably riskier in a lot of story games the risk of spoilers if the model isn't sandboxed properly, which I don't think is trivial to do as you don't know the training set which probably includes games and books games might be based on.
- kdkdjfjejd 23d ago
  Why would anyone want to have an LLM generate content in a game when people can barely stomach it generating awful pictures and generic prose? Who would want to have a non-deterministic, unreliable, out-of-your-control companion in a single player game? What kind of game designer would find it acceptable to have no control over large swathes of their own game? Who would like to play a game that no one really created? Games are where art meets engineering to create entertainment and awe. LLMs don’t fit.
rao-v 24d ago
I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though. If nothing else - this is how NPCs should work in games moving forward!
- pantelisk 24d ago
  Hello, author here. I was intentionally a little vague about this because this is kind of the thing that makes the whole approach work. An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening. This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago. ---- In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little). We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning... And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions. As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?" It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?
  - alwaysanoobie 24d ago
    What I don't understand is how are you passing the world state to the model? Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?
oscarmcdougall 24d ago
Love the concept of void mode. A companion that follows you across games, that's cool.
seabombs 24d ago
I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.
- MattDamonSpace 24d ago
  New mediums always absolutely knock the humanity organism for a bit
- deleted 23d ago
  [deleted]
- utopiah 23d ago
  I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an interesting exploration but the jagged frontier makes it really challenging to know what will consistently work and what will not, to the point that I bet (maybe pessimistically) one will be gradually less daring with creative plan with their "companion" simply because they can't trust it.
  - SamBam 23d ago
    I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do? The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.
usernomdeguerre 24d ago
>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it) I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old. unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?
- pantelisk 24d ago
  I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently. Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
- Zylokloto 24d ago
  As soon as the AI progress 'slows down' a little bit and its a little bit more clear what makes sense of converting into a hardware model, there will be for sure something. It either might be some neural engine like what Apple is doing and tensor units like what high end android phones already have or something dedicated. Based on the current capacity issues around the globe, perhaps 1-5 years? But we have for sure crossed a price point were you just might buy 10 ai credits and will be able to just play 100h without running ot of your ai credits. basic conversations etc. finetuned for a game, doesn't need a frontier model.
- giancarlostoro 24d ago
  > I do wonder if this is an avenue for console gaming that might be practical in a few years; If the AI Bubble bursts a little and the inflated prices for tech dwindle down to normal maybe, otherwise, it will be too expensive. Or some other advancement that challenges the giants financially but incentivizes companies to build for local AI usage.
walrus01 24d ago
I guess this means you can fus ro dah the LLM off a mountain, so there is that.
Evidlo 24d ago
The voice reminds me of Marcus the worm from VRChat https://www.youtube.com/watch?v=yAEMVVtnUqM https://www.youtube.com/watch?v=yAEMVVtnUqM
Havoc 24d ago
Wow that demo video is way better than expected. Well done
panki27 24d ago
Have you seen Mantella yet? https://art-from-the-machine.github.io/Mantella/ https://art-from-the-machine.github.io/Mantella/ tl;dr: let's you talk to all NPCs via LLM. Last time I tried it, latency was too much for it to be enjoyable, your approach seems to be much faster.
aussieguy1234 24d ago
Next up: A smart glasses companion that follows you around in the game known as life. Ive seen something like this in sci fi films.
- dosenbrot 24d ago
  Dennou Coil had this exact same concept.
- kdkdjfjejd 23d ago
  Hopefully it’s a copper arresting you for walking around pointing cameras at people’s faces without permission.
mihau 24d ago
I expected slop, but this is really good! (both demo and write-up)
mike_hearn 24d ago
This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game. What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.
- pantelisk 23d ago
  This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight. The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)
  - ajcp 23d ago
    Indeed, The Bitter Lesson comes at us quicker and quicker now
toilet 24d ago
Great concept and presentation. How much knowledge about the world does the agent have? Does Skyrim accurately model what their in game character can actually see and hear?
big_dave212 24d ago
awesome idea and implementation. feels like a game feature that you didn't know that you wanted.
nicedaylucas 24d ago
[flagged]
Xx_crazy420_xX 24d ago
Very promising project! The latency is great and looks pretty polished for solo work. > ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you > It creates embeddings from the full text as well as its extracted structure I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?
voidUpdate 24d ago
Does it have any agency of its own, or is it entirely reliant on you commanding it? > "Maybe it's because Varkos is a dog, and who doesn't like dogs" Me
- pantelisk 24d ago
  Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again. Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise. After play testing it I had to tone it down quite a bit as it caused continuity issues... In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.
  - debo_ 23d ago
    There are players in my D&D groups that act very similar to your bored dog companion.
crspatel 24d ago
[flagged]
igleria 24d ago
I imagine playing fallout with this and.. please keep at it!
Jach 24d ago
Nice demo, I'm always impressed by making these things low latency enough to be actually interactive. Were you inspired at all by the autonomous vtuber Neuro-sama recently getting some in-game integrations to allow her to play Skyrim on her own? (https://www.youtube.com/watch?v=3o7ORYqLrRw https://www.youtube.com/watch?v=3o7ORYqLrRw shows the highlights from the first stream.)
jhfdbkofdchk 23d ago
An 80% good enough solution would be to have it periodically say "Hey, you. You're finally awake."
hombre_fatal 23d ago
Wow, what a great demo video. And this is the worst this tech will ever be. Though maybe nobody will ever beat the dog's hilarious persona.
fsnovask 23d ago
>It creates embeddings from the full text as well as its extracted structure, then semantically combines and compares it with action prototypes. A separate classifier estimates whether the turn is a command, question, chat, clarification or complex request. Everything gets merged together. Does this mean the prototypes and classifications need to follow what you can actually do in the game? Were they all hand-coded or generated somehow?
arkensaw 23d ago
Pretty neat but a talking dog is so weird, especially one that sounds like Stellan Skaarsgard Would be better if he just barked
halflife 23d ago
This is excellent, I really want to connect this with fallout community edition. How is the personality evolution saved? Plain text?
csh0 23d ago
This is a fantastic application of an LLM. What’s great is that the LLMisms fold neatly into “dumb but lovable sidekick”. So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even.
khalic 23d ago
hahahaha that's awesome, good job man
Linekiller89 23d ago
[dead]
danielvaughn 23d ago
Some variant of this idea has played in the back of my mind over the last few years. I'd love to play Skyrim, but where each character builds their own context for their history, their personality, their goals, and their interactions with you. The world would feel incredibly alive.
robertoconnor 23d ago
[dead]
p0w3n3d 23d ago
The voice of the companion is awesome. I'm surprised how good the talk is (i.e. the voice recognition). My disabled brother is struggling with good native polish voice to text btw. If you people can recommend anything good or local
- jam 23d ago
  I'm really enjoying https://handy.computer/ https://handy.computer/ Runs local and has several models that should support Polish, though I can not personally validate this.
  - vekker 22d ago
    Can confirm this is a really excellent piece of OSS for local STT, I use it in both English and Dutch. The only thing it has trouble with are brand/product/tooling/person names. You can add custom vocabulary, but then it tends to over-correct and insert those custom words when I never said them. However this isn't unique to local STT, frontier cloud models also struggle with this.
streetsandsnaps 23d ago
This is hilariously delightful, thank you for this during my morning coffee.
vee-kay 18d ago
[dead]

399 points by pantelisk 24d ago

Hilliard_Ohiooo 24d ago

Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.

goda90 24d ago
It's expensive, early attempts didn't impress, and many gamers are hostile to the tech because it threatens the livelihood of game creators and impacts gaming hardware prices.
bbmatryoshka 24d ago
at the current token prices, far too expensive, but in medium/long term I think we will see a revolution in this field, the concepts are already there (and the expensive proofs of concept already work)
deleted 24d ago
[deleted]
numpad0 24d ago
Because games are entertainment, but outputs generated by current AI are rage inducing and felt hostile, not entertaining. It just isn't past the "make it work" stage.
pphysch 24d ago
LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.
- keyringlight 24d ago
  One challenge I see is that to work well and stand above 'easy' implementations the game would need to provide a rich context for the AI to exist in and interact with. Skyrim is a richly detailed world with an engine that allows that level of interaction, especially as it builds upon earlier elder scrolls games. Then there's the factor of whether commercial studios with the capacity to make that world in the first place (which has been a major draw for players) will invest further budget into AI to add more value, and if they have options to see a return on that investment via the main purchase price or additional costs.
  - igleria 24d ago
    just offer a separate subscription to host the companion model. I feel dirty typing this.
snowram 24d ago
It did, look at the game Suck Up or a variety of Skyrim mods. Problem is gamers are hostile to any obvious trace of IA in their games. Also, consumer hardware is overall too weak to run a convincing llm model.
nottorp 24d ago
It depends on if you want samey games or anything creative. You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but: One: do you really want that? There is such a thing as too much realism in a video game. Two: do you really want all NPCs in all games to speak nigerian business english? Reminder for 2: https://www.theguardian.com/technology/2024/apr/16/techscape-ai-gadgest-humane-ai-pin-chatgpt https://www.theguardian.com/technology/2024/apr/16/techscape...
- flaunf221 24d ago
  It's a simple thing: Who wants to read AI-generated articles and respond to AI-generated comments? Probably not many. For same reason gamers mostly don't want to chat with AI-backed NPCs. More is only better when it has meaning.
  - nottorp 23d ago
    > More is only better when it has meaning. Well there are a lot of AAA gaming companies that should learn that. But instead they feel they have to ad MOAR CONTENT and then complain of balooning costs...
brightstep 23d ago
I'm especially interested in how this can be applied to the world of the game. I think that creates much more interesting results than NPCs who can hold dynamic conversation. Imagine an MMO where quests you complete actually shape the world (its landscape, structures and enemies). Then dynamic quests can be created based on the new world, creating a living loop.
utopiah 23d ago
IMHO at least 2 reasons, the jagged frontier (you cant easily evaluate the capabilities of your companion, I expanded a bit on https://news.ycombinator.com/item?id=49419952 https://news.ycombinator.com/item?id=49419952 ) and probably riskier in a lot of story games the risk of spoilers if the model isn't sandboxed properly, which I don't think is trivial to do as you don't know the training set which probably includes games and books games might be based on.
kdkdjfjejd 23d ago
Why would anyone want to have an LLM generate content in a game when people can barely stomach it generating awful pictures and generic prose? Who would want to have a non-deterministic, unreliable, out-of-your-control companion in a single player game? What kind of game designer would find it acceptable to have no control over large swathes of their own game? Who would like to play a game that no one really created? Games are where art meets engineering to create entertainment and awe. LLMs don’t fit.

rao-v 24d ago

I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though. If nothing else - this is how NPCs should work in games moving forward!

pantelisk 24d ago
Hello, author here. I was intentionally a little vague about this because this is kind of the thing that makes the whole approach work. An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening. This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago. ---- In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little). We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning... And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions. As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?" It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?
- alwaysanoobie 24d ago
  What I don't understand is how are you passing the world state to the model? Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?

oscarmcdougall 24d ago

Love the concept of void mode. A companion that follows you across games, that's cool.

seabombs 24d ago

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as the player is attacked by a wolf ("F--- dude you could have warned me!") was great comedy.

MattDamonSpace 24d ago
New mediums always absolutely knock the humanity organism for a bit
deleted 23d ago
[deleted]
utopiah 23d ago
I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an interesting exploration but the jagged frontier makes it really challenging to know what will consistently work and what will not, to the point that I bet (maybe pessimistically) one will be gradually less daring with creative plan with their "companion" simply because they can't trust it.
- SamBam 23d ago
  I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do? The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

usernomdeguerre 24d ago

>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it) I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old. unfortunate that the "ALE" design wasn't opensourced (couldn't find a link in their post) but I would be interested in learning more about the design, in particular what sort of data pipeline was necessary from skyrim to give this sort of action flexibility?

pantelisk 24d ago
I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently. Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).
Zylokloto 24d ago
As soon as the AI progress 'slows down' a little bit and its a little bit more clear what makes sense of converting into a hardware model, there will be for sure something. It either might be some neural engine like what Apple is doing and tensor units like what high end android phones already have or something dedicated. Based on the current capacity issues around the globe, perhaps 1-5 years? But we have for sure crossed a price point were you just might buy 10 ai credits and will be able to just play 100h without running ot of your ai credits. basic conversations etc. finetuned for a game, doesn't need a frontier model.
giancarlostoro 24d ago
> I do wonder if this is an avenue for console gaming that might be practical in a few years; If the AI Bubble bursts a little and the inflated prices for tech dwindle down to normal maybe, otherwise, it will be too expensive. Or some other advancement that challenges the giants financially but incentivizes companies to build for local AI usage.

walrus01 24d ago

I guess this means you can fus ro dah the LLM off a mountain, so there is that.

Evidlo 24d ago

The voice reminds me of Marcus the worm from VRChat https://www.youtube.com/watch?v=yAEMVVtnUqM https://www.youtube.com/watch?v=yAEMVVtnUqM

Havoc 24d ago

Wow that demo video is way better than expected. Well done

panki27 24d ago

Have you seen Mantella yet? https://art-from-the-machine.github.io/Mantella/ https://art-from-the-machine.github.io/Mantella/ tl;dr: let's you talk to all NPCs via LLM. Last time I tried it, latency was too much for it to be enjoyable, your approach seems to be much faster.

aussieguy1234 24d ago

Next up: A smart glasses companion that follows you around in the game known as life. Ive seen something like this in sci fi films.

dosenbrot 24d ago
Dennou Coil had this exact same concept.
kdkdjfjejd 23d ago
Hopefully it’s a copper arresting you for walking around pointing cameras at people’s faces without permission.

mihau 24d ago

I expected slop, but this is really good! (both demo and write-up)

mike_hearn 24d ago

This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools asynchronously while speaking which would be ideal for a game. What you'd want is maybe some kind of Live model with voice warping so it can be given different Skyrim themed 'Nordic' voices, and then custom tools to interact with the game engine.

pantelisk 23d ago
This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight. The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)
- ajcp 23d ago
  Indeed, The Bitter Lesson comes at us quicker and quicker now

toilet 24d ago

Great concept and presentation. How much knowledge about the world does the agent have? Does Skyrim accurately model what their in game character can actually see and hear?

big_dave212 24d ago

awesome idea and implementation. feels like a game feature that you didn't know that you wanted.

nicedaylucas 24d ago

[flagged]

Xx_crazy420_xX 24d ago

Very promising project! The latency is great and looks pretty polished for solo work. > ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you > It creates embeddings from the full text as well as its extracted structure I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?

voidUpdate 24d ago

Does it have any agency of its own, or is it entirely reliant on you commanding it? > "Maybe it's because Varkos is a dog, and who doesn't like dogs" Me

pantelisk 24d ago
Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again. Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise. After play testing it I had to tone it down quite a bit as it caused continuity issues... In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.
- debo_ 23d ago
  There are players in my D&D groups that act very similar to your bored dog companion.

crspatel 24d ago

[flagged]

igleria 24d ago

I imagine playing fallout with this and.. please keep at it!

Jach 24d ago

Nice demo, I'm always impressed by making these things low latency enough to be actually interactive. Were you inspired at all by the autonomous vtuber Neuro-sama recently getting some in-game integrations to allow her to play Skyrim on her own? (https://www.youtube.com/watch?v=3o7ORYqLrRw https://www.youtube.com/watch?v=3o7ORYqLrRw shows the highlights from the first stream.)

jhfdbkofdchk 23d ago

An 80% good enough solution would be to have it periodically say "Hey, you. You're finally awake."

hombre_fatal 23d ago

Wow, what a great demo video. And this is the worst this tech will ever be. Though maybe nobody will ever beat the dog's hilarious persona.

fsnovask 23d ago

>It creates embeddings from the full text as well as its extracted structure, then semantically combines and compares it with action prototypes. A separate classifier estimates whether the turn is a command, question, chat, clarification or complex request. Everything gets merged together. Does this mean the prototypes and classifications need to follow what you can actually do in the game? Were they all hand-coded or generated somehow?

arkensaw 23d ago

Pretty neat but a talking dog is so weird, especially one that sounds like Stellan Skaarsgard Would be better if he just barked

halflife 23d ago

This is excellent, I really want to connect this with fallout community edition. How is the personality evolution saved? Plain text?

csh0 23d ago

This is a fantastic application of an LLM. What’s great is that the LLMisms fold neatly into “dumb but lovable sidekick”. So even when it chokes or stumbles on a command, the kind of frustration the user expresses when correcting it feels natural and part of the game even.

khalic 23d ago

hahahaha that's awesome, good job man

Linekiller89 23d ago

[dead]

danielvaughn 23d ago

Some variant of this idea has played in the back of my mind over the last few years. I'd love to play Skyrim, but where each character builds their own context for their history, their personality, their goals, and their interactions with you. The world would feel incredibly alive.

robertoconnor 23d ago

[dead]

p0w3n3d 23d ago

The voice of the companion is awesome. I'm surprised how good the talk is (i.e. the voice recognition). My disabled brother is struggling with good native polish voice to text btw. If you people can recommend anything good or local

jam 23d ago
I'm really enjoying https://handy.computer/ https://handy.computer/ Runs local and has several models that should support Polish, though I can not personally validate this.
- vekker 22d ago
  Can confirm this is a really excellent piece of OSS for local STT, I use it in both English and Dutch. The only thing it has trouble with are brand/product/tooling/person names. You can add custom vocabulary, but then it tends to over-correct and insert those custom words when I never said them. However this isn't unique to local STT, frontier cloud models also struggle with this.

streetsandsnaps 23d ago

This is hilariously delightful, thank you for this during my morning coffee.

vee-kay 18d ago

[dead]