5 ms·

how quick does it respond? what are specs of your laptop?

chorlton2080 26d ago
Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?
- jonplackett 26d ago
  It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that
- dominotw 26d ago
  if you are so sure about what the final shape of your output is then its prbly not a common use of ai
  - reverius42 23d ago
    If you are sure about the final shape of your output it's a great use case for AI, as you can define what you want in your prompt and refine towards it! It's where you don't know the end state you're looking for that you'll end up generating slop on top of slop and creating a whole Gastown just to power your Gastown.
Gareth321 26d ago
I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.
- alexpotato 26d ago
  Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
  - b112 26d ago
    Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. (persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
    - alexpotato 25d ago
      > the only difference is I setup a persistent drive, Going to try this out vs the snapshot! I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
  - embedding-shape 26d ago
    > Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2 https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
    - alexpotato 25d ago
      It's actually a bit less than $2 but I rounded up just to make the math easier. And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.

chorlton2080 26d ago

Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?

jonplackett 26d ago
It requires patience but it’s more like waiting 5 mins for it to do tasks. You need to be much more involved though and do things slower than Claude where you can trust it to do a lot of tasks at once. It doesn’t have the context for that
dominotw 26d ago
if you are so sure about what the final shape of your output is then its prbly not a common use of ai
- reverius42 23d ago
  If you are sure about the final shape of your output it's a great use case for AI, as you can define what you want in your prompt and refine towards it! It's where you don't know the end state you're looking for that you'll end up generating slop on top of slop and creating a whole Gastown just to power your Gastown.

Gareth321 26d ago

I tried it on my M1 MacBook Pro. It's slow but surprisingly smart as a general purpose LLM. Maybe GPT-5.3 level. I gave it a bunch of tools and it can search the internet, make product recommendations, document, code, etc.

alexpotato 26d ago
Had the same reaction so had Grok create a script to: - find a free GPU droplet on digital ocean - fire it up - pull in a snapshot of the model + extra files/packages etc - set up a ssh tunnel so that the localhost:8000 routes to the above Then I just configured OpenCode to use the above and was off to the races. Works out to be about ~$2/hr all said and done which isn't bad as I only pay when I'm using it (but could get expensive with 24/7 running)
- b112 26d ago
  Awesome, I used Claude to write a small python script to do the same with Linode's API. The only difference is I setup a persistent drive, and with Linode you can boot off of it. So my biggest start up lag is ~ 2 minutes to deploy + boot, then maybe 2 more to warm the model. I actually dislike LLMs. But I'm a realist, and on-demand compute like this is massive cost saving measure. (persistent drives are relatively cheap, compared to a box with several GPUs.. or even one. I find it worth the expense)
  - alexpotato 25d ago
    > the only difference is I setup a persistent drive, Going to try this out vs the snapshot! I also really like this experiment b/c it's a mix of LLMs and old school IaaC/DevOps.
- embedding-shape 26d ago
  > Works out to be about ~$2/hr all said What GPU you end up with for that price? Vast.ai (https://cloud.vast.ai/?priceInstanceHourlyMax=2 https://cloud.vast.ai/?priceInstanceHourlyMax=2) has a bunch of setups available to reach 192GB VRAM under $2 :) Quick skim showed 4x48, 2x96 and 8x24, all for under 2 buckaroos or around there.
  - alexpotato 25d ago
    It's actually a bit less than $2 but I rounded up just to make the math easier. And yeah, did a lot of work with Vast AI at a past job and it's pretty wild the variety of prices/hardware that they have.