6 ms·

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running m

by ttul 15d ago

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that? If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?

wilj 15d ago
runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.
strobe 15d ago
yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).
Youden 14d ago
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention". The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
- ljlolel 14d ago
  These are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now. Phala isn't verifying all the way down but NEAR is and I know the CEO
  - Youden 11d ago
    NEAR AI does a lot of things right but with this kind of thing, it only takes one mistake to completely break security. If you look at [0] (the code they run in the CVM), there are a couple of things that worry me: - They ship logs out of the CVM and worse, they send them to third parties (DataDog). Even if we could verify every bit of code running in the TEE, it's not enough to know the code doesn't maliciously ship prompts to a third party, we also need to audit what each binary logs. - SGLang, the core inference engine, isn't reproducibly built. We have no way to verify that the thing they call "SGLang" is what they claim it is. Really, it's the log shipping processes that worry me the most. Ideally, NEAR would minimise how much auditing needs to be done by having the minimal open-source proxy be the only thing with network access, making it much easier to audit potential exfiltration routes. [0]: https://github.com/nearai/cvm-compose-files/blob/main/prod/GLM-5.3-Flash-SGL-TP4.yaml https://github.com/nearai/cvm-compose-files/blob/main/prod/G...
ljlolel 14d ago
i've got a "Router of GPUs" end-to-end encrypted: TrustedRouter.com.

by ttul 15d ago

wilj 15d ago

runpod.io is essentially this. You can rent the hardware for cheap in small time slices. I do this whenever I need to do a lot of embeddings, fast. I have an agent skill that will estimate the optimum hardware to reserve for the time/price constraints of the job, and you can spin up temporary inference for cheap via their API as well.

strobe 15d ago

yes, and it's already some offerings like that but they all cost a lot because they only good for "I have some idea of workload for N hours or days" lets rent it and run. That fine for some experimentation but if you think about renting something 24/7 even for example to share it with the friends that will cost at least 4x from any API prices as result (something like rtx 6000 48gb will cost ~$470/m).

Youden 14d ago

Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention". The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.

ljlolel 14d ago
These are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now. Phala isn't verifying all the way down but NEAR is and I know the CEO
- Youden 11d ago
  NEAR AI does a lot of things right but with this kind of thing, it only takes one mistake to completely break security. If you look at [0] (the code they run in the CVM), there are a couple of things that worry me: - They ship logs out of the CVM and worse, they send them to third parties (DataDog). Even if we could verify every bit of code running in the TEE, it's not enough to know the code doesn't maliciously ship prompts to a third party, we also need to audit what each binary logs. - SGLang, the core inference engine, isn't reproducibly built. We have no way to verify that the thing they call "SGLang" is what they claim it is. Really, it's the log shipping processes that worry me the most. Ideally, NEAR would minimise how much auditing needs to be done by having the minimal open-source proxy be the only thing with network access, making it much easier to audit potential exfiltration routes. [0]: https://github.com/nearai/cvm-compose-files/blob/main/prod/GLM-5.3-Flash-SGL-TP4.yaml https://github.com/nearai/cvm-compose-files/blob/main/prod/G...

ljlolel 14d ago

i've got a "Router of GPUs" end-to-end encrypted: TrustedRouter.com.