5 ms·
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention". The pr
by Youden 16d ago
Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention".
The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.
- ljlolel 16d agoThese are all on my router TrustedRouter, and more providers coming. Tinfoil has some newer ones too like GLM 5.3 now. Phala isn't verifying all the way down but NEAR is and I know the CEO
- Youden 13d agoNEAR AI does a lot of things right but with this kind of thing, it only takes one mistake to completely break security. If you look at [0] (the code they run in the CVM), there are a couple of things that worry me: - They ship logs out of the CVM and worse, they send them to third parties (DataDog). Even if we could verify every bit of code running in the TEE, it's not enough to know the code doesn't maliciously ship prompts to a third party, we also need to audit what each binary logs. - SGLang, the core inference engine, isn't reproducibly built. We have no way to verify that the thing they call "SGLang" is what they claim it is. Really, it's the log shipping processes that worry me the most. Ideally, NEAR would minimise how much auditing needs to be done by having the minimal open-source proxy be the only thing with network access, making it much easier to audit potential exfiltration routes. [0]: https://github.com/nearai/cvm-compose-files/blob/main/prod/GLM-5.3-Flash-SGL-TP4.yaml https://github.com/nearai/cvm-compose-files/blob/main/prod/G...