8 ms·

+1 - I work in finance, and there's no way we're sending our data and code outside the organization. We have our own H100s.

deleted 1y ago
[deleted]
filoleg 1y ago
Add big law to the list as well. There are at least a few firms here that I am just personally aware of running their models locally. In reality, I bet there are way more.
- atlasunshrugged 1y ago
  Add government here too (along with all the firms that service government customers)
  - rasmus1610 1y ago
    Add healthcare. Cannot send our patients data to a cloud provider
    - nixgeek 1y ago
      A ton of EMR systems are cloud-hosted these days. There’s already patient data for probably a billion humans in the various hyperscalers. Totally understand that approaches vary but beyond EMR there’s work to augment radiologists with computer vision to better diagnose, all sorts of cloudy things. It’s here. It’s growing. Perhaps in your jurisdiction it’s prohibited? If so I wonder for how long.
      - londons_explore 1y ago
        Even if it's possible, there is typically a lot of paperwork to get that stuff approved. There might be a lot less paperwork to just buy 50 decent GPU's and have the IT guy self-host.
      - fineIllregister 1y ago
        In the US, HIPAA requires that health care providers complete a Business Associate Agreement with any other orgs that receive PHI in the course of doing business [1]. It basically says they understand HIPAA privacy protections and will work to fulfill the contracting provider's obligations regarding notification of breaches and deletion. Obviously any EMR service will include this by default. Most orgs charge a huge premium for this. OpenAI offers it directly [2]. Some EMR providers are offering it as an add-on [3], but last I heard, it's wicked expensive. 1: https://www.hhs.gov/hipaa/for-professionals/covered-entities/sample-business-associate-agreement-provisions/index.html https://www.hhs.gov/hipaa/for-professionals/covered-entities... 2: https://help.openai.com/en/articles/8660679-how-can-i-get-a-business-associate-agreement-baa-with-openai-for-the-api-services https://help.openai.com/en/articles/8660679-how-can-i-get-a-... 3: https://www.ntst.com/carefabric/careguidance-solutions/ai-documentation-assistant https://www.ntst.com/carefabric/careguidance-solutions/ai-do...
        dragonwriter 1y ago
        > Most LLM companies might not even offer it. I'm pretty sure the LLM services of the big general-purpose cloud providers do (I know for sure that Amazon Bedrock is a HIPAA Eligible Service, meaning it is covered within their standard Business Associate Addendum [their name for the Business Associate Agreeement as part of an AWS contract].) https://aws.amazon.com/compliance/hipaa-eligible-services-reference/ https://aws.amazon.com/compliance/hipaa-eligible-services-re...
        fineIllregister 1y ago
        Sorry to edit snipe you; I realized I hadn't checked in a while so I did a search and updated my comment. It appears OpenAI, Google, and Anthropic also offer BAAs for certain LLM services.
        linuxftw 1y ago
        I worked a big health care company recently. We were using Azure's private instances of the GPT models. Fully industry compliant.
    - kakoni 1y ago
      Europe? US? In Finland doctors can send live patient encounters to azure openai for transcription and summarization.
      - filoleg 1y ago
        In the US, it would be unthinkable for a hospital to send patient data to something like ChatGPT or any other public services. Might be possible with some certain specific regions/environments of Azure tho, because iirc they have a few that support government confidentiality type of stuff, and some that tout HIPAA compliance as well. Not sure about details of those though.
Foobar8568 1y ago
Look at (private) banks in Switzerland, there are enough press release, and I can confirm most of them. Managing private clients direct data is still a concern if it can be directly linked to them. Only JB I believe have on premise infrastructure for these use cases.
helsinki 1y ago
This is not a shared sentiment across the buy side. I’m guessing you work at a bank?
LinXitoW 1y ago
Possibly stupid question, but does this apply to things like M365 too? Because just like with Inference providers, the only thing keeping them from reading/abusing your data is a pinky promise contract. Basically, isn't your data as safe/unsafe in a sharepoint folder as it is sending it to a paid inference provider?
- Bombthecat 1y ago
  Yap, companies are just paranoid, because it's new. Just like the cload back then. Sooner or later everyone will use an ai provider
  - megaloblasto 1y ago
    A lot of people and companies use local storage and compute instead of the cloud. Cloud data is leaked all the time.
undefuser 1y ago
Does it mean that renting a Bare metal server with H100s is also out of question for your org?
arkonrad 1y ago
Do you have your own platform to run inference?

deleted 1y ago

[deleted]

filoleg 1y ago

Add big law to the list as well. There are at least a few firms here that I am just personally aware of running their models locally. In reality, I bet there are way more.

atlasunshrugged 1y ago
Add government here too (along with all the firms that service government customers)
- rasmus1610 1y ago
  Add healthcare. Cannot send our patients data to a cloud provider
  - nixgeek 1y ago
    A ton of EMR systems are cloud-hosted these days. There’s already patient data for probably a billion humans in the various hyperscalers. Totally understand that approaches vary but beyond EMR there’s work to augment radiologists with computer vision to better diagnose, all sorts of cloudy things. It’s here. It’s growing. Perhaps in your jurisdiction it’s prohibited? If so I wonder for how long.
    - londons_explore 1y ago
      Even if it's possible, there is typically a lot of paperwork to get that stuff approved. There might be a lot less paperwork to just buy 50 decent GPU's and have the IT guy self-host.
    - fineIllregister 1y ago
      In the US, HIPAA requires that health care providers complete a Business Associate Agreement with any other orgs that receive PHI in the course of doing business [1]. It basically says they understand HIPAA privacy protections and will work to fulfill the contracting provider's obligations regarding notification of breaches and deletion. Obviously any EMR service will include this by default. Most orgs charge a huge premium for this. OpenAI offers it directly [2]. Some EMR providers are offering it as an add-on [3], but last I heard, it's wicked expensive. 1: https://www.hhs.gov/hipaa/for-professionals/covered-entities/sample-business-associate-agreement-provisions/index.html https://www.hhs.gov/hipaa/for-professionals/covered-entities... 2: https://help.openai.com/en/articles/8660679-how-can-i-get-a-business-associate-agreement-baa-with-openai-for-the-api-services https://help.openai.com/en/articles/8660679-how-can-i-get-a-... 3: https://www.ntst.com/carefabric/careguidance-solutions/ai-documentation-assistant https://www.ntst.com/carefabric/careguidance-solutions/ai-do...
      - dragonwriter 1y ago
        > Most LLM companies might not even offer it. I'm pretty sure the LLM services of the big general-purpose cloud providers do (I know for sure that Amazon Bedrock is a HIPAA Eligible Service, meaning it is covered within their standard Business Associate Addendum [their name for the Business Associate Agreeement as part of an AWS contract].) https://aws.amazon.com/compliance/hipaa-eligible-services-reference/ https://aws.amazon.com/compliance/hipaa-eligible-services-re...
        fineIllregister 1y ago
        Sorry to edit snipe you; I realized I hadn't checked in a while so I did a search and updated my comment. It appears OpenAI, Google, and Anthropic also offer BAAs for certain LLM services.
      - linuxftw 1y ago
        I worked a big health care company recently. We were using Azure's private instances of the GPT models. Fully industry compliant.
  - kakoni 1y ago
    Europe? US? In Finland doctors can send live patient encounters to azure openai for transcription and summarization.
    - filoleg 1y ago
      In the US, it would be unthinkable for a hospital to send patient data to something like ChatGPT or any other public services. Might be possible with some certain specific regions/environments of Azure tho, because iirc they have a few that support government confidentiality type of stuff, and some that tout HIPAA compliance as well. Not sure about details of those though.

Foobar8568 1y ago

Look at (private) banks in Switzerland, there are enough press release, and I can confirm most of them. Managing private clients direct data is still a concern if it can be directly linked to them. Only JB I believe have on premise infrastructure for these use cases.

helsinki 1y ago

This is not a shared sentiment across the buy side. I’m guessing you work at a bank?

LinXitoW 1y ago

Possibly stupid question, but does this apply to things like M365 too? Because just like with Inference providers, the only thing keeping them from reading/abusing your data is a pinky promise contract. Basically, isn't your data as safe/unsafe in a sharepoint folder as it is sending it to a paid inference provider?

Bombthecat 1y ago
Yap, companies are just paranoid, because it's new. Just like the cload back then. Sooner or later everyone will use an ai provider
- megaloblasto 1y ago
  A lot of people and companies use local storage and compute instead of the cloud. Cloud data is leaked all the time.

undefuser 1y ago

Does it mean that renting a Bare metal server with H100s is also out of question for your org?

arkonrad 1y ago

Do you have your own platform to run inference?