5 ms·

Price. Data privacy. Controlled latency. Plenty of reasons to not send arbitrary data to a third party service.

tycoon177 3y ago
There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.
- lolinder 3y ago
  Another side of availability is that they'll make changes to the model without warning, which alters the results of the prompts you already have written. Developing against their API is developing against a moving target.
- phillipcarter 3y ago
  FWIW, OpenAI's availability seems to have gotten significantly better since May when we launched with them. I monitor our availability Service Level Objective and we keep needing to increase the success rate because they keep improving things. This doesn't take away from high availability being a legitimate need to host your own LLM, though.
LewisVerstappen 3y ago
Can you elaborate on the pricing difference?
- alrlroipsp 3y ago
  Free vs monthly cost. What is there to elaborate on?
  - rimeice 3y ago
    It’s gona have to be hosted and run from somewhere…
  - phillipcarter 3y ago
    Hosting your own LLM is anything but free. Aside from the constant operational expense with people monitoring and fixing issues, you need to provision enough resources and run your own inference server, which is both nontrivial and likely to perform far worse than OpenAI. There's legitimate reasons to host an LLM yourself, but it's not a "make this cheaper" button.
    - kuchenbecker 3y ago
      There may be a tipping point where you're burning XXM/year in API costs and the maintenance cost of rolling your own can be justified. In the short term I agree, and one thing to consider is how rapidly the space is evolving and whether your team can even keep up with the latest advancements. However, there will come a time when the bill comes due after launch and it will be very tempting to hire people to reduce the CapEx on the API.

There's also the availability factor. OpenAI has been known to go down on occasion and without warning. If a product relies on an LLM, I wouldn't feel great about the observed uptime of OpenAI APIs.

lolinder 3y ago
Another side of availability is that they'll make changes to the model without warning, which alters the results of the prompts you already have written. Developing against their API is developing against a moving target.
phillipcarter 3y ago
FWIW, OpenAI's availability seems to have gotten significantly better since May when we launched with them. I monitor our availability Service Level Objective and we keep needing to increase the success rate because they keep improving things. This doesn't take away from high availability being a legitimate need to host your own LLM, though.

LewisVerstappen 3y ago

Can you elaborate on the pricing difference?

alrlroipsp 3y ago
Free vs monthly cost. What is there to elaborate on?
- rimeice 3y ago
  It’s gona have to be hosted and run from somewhere…
- phillipcarter 3y ago
  Hosting your own LLM is anything but free. Aside from the constant operational expense with people monitoring and fixing issues, you need to provision enough resources and run your own inference server, which is both nontrivial and likely to perform far worse than OpenAI. There's legitimate reasons to host an LLM yourself, but it's not a "make this cheaper" button.
  - kuchenbecker 3y ago
    There may be a tipping point where you're burning XXM/year in API costs and the maintenance cost of rolling your own can be justified. In the short term I agree, and one thing to consider is how rapidly the space is evolving and whether your team can even keep up with the latest advancements. However, there will come a time when the bill comes due after launch and it will be very tempting to hire people to reduce the CapEx on the API.