6 ms·

There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve

CuriouslyC 1y ago
LiteLLM is kind of a mess TBH, I guess it's ok if you just want a docker container to proxy to for personal projects, but actually using it in production isn't great.
- dlojudice 1y ago
  > but actually using it in production isn't great. I only use it in development. Could you elaborate on why you don't recommend using it in production?
- honorable_coder 1y ago
  the people behind envoy proxy built: https://github.com/katanemo/archgw https://github.com/katanemo/archgw - has the learnings of Envoy but natively designed to process/route prompts to agents and LLMs. Would be curious about your thoughts
- tom_usher 1y ago
  I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic (https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L3160 https://github.com/BerriAI/litellm/blob/main/litellm/utils.p...)
  - swyx 1y ago
    can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
    - otabdeveloper4 1y ago
      LiteLLM is the worst code I have ever read in my life. Quite an accomplishment, lol.
      - swyx 1y ago
        ok still not helpful in giving substantial criticism
        honorable_coder 1y ago
        and you say you aren't "vested" in liteLLM?
        swyx 1y ago
        yes, green text hn account, i am not. i just want help in properly identifying flaws in litellm. clearly nobody here is offering actual analysis.
        otabdeveloper4 1y ago
        Sorry if this sounds harsh, but I'm not really interested in spending time to code review the worst code I've ever seen in 30 years of programming. Is LiteLLM's code written by an LLM?
ieuanking 1y ago
we are trying to apply model-routing to academic work and pdf chat with ubik.studio -- def lmk what you think
swyx 1y ago
portkey as well which is both js and open source https://www.latent.space/p/gateway https://www.latent.space/p/gateway
- pzo 1y ago
  why provide link if there is not a single portkey keyword there?
  - swyx 1y ago
    its my interview w portkey folks which has more thoughts on the category
wongarsu 1y ago
And all of them despite 80% of model providers offering an OpenAI compatible endpoint
- troyvit 1y ago
  I think Mozilla of all people would understand why standardizing on one private organization's way of doing things might not be best for the overall ecosystem. Building a tool that meets LLM providers where they are instead of relying on them to homogenize on OpenAI's choices seems like a great reason for this project.

CuriouslyC 1y ago

LiteLLM is kind of a mess TBH, I guess it's ok if you just want a docker container to proxy to for personal projects, but actually using it in production isn't great.

dlojudice 1y ago
> but actually using it in production isn't great. I only use it in development. Could you elaborate on why you don't recommend using it in production?
honorable_coder 1y ago
the people behind envoy proxy built: https://github.com/katanemo/archgw https://github.com/katanemo/archgw - has the learnings of Envoy but natively designed to process/route prompts to agents and LLMs. Would be curious about your thoughts
tom_usher 1y ago
I definitely appreciate all the work that has gone in to LiteLLM but it doesn't take much browsing through the 7000+ line `utils.py` to see where using it could become problematic (https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L3160 https://github.com/BerriAI/litellm/blob/main/litellm/utils.p...)
- swyx 1y ago
  can you double click a little bit? many files in professional repos are 1000s of lines. LoC in it self is not a code smell.
  - otabdeveloper4 1y ago
    LiteLLM is the worst code I have ever read in my life. Quite an accomplishment, lol.
    - swyx 1y ago
      ok still not helpful in giving substantial criticism
      - honorable_coder 1y ago
        and you say you aren't "vested" in liteLLM?
        swyx 1y ago
        yes, green text hn account, i am not. i just want help in properly identifying flaws in litellm. clearly nobody here is offering actual analysis.
      - otabdeveloper4 1y ago
        Sorry if this sounds harsh, but I'm not really interested in spending time to code review the worst code I've ever seen in 30 years of programming. Is LiteLLM's code written by an LLM?

ieuanking 1y ago

we are trying to apply model-routing to academic work and pdf chat with ubik.studio -- def lmk what you think

swyx 1y ago

portkey as well which is both js and open source https://www.latent.space/p/gateway https://www.latent.space/p/gateway

pzo 1y ago
why provide link if there is not a single portkey keyword there?
- swyx 1y ago
  its my interview w portkey folks which has more thoughts on the category

wongarsu 1y ago

And all of them despite 80% of model providers offering an OpenAI compatible endpoint

troyvit 1y ago
I think Mozilla of all people would understand why standardizing on one private organization's way of doing things might not be best for the overall ecosystem. Building a tool that meets LLM providers where they are instead of relying on them to homogenize on OpenAI's choices seems like a great reason for this project.