8 ms·

This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept. The only thing I take i

This is a cool project, and the idea of using LLMs to selectively extract features from open source projects is an interesting concept. The only thing I take issue with is the phrase "LiteLLM Without the Bloat." A lot of the features that have been removed (like cost tracking, streaming, caching) are... kind of the core value proposition of LiteLLM for many of their users.

OutOfHere 6d ago
LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name. Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.
- sv123 6d ago
  But imagine Sqlite not supporting joins or window functions... sure they are useful but look how many LOC it adds! Who is the arbiter of what Lite actually means?
- datadrivenangel 6d ago
  LiteLLM's problem isn't really features, it's how bloated all the features are, and specifically how AI maximalist and janky their dev practices are.
  - yujonglee 6d ago
    thanks for the feedback. genuinely curious what you think we could be doing better, especially around our dev practices. Would love to hear specifics.
    - OutOfHere 6d ago
      Didn't yall have a release of a package that stole user keys or such?
      - otabdeveloper4 6d ago
        Don't worry about it. Claude did a thorough security review of their code.
    - deleted 6d ago
      [deleted]
    - miki123211 5d ago
      Maybe this has improved since I last used it, but I was dismayed at not being able to do something like: litellm.register("foo", CustomAIProvider) litellm.do_whatever("foo/my-cool-model", "what is 2+2")
- mpyne 6d ago
  We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.
  - blazarquasar 6d ago
    It may not be a huge resource hog, but it adds a ton of latency. https://www.getmaxim.ai/bifrost/resources/benchmarks https://www.getmaxim.ai/bifrost/resources/benchmarks Having ran both LiteLLM and Bifrost for months, I can largely confirm the numbers from those benchmarks for myself.
    - mpyne 5d ago
      It may, but the latency it contributes to the end-to-end AI processing has been not noticeable in practice for our users. That's not to say Bifrost wouldn't have been better, but the choice to use LiteLLM was arrived at after a fair bit of internal discussion (most of which predated my addition to the team), and so far we've seen nothing from LiteLLM that has been contradictory to the pros/cons they thought would be the case when LiteLLM was adopted. Or in other words, the org will be happy indeed when they have solved so many of the rest of the problems we've had in AI uptake that the difference in latency between one AI gateway or the other becomes a problem to be solved.
- otabdeveloper4 6d ago
  LiteLLM is vibecoded trash.
nylonstrung 6d ago
IIRC it's 700mb with all deps included, to me that's clear bloat given you could accomplish it with a fraction of that
- raybb 6d ago
  I guess they got that message so they're porting to rust! https://docs.litellm.ai/blog/litellm-rust-launch https://docs.litellm.ai/blog/litellm-rust-launch
  - more-coffee 6d ago
    That blog post smells like Opus.
- znpy 6d ago
  We run litellm at work and not once anybody cared about those 700mb.
- geek_at 5d ago
  And let's not forget that they regularly break things. For example they had merged a pull request that was supposed to fix issues related to the openrouter/free models but in the process broke openrouter for all models except the free ones. The fix was deployed 2 weeks later (!!) to main (but you could downgrade of course). Or that other time they broke model selection if you had selected "this key can used all models of their team" than the only model in the auto-selection for harnesses was an invalid "all-team-models" entry. Fixes this one in 1 week though. all of them on the :latest docker tag btw
valcarvalho 5d ago
>> kind of the core value proposition of LiteLLM for many of their users. THIS! And it's way cheaper than others like Kong =)
taikon 5d ago
Where does it mention it selectively extract features from OSS projects?

OutOfHere 6d ago

LiteLLM doesn't quite live up to its name. With all those features, there is nothing "lite" about it. It is essential for a project to live up to its name. Imagine Sqlite adding heavy features from Postgresql, e.g. row-level security.

sv123 6d ago
But imagine Sqlite not supporting joins or window functions... sure they are useful but look how many LOC it adds! Who is the arbiter of what Lite actually means?
datadrivenangel 6d ago
LiteLLM's problem isn't really features, it's how bloated all the features are, and specifically how AI maximalist and janky their dev practices are.
- yujonglee 6d ago
  thanks for the feedback. genuinely curious what you think we could be doing better, especially around our dev practices. Would love to hear specifics.
  - OutOfHere 6d ago
    Didn't yall have a release of a package that stole user keys or such?
    - otabdeveloper4 6d ago
      Don't worry about it. Claude did a thorough security review of their code.
  - deleted 6d ago
    [deleted]
  - miki123211 5d ago
    Maybe this has improved since I last used it, but I was dismayed at not being able to do something like: litellm.register("foo", CustomAIProvider) litellm.do_whatever("foo/my-cool-model", "what is 2+2")
mpyne 6d ago
We run it at my org and it's never been a noticeable resource hog. It's actually the best performer between it, our AI observability stack and the front end.
- blazarquasar 6d ago
  It may not be a huge resource hog, but it adds a ton of latency. https://www.getmaxim.ai/bifrost/resources/benchmarks https://www.getmaxim.ai/bifrost/resources/benchmarks Having ran both LiteLLM and Bifrost for months, I can largely confirm the numbers from those benchmarks for myself.
  - mpyne 5d ago
    It may, but the latency it contributes to the end-to-end AI processing has been not noticeable in practice for our users. That's not to say Bifrost wouldn't have been better, but the choice to use LiteLLM was arrived at after a fair bit of internal discussion (most of which predated my addition to the team), and so far we've seen nothing from LiteLLM that has been contradictory to the pros/cons they thought would be the case when LiteLLM was adopted. Or in other words, the org will be happy indeed when they have solved so many of the rest of the problems we've had in AI uptake that the difference in latency between one AI gateway or the other becomes a problem to be solved.
otabdeveloper4 6d ago
LiteLLM is vibecoded trash.

nylonstrung 6d ago

IIRC it's 700mb with all deps included, to me that's clear bloat given you could accomplish it with a fraction of that

raybb 6d ago
I guess they got that message so they're porting to rust! https://docs.litellm.ai/blog/litellm-rust-launch https://docs.litellm.ai/blog/litellm-rust-launch
- more-coffee 6d ago
  That blog post smells like Opus.
znpy 6d ago
We run litellm at work and not once anybody cared about those 700mb.
geek_at 5d ago
And let's not forget that they regularly break things. For example they had merged a pull request that was supposed to fix issues related to the openrouter/free models but in the process broke openrouter for all models except the free ones. The fix was deployed 2 weeks later (!!) to main (but you could downgrade of course). Or that other time they broke model selection if you had selected "this key can used all models of their team" than the only model in the auto-selection for harnesses was an invalid "all-team-models" entry. Fixes this one in 1 week though. all of them on the :latest docker tag btw

valcarvalho 5d ago

>> kind of the core value proposition of LiteLLM for many of their users. THIS! And it's way cheaper than others like Kong =)

taikon 5d ago

Where does it mention it selectively extract features from OSS projects?