8 ms·
It sounds like you don’t need immediate llm responses and can batch process your data nightly? Have you considered running a local llm? May not need to pay for
by LTL_FTC 8mo ago
It sounds like you don’t need immediate llm responses and can batch process your data nightly? Have you considered running a local llm? May not need to pay for api calls. Today’s local models are quite good. I started off with cpu and even that was fine for my pipelines.
- queenkjuul 8mo agoAgreed, I'm pretty amazed at what I'm able to do locally just with an AMD 6700XT and 32GB of RAM. It's slow, but if you've got all night...
- kreetx 8mo agoThough haven't done any extensive testing then I personally could easily get by with current local models. The only reason I don't is that the hosted ones all have free tiers.
- ydu1a2fovb 8mo agoCan you suggest any good llms for cpu?
- R_D_Olivaw 8mo agoFollowing.
- Aerbil313 8mo agoHey Olivaw, saw a comment of yours asking about planners. Wanted to reply but it’s expired. Check out bullet journalling.
- R_D_Olivaw 8mo agoThanks for the reply! Bullet journaling is neat, but I'm far too whacky with my notes to stick to that kind of structure. I have various other structures I implement, but they're just hodge podges of things.
- LTL_FTC 8mo agoI started off using gpt-oss-120b on cpu. It uses about 60-65gb of memory or so but my workstation has 128gb of ram. If I had less ram, I would start off with the gpt-oss-20b model and go from there. Look for MoE models as they are more efficient to run.
- LTL_FTC 8mo agoI started off using gpt-oss-120b on cpu. It uses about 60-65gb of memory or so but my workstation has 128gb of ram. If I had less ram, I would start off with the gpt-oss-20b model and go from there. Look for MoE models as they are more efficient to run.
- ok_orco 8mo agoI haven't thought about that, but really want to dig in more now. Any places you recommend starting?
- LTL_FTC 8mo agoI started off using gpt-oss-120b on cpu. It uses about 60-65gb of memory or so but my workstation has 128gb of ram. If I had less ram, I would start off with the gpt-oss-20b model and go from there. Look for MoE models as they are more efficient to run. My old threadripper pro was seeing about 15tps, which was quite acceptable for the background tasks I was running.