6 ms·
One thing that evals are super important from the get go are where the harness+model inference is part of the product, e.g. if you are doing voice ai, building
by ghm2199 20d ago
One thing that evals are super important from the get go are where the harness+model inference is part of the product, e.g. if you are doing voice ai, building out a test harness to test the system is a non trivial first step.
- Urahandystar 19d agoDo you have any idea or resources as to where to look for a guide on one? Or what pitfalls to look out for?
- ghm2199 18d agoI think for voice-ai every one builds their own harness, and thats why there isn't a common one out there. Also if you are a serious company that has a llm in your production loop, you would have to build an in house thing because its so critical to your product. Its like performance-engineering, most performance engineer work is done in house and it varies wildly. Another reason is that because voice quality is a vibe measure(intonition, pitch, human variation), its impossible to make the whole thing deterministic. I use pi and built a harness for just an llm that calls a bunch of tools. That got me 50% of the way and it would be fast. Then build it for STT and TTS, this will be slower but it will get you far. There are a bunch of tools out there for building basic harnesses.