matt_lee

6 ms·

1.
Show HN: Auto-generate hard evaluation data for LLMs(talc.ai)
14 points by matt_lee 2y ago|1 comments
2.
by matt_lee 2y ago
When we got accepted into S23, our call started with Seibel saying something along the lines of "We like you, but we're pretty sure you're going to have to pivot." A non-trivial amount of my batch mates also joined with
3.
by matt_lee 2y ago
> Why not "why not" comments? Not why "not comments" I nearly exploded trying to grok this
4.
by matt_lee 3y ago
A few weeks back we shared a demo [1] of this on HN and got to chat with a lot of interesting folks but we weren't able to support everyone just because we're a small team. We're opening up a free developer tier of our CLI to
5.
Show HN: Talc (S23) Question and Answer Generation for AI Assistants(talc.ai)
3 points by matt_lee 3y ago|1 comments
6.
by matt_lee 3y ago
Thank you for the kind words! One of my regrets about the demo is that we paid a lot of attention to showing off our ability to generate high quality Q/A pairs, but not nearly as much to showing what a thoughtful and thorough grading r
7.
by matt_lee 3y ago
Not a pricing expert by any means, but here were some of the considerations we thought of when we made that decision: 1. There's plenty of usage based infrastructure/dev tools (ie. AWS, databricks), so I don't think we incent
8.
by matt_lee 3y ago
Thanks for giving it a whirl! I agree that the current grading is a bit harsh -- the rubric we're using in this demo is fairly rudimentary. What we've seen be more helpful is a range of grades along the lines of correct / cor
9.
by matt_lee 3y ago
Thanks! Max's cofounder chiming in here. 1) There's an interesting subtlety in the phrase "the correct question-answer pairs". While we don't often find factually incorrect pairs because of how we're running th
10.
LLMs are still bad at handling dates(talcai.substack.com)
3 points by matt_lee 3y ago|0 comments