8 ms·

I said the same thing to a previous company before I was let go. Confused why they were butchering their business strategy in favor of a gold rush. The main be

I said the same thing to a previous company before I was let go. Confused why they were butchering their business strategy in favor of a gold rush. The main benefit of LLMs was already abundantly clear: literally just chat with it in day to day work when you can. Ask it questions about accounting, other domains it knows, etc. That's like up to 10-20% performance increase on tasks if you align OK. Still, they were in search of a unicorn, and it was really tiring to be asked regularly how AI could help my workflows. They were not even spending a real budget on discovering "groundbreaking" use cases, meanwhile hounding us to shove a RAG-bot into every product they owned. The only thing that made sense was that it was a marketing strategy to promote visibility, but they would not acknowledge that or tell us that directly (but still--it was not their business strategy to get NEW customers).

bitcurious 2y ago
> The main benefit of LLMs was already abundantly clear In my industry the main benefit (so far) is taking all of our human-legible unstructured data and translating it into computer-legible structured data. Loving it.
- bronco21016 2y ago
  Are you able to talk more about that? I’m curious what costs are when you run this at scale. We paid a firm $60k to write a custom parser. We parse around 50,000 pages/month. The parser is 100% accurate and has near $0 continuing costs.
- rcarmo 2y ago
  How do you do quality control?
rsynnott 2y ago
> Ask it questions about accounting, other domains it knows Be very careful here if you're using it for anything important! LLMs are quite good at answering questions about accounting in ways which are superficially convincing-looking, yet also complete nonsense. "But the magic robot told me it was okay" will not fly in a tax audit, say.
- graemep 2y ago
  Exactly my immediate reaction. Accounting has to follow very strict rules and needs some application of judgement. It might answer questions in a useful way, but you have to make sure you understand the answers and that they match accounting standards or tax rules (and one danger, at least in some places, is that they are different and you might apply the wrong one).
  - cyberpunk 2y ago
    I couldn’t be arsed typing a reference number into my online banking for a bill payment the other and it was a copy protected pdf, so I fired a screenshot into Claude and GPT and asked it to extract the details I need and both of them repeatedly got the OCR wrong. I don’t trust these at all for anything apart from code which I can at least read/rewrite. It’s quite nice for unit tests I guess. And weird k8s manifests you only write now again like batch/v1 CronJob or whatever. I’m not panicking about my job just yet..
    - rsynnott 2y ago
      Erk. I’d actually kind of assumed that the likes of ChatGPT would offload OCR to, well, conventional OCR, which is, basically, a solved problem (possibly the only ‘AI’ thing which can be considered so).
    - leoedin 2y ago
      I needed to normalise a big list of dates recently. I thought maybe GPT could help. It spat out a list of normalised dates which, after a bit of careful reading, were about 95% right. How can you trust a tool that's right 95% of the time? In the end I wrote a script which handled edge cases explicitly. That took a little bit longer, but the output is deterministic. It took less time than manually cross referencing the output and input would have. I tried asking GPT to write the conversion script instead, but the script it generated just didn't deal with the edge cases. After a few rounds of increasingly specific directions which didn't seem to be helping, I gave up. I've been using copilot for development work. It has some magic moments, and it can be great for boilerplate. But then it introduces subtle bugs which are really hard to catch in review, or suggests completely incorrect function signatures and I wonder if it's adding very much at all. The biggest problem with these tools is that they turn a fun problem solving exercise into an incredibly tedious reviewing exercise. I'd much rather do it myself and understand it fully than have to review the unreliable output of an LLM. I find it much simpler to be correct than to find flaws in other peoples work. Am I missing something?
jrflowers 2y ago
> The main benefit of LLMs was already abundantly clear: literally just chat with it “It is valuable because you can talk to it” is the same idea that drove a tidal wave of sales for Furby in 1998

bitcurious 2y ago

> The main benefit of LLMs was already abundantly clear In my industry the main benefit (so far) is taking all of our human-legible unstructured data and translating it into computer-legible structured data. Loving it.

bronco21016 2y ago
Are you able to talk more about that? I’m curious what costs are when you run this at scale. We paid a firm $60k to write a custom parser. We parse around 50,000 pages/month. The parser is 100% accurate and has near $0 continuing costs.
rcarmo 2y ago
How do you do quality control?

rsynnott 2y ago

> Ask it questions about accounting, other domains it knows Be very careful here if you're using it for anything important! LLMs are quite good at answering questions about accounting in ways which are superficially convincing-looking, yet also complete nonsense. "But the magic robot told me it was okay" will not fly in a tax audit, say.

graemep 2y ago
Exactly my immediate reaction. Accounting has to follow very strict rules and needs some application of judgement. It might answer questions in a useful way, but you have to make sure you understand the answers and that they match accounting standards or tax rules (and one danger, at least in some places, is that they are different and you might apply the wrong one).
- cyberpunk 2y ago
  I couldn’t be arsed typing a reference number into my online banking for a bill payment the other and it was a copy protected pdf, so I fired a screenshot into Claude and GPT and asked it to extract the details I need and both of them repeatedly got the OCR wrong. I don’t trust these at all for anything apart from code which I can at least read/rewrite. It’s quite nice for unit tests I guess. And weird k8s manifests you only write now again like batch/v1 CronJob or whatever. I’m not panicking about my job just yet..
  - rsynnott 2y ago
    Erk. I’d actually kind of assumed that the likes of ChatGPT would offload OCR to, well, conventional OCR, which is, basically, a solved problem (possibly the only ‘AI’ thing which can be considered so).
  - leoedin 2y ago
    I needed to normalise a big list of dates recently. I thought maybe GPT could help. It spat out a list of normalised dates which, after a bit of careful reading, were about 95% right. How can you trust a tool that's right 95% of the time? In the end I wrote a script which handled edge cases explicitly. That took a little bit longer, but the output is deterministic. It took less time than manually cross referencing the output and input would have. I tried asking GPT to write the conversion script instead, but the script it generated just didn't deal with the edge cases. After a few rounds of increasingly specific directions which didn't seem to be helping, I gave up. I've been using copilot for development work. It has some magic moments, and it can be great for boilerplate. But then it introduces subtle bugs which are really hard to catch in review, or suggests completely incorrect function signatures and I wonder if it's adding very much at all. The biggest problem with these tools is that they turn a fun problem solving exercise into an incredibly tedious reviewing exercise. I'd much rather do it myself and understand it fully than have to review the unreliable output of an LLM. I find it much simpler to be correct than to find flaws in other peoples work. Am I missing something?

jrflowers 2y ago

> The main benefit of LLMs was already abundantly clear: literally just chat with it “It is valuable because you can talk to it” is the same idea that drove a tidal wave of sales for Furby in 1998