9 ms·

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstr

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are definitely a big problem. Summarizing is pretty rock solid in my testing, but reasoning is really hard. Action models, where you ask the llm to take in a user input and try to get the llm to decide what to do next, is just really hard, specifically it's hard to get the llm to understand the context and get it to say when it's not sure. That said, it's still a gamechanger that I can do it at all. 3. I am a bit more hyped than the author that this is a game changer, but like them, I don't think it's going to be the end of the world. There are some jobs that are going to be heavily impacted and I think we are going to have a rough few years of bots astroturfing platforms. But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. Similarly, certain ML tasks are now easy to implement even for dumb dumb web devs like me.

mirekrusin 2y ago
Regarding null hypothesis and negation problems - I find it personally interesting because similar fenomenon happens in our brains. Dreams, emotions, affirmations etc. process inner dialogue more less by ignoring negations and amplifying emotionally rich parts.
airstrike 2y ago
> Summarizing is pretty rock solid in my testing Yet, for some reason, ChatGPT is still pretty bad at generating titles for chats, and I didn't have better luck with the API even after trying to engineer the right prompt for quite a while... For some odd reason, once in a while I get things in different languages. It's funny when it's in a language I can speak, but I recently got "Relm4 App Yenileştirme Titizliği" which ChatGPT tells me means "Relm4 App Renewal Thoroughness" when I actually was asking it to adapt a snippet of gtk-rs code to relm4, so not particularly helpful
devdiary 2y ago
> at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call They are also dull (higher latency for same resources) APIs if you're self-hosting LLM. Special attention needed to plan the capacity.
tmpz22 2y ago
> IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. I advocate for these metaphors to help people better understand a reasonable expectation for LLMs in modern development workflows. Mostly because they show it as a trade-off versus a silver bullet. There were trade-offs to the evolution of devops, consider for example the loss of key skillsets like database administration as a direct result of "just use AWS RDS" and the explosion in cloud billing costs (especially the OpEx of startups who weren't even dealing with that much data or regional complexity!) - and how it indirectly led to Gitlabs big outage and many like it.
gopher_space 2y ago
> Summarizing is pretty rock solid in my testing, but reasoning is really hard. Asking for analogies has been interesting and surprisingly useful.
- eru 2y ago
  Could you elaborate, please?
  - gopher_space 2y ago
    Instead of `if X == Y do ...` it's more like `enumerate features of X in such a manner...` and then `explain feature #2 of X in terms that Y would understand` and then maybe `enumerate the manners in which Y might apply X#2 to TASK` and then have it do the smartest number. The most lucid explanation for SQL joins I've seen was in a (regrettably unsaved) exchange where I asked it to compare them to different parts of a construction project and then focused in on the landscaping example. I felt like Harrison Ford panning around a still image in the first Blade Runner. "Go back a point and focus in on the third paragraph".
    - eru 2y ago
      Thanks. That sounds interesting.
motoxpro 2y ago
Devops is such an amazing analogy.
lordofmoria 2y ago
OP here - I had never thought of the analogy to DevOps before, that made something click for me, and I wrote a post just now riffing off this notion: https://kenkantzer.com/gpt-is-the-heroku-of-ai https://kenkantzer.com/gpt-is-the-heroku-of-ai Basically, I think we’re using GPT as the PaaS/heroku/render equivalent of AI ops. Thank you for the insight!!
- harryp_peng 2y ago
  You only processed 500m tokens, which is shockingly little. perhaps only 2k in incurred costs?
ryoshu 2y ago
> But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. Thank you. Seeing similar things. Clients are also seeing sticker shock on how much the big models cost vs. the output. That will all come down over time.
- nineteen999 2y ago
  > That will all come down over time. So will interest, as more and more people realise theres nothing "intelligent" about the technology, it's merely a Markov-chain-word-salad generator with some weights to improve the accuracy somewhat. I'm sure some people (other than AI investors) are getting some value out of it, but I've found it to be most unsuited to most of the tasks I've applied it to.
  - mediaman 2y ago
    The industry is troubled both by hype marketers who believe LLMs are superhuman intelligence that will replace all jobs, and cynics who believe they are useless word predictors. Some workloads are well-suited to LLMs. Roughly 60% of applications are for knowledge management and summarization tasks, which is a big problem for large organizations. I have experience deploying these for customers in a niche vertical, and they work quite well. I do not believe they're yet effective for 'agentic' behavior or anything using advanced reasoning. I don't know if they will be in the near future. But as a smart, fast librarian, they're great. A related area is tier one customer service. We are beginning to see evidence that well-designed applications (emphasis on well-designed -- the LLM is just a component) can significantly bring down customer service costs. Most customer service requests do not require complex reasoning. They just need to find answers to a set of questions that are repeatedly asked, because the majority of service calls are from people who do not read docs. People who read documentation make fewer calls. In most cases around 60-70% of customer service requests are well-suited to automating with a well-designed LLM-enabled agent. The rest should be handled by humans. If the task does not require advanced reasoning and mostly involves processing existing information, LLMs can be a good fit. This actually represents a lot of work. But many tech people are skeptical, because they don't actually get much exposure to this type of work. They read the docs before calling service, are good at searching for things, and excel at using computers as tools. And so, to them, it's mystifying why LLMs could still be so valuable.
weatherlite 2y ago
> Similarly, certain ML tasks are now easy to implement even for dumb dumb web devs like me For example?
- aerhardt 2y ago
  Anything involving classification, extraction, or synthesis.
- spunker540 2y ago
  Lots of applied NLP tasks used to require paying annotators to compile a golden dataset and then train an efficient model on the dataset. Now, if cost is little concern you can use zero shot prompting on an inefficient model. If cost is a concern, you can use GPT4 to create your golden dataset way faster and cheaper than human annotations, and then train your more efficient model. Some example NLP tasks could be classifiers, sentiment, extracting data from documents. But I’d be curious which areas of NLP __weren’t__ disrupted by LLMs.
  - teleforce 2y ago
    > But I’d be curious which areas of NLP __weren’t__ disrupted by LLMs Essentially come up with a potent generic model using human feedback, label and annotation for LLM e.g GPT 4, then use it to generate golden dataset for other new models without human in the loop, very innovative indeed.
  - saaaaaam 2y ago
    I’m interested by your comment that you can “use GPT4 to create your golden dataset”. Would you be willing to expand a little and give a brief example please? It would be really helpful for me to understand this a little better!
checkyoursudo 2y ago
> get it to say when it's not sure This is a function of the language model itself. By the time you get to the output, the uncertainty that is inherent in the computation is lost to the prediction. It is like if you ask me to guess heads or tails, and I guess heads, I could have stated my uncertainty (e.g. Pr [H] = .5) before hand, but in my actual prediction of heads, and then the coin flip, that uncertainty is lost. It's the same with LLMs. The uncertainty in the computation is lost in the final prediction of the tokens, so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think), then you should not find an LLM output really ever to say it does not understand. But that is because it never understands, it just predicts.
- xiphias2 2y ago
  > so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think) Why shouldn't you ask for uncertainaty? I love asking for scores / probabilities (usually give a range, like 0.0 to 1.0) whenever I ask for a list, and it makes the output much more usable
  - dollo_7 2y ago
    I'm not sure if that is a metric you can rely on. LLMs are very sensitive to the position of your item lists along the context, paying extra attention at the beginning and the end of those list. See the listwise approach at "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting", https://arxiv.org/abs/2306.17563 https://arxiv.org/abs/2306.17563
- taneq 2y ago
  It's not just loss of the uncertainty in prediction, it's also that an LLM has zero insight into its own mental processes as a separate entity from its training data and the text it's ingested. If you ask it how sure it is, the response isn't based on its perception of its own confidence in the answer it just gave, it's based on how likely it is for an answer like that to be followed by a confident affirmation in its training data.
- 2y ago

mirekrusin 2y ago

Regarding null hypothesis and negation problems - I find it personally interesting because similar fenomenon happens in our brains. Dreams, emotions, affirmations etc. process inner dialogue more less by ignoring negations and amplifying emotionally rich parts.

airstrike 2y ago

> Summarizing is pretty rock solid in my testing Yet, for some reason, ChatGPT is still pretty bad at generating titles for chats, and I didn't have better luck with the API even after trying to engineer the right prompt for quite a while... For some odd reason, once in a while I get things in different languages. It's funny when it's in a language I can speak, but I recently got "Relm4 App Yenileştirme Titizliği" which ChatGPT tells me means "Relm4 App Renewal Thoroughness" when I actually was asking it to adapt a snippet of gtk-rs code to relm4, so not particularly helpful

devdiary 2y ago

> at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call They are also dull (higher latency for same resources) APIs if you're self-hosting LLM. Special attention needed to plan the capacity.

tmpz22 2y ago

> IMHO it's similar to what happened to DevOps in the 2000s, you just don't need a big special team to help you deploy anymore, you hire a few specialists and mostly buy off the shelf solutions. I advocate for these metaphors to help people better understand a reasonable expectation for LLMs in modern development workflows. Mostly because they show it as a trade-off versus a silver bullet. There were trade-offs to the evolution of devops, consider for example the loss of key skillsets like database administration as a direct result of "just use AWS RDS" and the explosion in cloud billing costs (especially the OpEx of startups who weren't even dealing with that much data or regional complexity!) - and how it indirectly led to Gitlabs big outage and many like it.

gopher_space 2y ago

> Summarizing is pretty rock solid in my testing, but reasoning is really hard. Asking for analogies has been interesting and surprisingly useful.

eru 2y ago
Could you elaborate, please?
- gopher_space 2y ago
  Instead of `if X == Y do ...` it's more like `enumerate features of X in such a manner...` and then `explain feature #2 of X in terms that Y would understand` and then maybe `enumerate the manners in which Y might apply X#2 to TASK` and then have it do the smartest number. The most lucid explanation for SQL joins I've seen was in a (regrettably unsaved) exchange where I asked it to compare them to different parts of a construction project and then focused in on the landscaping example. I felt like Harrison Ford panning around a still image in the first Blade Runner. "Go back a point and focus in on the third paragraph".
  - eru 2y ago
    Thanks. That sounds interesting.

motoxpro 2y ago

Devops is such an amazing analogy.

lordofmoria 2y ago

OP here - I had never thought of the analogy to DevOps before, that made something click for me, and I wrote a post just now riffing off this notion: https://kenkantzer.com/gpt-is-the-heroku-of-ai https://kenkantzer.com/gpt-is-the-heroku-of-ai Basically, I think we’re using GPT as the PaaS/heroku/render equivalent of AI ops. Thank you for the insight!!

harryp_peng 2y ago
You only processed 500m tokens, which is shockingly little. perhaps only 2k in incurred costs?

ryoshu 2y ago

> But all in all I think it's more of a force multiplier rather than a breakthrough like the internet. Thank you. Seeing similar things. Clients are also seeing sticker shock on how much the big models cost vs. the output. That will all come down over time.

nineteen999 2y ago
> That will all come down over time. So will interest, as more and more people realise theres nothing "intelligent" about the technology, it's merely a Markov-chain-word-salad generator with some weights to improve the accuracy somewhat. I'm sure some people (other than AI investors) are getting some value out of it, but I've found it to be most unsuited to most of the tasks I've applied it to.
- mediaman 2y ago
  The industry is troubled both by hype marketers who believe LLMs are superhuman intelligence that will replace all jobs, and cynics who believe they are useless word predictors. Some workloads are well-suited to LLMs. Roughly 60% of applications are for knowledge management and summarization tasks, which is a big problem for large organizations. I have experience deploying these for customers in a niche vertical, and they work quite well. I do not believe they're yet effective for 'agentic' behavior or anything using advanced reasoning. I don't know if they will be in the near future. But as a smart, fast librarian, they're great. A related area is tier one customer service. We are beginning to see evidence that well-designed applications (emphasis on well-designed -- the LLM is just a component) can significantly bring down customer service costs. Most customer service requests do not require complex reasoning. They just need to find answers to a set of questions that are repeatedly asked, because the majority of service calls are from people who do not read docs. People who read documentation make fewer calls. In most cases around 60-70% of customer service requests are well-suited to automating with a well-designed LLM-enabled agent. The rest should be handled by humans. If the task does not require advanced reasoning and mostly involves processing existing information, LLMs can be a good fit. This actually represents a lot of work. But many tech people are skeptical, because they don't actually get much exposure to this type of work. They read the docs before calling service, are good at searching for things, and excel at using computers as tools. And so, to them, it's mystifying why LLMs could still be so valuable.

weatherlite 2y ago

> Similarly, certain ML tasks are now easy to implement even for dumb dumb web devs like me For example?

aerhardt 2y ago
Anything involving classification, extraction, or synthesis.
spunker540 2y ago
Lots of applied NLP tasks used to require paying annotators to compile a golden dataset and then train an efficient model on the dataset. Now, if cost is little concern you can use zero shot prompting on an inefficient model. If cost is a concern, you can use GPT4 to create your golden dataset way faster and cheaper than human annotations, and then train your more efficient model. Some example NLP tasks could be classifiers, sentiment, extracting data from documents. But I’d be curious which areas of NLP __weren’t__ disrupted by LLMs.
- teleforce 2y ago
  > But I’d be curious which areas of NLP __weren’t__ disrupted by LLMs Essentially come up with a potent generic model using human feedback, label and annotation for LLM e.g GPT 4, then use it to generate golden dataset for other new models without human in the loop, very innovative indeed.
- saaaaaam 2y ago
  I’m interested by your comment that you can “use GPT4 to create your golden dataset”. Would you be willing to expand a little and give a brief example please? It would be really helpful for me to understand this a little better!

checkyoursudo 2y ago

> get it to say when it's not sure This is a function of the language model itself. By the time you get to the output, the uncertainty that is inherent in the computation is lost to the prediction. It is like if you ask me to guess heads or tails, and I guess heads, I could have stated my uncertainty (e.g. Pr [H] = .5) before hand, but in my actual prediction of heads, and then the coin flip, that uncertainty is lost. It's the same with LLMs. The uncertainty in the computation is lost in the final prediction of the tokens, so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think), then you should not find an LLM output really ever to say it does not understand. But that is because it never understands, it just predicts.

xiphias2 2y ago
> so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think) Why shouldn't you ask for uncertainaty? I love asking for scores / probabilities (usually give a range, like 0.0 to 1.0) whenever I ask for a list, and it makes the output much more usable
- dollo_7 2y ago
  I'm not sure if that is a metric you can rely on. LLMs are very sensitive to the position of your item lists along the context, paying extra attention at the beginning and the end of those list. See the listwise approach at "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting", https://arxiv.org/abs/2306.17563 https://arxiv.org/abs/2306.17563
taneq 2y ago
It's not just loss of the uncertainty in prediction, it's also that an LLM has zero insight into its own mental processes as a separate entity from its training data and the text it's ingested. If you ask it how sure it is, the response isn't based on its perception of its own confidence in the answer it just gave, it's based on how likely it is for an answer like that to be followed by a confident affirmation in its training data.
2y ago