8 ms·

The "spreadsheet" example video is kind of funny: guy talks about how it normally takes him 4 to 8 hours to put together complicated, data-heavy reports. Now he

• anentropic 1y ago
  yes... and arguably the last 5% is harder now because you didn't spend the time yourself to get to that point so you're not really 'up to speed' on what has been produced so far
• rvz 1y ago
  > It feels like either finding that 2% that's off (or dealing with 2% error) will be the time consuming part in a lot of cases. The last '2%' (and in some benchmarks 20%) could cost as much as $100B+ more to make it perfect consistently without error. This requirement does not apply to generating art. But for agentic tasks, errors at worst being 20% or at best being 2% for an agent may be unacceptable for mistakes. As you said, if the agent makes an error in either of the steps in an agentic flow or task, the entire result would be incorrect and you would need to check over the entire work again to spot it. Most will just throw it away and start over; wasting more tokens, money and time. And no, it is not "AGI" either.
  • deleted 1y ago
    [deleted]
• maccard 1y ago
  I’ve worked at places that sre run on spreadsheets. You’d be amazed at how often they’re wrong IME
  • pyman 1y ago
    It takes my boss seven hours to create that spreadsheet, and another eight to render a graph.
  • eboynyc32 1y ago
    Exciting stuff
  • ants_everywhere 1y ago
    There is a literature on this. The usual estimate you see is that about 2-5% of spreadsheets used for running a business contain errors.
  • davedx 1y ago
    The entire financial sector, for one thing
• apwell23 1y ago
  Lol the music and presentation made it sound like that guy was going to talk about something deep and emotional not spreadsheets and expense reports.
• travelalberta 1y ago
  I think this is my favorite part of the LLM hype train: the butterfly effect of dependence on an undependable stochastic system propagates errors up the chain until the whole system is worthless. "I think it got 98% of the information correct..." how do you know how much is correct without doing the whole thing properly yourself? The two options are: - Do the whole thing yourself to validate - Skim 40% of it, 'seems right to me', accept the slop and send it off to the next sucker to plug into his agent. I think the funny part is that humans are not exempt from similar mistakes, but a human making those mistakes again and again would get fired. Meanwhile an agent that you accept to get only 98% of things right is meeting expectations.
  • tibbar 1y ago
    This depends on the type of work being done. Sometimes the cost of verification is much lower than the cost of doing the work, sometimes it's about the same, and sometimes it's much more. Here's some recent discussion [0] [0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law https://www.jasonwei.net/blog/asymmetry-of-verification-and-...
  • groby_b 1y ago
    > how do you know how much is correct Because it's a budget. Verifying them is _much_ cheaper than finding all the entries in a giant PDF in the first place. > the butterfly effect of dependence on an undependable stochastic system We're using stochastic systems for a long time. We know just fine how to deal with them. > Meanwhile an agent that you accept to get only 98% of things right is meeting expectations. There are very few tasks humans complete at a 98% success rate either. If you think "build spreadsheet from PDF" comes anywhere close to that, you've never done that task. We're barely able to recognize objects in their default orientation at a 98% success rate. (And in many cases, deep networks outperform humans at object recognition) The task of engineering has always been to manage error rates and risk, not to achieve perfection. "butterfly effect" is a cheap rhetorical distraction, not a criticism.
    • michaelmrose 1y ago
      There are in fact lots of tasks people complete immediately at 99.99% success rate at first iteration or 99.999% after self and peer checking work Perhaps importantly checking is a continual process and errors are identified as they are made and corrected whilst in context instead of being identified later by someone completely devoid of any context a task humans are notably bad at. Lastly it's important to note the difference between a overarching task containing many sub tasks and the sub tasks. Something which fails at a sub task comprising 10 sub tasks 2% of the time per task has a miserable 18% failure rate at the overarching task. By 20 it's failed at 1 in 3 attempts worse a failing human knows they don't know the answer the failing AI produces not only wrong answers but convincing lies Failure to distinguish between human failure and AI failure in nature or degree of errors is a failure of analysis.
• ricardobayes 1y ago
  Of course, Pareto principle is at work here. In an adjacent field, self-driving, they are working on the last "20%" for almost a decade now. It feels kind of odd that almost no one is talking about self-driving now, compared to how hot of a topic it used to be, with a lot of deep, moral, almost philosophical discussions.
  • satvikpendem 1y ago
    > The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time. — Tom Cargill, Bell Labs https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule
    • stpedgwdgfhgdd 1y ago
      In my experience for enterprise software engineering, in this stage we are able to shrink the coding time with ~20%, depending on the kind of code/tests. However CICD remains tricky. In fact when AI agents start building autonomous, merge trains become a necessity…
    • baxtr 1y ago
      Love this quote. Kinda sad I didn’t see this earlier in my life.
  • danny_codes 1y ago
    It’s past the hype curve and into the trough of disillusionment. Over the next 5,10,15 years (who can say?) the tech will mature out of the trough into general adoption. GenAI is the exciting new tech currently riding the initial hype spike. This will die down into the trough of disillusionment as well, probably sometime next year. Like self-driving, people will continue to innovate in the space and the tech will be developed towards general adoption. We saw the same during crypto hype, though that could be construed as more of a snake oil type event.
    • 1y ago
• samtp 1y ago
  This is the exact same issue that I've had trying to use LLMs for anything that needs to be precise such as multi-step data pipelines. The code it produces will look correct and produce a result that seems correct. But when you do quality checks on the end data, you'll notice that things are not adding up. So then you have to dig into all this overly verbose code to identify the 3-4 subtle flaws with how it transformed/joined the data. And these flaws take as much time to identify and correct as just writing the whole pipeline yourself.
  • nemomarx 1y ago
    I think it's basically equivalent to giving that prompt to a low paid contractor coder and hoping their solution works out. At least the turnaround time is faster? But normally you would want a more hands on back and forth to ensure the requirements actually capture everything, validation and etc that the results are good, layers of reviews right
    • samtp 1y ago
      It seems to be a mix between hiring an offshore/low level contractor and playing a slot machine. And by that I mean at least with the contractor you can pretty quickly understand their limitations and see a pattern in the mistakes they make. While an LLM is obviously faster, the mistakes are seemingly random so you have to examine the result much more than you would with a contractor (if you are working on something that needs to be exact).
      • dingnuts 1y ago
        the slot machine is apt. insert tokens, pull lever, ALMOST get a reward. Think: I can start over, manually, or pull the lever again. Maybe I'll get a prize if I pull it again... and of course, you pay whether the slot machine gives a prize or not. Between the slot machine psychological effect and sunk cost fallacy I have a very hard time believing the anecdotes -- and my own experiences -- with paid LLMs. Often I say, I'd be way more willing to use and trust and pay for these things if I got my money back for output that is false.
    • 1y ago
• mclau157 1y ago
  the bigger takeaway here is will his boss allow him to walk his dog or will he see available downtime and try to fill it with more work?
  • kingnothing 1y ago
    95% of people doing his job will lose them. 1 person will figure out the 2% that requires a human in the loop.
    • fkyoureadthedoc 1y ago
      I don't know why everyone is so confident that jobs will be lost. When we invented power tools did we fire everyone that builds stuff, or did we just build more stuff?
      • skeeter2020 1y ago
        if you replace "power tools" with industrial automation it's easy to cherry pick extremes from either side. Manufacturing? a lot of jobs displaced, maybe not lost.
        • fkyoureadthedoc 1y ago
          That would be analogous to RPA maybe, and sure that has eliminated many roles. But software development and other similarly complex ever changing tasks have not been automated in the same way, and it's not even close to happening. Rote repetitive tasks with some decision making involved, probably on the chopping block.
    • satyrun 1y ago
      I do this kind of job and there is no way I am doing this job in 5-10 years. I don't even think it is my company that is going to adapt to let me go but it is going to be an AI first competitor that puts the company I work for out of business completely. There are all these massively inefficient dinosaur companies in the economy that are running digitized versions of paper shuffling and a huge number of white collar bullshit jobs built on top of digitized paper shuffling. Wage inflation has been eating away at the bottom line on all these businesses since Covid and we are going to have a dinosaur company mass extinction event in the next recession. IMO the category error being made is that LLMs are going to agentically do digitized paper shuffling and put digitized paper shufflers out of work. That is not the problem for my job. The issue is agentically from the ground up making the concept of digitized paper shuffling null and void. A relic of the past that can't compete in the economy.
• jstummbillig 1y ago
  I am looking forward to learning why this is entirely unlike working with humans, who in my experience commit very silly and unpredictable errors all the time (in addition to predictable ones), but additionally are often proud and anxious and happy to deliberately obfuscate their errors.
  • exitb 1y ago
    You can point out the errors to people, which will lead to less issues over time, as they gain experience. The models however don’t do that.
    • jstummbillig 1y ago
      I think there is a lot of confusion on this topic. Humans as employees have the same basic problem: You have to train them, and at some point they quit, and then all that experience is gone. Only: The teaching takes much longer. The retention, relative to the time it takes to teach, is probably not great (admittedly I have not done the math). A model forgets "quicker" (in human time), but can also be taught on the spot, simply by pushing necessary stuff into the ever increasing context (see claude code and multiple claude.md on how that works at any level). Experience gaining is simply not necessary, because it can infer on the spot, given you provide enough context. In both cases having good information/context is key. But here the difference is of course, that an AI is engineered to be competent and helpful as a worker, and will be consistently great and willing to ingest all of that, and a human will be a human and bring their individual human stuff and will not be very keen to tell you about all of their insecurities.
    • 8note 1y ago
      but the person doing the job changes every month or two. theres no persistent experience being built, and each newcomer to the job screws it up in their own unique way
    • closewith 1y ago
      The models do do that, just at the next iteration of the model. And everyone gains from everyone's mistakes.
• iwontberude 1y ago
  I call it a monkey's paw for this exact reason.
• LandoCalrissian 1y ago
  In the context of a budget that's really funny too. If you make a 18 trillion dollar error just once, no big deal, just one error right?
• Aurornis 1y ago
  > how it normally takes him 4 to 8 hours to put together complicated, data-heavy reports. Now he fires off an agent request, goes to walk his dog, and comes back to a downloadable spreadsheet of dense data, which he pulls up and says "I think it got 98% of the information correct... This is where the AI hype bites people. A great use of AI in this situation would be to automate the collection and checking of data. Search all of the data sources and aggregate links to them in an easy place. Use AI to search the data sources again and compare against the spreadsheet, flagging any numbers that appear to disagree. Yet the AI hype train takes this all the way to the extreme conclusion of having AI do all the work for them. The quip about 98% correct should be a red flag for anyone familiar with spreadsheets, because it’s rarely simple to identify which 2% is actually correct or incorrect without reviewing everything. This same problem extends to code. People who use AI as a force multiplier to do the thing for them and review each step as they go, while also disengaging and working manually when it’s more appropriate have much better results. The people who YOLO it with prompting cycles until the code passes tests and then submit a PR are causing problems almost as fast as they’re developing new features in non-trivial codebases.
  • ivape 1y ago
    ”The people who YOLO it with prompting cycles until the code passes tests and then submit a PR are causing problems almost as fast as they’re developing new features in non-trivial codebases.” This might as well be the new definition of “script kiddie”, and it’s the kids that are literally going to be the ones birthed into this lifestyle. The “craft” of programming may not be carried by these coming generations and possibly will need to be rediscovered at some point in the future. The Lost Art of Programming is a book that’s going to need to be written soon.
    • NortySpock 1y ago
      Oh come on, people have been writing code with bad, incomplete, flaky, or absent tests since automated testing was invented (possibly before). It's having a good, useful and reliable test suite that separates the sheep from the goats.* Would you rather play whack-a-mole with regressions and Heisenbugs, or ship features? * (Or you use some absurdly good programing language that is hard to get into knots with. I've been liking Elixir. Gleam looks even better...)
• deleted 1y ago
  [deleted]
• taf2 1y ago
  I think the question then is what's the human error rate... We know we're not perfect... So if you're 100% rested and only have to find the edge case bug, maybe you'll usually find it vs you're burned out getting it 98% of the way there and fail to see the 2% of the time bugs... Wording here is tricky to explain but I think what we'll find is this helps us get that much closer... Of course when you spend your time building out 98% of the thing you have sometimes a deeper understanding of it so finding the 2% edge case is easier/faster but only time will tell
  • sebasvisser 1y ago
    Would be insane to expect an ai to just match us right…nooooo if it pertains computers/automation/ai it needs to be beyond perfect.
    • aniforprez 1y ago
      Right? Why are we giving grace to a damn computer as if it's human? How are people defending this? If it's a computer, I don't care how intelligent it is. 98% right is actually unacceptable.
  • hiq 1y ago
    The problem with this spreadsheet task is that you don't know whether you got only 2% wrong (just rounded some numbers) or way more (e.g. did it get confused and mistook a 2023 PDF with one from 1993?), and checking things yourself is still quite tedious unless there's good support for this in the tool. At least with humans you have things like reputation (has this person been reliable) or if you did things yourself, you have some good idea of how diligent you've been.
• ncr100 1y ago
  2% wrong is $40,000 on a $2m budget.
• deleted 1y ago
  [deleted]
• thorum 1y ago
  People say this, but in my experience it’s not true. 1) The cognitive burden is much lower when the AI can correctly do 90% of the work. Yes, the remaining 10% still takes effort, but your mind has more space for it. 2) For experts who have a clear mental model of the task requirements, it’s generally less effort to fix an almost-correct solution than to invent the entire thing from scratch. The “starting cost” in mental energy to go from a blank page/empty spreadsheet to something useful is significant. (I limit this to experts because I do think you have to have a strong mental framework you can immediately slot the AI output into, in order to be able to quickly spot errors.) 3) Even when the LLM gets it totally wrong, I’ve actually had experiences where a clearly flawed output was still a useful starting point, especially when I’m tired or busy. It nerd-snipes my brain from “I need another cup of coffee before I can even begin thinking about this” to “no you idiot, that’s not how it should be done at all, do this instead…”
  • Forgeties79 1y ago
    >The cognitive burden is much lower when the AI can correctly do 90% of the work. Yes, the remaining 10% still takes effort, but your mind has more space for it. I think their point is that 10%, 1%, whatever %, the type of problem is a huge headache. In something like a complicated spreadsheet it can quickly become hours of looking for needles in the haystack, a search that wouldn't be necessary if AI didn't get it almost right. In fact it's almost better if it just gets some big chunk wholesale wrong - at least you can quickly identify the issue and do that part yourself, which you would have had to in the first place anyway. Getting something almost right, no matter how close, can often be worse than not doing it at all. Undoing/correcting mistakes can be more costly as well as labor intensive. "Measure twice cut once" and all that. I think of how in video production (edits specifically) I can get you often 90% of the way there in about half the time it takes to get it 100%. Those last bits can be exponentially more time consuming (such as an intense color grade or audio repair). The thing is with a spreadsheet like that, you can't accept a B+ or A-. If something is broken, the whole thing is broken. It needs to work more or less 100%. Closing that gap can be a huge process. I'll stop now as I can tell I'm running a bit in circles lol
    • 1y ago
• colinnordin 1y ago
  Totally agree. Also, do you really understand what the numbers in that spreadsheet mean if you have not been participating in pulling them together?
• chrisgd 1y ago
  Great point. Plus, working on your laptop on a couch is not ideal for deep excel work
• maxlin 1y ago
  The act of trying to make that 2% appear like "minimal, dismissable" is almost a mass psychosis in the AI world at times it seems like. A few comparisons: >Pressing the button: $1 >Knowing which button to press: $9,999 Those 2% copy-paste changes are the $9.999 and might take as long to find as rest of the work. Also: SCE to AUX.
  • hx8 1y ago
    I also find that validating data can be much faster than calculating data. It's like when you're in algebra class and you're told to "solve for X". Once you find the value for X you plug it into the equation to see if it fits, and it's 10x faster than solving for X originally. Regardless of if AI generates the spreadsheet or if I generate the spreadsheet, I'm still going to do the same validation steps before I share it with anyone. I might have a 2% error rate on a first draft.
• lossolo 1y ago
  I have a friend who's vibe-coding apps. He has a lot of them, like 15 or more, but most are only 60–90% complete (almost every feature is only 60-90% complete), which means almost nothing works properly. Last time he showed me something, it was sending the Supabase API key in the frontend with write permissions, so I could edit anything on his site just by inspecting the network tab in developer tools. The amount of technical debt and security issues building up over the coming years is going to be massive.
• chairmansteve 1y ago
  Yes. Any success I have had with LLMs has been by micromanaging them. Lots of very simple instructions, look at the results, correct them if necessary, then next step.
• deleted 1y ago
  [deleted]
• Fomite 1y ago
  98% correct spreadsheets are going to get so many papers retracted.
• fsndz 1y ago
  By that definition, the ChatGPT app is now an AI agent. When you use ChatGPT nowadays, you can select different models and complement these models with tools like web search and image creation. It’s no longer a simple text-in / text-out interface. It looks like it is still that, but deep down, it is something new: it is agentic… https://medium.com/thoughts-on-machine-learning/building-ai-agents-is-not-enough-bb9d8d8b6887?sk=8a77f8adddd432bdc3fcba0a821af804 https://medium.com/thoughts-on-machine-learning/building-ai-...
• guluarte 1y ago
  it now will take him 4-8hours plus a 200usd monthly bill, a win-win for everybody.
• FridgeSeal 1y ago
  It compounds too: At a certain point, relentlessly checking for whether the model has got everything is more effort in turn than…doing it. Moreover, is it actually a 4-8 hour job? Or is the person not using the right tool, is the better tool a sql query? Half these “wow ai” examples feel like “oh my plates are dirty, better just buy more”.
• dkga 1y ago
  Yes - and that is especially true for high-stakes processes in organizations. For example, accounting, HR benefits, taxation needs to be exactly right.
• mentalpiracy 1y ago
  > "I think it got 98% of the information correct... I just needed to copy / paste a few things. If it can do 90 - 95% of the time consuming work, that will save you a ton of time" "Hello, yes, I would like to pollute my entire data store" is an insane a sales pitch. Start backing up your data lakes on physical media, there is going to be an outrageous market for low-background data in the future. semi-related: How many people are going to get killed because of this?
  • vidarh 1y ago
    How often will 98% correct data actually be worse? How often will it be better? 98% might well be disastrous, but I've seen enough awful quality human-produced data that without some benchmarks I'm not confident we know whether this would be better or worse.
• positron26 1y ago
  Distinguishing whether a problem is 0.02 ^ n for error or 0.98 ^ n for accuracy is emerging as an important skill. Might explain why some people grind up a billion tokens trying to make code work only to have it get worse while others pick apart the bits of truth and quickly fill in their blind spots. The skillsets separating wheat from chaff are things like honest appreciation for corroboration, differentiating subjective from objective problems, and recognizing truth-preserving relationships. If you can find the 0.02 ^ n sub-problems, you can grind them down with AI and they will rapidly converge, leaving the 0.98 ^ n problems to focus human touch on.
• mdale 1y ago
  How well does the average employee do it? The baseline is not what you would do but what it would take to task someone to do it.
• j_timberlake 1y ago
  "It feels like either finding that 2% that's off (or dealing with 2% error) will be the time consuming part in a lot of cases." This is the part you have wrong. People just won't do that. They'll save the 8 hours and just deal with 2% error in their work (which reduces as AI models get better). This doesn't work with something with a low error tolerance, but most people aren't building the next Golden Gate Bridge. They'll just fix any problems as they crop up. Some of you will be screaming right now "THAT'S NOT WORTH IT", as if companies don't already do this to consumers constantly, like losing your luggage at the airport or getting your order wrong. Or just selling you something defective, all of that happens >2% of the time, because companies know customers will just deal-with-it.
  • camdenreslink 1y ago
    It’s not worth it because of the compounding effect when it is a repeated process. 98% accuracy might be fine for a single iteration, but if you run your process 365 times (maybe once a day for a year) whatever your output is will be so wrong that it is unusable.
    • j_timberlake 1y ago
      Can you name a single job like this? It's much easier to name jobs where the accuracy doesn't compound, like daily customer service chatbots, or personal-tutor bots, or news-aggregator bots, or the inevitable (and somewhat dubious) do-my-tax-returns bot. All I can think of is vibe-coding, and vibe-coding jobs aren't a thing.
      • econ 1y ago
        Doctors get the diagnosis wrong 10-23% of the time (depending on who you ask)
• d--b 1y ago
  I see it as a good reason why people aren’t going to lose their jobs that much. It just make people quite faster at what they’re already doing.
• eitally 1y ago
  Honestly, though, there are far more use cases where 98% correct is equivalent to perfect than situations that require absolute correctness, both in business and for personal use.
• sensanaty 1y ago
  My favorite part is people taking the 98% number to heart as if there's any basis to it whatsoever and isn't just a number they pulled out of their ass in this marketing material made by an AI company trying to sell you their AI product. In my experience it's more like a 70% for dead simple stuff, and dramatically lower for anything moderately complex. And why 98%? Why not 99% right? Or 99.9% right? I know they can't outright say 100% because everyone knows that's a blatant lie, but we're okay with them bullshitting about the 98% number here? Also there's no universe in which this guy gets to walk his dog while his little pet AI does his work for him, instead his boss is going to hound him into doing quadruple the work because he's now so "efficient" that he's finishing his spreadsheet in an hour instead of 8 or whatever. That, or he just gets fired and the underpaid (or maybe not even paid) intern shoots off the same prompt to the magic little AI and does the same shoddy work instead of him. The latter is definitely what the C-suite is aiming for with this tech anyway.
• stingraycharles 1y ago
  To be fair, this is also the case with humans: humans make errors as well, and you still need to verify the results. I once was managing a team of data scientists and my boss kept getting frustrated about some incorrectnesses she discovered, and it was really difficult to explain that this is just human error and it would take lots of resources to ensure 100% correctness. The same with code. It’s a cost / benefits balance that needs to be found. AI just adds another opportunity into this equation.
• NoboruWataya 1y ago
  Also... he "thinks" it got 98% of the data correct. How does he know?
• vonneumannstan 1y ago
  >It feels like either finding that 2% that's off (or dealing with 2% error) will be the time consuming part in a lot of cases. People act like this is some new thing but this exactly what supervising a more junior coworker is like. These models won't stay performing at Jr. levels for long. That is clear
  • LgLasagnaModel 1y ago
    It is not exactly like that. Junior coworkers mess up in fairly predictable ways.
• casperb 1y ago
  This reminds me of the story where Barclays had to buy bad assets from the Lehman bankruptcy because they only hid the rows of assets they did not want, but the receiver saw all the rows due to a mistake somewhere. The kind of 2% fault rate in Excel that could tank a big bank. https://www.computerworld.com/article/1561181/excel-error-leaves-barclays-with-more-lehman-assets-than-it-bargained-for.html https://www.computerworld.com/article/1561181/excel-error-le...