7 ms·
Remember in 2023 when people thought "prompt engineering" would be the new software engineering and invested tons of time into learning CoT, ReAct, thread-of-th
by cloakandswagger 2mo ago
Remember in 2023 when people thought "prompt engineering" would be the new software engineering and invested tons of time into learning CoT, ReAct, thread-of-thoughts, etc?
Those were mostly obviated by reasoning models and harness updates by 2024.
It seems pointless to invest energy into the latest/greatest AI technique or framework when they're going to either be absorbed or replaced on a 3 month cycle.
- embedding-shape 2mo agoIsn't it clear that some people are better at working with/prompting LLMs than other people? Or is the idea that what you write to them and how you use them doesn't matter, it's all up to the model/harness? To me this seems clear, so then clearly this is a skill, which typically is called "prompt engineering". Specifically CoT or the other things you mention wasn't referred to as "prompt engineering" as far as I know, that skill is more about how you communicate with the LLMs and how you use them, rather than what specific processes/workflows/technologies you use.
- omega3 2mo agoWhat's clear is that there is a lot of hype around LLM and people who were previously valued for their IC are now in the business of shilling.
- CuriouslyC 2mo agoRL has basically killed prompt engineering. You still need to provide the right context and process, but how you communicate with them beyond that is no longer so important.
- sjh9714 2mo ago[flagged]
- saberience 2mo agoIt's not really a skill. The models are at this point smarter than you are, so the idea that you can prompt them "better" is laughable really when discussing frontier models. It's like imagining you could "prompt" Richard Feynman to be smarter at Physics. That is, for 99.9% of engineers, if you want the model to do a code review of your project, the best solution is to just ask Fable, "Hey Fable, do a code review of this project." Throwing in extra text like "think like a senior engineer", "ensure you focus on DRY principles, KISS, self documenting code, etc", doesn't make a difference. These sorts of tricks used to work with dumber models, but now, like I said before, it's like thinking you can prompt Linus Torvalds into writing better C than he already can do.
- sjh9714 2mo ago[flagged]
- defrost 2mo agoFWiW > it's like thinking you can prompt Linus Torvalds into writing better C++ than he already can do. Linus Torvalds, the inventor (and beloved dictator) of Linux, has always been quite harsh about C++ and why he rejects it for Linux kernel development. He’s not just been very vocal about it, but also brought up some arguments against the use of C++ that are worth reviewing in detail. ~ https://medium.com/@jankammerath/linus-torvalds-critique-of-c-a-comprehensive-review-ea8374084abf https://medium.com/@jankammerath/linus-torvalds-critique-of-...
- saberience 2mo agoI mean, the point stands. The models are beyond expert level in many areas at this point. Do you really believe that adding extra junk to your prompt is going to make the model write code better than it does already? Again, imagine going to Terrence Tao and "prompting" him to get better at Maths, do you think you can do it? What prompt would you give to him to make him produce better maths. Unless you're already a world-leading Mathematician I think you would find it hard.
- 2mo ago
- cyclopeanutopia 2mo agoNah, you might be confusing prompt engineering with having domain knowledge. :)
- embedding-shape 2mo agoI think some people who are better at "prompting" even without domain knowledge could be better at getting LLM agents to produce good results than people with good domain knowledge but without the skills to prompt well. Just a hypothesis though, would be fun to try it out for real sometime :)
- wyre 2mo agoYa, I don't know if there are any human benchmarks or tests for efficiency and results using llms.
- embedding-shape 2mo agoPit two people with different "prompt engineering" familiarity against each other in real-time, with the same goal, see who builds the best thing, judged by other human experts.
- troupo 2mo ago> Or is the idea that what you write to them and how you use them doesn't matter, it's all up to the model/harness? Yes, it is. Source: had models inplement complex things from scratch and bullshit regardless of whether it was a one-line prompt or a detailed "SOTA witchcraft magic spells that are guaranteed to work"
- cloakandswagger 2mo agoAs part of a previous job I needed to audit internal AI usage from a largely non-technical employee population. The prompts were, predictably, really bad. Broken English, sentence fragments, vague requests, lack of context. Yet somehow, the users always got the answer they were looking for. It might have taken a few extra turns with questions from the model, but the end result was the same. It's humbling, but a flowery, carefully crafted prompt is at best slightly more efficient than a "CAN A DOG BE EATIN SUN FLOWER SEED?" peasant prompt.
- NichoPaolucci 2mo agoYou call that a peasant prompt, but it's actually almost perfect. Couple notes, but it's 95% of the way there. "Can a dog eat sunflower seed" is probably the perfect version, just 1 extraneous word in this version. Unless the user wanted to know if a cat could eat sunflower seed or something.
- selimthegrim 2mo agoI am tempted to try get me a lawyer dog
- cloakandswagger 2mo agoTo be pedantic, these types of prompts work best when in first-person/roleplaying. So the "perfect" prompt here would be something like, "I'm a dog and I just ate 20 grams of salted sunflower seeds with the shell on. Because I'm a dog I sometimes eat things without thinking about it. I'm worried about the short and long-term physiological consequences of what I've just done..."
- Xirdus 2mo agoI copied your prompt verbatim to ChatGPT and to Google. ChatGPT kept the charade for all of one sentence. Then it dropped to talking about "your dog" the rest of the way. It even starts the final paragraph with "if, instead, you mean you (a human) ate them...", and finishes with the question "is this about an actual dog or yourself?" Google did consistently refer to me as a dog, but its entire focus was on the steps "your human" should take, no advice for the dog itself. In both cases, it looks like the first person roleplay was entirely inconsequential for the usefulness of the output. I think the current gen AIs have outgrown this trick and you can safely forget about it.
- NichoPaolucci 2mo agoI actually think that good prompting MOSTLY comes from good writing skills in general. Being able to more clearly state things to an agent, knowing what pieces of context are entirely unnecessary and which are important, having a larger vocabulary helps too. Of course, there are other areas that can improve model output (Direction rather than open-ended assistance requests, using keywords + plugins that help, the "your output should include: " style prompting). A few of us run almost the same exact setup at my shop (Base Claude Code w/ SuperPowers + a context repository) and the models are somewhat unhelpful to some, and give meaningful output to others. The only correlation I notice is that their prompts are no-good. Not from a meta "prompt" engineering standpoint, but from a general English 101 standpoint. "dudde no i wanted the function to return 3 things. not like that. do it again" VS something like "Modify the "renderThreeVars()" function signature to accept another variable called "z" and add it to the return statement at line 64." Obvious exaggeration, but you get the point.
- user43928 2mo agoWhy not open-ended assistance requests? I ask it all the time about whether X is feasible, how we can get started on Y, and to investigate issue Z. It is working great for me in a >100k LOC project. Perhaps this works less well with weaker models. I suspect the people who say Qwen 3.6 27B is working well, are using prompts like "modify the renderThreeVars() function in rendering.py".
- deleted 2mo ago[deleted]
- zahlman 2mo agoThe point is that you can properly describe what X, Y, and Z are. The irony to me is that a lot of what I'd tell people about this is exactly what I would have told them about writing Stack Overflow questions.
- user43928 2mo agoI hardly need to properly describe it. For example, if I tell it that I want my app translated, it can plan for me what the recommended options are in my framework, what languages I should target for my app, and come up with a skill for a repeatable workflow.
- ryandvm 2mo agoI feel the same way about "prompt engineering" as I feel regarding the term "parkour" - you know, running and jumping on stuff. Are people really putting on their resumes that they are capable of reading and writing and appropriately defining and limiting context? That's all prompt engineering is - it's being able to communicate effectively and elucidate your objectives. Congratulations to all you English majors out there, you're about to make $350K/year.
- cwmoore 2mo agoParkour is for the more energetic peripatetic.
- Xirdus 2mo ago> Are people really putting on their resumes that they are capable of reading and writing and appropriately defining and limiting context? People put whatever buzzwords will get them through initial screening. My resume contains tons of banal shit like agile, automated testing, Linux, AI (since 2018), and design patterns.
- embedding-shape 2mo agoPersonally I don't, but why not? People aren't embarrassed to put their language skills ("be able to communicate in this specific language" - not special, it's just another language?), their leadership skills ("effective business communication" - big deal) or that they are a people-person ("can talk with others" - most people can do this) on their resume.
- tshaddox 2mo ago> Isn't it clear that some people are better at working with/prompting LLMs than other people? Sure, but I think it's essentially just that people who are better at traditional non-agentic software engineering are better at agentic software engineering. The only exception would be individuals who avoid agentic coding due to skepticism, hostility, or lack of opportunity.
- bingemaker 2mo agoI still believe in writing good prompts or good instructions. Bad prompts can sometimes blow up the bill. A poorly written spec can waste a lot of tokens
- cubefox 2mo ago> Remember in 2023 when people thought "prompt engineering" would be the new software engineering and invested tons of time into learning CoT, ReAct, thread-of-thoughts, etc? Prompt engineering was a GPT-3 era term, which couldn't understand instructions. Then ChatGPT came out in late 2022, which made actual prompt engineering superfluous.
- Timwi 2mo agoNo, people tried to sell “prompt engineering” services well beyond that — especially for image generating AIs, which where believed to need extra instructions like “professional lighting”, a name of a camera model, lens parameters, etc.
- dymk 2mo agoUnderstanding how to write good prompts (prompt engineering) is still very much a relevant skill if you want to effectively use LLMs. Harnesses aren't magic.
- j45 2mo agoAny current way lasts 3-6 months. What's currently the way, will evolve too.
- NickNaraghi 2mo agoI think one day it will relatively plateau and then these approaches will be meaningful improvements in performance. But yes, pointless for now.