7 ms·
Fast AI seems genuinely exciting and somewhat unsettling to me. Right now Claude is faster than me on some tasks but we’re at least close. I have a prompt to cl
by goyozi 3mo ago
Fast AI seems genuinely exciting and somewhat unsettling to me. Right now Claude is faster than me on some tasks but we’re at least close. I have a prompt to clean up a PR that’s been running for 1h now and I expect it to take another few. It’s hard to imagine how the workflow would look like if it was near-instant. On the one hand, it might be easier to focus. Some prompts take so long that I start to multitask and regret it later. On the other, AI that takes a few seconds to max few minutes to solve what used to take hours or days? That’s a game changer and I don’t even know where we fit in.
- pianopatrick 3mo agoWe fit in for the things that are not artificial. So long as AI lives in server farms, humans will be needed for tasks in the physical world. It's only if we combine AI with robots that things get really dicey.
- fartfeatures 3mo agoThis is very dystopian in my opinion. I'm not the arms, legs, sensors and actuators for a machine super intelligence. I wouldn't treat another human as my slave because they aren't as intelligent as I am any more than I would expect to become a slave for a machine. This is our world (for now) and that is why we fit in. Not because we can serve.
- davedx 3mo agoAgree https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Scream https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Sc...
- fartfeatures 3mo agoSounds like snuff porn, not my sort of thing but thanks though.
- ionwake 3mo ago"It seeks revenge on humanity for its own creation." This is brilliant as it reminded me of a famous hitchikers quote: "In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move. — From The Restaurant at the End of the Universe (Book 2)" Maybe we are stuck in an eternal loop
- cicko 3mo ago"This is our world" sounds a bit exclusive towards other living and sentient beings on this planet.
- nativeit 3mo agoIt depends on what’s included in “our”.
- throwaway67678 3mo agoNever read Asimov's Multivac novels? Admittedly not all of them are stellar examples of a future to follow
- Muromec 3mo agoYou don't need ai superintelligence, just plain capitalism is enough
- ipkstef 3mo agoasking for curiosities sake. What kind of PR loop are you running that takes a few hours?
- ketzo 3mo agonot OP but usually for me this means long verification loop; waiting 10min on CI checks, that kind of thing, rather than actual 1hr wall clock of token generation
- devmor 3mo agoOr slow MCP servers that are waiting on HTTP calls from APIs, playwright/other UI instrumentation, etc.
- RussianCow 3mo agoBut those things won't be sped up by a faster LLM, so I feel like that's not what the OP is talking about.
- goyozi 3mo agoWell, I used an extreme example. OTOH, I’ve done quite a few of those „fix CI” or „migrate X” prompts recently and while there is a fixed component like running CI / builds, I’d say the LLM time is still around or above 50%, especially at the beginning of the project. Then there’s also regular tasks that now take minutes per message which completely get me out of the zone. I imagine iterating on those in near real time would be a big change.
- goyozi 3mo agoI’m rewriting our integration test suite to run tests in parallel. I have the changes split across 7 branches, and each needs to be fixed to have no flaky tests. I told it I want 3 consecutive CI runs with no flakes and no artificial fixes / assert removals etc. We’ll see what comes out; it’s almost a side project so there’s not much to lose other than some of my weekly limit that resets soon.
- flexagoon 3mo agoI'm using Deepseek-v4-pro as my main model and this is sometimes pretty annoying, I have to do some easy boring task, think "I'll just leave the agent to do it and go take a nap", but it's already done writing the code before I even walk away from the computer
- RussianCow 3mo agoDo you mean Flash and not Pro? I haven't tried it personally, but according to OpenRouter, the fastest DeekSeep V4 Pro providers are only ~50tps. That's slower than Claude Opus. https://openrouter.ai/deepseek/deepseek-v4-pro?sort=throughput https://openrouter.ai/deepseek/deepseek-v4-pro?sort=throughp...
- specproc 3mo agoYeah, flash is crazy fast, but I've found performance variable.
- binary0010 3mo agoFlash is amazing if you know the domain really well. E.g. occasionally it makes the dumbest mistakes you've ever seen and can't correct them. However it's fairly rare, and if you know the domain really well, occasionally popping in the code and pushing it towards the correct solution takes like 20seconds or whatever. So the speed you can move with flash + high domain knowledge beats opus by a mile in my experience. I tried to switch back to 4.8 for a bit when it came out, feels so bad waiting 20mins for a mediocre solution when I could have had everything complete - with multiple iteration cycles - in flash in like 3-5mins.
- addozhang 3mo agoYes, you don't need much domain knowledge to use Opus, but it's just way too expensive.
- 3mo ago
- recroad 3mo agoWoah - what’s the prompt and what’s the PR?
- goyozi 3mo agoI replied in more detail under another comment. TLDR: fixing flaky CI across multiple branches
- HarHarVeryFunny 3mo agoI don't see many companies being willing to pay 3x more for faster code generation. Cloud-based AI code generation is already extremely fast, and hardly the bottleneck for most software product development. There can't be many normal use cases where there'd be any cost benefit.
- fragmede 3mo agoThe "traditional" way we vibe code is human software developer prompts AI -> AI generates code -> (human checks code) -> code gets compiled/deployed/etx -> users use "binary". At the speed of 1000 tok/sec, user prompts obliquely -> AI vets generated code -> code deployed -> user gets response from deployed code. It's a cute toy right now, but you can tell an LLM that it's an http server, and have it respond directly to a web browser hitting it. It generates headers in response, as well as page contents. As 1000 tok/sec becomes three new normal, we will come up with newer ways to use it outside of toy fiction encyclopedias.
- HarHarVeryFunny 3mo ago1000 tokens per sec is still massively slower than serving a normal web page - if something doesn't respond in a few seconds many people give up. I'm not saying there aren't any use cases for super-fast (and super-expensive) generation, but it does seem a bit niche. If it was free then sure faster is better, but what are the mainstream use cases where people might pay 3x more for a faster version of something that is already fast? I think it would have to be an application where it paid for itself - where the 10x faster response was actually worth more than 3x the cost to you - where the extra speed was worth the extra cost.
- efromvt 3mo agoI'd be very curious about the bottleneck breakdown in most current software dev - I suspect inference is far from the bottleneck in most things I do, though driving it to 0 would still be nice. I do agree that if it was 0 we'd probably change development approaches to reduce the new bottlenecks more, but it'll take full-process innovation to really get something near-instant. (I should go measure this now, I'm curious)
- ilaksh 3mo agoUse Claude fast mode and turn off thinking. Tell it to just explain what it's plan is to you at a high level. It will go much faster.
- skybrian 3mo agoIf we get low enough latency, there's no reason to multitask. You can ask it to do one thing at a time and immediately see what it did. That's a nice way to work! This is normal interactive UI for tasks that aren't compute-intensive. Programs spend most of their time idle, waiting for us to click a button. We shouldn't be waiting for them or spinning more plates to keep them busy. However, a faster llm isn't enough. You also need fast compiles and fast tests.
- binyu 3mo ago> Right now Claude is faster than me on some tasks but we’re at least close. I dont doubt it, but I don't think you can spawn 10 copies of yourself working simultaneously.
- AlecSchueler 3mo agoNo, but nor can you keep track of what 10 agents are doing simultaneously. Hence the multitasking regret.
- pixel_popping 3mo agoAn agent can, you don't need to watch tasks, you can have a live digest with another tool.
- logankeenan 3mo agoDo you have any recommendations for a live digest tool?
- AlecSchueler 3mo agoWho watches the watchers?
- UncleOxidant 3mo agoHave you tried Gemini 3.5 Flash? It's quite fast. Amazing how fast it finishes tasks. Much faster than Claude.
- switchbak 3mo agoNow the next bottleneck is the compiler - which we can model in an LLM! It's only wrong 15% of the time :) But truly, using Cerebras at ~2k tokens/s, with very low latency is like a vision into the future. You start to rework your workflow around things that can happen without onerous manual review - stating the conditions for success, etc. It's rare that I have a problem that maps well to that, but I expect this is where things are headed. Of course the fast models tend to not be the SOTA ones, but if that was the case - high quality and near-instant thinking, that's a game changer that I don't think we're really prepared for. The things that get unlocked with higher-than-reasonable speed become very interesting.
- lhoff 3mo agoHave you tried https://chatjimmy.ai/ https://chatjimmy.ai/ it’s only a demo but it blew my mind. I had the sudden feeling that this is the future.
- colordrops 3mo agoWhat do you mean "demo"? Seems to work... Who is behind this?
- alfiopuglisi 3mo agoThese guys: https://taalas.com/products/ https://taalas.com/products/
- Silphendio 3mo agoIt's a 3 bit quant of Llama3-8B. I'm sure there are use-cases for that, but it's useless when it comes to tool calls or coding and I wouldn't trust it's factual accuracy either.
- coderbants 3mo agoIt cuts both ways. Sometimes I ask Gemini 3.5 Flash to do something for me and it kicks it out almost instantly and it works great, and it's a bit scary how quickly it can do that. Then I ask it to do something else and it goes off-road and where I used to be able to interject with a "wow wow wow, that's not right", by the time I see the text on screen and react it's already made massive changes. Short of making it commit between every edit it's hard to prevent it from going wrong as quickly as it goes right (and even then, it can make a boo-boo on a remote API too depending on how much privilege it has).
- bendangelo 3mo agoI use planning mode in opencode. It has a prompt to tell it to plan it out etc. Then I execute with a smaller model. it works well
- Bombthecat 3mo agoLiving on the street or cave lol
- dkersten 3mo agoI’ve been playing around with groq and GPT OSS which they run at 1000 TPS (20B) or 800 TPS (120B) and the speed feels quite magical. I haven’t tried cerebras’ 3000 TPS yet but I did try the demo of that 15,000 TPS model whose name escapes me right now. I’m not sure if it makes a meaningful difference for my actual work, but it sure is amazing to watch it generate a screen full of text in the blink of an eye. I do think it’s super useful for rubbing little validation checks like showing it a diff to ensure that the changes are on task, and being able to do those quicker really helps because it means you can do many focused checks without them getting in the way.
- robberth 3mo agohttps://chatjimmy.ai/ https://chatjimmy.ai/ ?
- msdz 3mo agoAFAIK Taalas, the company behind this demo, still only have their initially "hardwarized" model available to test in ChatJimmy, which IIRC is a rather stupid Llama 3ish 8b. Don't get me wrong though, that demo is still incredibly impressive & makes me very much excited for the hardware-based model era (potentially) ahead. Once you've experienced those speeds, you really start to think about the whole class of things that becomes possible; massively parallel decode paths, extensive reasoning loops, etc…
- hedgehog 3mo agoFor scale though if three or four chips that size can replicate a Qwen 27B experience that'll be quite useful.
- dkersten 3mo agoThat’s the one. The speed is incredible and fun to see, but the model is rather weak to the point where I’m not sure it’s particularly useful for most people.
- ayewo 3mo ago
- joshcreates 3mo ago[flagged]
- OtomotO 3mo ago> That’s a game changer and I don’t even know where we fit in. Doing non trivial work.
- giancarlostoro 3mo agoYou can run Claude in "fast" mode it costs you more on your compute use, but its reasonably fast. I'm not sure I care to go "faster" than where things are now, otherwise you start losing on manual review and testing time. I would argue that Claude can poop out weeks (if not months) of coding effort in a few hours, and get you insanely close to a good product if you define the tech stack, and the business rules. Can it goof here and there? Sure. You can also make it refactor all the code on a whim faster than any intern could. I think it's good enough to avoid you mundane stupid bugs in most cases. I don't know what people who hate it are doing, maybe they're not even trying at all or are dismissing it from the first output (as though everyone writes perfect code in one shot right?) or maybe its just pride getting in the way of them using a decent tool to its true potential.
- cman1444 3mo agoReminds me of the doherty threshold. When will AI respond in less than 400 milliseconds?
- fnordpiglet 3mo agoI’ve used codex code optimized for a few projects and it’s unsettling how fast it is. It’s hard to think fast enough to keep up with it. Mental fatigue was a real challenge because the decisions that required my input were rapid fire and legitimate ambiguities that were appropriate escalations. I am too much a geezer for the intensity of it. But I’ll take it!
- noisy_boy 3mo agoThe first wave was just getting half decent answers. The second wave was being able to choose between actually getting reasonably ok coding results OR getting not so great results very fast. The third wave would be getting good results fast. We need to really worry when we get amazing results very fast.
- lukan 3mo ago"I don’t even know where we fit in." Giving directions and verifying its output? But my mental capacity is still limited. I can make way more prompts, than I can read code.