6 ms·
How has DeepSeek improved the Transformer architecture?
- doener 2y agoI hate it so much that HN automatically removes some words in headlines like „how.“ You can add them after posting though for a while by editing the headline.
- dboreham 2y agoPerhaps an faq, but why the weird quote characters?
- vanderZwan 2y agoI'm guessing they're using a mobile device with a keyboard that does this automatically.
- doener 2y agoBecause I‘m German and that‘s the way we use them in Germany. So my German mobile keyboard does this automatically, yes. Oftentimes I change it in English messages, sometimes it slips.
- bflesch 2y agoI'm also German and have never seen such weird quotes. Maybe this is some weird windows charset issue but definitely not widespread way of quoting text.
- dxyms 2y agoIt definitely exists in German as well as some other European countries, https://en.m.wikipedia.org/wiki/Quotation_mark https://en.m.wikipedia.org/wiki/Quotation_mark I checked a few German newspapers, some use it, some don’t
- actionfromafar 2y agoI don't know why, but in my browser, the closing quote is showed correctly (forward slanting) in the edit box, but backwards slanting when submitted. Weird. „how“
- vanderZwan 2y agoDifferent fonts, most likely (I can't say for sure because I customized the CSS for HN using the Stylus browser extension)
- deleted 2y ago[deleted]
- iamacyborg 2y agoWait until you see how the French do quotes
- cyberax 2y agoMandarin Chinese keyboards「have entered the chat」.
- numpad0 2y agoDo Chinese people actually do that? I thought 「」 has ever so slightly different bearing than "" but then again I don't speak Chinese
- cyberax 2y agoSometimes. It's formal, but text autocomplete systems can sometimes insert them. The other form of quotes is similar to English quotes, but looks a bit different.
- whimsicalism 2y agonone of these techniques except MLA are new
- WithinReason 2y agoThere is a big difference between inventing a technique and productising it.
- anonymousDan 2y agoOne issue is that a lot of techniques proposed (especially from academic research) are hard to validate at scale given the resources required. At least DeepSeek helps a little in that regard.
- grazing_fields 2y ago[dead]
- eldenring 2y agoThey're not new in the same way Attention wasn't new when the transformer paper was written. No one (publically) had really pushed any of these techniques far, especially not for such a big run.
- whimsicalism 2y agono one publicly pushes any techniques very far except for meta and it’s true they continue to train dense models for whatever reason. the transformer was an entirely new architecture, very different step change than this e: and alibaba
- deleted 2y ago[deleted]
- leetharris 2y agoThey likely continue to train dense models because they are far easier to fine tune and this is a huge use case for the Llama models
- juancn 2y agoThe compute scheduling part of the paper is also vey good, the way they balanced load to keep compute and communication in check. There is also a lot of thought put into all the tiny bits of optimization to reduce memory usage, using FP8 effectively without significant loss of precision nor dynamic range. None of the techniques by themselves are really mind blowing, but the whole of it is very well done. The DeepSeekV3 paper is really a good read: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSee...
- ahartmetz 2y agoWhen everyone kind of ignores performance because compute is cheap and speed will double anway in 18 months (note: hasn't been true for 15 years), the willingness to optimize is almost a secret weapon. The first 50% or so are usually not even difficult because there is so much low-hanging fruit, and in most environments there's a lot of helpful tooling to measure exactly which parts are slow.
- steve_adams_86 2y agoAt my org this is a crazy problem. Before I arrived, people would throw all kinds of compute at problems. They still do. When you've got AWS over there ready to gobble up whatever tasks you've got, and the org is willing to pay, things get really sloppy. It's also a science-based organization like OpenAI. Very intelligent people, but they aren't programmers first.
- sgt101 2y agoCompute has been more than doubling because people have been spending silly money on it. How long ago would a proposal for a $10m cluster for ML have been thought surreal by any funding agency? Certainly less than 10 years ago. Now people are talking of spending billions and billions. Madness.
- HarHarVeryFunny 2y agoWhen people are talking about $100M-$1B frontier model training runs, then obviously efficiency matters! Sure training cost will go down with time, but if you are only using 10% of the compute of your competition (TFA: DeepSeek vs LLaMa) then you could be saving 100's of millions per training run!
- 1970-01-01 2y agoHas DeepSeek challenged the very weird hallucination problem? Reducing hallucinations now seems to be the remaining fundamental issue that needs scientific research. Everything else feels like an engineering problem.
- whimsicalism 2y agohallucinations decrease with scale and reasoning, the model just gets better and stops making stuff up.
- littlestymaar 2y agoo1 still hallucinates badly though.
- Jerrrry 2y agoFalse, facts only need to be seen once, and one mis-step in reasoning and your CoT is derailed.
- whimsicalism 2y ago> one mis-step in reasoning and your CoT is derailed. tell me you've never seen reasoning traces without telling me
- GaggiX 2y agoFrom what I see, the Deepseek R1 model seems to be better calibrated (knowing what it knows) than any other model, at least on the HLE benchmark: https://lastexam.ai/ https://lastexam.ai/
- bane 2y agoTo me, the second biggest problem is that the models aren't really conversational yet. They can maintain some state between prompt and response, but in normal human-human interactions responses can be interrupted by either party with additional detail or context provided. "Write Python code for the game of Tetris" resulting in working code that resembles Tetris is great. But the back and forth asking for clarification or details (or even post-solution adjustments) isn't there. The models dwell and draw almost entirely self-referentially from their own reasoning through the entire exchange. "Do you want to keep score?" "How should scoring work?" "Do you want aftertouch?" "What about pushing down, should it be instantaneous or at some multiple of the normal drop speed?" "What should that multiple be?" as well as questions from the prompter that inquire about capabilities and possibilities. "Can you add one 5-part piece that shows up randomly on average every 100 pieces on average?" or "Is it possible to make the drop speed function as an acceleration rather than a linear drop speed?"....these are somewhat possible, but sometimes require the model to re-reason the entire solution again. So right now, even the best models may or may not provide working code that generates something that resembles a Tetris game, but have no specifics beyond what some internal self-referential reasoning provides, even if that reasoning happens in stages. Such a capability would help users of these models troubleshoot or fix specific problems or express specific desires....the Tetris game works but has no left-hand L blocks for example. Or the scoring makes no sense. Everything happens in a sort of highly superficial approach where the reasoning is used to fill in gaps in the top-down understand of the problem the model is working on.
- ilaksh 2y agoWhy is it that the larger models are better at understanding and following more and more complex instructions. And generally just smarter? With DeepSeek we can now run on non-GPU servers with a lot of RAM. But surely quite a lot of the 671 GB or whatever is knowledge that is usually irrelevant? I guess what I sort of am thinking of is something like a model that comes with its own built in vector db and search as part of every inference cycle or something. But I know that there is something about the larger models that is required for really intelligent responses. Or at least that is what it seems because smaller models are just not as smart. If we could figure out how to change it so that you would rarely need to update the background knowledge during inference and most of that could live on disk, that would make this dramatically more economical. Maybe a model could have retrieval built in, and trained on reducing the number of retrievals the longer the context is. Or something.
- joshuakogut 2y agoYesterday when I started evaluating Deepseek-R1 V3 it was insanely better at code generation using elaborate prompts, I asked it to write me some boilerplate code in python using the ebaysdk library to pull a list of all products sold by user with $name and it spit it out, just a few tweaks and it was ready to go. I tried the same thing on the 7B and 32B model today, neither are as effective as codellama.
- A4ET8a8uTh0_v2 2y agoI am intrigued. What did you use to run your deepseek instance?
- ilaksh 2y agoI think people didn't understand my comment. I am very aware of this already.
- seba_dos1 2y agoI think you failed to convey what you meant to with your comment. If you want your contribution to the discussion to be meaningful, you may want to give it another go.