8 ms·
DeepSeek Coder: Let the Code Write Itself
- hackerlight 3y agoIn the benchmarks, are they using the base GPT-4, or are they using a GPT like Grimoire which will be better at coding? If they aren't using Grimoire, isn't it unfair to compare their fine tuned model to base GPT-4?
- BugsJustFindMe 3y agoIs it unfair if GPT4 still beats them handily?
- anakaine 3y agoIt's probably unfair to the readers that they are only presenting performance against GPT3.5 turbo and not against GPT 4. We could all make our numbers look excellent if we compared them to lesser models.
- BugsJustFindMe 3y ago> they are only presenting performance against GPT3.5 turbo and not against GPT 4 GPT4 is represented in all of the images as the top performer. Are you blind and using a screen reader that doesn't OCR images, by chance? If so you may need to access the linked PDF where the charts are given as text.
- byyoung3 3y agolooks like code llama 70B outperforms on humaneval I believe
- SparkyMcUnicorn 3y agoLooks like even deepseek-coder 1.3b benchmarks higher than CodeLlama 70b. https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html
- _boffin_ 3y agoBeen using DeepSeek Coder 33B Q8 on my work laptop for a bit now. I like it, but am still finding myself going to GPT-4's API for the more nuanced things. They just released a v1.5 (https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5 https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruc...), but for some reason, they reduced the context length from ~16k to ~4k.
- busfahrer 3y agoInteresting, what are your laptop specs? I imagine the 33B Q8 to need a bit of memory - are you sharing VRAM and RAM?
- elwebmaster 3y agoMixtral > Codellama > DeepSeek Coder. Very weird model, writes super long comments on one line, definitely not at the level of Codellama, benchmarks be damned.
- findjashua 3y agodo you find Mixtral to be better than the new 70B one that Meta released a couple days back as well?
- findjashua 3y agodid a comparison on LM Studio - the answers are eerily similar, but Mixtral is way way faster. Codellama-70B is slow to the point of being unusable. (M1 Max, 64 GB RAM)
- _boffin_ 3y agoReduce the repetition penalty to 1 and that should fix it.
- chrisweekly 3y agotangent: I often have a hard time disambiguating the ">" in comparisons like yours: (A) greater than (ie, Mixtral superior to DeepSeek, w/ Codellama in between) vs (B) arrow/sequence (ie, start w/ Mixtral, progress to Codellama, finally land on DeepSeek as the culmination). I'd love to hear of a less ambiguous way to represent these.
- mooreds 3y agoHow does Mixtral >= Codellama >= DeepSeek Work for you?
- illusive4080 3y agoSadly that implies that all 3 could be equal or very close to equal.
- sestinj 3y agoWe've been playing with the 1.3b model for continue.dev's autocomplete and it's quite impressive. One unclear part is whether the license really permits commercial usage, but regardless it's exciting to see the construction of more complex datasets. They mention that training on multiple tasks (FIM + normal completion) improves performance...wonder whether training to output diffs would be equally helpful (this is the holy grail needed to generate changes in O(diff length) time)
- explorigin 3y ago> This code repository is licensed under the MIT License. The use of DeepSeek Coder models is subject to the Model License. DeepSeek Coder supports commercial use. Says so on https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct#4-license https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instr... They have their own license to prevent things like propaganda or military use.
- DeepSeek 3y agoHello sestinj, I work at DeepSeek, and I'm glad to hear that our models work for you! DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use. We claim no rights on and take no responsibility from the output generated by the model induced by user prompts. Feel free to deploy and use DeepSeek models for any creative projects. We are a starup company and would like to concentrate on building better models, so it would be best if the users can help create a healthy ecosystem. Should you have any questions or requirements, I'm always happy to support.
- objactivate 3y agoThe latest 7B model demonstrated impressive performance on benchmarks. However, I have a question regarding its specific deployment method: How can I merge the parts of the Safetensors format? Specifically, I'm referring to files named 'model-00001-of-00002.safetensors' and 'model-00002-of-00002.safetensors'. My motivation is straightforward: I aim to combine the Safetensor 'shards' and then utilize the 'convert.py' script from the llama.cpp project to transform a single .safetensors file into the GGUF format. This conversion facilitates running the models on WasmEdge. I appreciate any guidance on this matter. Thank you.
- Havoc 3y agoI’ve been using their 7B with tabbyML. Works well but closer to a very smart code complete rather than generating much novel blocks of code
- illusive4080 3y agoFells like all Gen AI is ‘very good code complete’ because if you give it a broad problem it’ll make mistakes.
- rickstanley 3y agoHello, I would like to take this opportunity and ask for help here, about using A.I. with my own codebase. Context: I missed [almost] the entire A.I. wave, but I knew that one day I would have to learn something about and/or use it. That day has come. I'm allocated in one team, that is migrating to another engine, let's say "engine A → engine B". We are looking from the perspective of A, to map the entries for B (inbound), and after the request to B is returned, we map back to A's model (outbound). This is a chore, and much of the work is repetitive, but it comes with its edge cases that we need to look out for and unfortunately there isn't a solid foundation of patterns apart from the Domain-driven design (DDD) thing. It seemed like a good use case for an A.I. Attempts: I began by asking to ChatGPT and Bard, with questions similar to: "how to train LLM on own codebase" and "how to get started with prompt engineering using own codebase". I concluded that, fine-tuning is expensive, for large models, unrealistic for my RTX 3060 with 6Gb VRAM, no surprise there; so, I searched here, in Hacker News, for keywords like "llama", "fine-tuning", "local machine", etc, and I found out about ollama and DeepSeek. I tried both ollama and DeepSeek, the former was slow but not as slow as the latter, which was dead slow, using a 13B model. I tried the 6/7B model (I think it was codellama) and I got reasonable results and speed. After feeding it some data, I was on my way to try and train on the codebase when a friend of mine came and suggested that I use Retrieval-Augmented Generation (RAG), I have yet to try it, with a setup Langchain + Ollama. Any thoughts, suggestions or experiences to share? I'd appreciate it.
- mhb 3y agoMaybe this is not relevant to you, but would it make any sense to first try Copilot with IntelliJ or Visual Studio?
- high_priest 3y agoCopilot has such a narrow input space, that it is not going to help in this case. Here, just saved you $$
- imp0cat 3y agoYou can use @workspace to tell him to ingest more of your workspace as required.
- phantomathkg 3y ago[flagged]
- deleted 3y ago[deleted]
- chii 3y agoJust tried it by asking how to create a game that is turn based, using an ECS system, and how to add a decision tree, and a save/load system, in the language Haxe. It outputs relatively correct haxe code, but it did halucinate that there is a library called 'haxe-tiled' to read tmx map files...