8 ms·

Even the 7B model of code llama seems to be competitive with Codex, the model behind copilot https://ai.meta.com/blog/code-llama-large-language-model-coding/ h

by up6w6 3y ago

Even the 7B model of code llama seems to be competitive with Codex, the model behind copilot https://ai.meta.com/blog/code-llama-large-language-model-coding/ https://ai.meta.com/blog/code-llama-large-language-model-cod...

ramesh31 3y ago
>Even the 7B model of code llama seems to be competitive with Codex, the model behind copilot It's extremely good. I keep a terminal tab open with 7b running for all of my "how do I do this random thing" questions while coding. It's pretty much replaced Google/SO for me.
- coder543 3y ago
  You've already downloaded and thoroughly tested the 7B parameter model of "code llama"? I'm skeptical.
  - lddemi 3y ago
    Likely meta employee?
    - MertsA 3y ago
      I've been using this or something similar internally for months and love it. The thing that gets downright spooky is the comments believe it or not. I'll have some method with a short variable name in a larger program and not only does it often suggest a pretty good snippet of code the comments will be correct and explain what the intent behind the code is. It's just a LLM but you really start to get the feeling the whole is greater than the sum of the parts.
      - coder543 3y ago
        I just don’t understand how anyone is making practical use of local code completion models. Is there a VS Code extension that I’ve been unable to find? HuggingFace released one that is meant to use their service for inference, not your local GPU. The instruct version of code llama could certainly be run locally without trouble, and that’s interesting too, but I keep wanting to test out a local CoPilot alternative that uses these nice, new completion models.
        fredoliveira 3y ago
        There are a bunch of VSCode extensions that make use of local models. Tabby seems to be the most friendly right now, but I admittedly haven't tried it myself: https://tabbyml.github.io/tabby/ https://tabbyml.github.io/tabby/
        kateklink 3y ago
        there's also Refact (https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/) with support of several open-source code LLMs and extension for VS Code and Jetbrains
  - realce 3y ago
    Just sign up at meta and you'll get an email link in like 5 minutes
    - coder543 3y ago
      Yes, that's not a response to my comment. No one who has been using any model for just the past 30 minutes would say that it has "pretty much replaced Google/SO" for them, unless they were being facetious.
      - tyre 3y ago
        They said 7b llama which I read as the base LLaMa model, not this one specifically. All of these LLMs are trained on Stack Overflow so it makes sense that they’d be good out of the box.
        brandall10 3y ago
        The top level comment is specifically citing performance of code llama against codex.
      - dataangel 3y ago
        GPT4 has replaced SO for me and I've been using it for months.
      - deleted 3y ago
        [deleted]
  - Eddygandr 3y ago
    Maybe confused Code Llama with Llama 2?
  - bbor 3y ago
    It was made available internally, I believe. So this is one of the many Meta engineers on this site —- after all, Facebook is now less hated than Google here ;)
    - deleted 3y ago
      [deleted]
- solarkraft 3y ago
  Huh? Do you perhaps mean standard Llama?
- ohyes 3y ago
  What hardware do you have that lets you run 7b and do other stuff at the same time?
  - hmottestad 3y ago
    Maybe a MacBook Pro. The Apple silicon chops can offload a special AI inference engine, and all ram is accessible by all parts of the chip.
  - brucethemoose2 3y ago
    Pretty much any PC with 16GB+ of fast RAM can do this, any PC with a dGPU can do it well.
  - _joel 3y ago
    If you're willing to sacrifice token/s you can even run these on your phone.
  - gzer0 3y ago
    An M1 Max with 64GB of RAM allows me to run multiple models simultaneously, on top of stable diffusion generating images non-stop + normal chrome, vscode, etc. Definitely feeling the heat, but it's working. Well worth the investment.
  - selfhoster11 3y ago
    A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.
    - FrozenSynapse 3y ago
      how's the generation speed on CPU?
      - selfhoster11 3y ago
        On Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.
SparkyMcUnicorn 3y ago
I'm not sure copilot is using codex anymore[0]. They've also been talking about a shift towards GPT-4 with "Copilot X" a few times now[1][2]. [0] https://github.blog/2023-07-28-smarter-more-efficient-coding-github-copilot-goes-beyond-codex-with-improved-ai-model/ https://github.blog/2023-07-28-smarter-more-efficient-coding... [1] https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x [2] https://github.blog/2023-07-20-github-copilot-chat-beta-now-available-for-every-organization/ https://github.blog/2023-07-20-github-copilot-chat-beta-now-...
- up6w6 3y ago
  True. The results from codex are actually from code-cushman-001 (Chen et al. 2021), which is an older model that Copilot was based on.
- zarzavat 3y ago
  Copilot X is just their name for their project to bring AI to more areas of VSCode. I don’t believe they can use GPT-4 for completions because it’s a chat-optimized model. It seems that they are using something else, that blog post seems to imply it’s a custom-trained model.
  - cosmojg 3y ago
    I use GPT-4 for code completion all the time! There are many Neovim extensions[1][2][3] (and I'm sure there are many VSCode extensions) which call the GPT-4 API directly for code completion. I'm pretty sure the only reason that Microsoft might avoid using GPT-4 for Copilot is cost. [1] https://github.com/cosmojg/nvim-magic https://github.com/cosmojg/nvim-magic [2] https://github.com/dpayne/CodeGPT.nvim https://github.com/dpayne/CodeGPT.nvim [3] https://github.com/aduros/ai.vim https://github.com/aduros/ai.vim

by up6w6 3y ago

ramesh31 3y ago

>Even the 7B model of code llama seems to be competitive with Codex, the model behind copilot It's extremely good. I keep a terminal tab open with 7b running for all of my "how do I do this random thing" questions while coding. It's pretty much replaced Google/SO for me.

coder543 3y ago
You've already downloaded and thoroughly tested the 7B parameter model of "code llama"? I'm skeptical.
- lddemi 3y ago
  Likely meta employee?
  - MertsA 3y ago
    I've been using this or something similar internally for months and love it. The thing that gets downright spooky is the comments believe it or not. I'll have some method with a short variable name in a larger program and not only does it often suggest a pretty good snippet of code the comments will be correct and explain what the intent behind the code is. It's just a LLM but you really start to get the feeling the whole is greater than the sum of the parts.
    - coder543 3y ago
      I just don’t understand how anyone is making practical use of local code completion models. Is there a VS Code extension that I’ve been unable to find? HuggingFace released one that is meant to use their service for inference, not your local GPU. The instruct version of code llama could certainly be run locally without trouble, and that’s interesting too, but I keep wanting to test out a local CoPilot alternative that uses these nice, new completion models.
      - fredoliveira 3y ago
        There are a bunch of VSCode extensions that make use of local models. Tabby seems to be the most friendly right now, but I admittedly haven't tried it myself: https://tabbyml.github.io/tabby/ https://tabbyml.github.io/tabby/
      - kateklink 3y ago
        there's also Refact (https://github.com/smallcloudai/refact/ https://github.com/smallcloudai/refact/) with support of several open-source code LLMs and extension for VS Code and Jetbrains
- realce 3y ago
  Just sign up at meta and you'll get an email link in like 5 minutes
  - coder543 3y ago
    Yes, that's not a response to my comment. No one who has been using any model for just the past 30 minutes would say that it has "pretty much replaced Google/SO" for them, unless they were being facetious.
    - tyre 3y ago
      They said 7b llama which I read as the base LLaMa model, not this one specifically. All of these LLMs are trained on Stack Overflow so it makes sense that they’d be good out of the box.
      - brandall10 3y ago
        The top level comment is specifically citing performance of code llama against codex.
    - dataangel 3y ago
      GPT4 has replaced SO for me and I've been using it for months.
    - deleted 3y ago
      [deleted]
- Eddygandr 3y ago
  Maybe confused Code Llama with Llama 2?
- bbor 3y ago
  It was made available internally, I believe. So this is one of the many Meta engineers on this site —- after all, Facebook is now less hated than Google here ;)
  - deleted 3y ago
    [deleted]
solarkraft 3y ago
Huh? Do you perhaps mean standard Llama?
ohyes 3y ago
What hardware do you have that lets you run 7b and do other stuff at the same time?
- hmottestad 3y ago
  Maybe a MacBook Pro. The Apple silicon chops can offload a special AI inference engine, and all ram is accessible by all parts of the chip.
- brucethemoose2 3y ago
  Pretty much any PC with 16GB+ of fast RAM can do this, any PC with a dGPU can do it well.
- _joel 3y ago
  If you're willing to sacrifice token/s you can even run these on your phone.
- gzer0 3y ago
  An M1 Max with 64GB of RAM allows me to run multiple models simultaneously, on top of stable diffusion generating images non-stop + normal chrome, vscode, etc. Definitely feeling the heat, but it's working. Well worth the investment.
- selfhoster11 3y ago
  A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.
  - FrozenSynapse 3y ago
    how's the generation speed on CPU?
    - selfhoster11 3y ago
      On Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.

SparkyMcUnicorn 3y ago

I'm not sure copilot is using codex anymore[0]. They've also been talking about a shift towards GPT-4 with "Copilot X" a few times now[1][2]. [0] https://github.blog/2023-07-28-smarter-more-efficient-coding-github-copilot-goes-beyond-codex-with-improved-ai-model/ https://github.blog/2023-07-28-smarter-more-efficient-coding... [1] https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x [2] https://github.blog/2023-07-20-github-copilot-chat-beta-now-available-for-every-organization/ https://github.blog/2023-07-20-github-copilot-chat-beta-now-...

up6w6 3y ago
True. The results from codex are actually from code-cushman-001 (Chen et al. 2021), which is an older model that Copilot was based on.
zarzavat 3y ago
Copilot X is just their name for their project to bring AI to more areas of VSCode. I don’t believe they can use GPT-4 for completions because it’s a chat-optimized model. It seems that they are using something else, that blog post seems to imply it’s a custom-trained model.
- cosmojg 3y ago
  I use GPT-4 for code completion all the time! There are many Neovim extensions[1][2][3] (and I'm sure there are many VSCode extensions) which call the GPT-4 API directly for code completion. I'm pretty sure the only reason that Microsoft might avoid using GPT-4 for Copilot is cost. [1] https://github.com/cosmojg/nvim-magic https://github.com/cosmojg/nvim-magic [2] https://github.com/dpayne/CodeGPT.nvim https://github.com/dpayne/CodeGPT.nvim [3] https://github.com/aduros/ai.vim https://github.com/aduros/ai.vim