6 ms·

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. Release thread: https://twitter.com/BigCodePr

by enum 3y ago

StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. Release thread: https://twitter.com/BigCodeProject/status/1654174941976068119 https://twitter.com/BigCodeProject/status/165417494197606811... Model: https://huggingface.co/bigcode/starcoder https://huggingface.co/bigcode/starcoder Paper: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Also available in HuggingChat: https://huggingface.co/chat https://huggingface.co/chat

espadrine 3y ago
A similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b The performance of StarCoder seems superior on HumanEval pass@1: • replit-code-v1-3b: 21.9% • replit-finetuned-v1-3b: 30.5% • StarCoder: 33.6% • StarCoder prompted: 40.8%
- enum 3y ago
  Also note that StarCoder itself is open (but not replit-finetuned).
- kartayyar 3y ago
  The quality delta is so huge that it's in a different league. Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.
  - sillysaurusx 3y ago
    If so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.
- sauwan 3y ago
  I'm assuming higher is better? How does this compare to GPT-3.5/4?
  - qwerty3344 3y ago
    I think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion
    - enum 3y ago
      Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.
      - manojlds 3y ago
        Aren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?
        dpf 3y ago
        code-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers
  - qwerty3344 3y ago
    https://newatlas.com/technology/gpt-4-reflexion/ https://newatlas.com/technology/gpt-4-reflexion/

by enum 3y ago

espadrine 3y ago

A similar model from Replit was released this week: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b The performance of StarCoder seems superior on HumanEval pass@1: • replit-code-v1-3b: 21.9% • replit-finetuned-v1-3b: 30.5% • StarCoder: 33.6% • StarCoder prompted: 40.8%

enum 3y ago
Also note that StarCoder itself is open (but not replit-finetuned).
kartayyar 3y ago
The quality delta is so huge that it's in a different league. Replit's model seems to have focused on being cheap to train and run. StarCoder seems to be vastly better on quality.
- sillysaurusx 3y ago
  If so, it’s a shame this post is getting comparatively little attention. Hopefully the quality will be enough for it to make a big splash somewhere.
sauwan 3y ago
I'm assuming higher is better? How does this compare to GPT-3.5/4?
- qwerty3344 3y ago
  I think it's still significantly behind GPT 3.5/4, both of which can get 67% on HumanEval, and 88% with Reflexion
  - enum 3y ago
    Keep in mind that StarCoder(Base) is just a pretrained LM. The extra stuff that makes 3.5/4 like RLHF gets built on this.
    - manojlds 3y ago
      Aren't GPT-3 etc base LM and ChatGPT the instruction tuned? Or am I wrong?
      - dpf 3y ago
        code-davinci-002 is a base LM, and the other 3.5 models (text-davinci-{002,003}, gpt-3.5-turbo, and ChatGPT) use instruction tuning and/or RLHF. Source: https://platform.openai.com/docs/model-index-for-researchers https://platform.openai.com/docs/model-index-for-researchers
- qwerty3344 3y ago
  https://newatlas.com/technology/gpt-4-reflexion/ https://newatlas.com/technology/gpt-4-reflexion/