8 ms·

StableCode

305 points by kpozin 3y ago

rvz 3y ago
Either way, the race to zero has been further accelerated. Stability AI, Apple, Meta, etc are clearly at the finish line putting pressure on cloud only AI models and cannot raise prices or compete with free.
- brucethemoose2 3y ago
  Hardware is still a limiting factor. Cloud AI providers get a big advantage from batching/pipelining and fancy ASICs. The question is how much they are willing to lower the tax.
- empath-nirvana 3y ago
  Open Source doesn't mean free. It costs a lot of money to run models and keep models up to date, and maybe a "good enough" model runs relatively cheaply, but there's always going to be a "state of the art" that people are willing to pay for.
- thewataccount 3y ago
  > Stability AI, Apple, Meta, etc are clearly at the finish line I'm very optimistic and expect them to catch up. I've used the open models a lot, to be clear they are starting to compare to GPT3.5Turbo right now, they can't compete with GPT4 at all. GPT4 is almost a year old from when it finished training I think? I expect open source models to stay ~1.5 years behind. That said they will eventually be "good enough". Keep in mind too though that using and scaling GPUs is not free. You have to run the models somewhere. Most businesses will still prefer a simple api to call instead of managing the infrastructure. On top of this many business (medium and smaller) will likely find models like GPT4 to be sufficient for their workload, and will appreciate the built in "rails" for their specific usecases. tl;dr - open models don't even compare to GPT4 yet (I use them all daily), they aren't free to run, and a API option is still preferably to a massive if not most companies.
  - nwoli 3y ago
    > Keep in mind too though that using and scaling GPUs is not free. You have to run the models somewhere. Long or medium term these will probably be dirt cheap to just run in the background though. It might be within 3-5 years since parallel compute is still growing and isn’t as bounded by moores law stagnation
    - thewataccount 3y ago
      I get decent performance with my 4090, enough that LLMs with exllama at 30B quantitized are very usable. But we're severely VRAM limited, especially on lower end hardware which rarely sees > 10GB of VRAM. I don't know how much slower it could be and still be useful though. The big thing is we need more VRAM, 30B is context length limited with only 24GB of vram, I've only barely made it above 3.2k tokens before running out. I hope you're right, that it becomes common for systems to have either dedicated TPU type stuff similar to smartphones, and that they absolutely load the crap with VRAM (which I don't think is even that expensive?) Models will also get smaller but I'm skeptical we'll get GPT4 performance with any useful context length under 24GB VRAM any time soon.
- _pdp_ 3y ago
  Lots of folks out there would rather skip the hassle of running their own models, and that's totally understandable. Similarly, you've got plenty of folks who'd rather pay for managed hosting services instead of dealing with the nitty-gritty of setting up everything themselves using free tools. This opens up exciting opportunities for successful companies to offer some real perks – think convenience, a smoother user experience, and lightning-fast speeds, just to name a few! All of these things save time and are worth paying for.
sebzim4500 3y ago
Hard to believe it can work that well when it only has 3B parameters, but I'd love to be proven wrong.
- politelemon 3y ago
  But it does mean, hopefully, it is easier to run on small hardware. Making it much more accessible.
- thewataccount 3y ago
  I was impressed enough by replit's 2.7B model that I'm convinced it's doable. I have a 4090 and consider that the "max expected card for a consumer to own". Also exllama doesn't support non-llama models and the creator doesn't seem interested in adding support for wizardcoder/etc. Because of this, using the alternatives are prohibitively slow to use a quantized 16B model on a 4090 (if the exllama author reads this _please_ add support for other model types!). 3B models with refact are pretty snappy with Refact, about as fast as github copilot. The other benefit is more context space, which will be a limiting factor for 16B models. tl;dr - I think we need ~3B models if we want any chance of consumer hardware to reasonably run coding models akin to github copilot with decent context length. And I think it's doable.
  - eyegor 3y ago
    I'm fairly confident a coding specific model should be a lot smaller - 3b should be plenty if not 1b or less. As it stands, there are quite a few 7-13b model sizes that can predict natural language quite well. Code seems at its surface a much simpler language, strict grammars, etc so I wouldn't think it needs to be anywhere near as large as the nlp models. Right now people are retraining nlp models to work with code, but I think the best code helper models in the future will be trained primarily on code and maybe fine tuned on some language. I'm thinking less of a chat bot api and more of a giant leap in "intellisense" services.
    - gsuuon 3y ago
      I'd really like to see smaller models trained on only one specific language, with it's own language specific tokenizer. I imagine the reduction in vocab size would translate to handling more context easier?
runako 3y ago
Is this a "product" that one could install and use or a model that one should expect an OEM to integrate into a product before programmers can use it? I'm asking because I don't see any links that would help me figure out how to try it out.
- yohannparis 3y ago
  To be honest, you’d better buy GitHub co-pilot and enjoy the productivity boost at a cheap price. Trying to download/install/setup/use StableCode is worth it only if you want to learn all those steps as well. If what you care is the final result, just buy an existing service.
  - arcanemachiner 3y ago
    I may put all my open source stuff on GitHub, but hell will freeze over before I willingly let Microsoft get a whiff of my private data, no matter how irrelevant it may be. GitHub Copilot sounds pretty neat though, I will admit that.
  - hmottestad 3y ago
    I have bought into co-pilot, but I can’t say it’s that much of a productivity boost. More often than not it recommends something completely wrong. I guess it might be more useful if I did more spring boot or maybe hibernate. I’ve found chat gpt to be more helpful in general. I can paste some code in and have a discussion about what I want it to fix for me.
  - fruit2020 3y ago
    I didn’t have a good experience with copilot. It was ok for some auto completions, but I found it very distracting to correct it many times when it didn’t do well. It’s like it interrupted my flow. Maybe there is a shortcut to enable it on demand rather than always on, but it wasn’t obvious in the Jetbrains plugin
  - selfhoster11 3y ago
    Yeah, why not upload all my employer's proprietary code to Microsoft? What could possibly go wrong? I get that consuming an API is far easier than setting up your own inference backend, but there are legitimate issues to consider before going in that direction.
monlockandkey 3y ago
Any performance metrics?
thewataccount 3y ago
I can't seem to find a demo, if anyone has a chance to test it, how does it compare to replit and wizardcoder?
- james2doyle 3y ago
  Looks like there is one on the Hugging Face page: https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b?text=%23%23%23Instruction%0AGenerate+a+python+function+to+find+number+of+CPU+cores%0A%23%23%23Response https://huggingface.co/stabilityai/stablecode-instruct-alpha... Not very promising based on this lame test
  - politelemon 3y ago
    I ran it locally and it seemed to do better. I switched Python to Bash and it also gave a good answer (nproc).
nwoli 3y ago
I love stability AI
3rd3 3y ago
How does it compare to GitHub Copilot?
- jstummbillig 3y ago
  When they don't voluntarily answer the question, you know the answer.
  - sebzim4500 3y ago
    It's not easy to compare them, to be fair. I guess you could come up with a thousand example prompts and pay some students to pick which output is better, but I can also see why you wouldn't bother. It probably depends on language, type of prompt, etc.
    - erwald 3y ago
      Sure it's easy -- you can use benchmarks like HumanEval, which Stability did. They just didn't compare to Codex or GPT-4. Of course such benchmarks don't capture all aspects of an LLM's capabilities, but they're a lot better than nothing!
    - maaaaattttt 3y ago
      One could team up with Hackerrank/leetcode, let the model code in the interface (maybe there's an API for that already, no idea), execute their code verbatim and see how many test cases they get right the first time around. Then, like for humans, give them a clue about one of the tests no passing (or code not working, too slow, etc.). Give points based on the difficulty of the question and the number of clues needed. I guess the obvious caveat is that these model are probably overfitted on these types of questions. But a specific benchmark could be made containing question kept secret for models. Time to build "Botrank" I guess.
- miohtama 3y ago
  The model, source, etc. are available under permissive terms https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b https://huggingface.co/stabilityai/stablecode-instruct-alpha... You can “run it locally”. Very handy if you do not trust automatically sending all your code to someone in the United States.
cutler 3y ago
Yet another site whose data privacy policy amounts to nothing more than an Accept button. Refuse to use such sites.
- capableweb 3y ago
  It's a model you download and run yourself, on your own hardware. No privacy policy needed.
  - barrotes 3y ago
    He's referring to the actual website, which doesn't give you the option of reject profilation cookies (mandatory in Europe). I commented about another website posted here few days ago. It gets me mad too
- smcleod 3y ago
  Use uBlock Origin and then you won't have to see them ;)
  - cutler 3y ago
    But won't that just default you to agreeing to whatever.
    - smcleod 3y ago
      The ublock cookie ones will decline if it’s possible I believe. IANAL but I doubt that most of the things that pop up websites, get you to agree to would really hold up in court unless they had evidence that you specifically agreed to something. From what I’ve seen most of them are like it or lump it where if you don’t agree, they exit you from the website anyway.
RomanPushkin 3y ago
Is it good at algos? From interviews: Implement queue that supports three methods: * push * pop * peek(i) peek returns element by its index. All three methods should have O(1) complexity [write code in Ruby]. ChatGPT wasn't able to solve that last time I tried https://twitter.com/romanpushkin/status/1617037136364199938 https://twitter.com/romanpushkin/status/1617037136364199938
- voxl 3y ago
  in what world is a hashtable lookup worst case O(1)? Your own solution doesn't match your requirements. If you want amortized complexity then a simple vector suffices.
  - RomanPushkin 3y ago
    1. I like toxic comments like that saying something is simple without actually solving the problem, you're the best. 2. The average complexity to search, insert, and delete data in a hash table is O(1), for interviews it works 99% of the time. 3. There is alternative O(1) solution you're looking for, I'll leave this exercise to you, bro. As well as the other exercise of being less toxic and a bit more respectful to people you don't know online lol.
    - Mystery-Machine 3y ago
      Average and big-O notation don't go together... Yes, it works 99% of the time, but parent explained to you why you won't get that 1% of the offers. That, plus taking offence by his comment, which makes you not pass behavioral part of the interview.
      - RomanPushkin 3y ago
        > which makes you not pass behavioral part of the interview I'm okay not getting 1% of the offers. I'm not $100 bill so everyone likes me. I appreciate an attempt to educate me though, I wanted to make clear that any discussion like that is useless without solution to a problem. Post your solution, we'll discuss downsides. You can see it from my side, and I have another one. The parent commenter ain't got no solution, but keeps insisting he can implement that easily with this and that... Good luck passing interviews with that attitude...
gaogao 3y ago
Its metrics on HumanEval seem not particularly good (26.89 Pass@1 for it vs. 61.64 for PanGu-Coder2 15b). Is it targeting a very specific latency for responses? I'd think a 15b quantization should run fast enough for most use cases? Even phi-1 1.3B has better performance at 50.6.
Uptrenda 3y ago
> People of every background will soon be able to create code to solve their everyday problems and improve their lives using AI, and we’d like to help make this happen Yeah, this is not going to happen. Anyone who has ever tried to gather requirements for software knows that users don't know what they want (clients especially lmao.) The language they use won't be detailed enough to create anything meaningful. Do you know what language would be? Code... Unironically, the best language for software isn't English. It's code. Should you specify what you want in enough detail for it to be meaningful suddenly you're doing something quite peculiar. Where have I heard it before? Oh yeah, you're writing code. These tools are all annoying AF. Developers don't need half-baked hints to write basic statements and regular people don't have the skills to hobble together whatever permutations these things spit out. Which rather begs the question: who the hell is the audience for this?
- deleted 3y ago
  [deleted]
- pylua 3y ago
  It’s about giving the domain experts who understand what the requirements should be a way to build something without having to have the domain knowledge of code.
  - smoldesu 3y ago
    I think the problem with that is the same reason why "no code" platforms struggle to succeed. Writing software without understanding control flow or libraries or APIs is practically impossible. Instead of being a liberating experience like it should be, it becomes a confining and frustrating one where you don't understand what is and isn't possible. LLMs will work really well when developers know what they want and how to ask for it, same with many no-code platforms. If you don't understand programming though, you can't even know if your request is possible.
    - pylua 3y ago
      That is true. I have seen this first hand with bpmn tools such as camunda, where the analysts simply cannot wrap their head around it. However, over time you will need to describe less and less of the code for a large majority of use cases. I expect Generative AI will be able to take more generic prompts based on a specific vendor and really generate more with less prompting given context of whatever you are targeting. Ie azure , camunda, etc
ethereal_ai 3y ago
As a user who cares more about the product, how does it compare to the gpt-4 code capability? gpt-4 is good enough for me, if it works better than gpt-4 I would love to try it!
whimsicalism 3y ago
> ~120,000 code instruction/response pairs in Alpaca format were trained on the base model to achieve this result. Very curious where they are getting this data from. In other open source papers, usually this comes from a GPT-4 output, but presumably Stability would not do that?
dragonwriter 3y ago
> People of every background will soon be able to create code to solve their everyday problems and improve their lives using AI, and we’d like to help make this happen Just like everytime people hyping a technology have said this with something else where “AI” is but otherwise an identical claim, no, it didn’t happen last time, its not happening this time, and there’s a pretty good chance its not happening next time, either.
jaimani_langoo 3y ago
AI Cannot magically read minds. Having said that It would be nicer to have complete solutions rather than code hints. Imagine having to write a detailed prompt rather than choosing a prediction. Something like : "Write a React/Node JS app that has authentication and a home page" and the AI model give you a complete project as the output. It would be great if it generates deterministic output for the Prompt. AI can really help increase the productivity of Programmers.
MyAccountYo 3y ago
I have thought about how these tools can be useful quite a lot. I have a prompt I can feed chat gpt and it will create whole feature "skeletons" with my naming rules and architecture quirks. Taking a lot of time from getting started when building something new. But with chat it is still too inconvenient, having something like this integrated in the ide via a script would he more convenient but still a very specific use case. I think what I want is this idea of "code completion" but not for writing the methods, which is the easy part. Instead the tool should structure classes and packages and modules and naming and suggest better ways to write certain things.
eduardocrs 3y ago
"People will never ..." Ai: "Hold my beer".
smcleod 3y ago
If I’m reading this correctly this could be an open source model that may compete with the likes of copilot? That is something I’d be very interested in if they can get the compute requirements down to those of say a standard 13B model. Then I could fine tune (correct term?) it on my offline data and hook it into something like fauxpilot and my IDE. I had a look at some of the recent code models (wizardcoder,strider etc) but it seemed that you need a really large model to be any good and quite a few of them were trying specifically for python.
- smcleod 3y ago
  Trained specifically for Python*

305 points by kpozin 3y ago

rvz 3y ago

Either way, the race to zero has been further accelerated. Stability AI, Apple, Meta, etc are clearly at the finish line putting pressure on cloud only AI models and cannot raise prices or compete with free.

brucethemoose2 3y ago
Hardware is still a limiting factor. Cloud AI providers get a big advantage from batching/pipelining and fancy ASICs. The question is how much they are willing to lower the tax.
empath-nirvana 3y ago
Open Source doesn't mean free. It costs a lot of money to run models and keep models up to date, and maybe a "good enough" model runs relatively cheaply, but there's always going to be a "state of the art" that people are willing to pay for.
thewataccount 3y ago
> Stability AI, Apple, Meta, etc are clearly at the finish line I'm very optimistic and expect them to catch up. I've used the open models a lot, to be clear they are starting to compare to GPT3.5Turbo right now, they can't compete with GPT4 at all. GPT4 is almost a year old from when it finished training I think? I expect open source models to stay ~1.5 years behind. That said they will eventually be "good enough". Keep in mind too though that using and scaling GPUs is not free. You have to run the models somewhere. Most businesses will still prefer a simple api to call instead of managing the infrastructure. On top of this many business (medium and smaller) will likely find models like GPT4 to be sufficient for their workload, and will appreciate the built in "rails" for their specific usecases. tl;dr - open models don't even compare to GPT4 yet (I use them all daily), they aren't free to run, and a API option is still preferably to a massive if not most companies.
- nwoli 3y ago
  > Keep in mind too though that using and scaling GPUs is not free. You have to run the models somewhere. Long or medium term these will probably be dirt cheap to just run in the background though. It might be within 3-5 years since parallel compute is still growing and isn’t as bounded by moores law stagnation
  - thewataccount 3y ago
    I get decent performance with my 4090, enough that LLMs with exllama at 30B quantitized are very usable. But we're severely VRAM limited, especially on lower end hardware which rarely sees > 10GB of VRAM. I don't know how much slower it could be and still be useful though. The big thing is we need more VRAM, 30B is context length limited with only 24GB of vram, I've only barely made it above 3.2k tokens before running out. I hope you're right, that it becomes common for systems to have either dedicated TPU type stuff similar to smartphones, and that they absolutely load the crap with VRAM (which I don't think is even that expensive?) Models will also get smaller but I'm skeptical we'll get GPT4 performance with any useful context length under 24GB VRAM any time soon.
_pdp_ 3y ago
Lots of folks out there would rather skip the hassle of running their own models, and that's totally understandable. Similarly, you've got plenty of folks who'd rather pay for managed hosting services instead of dealing with the nitty-gritty of setting up everything themselves using free tools. This opens up exciting opportunities for successful companies to offer some real perks – think convenience, a smoother user experience, and lightning-fast speeds, just to name a few! All of these things save time and are worth paying for.

sebzim4500 3y ago

Hard to believe it can work that well when it only has 3B parameters, but I'd love to be proven wrong.

politelemon 3y ago
But it does mean, hopefully, it is easier to run on small hardware. Making it much more accessible.
thewataccount 3y ago
I was impressed enough by replit's 2.7B model that I'm convinced it's doable. I have a 4090 and consider that the "max expected card for a consumer to own". Also exllama doesn't support non-llama models and the creator doesn't seem interested in adding support for wizardcoder/etc. Because of this, using the alternatives are prohibitively slow to use a quantized 16B model on a 4090 (if the exllama author reads this _please_ add support for other model types!). 3B models with refact are pretty snappy with Refact, about as fast as github copilot. The other benefit is more context space, which will be a limiting factor for 16B models. tl;dr - I think we need ~3B models if we want any chance of consumer hardware to reasonably run coding models akin to github copilot with decent context length. And I think it's doable.
- eyegor 3y ago
  I'm fairly confident a coding specific model should be a lot smaller - 3b should be plenty if not 1b or less. As it stands, there are quite a few 7-13b model sizes that can predict natural language quite well. Code seems at its surface a much simpler language, strict grammars, etc so I wouldn't think it needs to be anywhere near as large as the nlp models. Right now people are retraining nlp models to work with code, but I think the best code helper models in the future will be trained primarily on code and maybe fine tuned on some language. I'm thinking less of a chat bot api and more of a giant leap in "intellisense" services.
  - gsuuon 3y ago
    I'd really like to see smaller models trained on only one specific language, with it's own language specific tokenizer. I imagine the reduction in vocab size would translate to handling more context easier?

runako 3y ago

Is this a "product" that one could install and use or a model that one should expect an OEM to integrate into a product before programmers can use it? I'm asking because I don't see any links that would help me figure out how to try it out.

yohannparis 3y ago
To be honest, you’d better buy GitHub co-pilot and enjoy the productivity boost at a cheap price. Trying to download/install/setup/use StableCode is worth it only if you want to learn all those steps as well. If what you care is the final result, just buy an existing service.
- arcanemachiner 3y ago
  I may put all my open source stuff on GitHub, but hell will freeze over before I willingly let Microsoft get a whiff of my private data, no matter how irrelevant it may be. GitHub Copilot sounds pretty neat though, I will admit that.
- hmottestad 3y ago
  I have bought into co-pilot, but I can’t say it’s that much of a productivity boost. More often than not it recommends something completely wrong. I guess it might be more useful if I did more spring boot or maybe hibernate. I’ve found chat gpt to be more helpful in general. I can paste some code in and have a discussion about what I want it to fix for me.
- fruit2020 3y ago
  I didn’t have a good experience with copilot. It was ok for some auto completions, but I found it very distracting to correct it many times when it didn’t do well. It’s like it interrupted my flow. Maybe there is a shortcut to enable it on demand rather than always on, but it wasn’t obvious in the Jetbrains plugin
- selfhoster11 3y ago
  Yeah, why not upload all my employer's proprietary code to Microsoft? What could possibly go wrong? I get that consuming an API is far easier than setting up your own inference backend, but there are legitimate issues to consider before going in that direction.

monlockandkey 3y ago

Any performance metrics?

thewataccount 3y ago

I can't seem to find a demo, if anyone has a chance to test it, how does it compare to replit and wizardcoder?

james2doyle 3y ago
Looks like there is one on the Hugging Face page: https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b?text=%23%23%23Instruction%0AGenerate+a+python+function+to+find+number+of+CPU+cores%0A%23%23%23Response https://huggingface.co/stabilityai/stablecode-instruct-alpha... Not very promising based on this lame test
- politelemon 3y ago
  I ran it locally and it seemed to do better. I switched Python to Bash and it also gave a good answer (nproc).

nwoli 3y ago

I love stability AI

3rd3 3y ago

How does it compare to GitHub Copilot?

jstummbillig 3y ago
When they don't voluntarily answer the question, you know the answer.
- sebzim4500 3y ago
  It's not easy to compare them, to be fair. I guess you could come up with a thousand example prompts and pay some students to pick which output is better, but I can also see why you wouldn't bother. It probably depends on language, type of prompt, etc.
  - erwald 3y ago
    Sure it's easy -- you can use benchmarks like HumanEval, which Stability did. They just didn't compare to Codex or GPT-4. Of course such benchmarks don't capture all aspects of an LLM's capabilities, but they're a lot better than nothing!
  - maaaaattttt 3y ago
    One could team up with Hackerrank/leetcode, let the model code in the interface (maybe there's an API for that already, no idea), execute their code verbatim and see how many test cases they get right the first time around. Then, like for humans, give them a clue about one of the tests no passing (or code not working, too slow, etc.). Give points based on the difficulty of the question and the number of clues needed. I guess the obvious caveat is that these model are probably overfitted on these types of questions. But a specific benchmark could be made containing question kept secret for models. Time to build "Botrank" I guess.
miohtama 3y ago
The model, source, etc. are available under permissive terms https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b https://huggingface.co/stabilityai/stablecode-instruct-alpha... You can “run it locally”. Very handy if you do not trust automatically sending all your code to someone in the United States.

cutler 3y ago

Yet another site whose data privacy policy amounts to nothing more than an Accept button. Refuse to use such sites.

capableweb 3y ago
It's a model you download and run yourself, on your own hardware. No privacy policy needed.
- barrotes 3y ago
  He's referring to the actual website, which doesn't give you the option of reject profilation cookies (mandatory in Europe). I commented about another website posted here few days ago. It gets me mad too
smcleod 3y ago
Use uBlock Origin and then you won't have to see them ;)
- cutler 3y ago
  But won't that just default you to agreeing to whatever.
  - smcleod 3y ago
    The ublock cookie ones will decline if it’s possible I believe. IANAL but I doubt that most of the things that pop up websites, get you to agree to would really hold up in court unless they had evidence that you specifically agreed to something. From what I’ve seen most of them are like it or lump it where if you don’t agree, they exit you from the website anyway.

RomanPushkin 3y ago

Is it good at algos? From interviews: Implement queue that supports three methods: * push * pop * peek(i) peek returns element by its index. All three methods should have O(1) complexity [write code in Ruby]. ChatGPT wasn't able to solve that last time I tried https://twitter.com/romanpushkin/status/1617037136364199938 https://twitter.com/romanpushkin/status/1617037136364199938

voxl 3y ago
in what world is a hashtable lookup worst case O(1)? Your own solution doesn't match your requirements. If you want amortized complexity then a simple vector suffices.
- RomanPushkin 3y ago
  1. I like toxic comments like that saying something is simple without actually solving the problem, you're the best. 2. The average complexity to search, insert, and delete data in a hash table is O(1), for interviews it works 99% of the time. 3. There is alternative O(1) solution you're looking for, I'll leave this exercise to you, bro. As well as the other exercise of being less toxic and a bit more respectful to people you don't know online lol.
  - Mystery-Machine 3y ago
    Average and big-O notation don't go together... Yes, it works 99% of the time, but parent explained to you why you won't get that 1% of the offers. That, plus taking offence by his comment, which makes you not pass behavioral part of the interview.
    - RomanPushkin 3y ago
      > which makes you not pass behavioral part of the interview I'm okay not getting 1% of the offers. I'm not $100 bill so everyone likes me. I appreciate an attempt to educate me though, I wanted to make clear that any discussion like that is useless without solution to a problem. Post your solution, we'll discuss downsides. You can see it from my side, and I have another one. The parent commenter ain't got no solution, but keeps insisting he can implement that easily with this and that... Good luck passing interviews with that attitude...

gaogao 3y ago

Its metrics on HumanEval seem not particularly good (26.89 Pass@1 for it vs. 61.64 for PanGu-Coder2 15b). Is it targeting a very specific latency for responses? I'd think a 15b quantization should run fast enough for most use cases? Even phi-1 1.3B has better performance at 50.6.

Uptrenda 3y ago

> People of every background will soon be able to create code to solve their everyday problems and improve their lives using AI, and we’d like to help make this happen Yeah, this is not going to happen. Anyone who has ever tried to gather requirements for software knows that users don't know what they want (clients especially lmao.) The language they use won't be detailed enough to create anything meaningful. Do you know what language would be? Code... Unironically, the best language for software isn't English. It's code. Should you specify what you want in enough detail for it to be meaningful suddenly you're doing something quite peculiar. Where have I heard it before? Oh yeah, you're writing code. These tools are all annoying AF. Developers don't need half-baked hints to write basic statements and regular people don't have the skills to hobble together whatever permutations these things spit out. Which rather begs the question: who the hell is the audience for this?

deleted 3y ago
[deleted]
pylua 3y ago
It’s about giving the domain experts who understand what the requirements should be a way to build something without having to have the domain knowledge of code.
- smoldesu 3y ago
  I think the problem with that is the same reason why "no code" platforms struggle to succeed. Writing software without understanding control flow or libraries or APIs is practically impossible. Instead of being a liberating experience like it should be, it becomes a confining and frustrating one where you don't understand what is and isn't possible. LLMs will work really well when developers know what they want and how to ask for it, same with many no-code platforms. If you don't understand programming though, you can't even know if your request is possible.
  - pylua 3y ago
    That is true. I have seen this first hand with bpmn tools such as camunda, where the analysts simply cannot wrap their head around it. However, over time you will need to describe less and less of the code for a large majority of use cases. I expect Generative AI will be able to take more generic prompts based on a specific vendor and really generate more with less prompting given context of whatever you are targeting. Ie azure , camunda, etc

ethereal_ai 3y ago

As a user who cares more about the product, how does it compare to the gpt-4 code capability? gpt-4 is good enough for me, if it works better than gpt-4 I would love to try it!

whimsicalism 3y ago

> ~120,000 code instruction/response pairs in Alpaca format were trained on the base model to achieve this result. Very curious where they are getting this data from. In other open source papers, usually this comes from a GPT-4 output, but presumably Stability would not do that?

dragonwriter 3y ago

> People of every background will soon be able to create code to solve their everyday problems and improve their lives using AI, and we’d like to help make this happen Just like everytime people hyping a technology have said this with something else where “AI” is but otherwise an identical claim, no, it didn’t happen last time, its not happening this time, and there’s a pretty good chance its not happening next time, either.

jaimani_langoo 3y ago

AI Cannot magically read minds. Having said that It would be nicer to have complete solutions rather than code hints. Imagine having to write a detailed prompt rather than choosing a prediction. Something like : "Write a React/Node JS app that has authentication and a home page" and the AI model give you a complete project as the output. It would be great if it generates deterministic output for the Prompt. AI can really help increase the productivity of Programmers.

MyAccountYo 3y ago

I have thought about how these tools can be useful quite a lot. I have a prompt I can feed chat gpt and it will create whole feature "skeletons" with my naming rules and architecture quirks. Taking a lot of time from getting started when building something new. But with chat it is still too inconvenient, having something like this integrated in the ide via a script would he more convenient but still a very specific use case. I think what I want is this idea of "code completion" but not for writing the methods, which is the easy part. Instead the tool should structure classes and packages and modules and naming and suggest better ways to write certain things.

eduardocrs 3y ago

"People will never ..." Ai: "Hold my beer".

smcleod 3y ago

If I’m reading this correctly this could be an open source model that may compete with the likes of copilot? That is something I’d be very interested in if they can get the compute requirements down to those of say a standard 13B model. Then I could fine tune (correct term?) it on my offline data and hook it into something like fauxpilot and my IDE. I had a look at some of the recent code models (wizardcoder,strider etc) but it seemed that you need a really large model to be any good and quite a few of them were trying specifically for python.

smcleod 3y ago
Trained specifically for Python*