7 ms·

It's not just the ability to correctly answer this, but the consistency. I asked this exact question to the `oasst-sft-6-llama-30b` model and it was able to co

It's not just the ability to correctly answer this, but the consistency. I asked this exact question to the `oasst-sft-6-llama-30b` model and it was able to consistently get the correct answer. I then tried the smaller `vicuna-7b` model, and while it usually gave the correct answer, there was the occasional miss. Interestingly, `oasst-sft-6-llama-30b`'s ability to answer correctly seems to be fairly stable across multiple configurations. I tried various temperature settings from 0.2 up to 1.2, different topP configs, and they all answered correctly.

kordlessagain 3y ago
By adding dense vector search, the accuracy of smaller models can be improved, because the reference material has more hints in it than the frozen model…
zmmmmm 3y ago
reminds me of voice recognition On the one hand, the problem was nearly "solved" in early 2000's, getting to 95% accuracy. But the daily of experience of using something that makes mistakes at that rate is infuriating and well and truly outside of consideration for putting into any kind of critical pathway. So it's a combination of how difficult it is to close the last few percentage points of accuracy with how important they are for most of the high value use cases. For the forseeable future I see most use of this tech coming from applications where it aids humans and/or checks their decisions rather than running solo.
- deleted 3y ago
  [deleted]
- deafpolygon 3y ago
  > Computer, earl grey, hot. I think we're getting closer to something like this, out of Star Trek. Even in Star Trek, AI did not take over critical functions - but rather assisted the crew in manning the starship.
  - setr 3y ago
    I’ve never understood why voice recognition has always attempted to be complete understanding of arbitrary input, rather than follow a simple command language eg <subject> <parameters> <action>. It could be made completely reliable with current tech (even a decade ago, really), by just minimizing the possibility space… and I’m pretty sure consumers would trivially be able to learn it, as long as they don’t try to go full pseudo-programming-language mode And “Computer, execute program alpha beta seven” would be the power user version of it We should already be at “computer, earl gray, hot” today
    - jjeaff 3y ago
      Most voice assistants can work with simple phrases like that. Alexa, lights on. Hey Google, thermostat 70 degrees.
      - gnicholas 3y ago
        Not Siri, which thinks I'm talking to her all the time when I'm speaking to a family member whose name contains neither an "s" nor an "r".
        hgsgm 3y ago
        That's because letters aren't sounds.
        gnicholas 3y ago
        Way to jump to unjustified conclusions. The name also doesn’t contain either sound.
      - setr 3y ago
        The problem is that’s not the only format they work on, and because input format is largely unconstrained, when they misunderstand, they catastrophically misunderstand. It’s just like the image recognition ML issue, where it can correctly predict a cat, but change a specific three pixels and it has 99% confidence it’s an ostrich. Or JavaScript equality. If you do it right, it’s right, but otherwise anything goes. Or Perl, in its entirety
- visarga 3y ago
  > how difficult it is to close the last few percentage points of accuracy Like, after getting to 99% you are about half way, the last 1% is the hard part.
dr_dshiv 3y ago
Because randomly casting dust on a table sometimes says intelligent things, therefore there is a continuous function between dust and ChatGPT? While “dust” might be flippant, their approach does seem to suggest that even hierarchical Markov models would be able to demonstrate abilities on their continuous metrics.

kordlessagain 3y ago

By adding dense vector search, the accuracy of smaller models can be improved, because the reference material has more hints in it than the frozen model…

zmmmmm 3y ago

reminds me of voice recognition On the one hand, the problem was nearly "solved" in early 2000's, getting to 95% accuracy. But the daily of experience of using something that makes mistakes at that rate is infuriating and well and truly outside of consideration for putting into any kind of critical pathway. So it's a combination of how difficult it is to close the last few percentage points of accuracy with how important they are for most of the high value use cases. For the forseeable future I see most use of this tech coming from applications where it aids humans and/or checks their decisions rather than running solo.

deleted 3y ago
[deleted]
deafpolygon 3y ago
> Computer, earl grey, hot. I think we're getting closer to something like this, out of Star Trek. Even in Star Trek, AI did not take over critical functions - but rather assisted the crew in manning the starship.
- setr 3y ago
  I’ve never understood why voice recognition has always attempted to be complete understanding of arbitrary input, rather than follow a simple command language eg <subject> <parameters> <action>. It could be made completely reliable with current tech (even a decade ago, really), by just minimizing the possibility space… and I’m pretty sure consumers would trivially be able to learn it, as long as they don’t try to go full pseudo-programming-language mode And “Computer, execute program alpha beta seven” would be the power user version of it We should already be at “computer, earl gray, hot” today
  - jjeaff 3y ago
    Most voice assistants can work with simple phrases like that. Alexa, lights on. Hey Google, thermostat 70 degrees.
    - gnicholas 3y ago
      Not Siri, which thinks I'm talking to her all the time when I'm speaking to a family member whose name contains neither an "s" nor an "r".
      - hgsgm 3y ago
        That's because letters aren't sounds.
        gnicholas 3y ago
        Way to jump to unjustified conclusions. The name also doesn’t contain either sound.
    - setr 3y ago
      The problem is that’s not the only format they work on, and because input format is largely unconstrained, when they misunderstand, they catastrophically misunderstand. It’s just like the image recognition ML issue, where it can correctly predict a cat, but change a specific three pixels and it has 99% confidence it’s an ostrich. Or JavaScript equality. If you do it right, it’s right, but otherwise anything goes. Or Perl, in its entirety
visarga 3y ago
> how difficult it is to close the last few percentage points of accuracy Like, after getting to 99% you are about half way, the last 1% is the hard part.

dr_dshiv 3y ago

Because randomly casting dust on a table sometimes says intelligent things, therefore there is a continuous function between dust and ChatGPT? While “dust” might be flippant, their approach does seem to suggest that even hierarchical Markov models would be able to demonstrate abilities on their continuous metrics.