8 ms·

This is awesome, can't wait to get api access to the 32k token model. Rather than this approach of just converting the whole repo to a text file, what I'm think

by ftufek 4y ago

This is awesome, can't wait to get api access to the 32k token model. Rather than this approach of just converting the whole repo to a text file, what I'm thinking is, you can let the model decide the most relevant files. The initial prompt would be, "person wants to do x, here are the file list of this repo: ...., give me a list of files that you'd want to edit, create or delete" -> take the list, try to fit the contents of them into 32k tokens and re-prompt with "user is trying to achieve x, here's the most relevant files with their contents:..., give me a git commit in the style of git patch/diff output". From playing around with it today, I think this approach would work rather well and can be like a huge step up from AI line autocompletion.

alooPotato 4y ago
It's just so slow for the autocompletion use case to do it like that. Ideally, you're never chaining serial requests to the LLM. Even if you do stuff in all the data into a single prompt, the execution time seems to be superlinear with the number of tokens, again getting super slow.
- ftufek 4y ago
  Yeah I agree it's too slow for autocompletion at the moment, but this would be for full feature implementations, not just autocomplete. For example, if I have a repo I want to add a table and rest api implementation in, it can do this: https://imgur.com/a/mIJvaJr https://imgur.com/a/mIJvaJr (ignore the formatting errors in the UI, somehow parts of it show up in as code and others not, but api wouldn't have this issue, especially since you can use the system message to enforce output format). I'm happy to wait even 30-60 seconds for this which I can easily evaluate, criticize (and the model will correct it) and then proceed to just patch and move on. I think the results from this will be much better with the 32k model, but remains to be seen.
wokwokwok 4y ago
Just remember the API charge is 6c per input token [1]. If you push 32k input tokens in, you're looking at $2000 per API call just as input. You... might wanna consider a self hosted alternative for that use case, or at least do like, a `| wc` to get an idea of what you're potentially sending before calling the api. [1] - https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp...
- dangond 4y ago
  6c per thousand tokens, so $2 per maxed out API call
  - wokwokwok 4y ago
    oh, you're quite right, I didn't see that it was per /1k tokens. My bad!
  - holoduke 4y ago
    Still super expensive. You cant build a business arround it with these rates. It should be at least 1000 times cheaper if not more.
    - sharemywin 4y ago
      depends on if you can build something that can classify what api to call.
    - anonzzzies 4y ago
      Depends how good it is now, doesn’t it? If it actually writes code (it does), which is ‘good enough’ compared to a 25-150$/hr human (it does for the lower part of that scale), the $2 is definitely a good business case. For instance, I had to write 1500 lines of js yesterday because one of my colleagues who took the task could not manage it in the previous 2 days and the deadline was today. We would’ve saved about 500$ in total doing it with that expensive API from the start. Now it was free besides my hourly wage (it took me little over an hour) so it made even more sense.
      - cornholio 4y ago
        > which is ‘good enough’ compared to a 25-150$/hr human (it does for the lower part of that scale), I'm not exactly sure you define that scale, perhaps Minecraft bots or the like, where the damage after complete failure is self contained to perhaps a few dollars or a few hours of human annoyance. I'm sure there are many niches where a 50% success rate of mass generated programs can earn you big bucks. But in my experience Codex does very limited reasoning about code paths. For the current state of the art, you are almost guaranteed to have catastrophic bugs in any non-trivial programs engineered by prompt.
visarga 4y ago
Working with GPT becomes like coding in plain English.
- victorbjorklund 4y ago
  And we are back with Cobol haha full circle
- marginalia_nu 4y ago
  There's a reason we don't code in plain English though. Natural language has ambiguities. This is the reason we invented programming languages. It's best illustrated by the old joke: A programmer's wife told him "Go to the store and buy milk and if they have eggs, get a dozen." He came back a while later with 12 cartons of milk. A good chunk of all bugs in software are down to the requirements being insufficiently well specified. Further, many bugs are the discovery of new requirements when informal specification encounters reality. "Read from standard input into this byte array" doesn't specify what to do when the input exceeds the byte array. When you overflow the buffer, you get a "well obviously you're supposed to not do that"... that's wasn't stated at all. When the function keeps going after a newline or a null byte or whatever, there's another "well obviously you're supposed to stop at those points". That was also not specified. and so on. At the point you're specifying all these cases and what to do when, it's so specific and stilted, you might as well be using a programming language.
  - dilap 4y ago
    Actually, programming languages were invented because speaking machine code was too much of a pain in the ass! Programming in English is a natural next step. Ambiguity is not an issue -- you keep speaking until its resolved. (We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.)
    - marginalia_nu 4y ago
      This type of well-defined language actually predates computers by several thousand years. Even way back in antiquity they used "programming languages" like these to get around the inherent ambiguities of natural language. Originally as formulaic syllogisms and Aristotelian logic, but then onto other forms of codified language, formal logic etc. Adding more words often makes things less clear, not more so. What you need is well-defined terms with no overloaded meaning. > (We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.) Humans get it wrong all the time though. A great many bugs arise from quite simply misinterpreting the requirements. Which leads to requirements becoming more formulaic and resembling a programming language.
groestl 4y ago
Maybe someone can correct me, but my understanding is that you would calculate the embeddings of code chunks, and the embedding of the prompt, and take those chunks that are most similar to the embedding of the prompt as context. Edit: This, btw, is also the reason why I think that this here popped up on the hackernews frontpage a short while ago: https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector
- michaelmior 4y ago
  This sounds like a reasonable start. Eventually we need to get to the point where we can expose an API for models to request additional information on their own.
  - samstave 4y ago
    Exactly, but this has some scary implications in the future - imagine when it is common pratice to allow AIs API access as a matter of course... When giving a prompt, the prompt causes the crawling of many APIs to build the response - the power of such activity/features, will be scary power-to-authoratarian goals. Imagine if the prompt is "Select all users who have political beliefs, posts, comments, links from APIs A, B, C, etc where sentiment appears to dissent from [party line]"
    - groestl 4y ago
      Imagine that these prompts are not triggered by humans anymore, but by the AI invoking itself. Sam Altman takes comfort in the thought that their AI does nothing without a human prompting it, so it has a human in the loop, as a circuit breaker if you will. This assumption is rapidly becoming a mere hope, as right now probably hundreds of developers are working on systems which, when put into production and connected to other systems, might just come down to: the AI is calling itself, and giving itself orders.
      - samstave 4y ago
        Imagine that you just out-did me on how dystopian this AI cyberpunk future could get. Imagine I want you to shoot me.' Imagine I Want [whatever]GPT to make a cyberpunk Anime based on such...
wongarsu 4y ago
You would probably should also include all relevant imports. So in C/C++ add all non-standard headers referenced by those files, in other languages simulate the import system, maybe pruning imported files to just the important parts (type definitions etc)
capableweb 4y ago
AFAIK, only the older models allows you to do fine tuning, not sure GPT4 will allow to create your own fine tuned model so basically with the API it will work the same as with the chat gui.
wantsanagent 4y ago
Please see the following repos for tools in this area: https://github.com/jerryjliu/llama_index https://github.com/jerryjliu/llama_index and/or https://github.com/hwchase17/langchain https://github.com/hwchase17/langchain

by ftufek 4y ago

alooPotato 4y ago

It's just so slow for the autocompletion use case to do it like that. Ideally, you're never chaining serial requests to the LLM. Even if you do stuff in all the data into a single prompt, the execution time seems to be superlinear with the number of tokens, again getting super slow.

ftufek 4y ago
Yeah I agree it's too slow for autocompletion at the moment, but this would be for full feature implementations, not just autocomplete. For example, if I have a repo I want to add a table and rest api implementation in, it can do this: https://imgur.com/a/mIJvaJr https://imgur.com/a/mIJvaJr (ignore the formatting errors in the UI, somehow parts of it show up in as code and others not, but api wouldn't have this issue, especially since you can use the system message to enforce output format). I'm happy to wait even 30-60 seconds for this which I can easily evaluate, criticize (and the model will correct it) and then proceed to just patch and move on. I think the results from this will be much better with the 32k model, but remains to be seen.

wokwokwok 4y ago

Just remember the API charge is 6c per input token [1]. If you push 32k input tokens in, you're looking at $2000 per API call just as input. You... might wanna consider a self hosted alternative for that use case, or at least do like, a `| wc` to get an idea of what you're potentially sending before calling the api. [1] - https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp...

dangond 4y ago
6c per thousand tokens, so $2 per maxed out API call
- wokwokwok 4y ago
  oh, you're quite right, I didn't see that it was per /1k tokens. My bad!
- holoduke 4y ago
  Still super expensive. You cant build a business arround it with these rates. It should be at least 1000 times cheaper if not more.
  - sharemywin 4y ago
    depends on if you can build something that can classify what api to call.
  - anonzzzies 4y ago
    Depends how good it is now, doesn’t it? If it actually writes code (it does), which is ‘good enough’ compared to a 25-150$/hr human (it does for the lower part of that scale), the $2 is definitely a good business case. For instance, I had to write 1500 lines of js yesterday because one of my colleagues who took the task could not manage it in the previous 2 days and the deadline was today. We would’ve saved about 500$ in total doing it with that expensive API from the start. Now it was free besides my hourly wage (it took me little over an hour) so it made even more sense.
    - cornholio 4y ago
      > which is ‘good enough’ compared to a 25-150$/hr human (it does for the lower part of that scale), I'm not exactly sure you define that scale, perhaps Minecraft bots or the like, where the damage after complete failure is self contained to perhaps a few dollars or a few hours of human annoyance. I'm sure there are many niches where a 50% success rate of mass generated programs can earn you big bucks. But in my experience Codex does very limited reasoning about code paths. For the current state of the art, you are almost guaranteed to have catastrophic bugs in any non-trivial programs engineered by prompt.

visarga 4y ago

Working with GPT becomes like coding in plain English.

victorbjorklund 4y ago
And we are back with Cobol haha full circle
marginalia_nu 4y ago
There's a reason we don't code in plain English though. Natural language has ambiguities. This is the reason we invented programming languages. It's best illustrated by the old joke: A programmer's wife told him "Go to the store and buy milk and if they have eggs, get a dozen." He came back a while later with 12 cartons of milk. A good chunk of all bugs in software are down to the requirements being insufficiently well specified. Further, many bugs are the discovery of new requirements when informal specification encounters reality. "Read from standard input into this byte array" doesn't specify what to do when the input exceeds the byte array. When you overflow the buffer, you get a "well obviously you're supposed to not do that"... that's wasn't stated at all. When the function keeps going after a newline or a null byte or whatever, there's another "well obviously you're supposed to stop at those points". That was also not specified. and so on. At the point you're specifying all these cases and what to do when, it's so specific and stilted, you might as well be using a programming language.
- dilap 4y ago
  Actually, programming languages were invented because speaking machine code was too much of a pain in the ass! Programming in English is a natural next step. Ambiguity is not an issue -- you keep speaking until its resolved. (We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.)
  - marginalia_nu 4y ago
    This type of well-defined language actually predates computers by several thousand years. Even way back in antiquity they used "programming languages" like these to get around the inherent ambiguities of natural language. Originally as formulaic syllogisms and Aristotelian logic, but then onto other forms of codified language, formal logic etc. Adding more words often makes things less clear, not more so. What you need is well-defined terms with no overloaded meaning. > (We already program in English, in a sense, when we tell humans what we want, and they go code it. Now we'll just be telling machines.) Humans get it wrong all the time though. A great many bugs arise from quite simply misinterpreting the requirements. Which leads to requirements becoming more formulaic and resembling a programming language.

groestl 4y ago

Maybe someone can correct me, but my understanding is that you would calculate the embeddings of code chunks, and the embedding of the prompt, and take those chunks that are most similar to the embedding of the prompt as context. Edit: This, btw, is also the reason why I think that this here popped up on the hackernews frontpage a short while ago: https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector

michaelmior 4y ago
This sounds like a reasonable start. Eventually we need to get to the point where we can expose an API for models to request additional information on their own.
- samstave 4y ago
  Exactly, but this has some scary implications in the future - imagine when it is common pratice to allow AIs API access as a matter of course... When giving a prompt, the prompt causes the crawling of many APIs to build the response - the power of such activity/features, will be scary power-to-authoratarian goals. Imagine if the prompt is "Select all users who have political beliefs, posts, comments, links from APIs A, B, C, etc where sentiment appears to dissent from [party line]"
  - groestl 4y ago
    Imagine that these prompts are not triggered by humans anymore, but by the AI invoking itself. Sam Altman takes comfort in the thought that their AI does nothing without a human prompting it, so it has a human in the loop, as a circuit breaker if you will. This assumption is rapidly becoming a mere hope, as right now probably hundreds of developers are working on systems which, when put into production and connected to other systems, might just come down to: the AI is calling itself, and giving itself orders.
    - samstave 4y ago
      Imagine that you just out-did me on how dystopian this AI cyberpunk future could get. Imagine I want you to shoot me.' Imagine I Want [whatever]GPT to make a cyberpunk Anime based on such...

wongarsu 4y ago

You would probably should also include all relevant imports. So in C/C++ add all non-standard headers referenced by those files, in other languages simulate the import system, maybe pruning imported files to just the important parts (type definitions etc)

capableweb 4y ago

AFAIK, only the older models allows you to do fine tuning, not sure GPT4 will allow to create your own fine tuned model so basically with the API it will work the same as with the chat gui.

wantsanagent 4y ago

Please see the following repos for tools in this area: https://github.com/jerryjliu/llama_index https://github.com/jerryjliu/llama_index and/or https://github.com/hwchase17/langchain https://github.com/hwchase17/langchain