6 ms·

I also work at Google and I agree with the general sentiment that AI completion is not doing engineering per se, simply because writing code is just a small par

I also work at Google and I agree with the general sentiment that AI completion is not doing engineering per se, simply because writing code is just a small part of engineering. However in my experience the system is much more powerful than you described. Maybe this is because I'm mostly writing C++ for which there is a much bigger training corpus than JavaScript. One thing the system is already pretty good at is writing entire short functions from a comment. The trick is not to write: function getAc... But instead: // This function smargls the bleurgh // by flooming the trux. function getAc... This way the completion goes much farther and the quality improves a lot. Essentially, use comments as the prompt to generate large chunks of code, instead of giving minimum context to the system, which limits it to single line completion.

deleted 2y ago
[deleted]
miki123211 2y ago
This is a good idea even outside of Google, with tools like copilot and such. Often when I don't know exactly what function / sequence of functions I need to achieve a particular outcome, I put in a comment describing what I want to do, and Copilot does the rest. I then remove the comment once I make sure that the generated code actually works. I find it a lot less flow-breaking than stackoverflow or even asking an LLM. It doesn't work all of the time, and sometimes you do have to Google still, but for the cases it does work for, it's pretty nice.
- Aachen 2y ago
  Why remove the comment that summarises the intent for humans? The compiler will ignore your comment anyway, so it's only there for the next human who comes along and will help them understand the code
  - cryptonym 2y ago
    Next human will put the code in a prompt and ask what it does. Chinese Whispers.
    - lupire 2y ago
      (No need to Orientalize to defamiarize, especially when a huge fraction of the audience is Chinese, so Orientalizing doesn't defamiliarize. Game of Whispers or Telephone works fine.)
      - cryptonym 2y ago
        Pardon my French.
      - protomolecule 2y ago
        Do the Chinese call it English Whispers?
        ahoka 2y ago
        It’s all Greek to them.
        tessierashpool 2y ago
        Chinese-Americans, at least, call it a game of Telephone, like everyone else in the English-speaking world except for the actual English. We call it “Telephone” because “Chinese Whispers” not only sounds racist, it is also super confusing. You need a lot of cultural context to understand the particular way in which Chinese whispers would be different from any other set of whispers.
        tessierashpool 2y ago
        I happened to re-read this, and to be clear, I'm not Chinese-American. the "we" there means "everyone else in the English-speaking world except for the actual English."
    - Aachen 2y ago
  - qwertox 2y ago
    It's often unnecessarily verbose. If you read a comment and glance at the code that follows, you'll understand what it is supposed to do. But the comment you're giving as an instruction to an LLM usually contains information which will then be duplicated in the generated code.
    - Aachen 2y ago
      I see. Might still be good to have a verbose comment than no comment at all, as well as a marker of "this was generated" so (by the age of the code) you have some idea of what quality the LLM was in that year and whether to proofread it once more or not
      - lupire 2y ago
        External comments are API usage comments. LLM prompts are also implementation proposal. Implementation comments belong inside the implementation, so they should be over if not deleted.
  - randomdata 2y ago
    Like he says, the "comment" describes what he wants to do. That's not what humans are interested in. The human already knows "what he wants to do" when they read the code. It's the things like "why did he want to do this in the first place?" that is lacking in the code, and what information is available to add in a comment for the sake of humans. Remember, LLMs are just compilers for programming languages that just so happen to have a lot of similarities with natural language. The code is not the comment. You still need to comment your code for humans.
    - JohnFen 2y ago
      > Like he says, the "comment" describes what he wants to do. That's not what humans are interested in. When I'm maintaining other people's code, or my own after enough time has gone by, I'm very interested in that sort of comment. It gives me a chance to see if the code as written does what the comment says it was intended to do. It's not valuable for most of the code in a project, but is incredibly valuable for certain key parts. You're right that comments about why things were done the way they were are the most valuable ones, but this kind of comment is in second place in my book.
      - mithametacs 2y ago
        Or for something that needs like a quick mathematical lemma or a worked example. A comment on what is fantastic.
  - miki123211 2y ago
    Because the code, when written, is usually obvious enough. Something like: query = query.orderBy(field: "username", Ordering.DESC) Doesn't need an explanation, but when working in a language I don't know well, I might not remember whether I'm supposed to call orderBy on the query or on the ORM module and pass query as the argument, whether the kwarg is called "field" or "column", whether it wants a string or something like `User.name` as the column expression, how to specify the ordering and so on.
Aachen 2y ago
This type of not having to think about the implementation, especially in a language that we've by now well-established can't be written safely by humans (including by Google's own research into Android vulnerabilities if I'm not mistaken), at least with the current level of LLM, worries me the most Time will tell whether it outputs worse, equal, or better quality than skilled humans, but I'd be very wary of anything it suggests beyond obvious boilerplate (like all the symbols needed in a for loop) or naming things (function name and comment autocompletes like the person above you described)
- munksbeer 2y ago
  > worries me the most It isn't something I worry about at all. If it doesn't work and starts creating bugs and horrible code, the best places will adjust to that and it won't be used or will be used more judiciously. I'll still review code like I always do and prevent bad code from making it into our repo. I don't see why it's my problem to worry about. Why is it yours?
  - Aachen 2y ago
    Because I do security audits Functional bugs in edge cases are annoying enough, and I seem to run into these regularly as a user, but there's yet another class of people creating edge cases for their own purposes. The nonchalant "if it doesn't work"... I don't know whether that confirms my suspicion that not all developers are aware of (as a first step; let alone control for) the risks
    - twoWhlsGud 2y ago
      And especially if it generates bugs in ways different from humans - human review might be less effective at catching it...
      - xp84 2y ago
        It generates bugs in pretty similar ways. It’s based on human-written code, after all. Edge cases will usually be the ones to get through. Most developers don’t correctly write tests that exercise the limits of each input (or indeed have time to both unit test every function that way, and integration test to be sure the bigger stories are correctly working). Nothing about ai assist changes any of this. (If anybody starts doing significant fully unsupervised “ai” coding they would likely pay the price in extreme instability so I’m assuming here that humans still basically read/skim PRs the same as they always have)
cryptonym 2y ago
I find writing code to be almost relaxing plus that's really a tiny fraction of dev work. Not too excited about potential productivity gains based purely on authoring snippets. I find it much more interesting on boosting maintainability, robustness and other quality metrics (not focusing on quality of AI output, actual quality of the code base).
alickz 2y ago
Comment Driven Programming might be interesting, as an offshoot of Documentation Driven Programming
jappgar 2y ago
I can guarantee you there is more publicly accessible javascript in the world than C++. Copilot will autocomplete entire functions as well, sometimes without comments or even after just typing "f". It uses your previous edits as context and can assume what you're implementing pretty well.
- lupire 2y ago
  Google does not write much publicly available JavaScript. They wrote their own special flavor. (Same for any hugel legacy operation)
  - bilekas 2y ago
    Can we get some more info on what you're reffering to ?
    - jkaptur 2y ago
      They're probably talking about Closure Compiler type annotations [0], which never really took off outside Google, but (imo) were pretty great in the days before TypeScript. (Disclosure: Googler) 0. https://github.com/google/closure-compiler/wiki/Annotating-JavaScript-for-the-Closure-Compiler https://github.com/google/closure-compiler/wiki/Annotating-J...
- infecto 2y ago
  I can guarantee you that the author was referencing code within Google. That is, their tooling is trained off internal code bases. I am imagining c++ dwarfs javascript.
michaelbuckbee 2y ago
I don't work at Google, but I do something similar with my code: write comments, generate the code, and then have the AI tooling create test cases. AI coding assistants are generally really good at ramping up a base level of tests which you can then direct to add more specific scenario's to.
gniv 2y ago
That's pretty nice. Does it write modern C++, as I guess it's expected?
- nlehuen 2y ago
  Yes it does. Internally Google uses C++20 (https://google.github.io/styleguide/cppguide.html#C++_Version https://google.github.io/styleguide/cppguide.html#C++_Versio...) and the model picks the style from training, I suppose.
tomhallett 2y ago
Has anyone made a coding assistant which can do this based off audio which I’m saying out loud while I’m typing (interview/pairing style), so instead of typing the comment I can just say it?
- hecanjog 2y ago
  I had some success using this for basic input, but never took it very far. It's meant to be customizable for that sort of thing though: https://talon.wiki/quickstart/getting_started/ https://talon.wiki/quickstart/getting_started/ (Edit: just the voice input part)
xp84 2y ago
I frequently use copilot and also find that writing comments like you do, to describe what I expect each function/class/etc to do gives superb results, and usually eliminates most of the actual coding work. Obviously it adds significant specification work but that’s not usually a bad thing.

deleted 2y ago

[deleted]

miki123211 2y ago

This is a good idea even outside of Google, with tools like copilot and such. Often when I don't know exactly what function / sequence of functions I need to achieve a particular outcome, I put in a comment describing what I want to do, and Copilot does the rest. I then remove the comment once I make sure that the generated code actually works. I find it a lot less flow-breaking than stackoverflow or even asking an LLM. It doesn't work all of the time, and sometimes you do have to Google still, but for the cases it does work for, it's pretty nice.

Aachen 2y ago
Why remove the comment that summarises the intent for humans? The compiler will ignore your comment anyway, so it's only there for the next human who comes along and will help them understand the code
- cryptonym 2y ago
  Next human will put the code in a prompt and ask what it does. Chinese Whispers.
  - lupire 2y ago
    (No need to Orientalize to defamiarize, especially when a huge fraction of the audience is Chinese, so Orientalizing doesn't defamiliarize. Game of Whispers or Telephone works fine.)
    - cryptonym 2y ago
      Pardon my French.
    - protomolecule 2y ago
      Do the Chinese call it English Whispers?
      - ahoka 2y ago
        It’s all Greek to them.
      - tessierashpool 2y ago
        Chinese-Americans, at least, call it a game of Telephone, like everyone else in the English-speaking world except for the actual English. We call it “Telephone” because “Chinese Whispers” not only sounds racist, it is also super confusing. You need a lot of cultural context to understand the particular way in which Chinese whispers would be different from any other set of whispers.
        tessierashpool 2y ago
        I happened to re-read this, and to be clear, I'm not Chinese-American. the "we" there means "everyone else in the English-speaking world except for the actual English."
  - Aachen 2y ago
- qwertox 2y ago
  It's often unnecessarily verbose. If you read a comment and glance at the code that follows, you'll understand what it is supposed to do. But the comment you're giving as an instruction to an LLM usually contains information which will then be duplicated in the generated code.
  - Aachen 2y ago
    I see. Might still be good to have a verbose comment than no comment at all, as well as a marker of "this was generated" so (by the age of the code) you have some idea of what quality the LLM was in that year and whether to proofread it once more or not
    - lupire 2y ago
      External comments are API usage comments. LLM prompts are also implementation proposal. Implementation comments belong inside the implementation, so they should be over if not deleted.
- randomdata 2y ago
  Like he says, the "comment" describes what he wants to do. That's not what humans are interested in. The human already knows "what he wants to do" when they read the code. It's the things like "why did he want to do this in the first place?" that is lacking in the code, and what information is available to add in a comment for the sake of humans. Remember, LLMs are just compilers for programming languages that just so happen to have a lot of similarities with natural language. The code is not the comment. You still need to comment your code for humans.
  - JohnFen 2y ago
    > Like he says, the "comment" describes what he wants to do. That's not what humans are interested in. When I'm maintaining other people's code, or my own after enough time has gone by, I'm very interested in that sort of comment. It gives me a chance to see if the code as written does what the comment says it was intended to do. It's not valuable for most of the code in a project, but is incredibly valuable for certain key parts. You're right that comments about why things were done the way they were are the most valuable ones, but this kind of comment is in second place in my book.
    - mithametacs 2y ago
      Or for something that needs like a quick mathematical lemma or a worked example. A comment on what is fantastic.
- miki123211 2y ago
  Because the code, when written, is usually obvious enough. Something like: query = query.orderBy(field: "username", Ordering.DESC) Doesn't need an explanation, but when working in a language I don't know well, I might not remember whether I'm supposed to call orderBy on the query or on the ORM module and pass query as the argument, whether the kwarg is called "field" or "column", whether it wants a string or something like `User.name` as the column expression, how to specify the ordering and so on.

Aachen 2y ago

This type of not having to think about the implementation, especially in a language that we've by now well-established can't be written safely by humans (including by Google's own research into Android vulnerabilities if I'm not mistaken), at least with the current level of LLM, worries me the most Time will tell whether it outputs worse, equal, or better quality than skilled humans, but I'd be very wary of anything it suggests beyond obvious boilerplate (like all the symbols needed in a for loop) or naming things (function name and comment autocompletes like the person above you described)

munksbeer 2y ago
> worries me the most It isn't something I worry about at all. If it doesn't work and starts creating bugs and horrible code, the best places will adjust to that and it won't be used or will be used more judiciously. I'll still review code like I always do and prevent bad code from making it into our repo. I don't see why it's my problem to worry about. Why is it yours?
- Aachen 2y ago
  Because I do security audits Functional bugs in edge cases are annoying enough, and I seem to run into these regularly as a user, but there's yet another class of people creating edge cases for their own purposes. The nonchalant "if it doesn't work"... I don't know whether that confirms my suspicion that not all developers are aware of (as a first step; let alone control for) the risks
  - twoWhlsGud 2y ago
    And especially if it generates bugs in ways different from humans - human review might be less effective at catching it...
    - xp84 2y ago
      It generates bugs in pretty similar ways. It’s based on human-written code, after all. Edge cases will usually be the ones to get through. Most developers don’t correctly write tests that exercise the limits of each input (or indeed have time to both unit test every function that way, and integration test to be sure the bigger stories are correctly working). Nothing about ai assist changes any of this. (If anybody starts doing significant fully unsupervised “ai” coding they would likely pay the price in extreme instability so I’m assuming here that humans still basically read/skim PRs the same as they always have)

cryptonym 2y ago

I find writing code to be almost relaxing plus that's really a tiny fraction of dev work. Not too excited about potential productivity gains based purely on authoring snippets. I find it much more interesting on boosting maintainability, robustness and other quality metrics (not focusing on quality of AI output, actual quality of the code base).

alickz 2y ago

Comment Driven Programming might be interesting, as an offshoot of Documentation Driven Programming

jappgar 2y ago

I can guarantee you there is more publicly accessible javascript in the world than C++. Copilot will autocomplete entire functions as well, sometimes without comments or even after just typing "f". It uses your previous edits as context and can assume what you're implementing pretty well.

lupire 2y ago
Google does not write much publicly available JavaScript. They wrote their own special flavor. (Same for any hugel legacy operation)
- bilekas 2y ago
  Can we get some more info on what you're reffering to ?
  - jkaptur 2y ago
    They're probably talking about Closure Compiler type annotations [0], which never really took off outside Google, but (imo) were pretty great in the days before TypeScript. (Disclosure: Googler) 0. https://github.com/google/closure-compiler/wiki/Annotating-JavaScript-for-the-Closure-Compiler https://github.com/google/closure-compiler/wiki/Annotating-J...
infecto 2y ago
I can guarantee you that the author was referencing code within Google. That is, their tooling is trained off internal code bases. I am imagining c++ dwarfs javascript.

michaelbuckbee 2y ago

I don't work at Google, but I do something similar with my code: write comments, generate the code, and then have the AI tooling create test cases. AI coding assistants are generally really good at ramping up a base level of tests which you can then direct to add more specific scenario's to.

gniv 2y ago

That's pretty nice. Does it write modern C++, as I guess it's expected?

nlehuen 2y ago
Yes it does. Internally Google uses C++20 (https://google.github.io/styleguide/cppguide.html#C++_Version https://google.github.io/styleguide/cppguide.html#C++_Versio...) and the model picks the style from training, I suppose.

tomhallett 2y ago

Has anyone made a coding assistant which can do this based off audio which I’m saying out loud while I’m typing (interview/pairing style), so instead of typing the comment I can just say it?

hecanjog 2y ago
I had some success using this for basic input, but never took it very far. It's meant to be customizable for that sort of thing though: https://talon.wiki/quickstart/getting_started/ https://talon.wiki/quickstart/getting_started/ (Edit: just the voice input part)

xp84 2y ago

I frequently use copilot and also find that writing comments like you do, to describe what I expect each function/class/etc to do gives superb results, and usually eliminates most of the actual coding work. Obviously it adds significant specification work but that’s not usually a bad thing.