6 ms·

Learning Rust with ChatGPT, Copilot and Advent of Code

83 points by sean_lynch 4y ago

worldsayshi 4y ago
In a way of seeing it, ChatGPT can be a great teacher because it can both provide subject knowledge and force you to be very critical about content you consume.
- PartiallyTyped 4y ago
  Today I built the SSO flow for an application at work, and I learned a lot significantly faster than I could by just searching because it was guided. The generated code had issues, but it felt a lot like rustlings [1]. At many points I felt that my solution and approach was akin to doing gradient descent, with ChatGPT giving me the direction and me doing backtracking to avoid overshooting (glossing what it missed). [1] https://github.com/rust-lang/rustlings https://github.com/rust-lang/rustlings
  - simonw 4y ago
    This is something I find really interesting about it: I feel like if you have an intermediate-to-expert level of knowledge it can absolutely amplify that, and make you massively more productive. The open question for me is how much it can benefit people with a novice level of understanding - that's one of the reasons I'm exploring Rust with Advent of Code using it. I have 20+ years of non-Rust programming experience though, so I have no idea how well this would work for someone learning to program for the first time. I'd be fascinating to see how well it works (or doesn't) for complete programming newcomers.
    - PartiallyTyped 4y ago
      I recently stumbled on this video [1], where the host and an expert used diffusion models to generate art, had viewers blindly rank them. The tl:dw is that at the hands of an expert, it can be a massive productivity boost, while at the hands of a newbie, it reduces the skill floor. I think that is very similar to what we are observing, but while it drops the skill floor a bit, meaning that it's easy to make stuff, it also acts as a multiplier for rate of improvement, where people with pre-existing knowledge can quickly adapt to a particular domain by probing further. The pre-existing knowledge serves as the backbone on which new info is added and filled in. Funnily enough this is the same idea as pretraining a model e.g. through self-supervision, hah! > A skill floor is the counterpart to a skill ceiling. A skill ceiling is the level of play that’s possible with training and mastery. A skill floor is a way of describing how difficult it is to begin the process of mastery. [2] [1] https://www.youtube.com/watch?v=NiJeB2NJy1A https://www.youtube.com/watch?v=NiJeB2NJy1A [2] https://esportsedition.com/general/skill-ceiling-skill-floor-esports-terminology/ https://esportsedition.com/general/skill-ceiling-skill-floor...
sean_lynch 4y ago
Have been thinking about exactly this workflow and of course Simon is way out ahead. I recommend taking a look at Simon's daily notes logs too, really fascinating back and forths.
- simonw 4y ago
  Here are my issue notes so far: https://github.com/simonw/advent-of-code-2022-in-rust/issues?q=is%3Aissue+is%3Aclosed https://github.com/simonw/advent-of-code-2022-in-rust/issues...
bsaul 4y ago
The way it explains code, the error, and then gives the solution to this particular error, i really don’t understand how someone could pretend we’re not witnessing at least a first hint of true intelligence.
- torben-friis 4y ago
  We haven't figured out what conscience is, so it's going to be fun to deal with near perfect imitations of human behaviour, and the societal consequences...
- 29athrowaway 4y ago
  You are witnessing the end of many occupations.
- drekipus 4y ago
  because you don't understand intelligence
- simonw 4y ago
  I still think it's the world's most impressive party trick. It turns out if you build a model that can predict what word comes next after a sequence of previous words, then train it on TBs (PBs?) of data, you get something that really does appear to be "intelligent". And which is absurdly useful, to boot! Until you trip it up, and the illusion shatters.
  - PartiallyTyped 4y ago
    It's more than that though, it also involves ranking the results to account for noise in the sampling process.
  - graeber_28927 4y ago
    Okay, but then is Wikipedia no more than a big party trick? It's the most comprehensive knowledge base humankind has ever produced, made in collaboration of phylantropic people around the world. Of course some info is sometimes false, but I don't find myself getting disillusioned of it all. It is what it is, and knowing the limitations doesn't ruin it for me, It is still incredibly valuable, and irreplacable, imo.
jpe90 4y ago
I’ve been using copilot for advent of code as well, as well as in regular day to day coding. I have less luck with comment-based coding than the author, but in my usage it’s been handy for normal autocomplete. There’s been a number of instances where I would write a comment describing a simple operation and it would struggle or generate a lot of noise, but if I just start writing an implementation it would give me a good suggestion pretty quickly. I guess the extra context helped. I was disappointed that it wasn’t more useful as a discovery tool- when I’m not knowledgeable about a language or framework, it can be hard to judge whether its suggestions are subtly wrong. After a while, I started getting a sense for when suggestions are likely to be valuable, which is generally when I have a concrete idea of what I want and what it should look like. When i’m doing more exploratory development I often just ignore the completion. Using ChatGPT instead for learning new stuff is a great idea and i’ll definitely be trying that. Something I wasn't expecting about copilot is that it actually has been giving me pretty good completions for emacs lisp. I think it’s going to be a very valuable tool in the long run and I recommend giving it a second thought if you’ve dismissed its utility.
sorobahn 4y ago
This is so cool! I looked at the transcript for day 5 [1] and realized how I learned the same thing regarding Rust strings not being indexable with integers due to them being a series of grapheme clusters. I didn't use ChatGPT and had to dig through the crate documentation [2] and look at stackoverflow [3], but Simon was able to get an equally great explanation by simply asking "Why is this so hard?" which I could relate to very much coming from C++ land. Now, the ability to trust these explanations is another issue, but I think it's interesting to imagine a future where software documentation is context aware to an individual's situation. The Rust docs are amazing and you can see they bring up indexing in the "UTF-8" section, but it requires me reading a section of the doc which I may not have realized was the reason for my frustration with a compiler error regarding indexing. Even if ChatGPT is not "intelligent" (whatever that means), its ability to take a context and almost act like an expert who's read every page of documentation that can point to you into a productive direction is very helpful. [1]: https://github.com/simonw/advent-of-code-2022-in-rust/issues/6 https://github.com/simonw/advent-of-code-2022-in-rust/issues... [2]: https://doc.rust-lang.org/std/string/struct.String.html#utf-8 https://doc.rust-lang.org/std/string/struct.String.html#utf-... [3]: https://stackoverflow.com/a/24542502 https://stackoverflow.com/a/24542502
- 63 4y ago
  I don't know if this is helpful at all, but "strings are not indexable because of their underlying implementation and the complexity of UTF-8" is drilled into the reader very hard by the rust book. Obviously to each their own, but I found I had a much easier time understanding the language by working through the book than by treating rust like any other language and randomly guessing and googling my way through errors.
  - simonw 4y ago
    I've not looked at the book or any of the documentation at all yet, but I'm getting the distinct impression that it's a cut above documentation for many other languages. I'm going to start working through that too.
- tialaramex 4y ago
Cloudef 4y ago
ChatGPT finally told me the solution for question: "How do you force borrow checker in rust to allow reusing mutable reference when self is borrowed by a function call, but not returned back, using unsafe is okay"
- oconnor663 4y ago
  If you do this, make sure you find a way to test your code under Miri. Unsafe workarounds to this problem might appear to work in practice but are often unsound according to the (likely, future) formal memory model. The situation is similar to a strict aliasing violation in C/C++, where trivial-seeming changes to surrounding code (or flags, or the compiler version) can turn a "benign" or "latent" violation into observable UB.
  - Cloudef 4y ago
    One doesnt always have to care about async / future / threading. Especially if its store once, read after data. Unless you are saying rust can do whatever it wants with its memory without clear specification nor way for you to force it to certain direction, then my answer is that rust has failed as a low level language at that point.
    - oconnor663 4y ago
      It's really the exact same situation with strict aliasing violations in C and C++. If you break the rules (which are different depending on the language but in any case kind of complicated) all bets are off, even in totally single-threaded code, even if all you're doing is integer arithmetic, and in fact even if your code passes ASan and UBSan. If you'll pardon some shameless self-promotion, I have a talk about this that goes into a lot of examples: https://www.youtube.com/watch?v=DG-VLezRkYQ https://www.youtube.com/watch?v=DG-VLezRkYQ The nice thing about Rust is that you mostly don't have to worry about any of this if you don't write the "unsafe" keyword. Folks with previous C and C++ experience often come in with understandable but mistaken assumptions about how unsafe code works in Rust, and I think it's important to study the rules carefully before you start writing unsafe code for production use.
rychco 4y ago
I just tried something similar to generate a bunch of boilerplate for parsing some deeply nested & complex JSON using Rust/Serde. It gave me a handful of errors, but it was ~90% correct. Which, to be fair, is still shockingly good. I was already very optimistic about Copilot by itself being able to basically eliminate the need to check StackOverflow/Docs for basic questions. Combined with ChatGPT I can essentially offload all boilerplate I would ever need to write (provided I'm still able to identify & correct the few errors it spits out from time to time).
ofrzeta 4y ago
This is amazing. Nice to see how Simon uses various tools to achieve his goals in his daily writeups. It might not be "intelligence" but it's still a very subtle search engine for programmers with a human language interface. I always found it a bit tedious to have to type out everything when you know you want a loop over and array or something. Now you can just tell ChatGPT what you want and let it do the typing. Disregarding the mistakes it makes while appearing confident about its answers :-) Now programmers can finally be directors where ChatGPT is the film team or the cinematographer. Sometimes it's overconfident and you need to correct it to conform to your imagination. Also as a programmer/director you still need a lot of knowledge to ask the right questions, judge the answers and so on (as demonstrated in Simon's writeups).

83 points by sean_lynch 4y ago

In a way of seeing it, ChatGPT can be a great teacher because it can both provide subject knowledge and force you to be very critical about content you consume.

PartiallyTyped 4y ago
Today I built the SSO flow for an application at work, and I learned a lot significantly faster than I could by just searching because it was guided. The generated code had issues, but it felt a lot like rustlings [1]. At many points I felt that my solution and approach was akin to doing gradient descent, with ChatGPT giving me the direction and me doing backtracking to avoid overshooting (glossing what it missed). [1] https://github.com/rust-lang/rustlings https://github.com/rust-lang/rustlings
- simonw 4y ago
  This is something I find really interesting about it: I feel like if you have an intermediate-to-expert level of knowledge it can absolutely amplify that, and make you massively more productive. The open question for me is how much it can benefit people with a novice level of understanding - that's one of the reasons I'm exploring Rust with Advent of Code using it. I have 20+ years of non-Rust programming experience though, so I have no idea how well this would work for someone learning to program for the first time. I'd be fascinating to see how well it works (or doesn't) for complete programming newcomers.
  - PartiallyTyped 4y ago
    I recently stumbled on this video [1], where the host and an expert used diffusion models to generate art, had viewers blindly rank them. The tl:dw is that at the hands of an expert, it can be a massive productivity boost, while at the hands of a newbie, it reduces the skill floor. I think that is very similar to what we are observing, but while it drops the skill floor a bit, meaning that it's easy to make stuff, it also acts as a multiplier for rate of improvement, where people with pre-existing knowledge can quickly adapt to a particular domain by probing further. The pre-existing knowledge serves as the backbone on which new info is added and filled in. Funnily enough this is the same idea as pretraining a model e.g. through self-supervision, hah! > A skill floor is the counterpart to a skill ceiling. A skill ceiling is the level of play that’s possible with training and mastery. A skill floor is a way of describing how difficult it is to begin the process of mastery. [2] [1] https://www.youtube.com/watch?v=NiJeB2NJy1A https://www.youtube.com/watch?v=NiJeB2NJy1A [2] https://esportsedition.com/general/skill-ceiling-skill-floor-esports-terminology/ https://esportsedition.com/general/skill-ceiling-skill-floor...

sean_lynch 4y ago

Have been thinking about exactly this workflow and of course Simon is way out ahead. I recommend taking a look at Simon's daily notes logs too, really fascinating back and forths.

simonw 4y ago
Here are my issue notes so far: https://github.com/simonw/advent-of-code-2022-in-rust/issues?q=is%3Aissue+is%3Aclosed https://github.com/simonw/advent-of-code-2022-in-rust/issues...

bsaul 4y ago

The way it explains code, the error, and then gives the solution to this particular error, i really don’t understand how someone could pretend we’re not witnessing at least a first hint of true intelligence.

torben-friis 4y ago
We haven't figured out what conscience is, so it's going to be fun to deal with near perfect imitations of human behaviour, and the societal consequences...
29athrowaway 4y ago
You are witnessing the end of many occupations.
drekipus 4y ago
because you don't understand intelligence
simonw 4y ago
I still think it's the world's most impressive party trick. It turns out if you build a model that can predict what word comes next after a sequence of previous words, then train it on TBs (PBs?) of data, you get something that really does appear to be "intelligent". And which is absurdly useful, to boot! Until you trip it up, and the illusion shatters.
- PartiallyTyped 4y ago
  It's more than that though, it also involves ranking the results to account for noise in the sampling process.
- graeber_28927 4y ago
  Okay, but then is Wikipedia no more than a big party trick? It's the most comprehensive knowledge base humankind has ever produced, made in collaboration of phylantropic people around the world. Of course some info is sometimes false, but I don't find myself getting disillusioned of it all. It is what it is, and knowing the limitations doesn't ruin it for me, It is still incredibly valuable, and irreplacable, imo.

jpe90 4y ago

I’ve been using copilot for advent of code as well, as well as in regular day to day coding. I have less luck with comment-based coding than the author, but in my usage it’s been handy for normal autocomplete. There’s been a number of instances where I would write a comment describing a simple operation and it would struggle or generate a lot of noise, but if I just start writing an implementation it would give me a good suggestion pretty quickly. I guess the extra context helped. I was disappointed that it wasn’t more useful as a discovery tool- when I’m not knowledgeable about a language or framework, it can be hard to judge whether its suggestions are subtly wrong. After a while, I started getting a sense for when suggestions are likely to be valuable, which is generally when I have a concrete idea of what I want and what it should look like. When i’m doing more exploratory development I often just ignore the completion. Using ChatGPT instead for learning new stuff is a great idea and i’ll definitely be trying that. Something I wasn't expecting about copilot is that it actually has been giving me pretty good completions for emacs lisp. I think it’s going to be a very valuable tool in the long run and I recommend giving it a second thought if you’ve dismissed its utility.

sorobahn 4y ago

This is so cool! I looked at the transcript for day 5 [1] and realized how I learned the same thing regarding Rust strings not being indexable with integers due to them being a series of grapheme clusters. I didn't use ChatGPT and had to dig through the crate documentation [2] and look at stackoverflow [3], but Simon was able to get an equally great explanation by simply asking "Why is this so hard?" which I could relate to very much coming from C++ land. Now, the ability to trust these explanations is another issue, but I think it's interesting to imagine a future where software documentation is context aware to an individual's situation. The Rust docs are amazing and you can see they bring up indexing in the "UTF-8" section, but it requires me reading a section of the doc which I may not have realized was the reason for my frustration with a compiler error regarding indexing. Even if ChatGPT is not "intelligent" (whatever that means), its ability to take a context and almost act like an expert who's read every page of documentation that can point to you into a productive direction is very helpful. [1]: https://github.com/simonw/advent-of-code-2022-in-rust/issues/6 https://github.com/simonw/advent-of-code-2022-in-rust/issues... [2]: https://doc.rust-lang.org/std/string/struct.String.html#utf-8 https://doc.rust-lang.org/std/string/struct.String.html#utf-... [3]: https://stackoverflow.com/a/24542502 https://stackoverflow.com/a/24542502

63 4y ago
I don't know if this is helpful at all, but "strings are not indexable because of their underlying implementation and the complexity of UTF-8" is drilled into the reader very hard by the rust book. Obviously to each their own, but I found I had a much easier time understanding the language by working through the book than by treating rust like any other language and randomly guessing and googling my way through errors.
- simonw 4y ago
  I've not looked at the book or any of the documentation at all yet, but I'm getting the distinct impression that it's a cut above documentation for many other languages. I'm going to start working through that too.
tialaramex 4y ago

Cloudef 4y ago

ChatGPT finally told me the solution for question: "How do you force borrow checker in rust to allow reusing mutable reference when self is borrowed by a function call, but not returned back, using unsafe is okay"

oconnor663 4y ago
If you do this, make sure you find a way to test your code under Miri. Unsafe workarounds to this problem might appear to work in practice but are often unsound according to the (likely, future) formal memory model. The situation is similar to a strict aliasing violation in C/C++, where trivial-seeming changes to surrounding code (or flags, or the compiler version) can turn a "benign" or "latent" violation into observable UB.
- Cloudef 4y ago
  One doesnt always have to care about async / future / threading. Especially if its store once, read after data. Unless you are saying rust can do whatever it wants with its memory without clear specification nor way for you to force it to certain direction, then my answer is that rust has failed as a low level language at that point.
  - oconnor663 4y ago
    It's really the exact same situation with strict aliasing violations in C and C++. If you break the rules (which are different depending on the language but in any case kind of complicated) all bets are off, even in totally single-threaded code, even if all you're doing is integer arithmetic, and in fact even if your code passes ASan and UBSan. If you'll pardon some shameless self-promotion, I have a talk about this that goes into a lot of examples: https://www.youtube.com/watch?v=DG-VLezRkYQ https://www.youtube.com/watch?v=DG-VLezRkYQ The nice thing about Rust is that you mostly don't have to worry about any of this if you don't write the "unsafe" keyword. Folks with previous C and C++ experience often come in with understandable but mistaken assumptions about how unsafe code works in Rust, and I think it's important to study the rules carefully before you start writing unsafe code for production use.

rychco 4y ago

I just tried something similar to generate a bunch of boilerplate for parsing some deeply nested & complex JSON using Rust/Serde. It gave me a handful of errors, but it was ~90% correct. Which, to be fair, is still shockingly good. I was already very optimistic about Copilot by itself being able to basically eliminate the need to check StackOverflow/Docs for basic questions. Combined with ChatGPT I can essentially offload all boilerplate I would ever need to write (provided I'm still able to identify & correct the few errors it spits out from time to time).

ofrzeta 4y ago

This is amazing. Nice to see how Simon uses various tools to achieve his goals in his daily writeups. It might not be "intelligence" but it's still a very subtle search engine for programmers with a human language interface. I always found it a bit tedious to have to type out everything when you know you want a loop over and array or something. Now you can just tell ChatGPT what you want and let it do the typing. Disregarding the mistakes it makes while appearing confident about its answers :-) Now programmers can finally be directors where ChatGPT is the film team or the cinematographer. Sometimes it's overconfident and you need to correct it to conform to your imagination. Also as a programmer/director you still need a lot of knowledge to ask the right questions, judge the answers and so on (as demonstrated in Simon's writeups).