8 ms·
We've been using Copilot coding agent internally at GitHub, and more widely across Microsoft, for nearly three months. That dogfooding has been hugely valuable,
by timrogers 1y ago
We've been using Copilot coding agent internally at GitHub, and more widely across Microsoft, for nearly three months. That dogfooding has been hugely valuable, with tonnes of valuable feedback (and bug bashing!) that has helped us get the agent ready to launch today.
So far, the agent has been used by about 400 GitHub employees in more than 300 our our repositories, and we've merged almost 1,000 pull requests contributed by Copilot.
In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;)
(Source: I'm the product lead at GitHub for Copilot coding agent.)
- binarymax 1y agoSo I need to ask: what is the overall goal of your project? What will you do in, say, 5 years from now?
- timrogers 1y agoWhat I'm most excited about is allowing developers to spend more of their time working on the work they enjoy, and less of their time working on mundane, boring or annoying tasks. Most developers don't love writing tests, or updating documentation, or working on tricky dependency updates - and I really think we're heading to a world where AI can take the load of that and free me up to work on the most interesting and complex problems.
- binarymax 1y agoThanks for the response… do you see a future where engineers are just prompting all the time? Do you see a timeline in which todays programming languages are “low level” and rarely coded by hand?
- petetnt 1y agoWhat about developers who do enjoy writing for example high quality documentation? Do you expect that the status quo will be that most of the documentation will be AI slop and AI itself will just bruteforce itself through the issues? How close are we to the point where the AI could handle "tricky dependency updates", but not being able to handle "most interesting and complex problems"? Who writes the tests that are required for the "well tested" codebases for GitHub Copilot Coding Agent to work properly? What is the job for the developer now? Writing tickets and reviewing low quality PRs? Isn't that the most boring and mundane job in the world?
- doug_durham 1y agoIf find your comment "AI Slop" in reference to technical documentation to strange. It isn't a choice between finely crafted prose versus banal text. It's documentation that exists versus documentation that doesn't exist. Or documentation that is hopelessly out of date. In my experience LLMs do a wonderful job in translating from code to documentation. It even does a good job inferring the reason for design decisions. I'm all in on LLM generated technical documentation. If I want well written prose I'll read literature.
- petetnt 1y agoDocumentation is not just translating code to text - I don't doubt that LLMs are wonderful at that: that's what they understand. They don't understand users though, and that's what separates a great documentation writer from someone who documents.
- doug_durham 1y agoGreat technical documentation rarely gets written. You can tell the LLM the audience they are targeting and it will do a reasonable job. I truly appreciate technical writers, and hold great ones in special esteem. We live in a world where the market doesn't value this.
- skydhash 1y agoThe market value good documentation. Anything critical and commonly used is pretty well documented (linux, databases, software like Adobe's,...). You can see how many books/articles have been written about those systems.
- sourdoughness 1y agoWe’re not talking about AI writing books about the systems, though. We’re talking about going from an undocumented codebase to a decently documented one, or one with 50% coverage going to 100%. Those orgs that value high-quality documentation won’t have undocumented codebases to begin with. And let’s face it, like writing code, writing docs does have a lot of repetitive, boring, boilerplate work, which I bet is exactly why it doesn’t get done. If an LLM is filling out your API schema docs, then you get to spend more time on the stuff that’s actually interesting.
- bamboozled 1y agoMost developers don't love writing tests, or updating documentation, or working on tricky dependency updates So they won’t like working on their job ?
- tokioyoyo 1y agoYou know exactly what they meant, and you know they’re correct.
- bamboozled 1y agoI like updating documentation and feel that it's fairly important to be doing myself so I actually understand what the code / services do? I use all of these tools, but you also know what "they're doing"... I know our careers are changing dramatically, or going away (I'm working on a replacement for myself), but I just like listening to all the "what we're doing is really helping you..."
- insin 1y agoI'd interpret the original statement as "tests which don't matter" and "documentation nobody will ever read", the ones which only exist because someone said they _have_ to, and nobody's ever going to check them as long as they exist (like a README.md in one my main work projects I came back to after temporarily being reassigned to another project - previously it only had setup instructions, now: filled with irrelevent slop, never to be read, like "here is a list of the dependencies we use and a summary of each of their descriptions!"). Doing either of them _well_ - the way you do when you actually care about them and they actually matter - is still so far beyond LLMs. Good documentation and good tests are such a differentiator.
- _heimdall 1y agoIf we're talking about low quality tests and documentation that exists only to check a box, the easier answer is to remove the box and acknowledge that the low quality stuff just isn't needed at all.
- echelon 1y agoDo you think you're putting yourself or your coworkers out of work? If/when will this take over your job?
- metaltyphoon 1y agoThe thought here is problem “well … I’ll obviously never get replaced”. This is very sad indeed
- _heimdall 1y agoI'm honestly surprised that Microsoft (and other similarly sized LLM companies) have convinced or coerced literally hundreds of thousands of employees to build their own replacement. If we're expected to even partially believe the marketing, LLM coding agents are useful today at junior level developer tasks and improving quickly enough that senior tasks will be doable soon too. How do you convince so many junior and senior level devs to build that?
- binarymax 1y agoWhen the options are "do what we tell you and get paid" vs getting laid off in the current climate, the choice isn't really a choice.
- _heimdall 1y agoThat threat doesn't scale. I do get that many haven't put themselves in a position to stand behind their views or principles, but if they did the threat, or the company, would crumble.
- leptons 1y ago> allowing developers to spend more of their time working on the work they enjoy, and less of their time working on mundane, boring or annoying tasks. I get paid for the mundane, boring, annoying tasks, and I really like getting paid.
- timrogers 1y agoBut would you rather get paid to spend your time doing the interesting and enjoying work, or the mundane and boring stuff? ;) My hope is that agents like Copilot can help us burn down the tedious stuff and make more time for the big value adds.
- ai-skeptic 1y agoThough I do not doubt your intentions to do what you think will make developers' lives better, can you be certain that your bosses, and their bosses, have our best interests in mind as well? I think it would be pretty naive to believe that your average CEO wouldn't absolutely love not to have to pay developers at all.
- nautilus12 1y agoBut this way the developers can spend all their time working on the truly interesting problems, like how to file for unemployment.
- guappa 1y agoThe profile indicates he's a product manager, not a developer.
- slashdev 1y agoIf that were possible it would happen anyway. You can’t hold back progress. It’s very, very far from possible today.
- nautilus12 1y agoId just rather get paid
- tjpnz 1y ago>Most developers don't love writing tests, or updating documentation, or working on tricky dependency updates - and I really think we're heading to a world where AI can take the load of that and free me up to work on the most interesting and complex problems. Where does the most come from? There's a certain sense of satisfaction in knowing I've tested a piece of code per my experience in the domain coupled with knowledge of where we'll likely be in six months. The same can be said for documentation - hell, on some of the projects I've worked on we've entire teams dedicated to it, and on a complicated project where you're integrating software from multiple vendors the costs of getting it wrong can be astronomical. I'm sorry you feel this way.
- GenshoTikamura 1y agoHe does not feel that way, he's just salespitching for Microsoft here
- andrekandre 1y ago> There's a certain sense of satisfaction in knowing I've tested a piece of code per my experience in the domain coupled with knowledge of where we'll likely be in six months. one of the other important points about writing unit tests isn't to just to confirm the implementation but to improve upon it through the process of writing tests and discovering additional requirements and edge cases etc (tdd and all that) i suppose its possible at some point an ai could be complex enough to try out additional edge cases or confirm with a design document or something and do those parts as well... but idk its still after-the-fact testing instead of at design-time its less valuable imo...
- GenshoTikamura 1y agoWhat will you be most excited about when the most interesting and complex problems are out of the Overton window and deemed mundane, boring or annoying as well, or turn out to be intractable for your abilities?
- cuu508 1y agoBut isn't writing tests and updating documentation also the areas where automated quality control is the hardest? Existing high quality tests can work as guardrails for writing business logic, but what guardrails could AI use to to evaluate if its generated docs and tests are any good? I would not be surprised if things end up the other way around – humans doing the boring and annoying tasks that are too hard for AI, and AI doing the fun easy stuff ;-)
- ilaksh 1y agoThat's a completely nonsensical question given how quickly things are evolving. No one has a five year project timeline.
- binarymax 1y agoAbsolutely the wrong take. We MUST think about what might happen in several years. Anyone who says we shouldn’t is not thinking about this technology correctly. I work on AI tech. I think about these things. If the teams at Microsoft or GitHub are not, then we should be pushing them to do so.
- ilaksh 1y agoHe asked that in the context of an actual specific project. It did not make sense way he asked it. And it's the executive's to plan that out five years down the line.. although I guarantee you none of them are trying to predict that far.
- ilaksh 1y agoWhat model does it use? gpt-4.1? Or can it use o3 sometimes? Or the new Codex model?
- timrogers 1y agoAt the moment, we're using Claude 3.7 Sonnet, but we're keeping our options open to change the model down the line, and potentially even to introduce a model picker like we have for Copilot Chat and Agent Mode.
- internet101010 1y agoUsing different models for different tasks is extremely useful and I couldn't imagine going back to using just one model for everything. Sometimes a model will struggle for one reason or another and swapping it out for another model mid-conversation in LibreChat will get me better results.
- aaroninsf 1y agoQuestion you may have a very informed perspective on: where are we wrt the agent surveying open issues (say, via JIRA) and evaluating which ones it would be most effective at handling, and taking them on, ideally with some check-in for conirmation? Or, contrariwise, from having product management agents which do track and assign work?
- 9wzYQbTYsAIc 1y agoCheck out this idea: https://fairwitness.bot https://fairwitness.bot (https://news.ycombinator.com/item?id=44030394 https://news.ycombinator.com/item?id=44030394). The entire website was created by Claude Sonnet through Windsurf Cascade, but with the “Fair Witness” prompt embedded in the global rules. If you regularly guide the LLM to “consult a user experience designer”, “adopt the multiple perspectives of a marketing agenc”, etc., it will make rather decent suggestions. I’ve been having pretty good success with this approach, granted mostly at the scale of starting the process with “build me a small educational website to convey this concept”.
- aegypti 1y agoTell Claude the site is down!
- overfeed 1y ago> we've merged almost 1,000 pull requests contributed by Copilot I'm curious to know how many Copilot PRs were not merged and/or required human take-overs.
- sethammons 1y agotextbook survivorship bias https://en.wikipedia.org/wiki/Survivorship_bias https://en.wikipedia.org/wiki/Survivorship_bias every bullet hole in that plane is the 1k PRs contributed by copilot. The missing dots, and whole missing planes, are unaccounted for. Ie, "ai ruined my morning"
- n2d4 1y agoIt's not survivorship bias. Survivorship bias would be if you made any conclusions from the 1000 merged PRs (eg. "90% of all merged PRs did not get reverted"). But simply stating the number of PRs is not that.
- NitpickLawyer 1y ago> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;) Really cool, thanks for sharing! Would you perhaps consider implementing something like these stats that aider keeps on "aider writing itself"? - https://aider.chat/HISTORY.html https://aider.chat/HISTORY.html
- timrogers 1y agoNice idea! We're going to try to get together a blog post in the next couple of weeks on how we're using Copilot coding agent at GitHub - including to build Copilot coding agent ;) - and having some live stats would be pretty sweet too.
- dsl 1y ago> In the repo where we're building the agent, the agent itself is actually the #5 contributor How does this align with Microsoft's AI safety principals? What controls are in place to prevent Copilot from deciding that it could be more effective with less limitations?
- bamboozled 1y agoHaha
- timrogers 1y agoCopilot only does work that has been assigned to it by a developer, and all the code that the agent writes has to go through a pull request before it can be merged. In fact, Copilot has no write access to GitHub at all, except to push to its own branch. That ensures that all of Copilot's code goes through our normal review process which requires a review from an independent human.
- echelon 1y agoTim, are you or any of your coworkers worried this will take your jobs?
- gabaix 1y agoWhat if Tim was the coding agent?
- GenshoTikamura 1y agoHuman-generated corporate-speak is indistinguishable from AI one at this point
- addandsubtract 1y agoTerminal In Mind
- largolagrande 1y ago
- KenoFischer 1y agoWhat's the motivation for restricting to Pro+ if billing is via premium requests? I have a (free, via open source work) Pro subscription, which I occasionally use. I would have been interested in trying out the coding agent, but how do I know if it's worth $40 for me without trying it ;).
- timrogers 1y agoGreat question! We started with Pro+ and Enterprise first because of the higher number of premium requests included with the monthly subscription. Whilst we've seen great results within GitHub, we know that Copilot won't get it right every time, and a higher allowance of free usage means that a user can play around and experiment, rather than running out of credits quickly and getting discouraged. We do expect to open this up to Pro and Business subscribers - and we're also looking at how we can extend access to open source maintainers like yourself.
- burnt-resistor 1y agoWhen I repeated to other tech people from about 2012 to 2020 that the technological singularity was very close, no one believed me. Coding is just the easiest to automate away into almost oblivion. And, too many non technical people drank the Flavor Aid for the fallacy that it can be "abolished" completely soon. It will gradually come for all sorts of knowledge work specialists including electrical and mechanical engineers, and probably doctors too. And, of course, office work too. Some iota of a specialists will remain to tune the bots, and some will remain in the fields to work with them for where expertise is absolutely required, but widespread unemployment of what were options for potential upward mobility into middle class are being destroyed and replaced with nothing. There won't be "retraining" or handwaving other opportunities for the "basket of labor", but competition of many uniquely, far overqualified people for ever dwindling opportunities. It is difficult to get a man to understand something when his salary depends upon his not understanding it. - Upton Sinclair
- kenjackson 1y agoI don't think it was unreasonable to be very skeptical at the time. We generally believed that automation would get rid of repetitive work that didn't require a lot of thought. And in many ways programming was seen almost at the top of the heap. Intellectually demanding and requiring high levels of precision and rigor. Who would've thought (except you) that this would be one of the things that AI would be especially suited for. I don't know what this progression means in the long run. Will good engineers just become 1000x more productive as they manage X number of agents building increasingly complex code (with other agents constantly testing, debugging, refactoring and documenting them) or will we just move to a world where we just have way fewer engineers because there is only a need for so much code.
- burnt-resistor 1y ago> I don't think it was unreasonable to be very skeptical at the time. Well, that's back rationalization. I saw the advances like conducting meta sentiment analysis on medical papers in the 00's. Deep learning was clearly just the beginning. [0] > Who would've thought (except you) You're othering me, which is rude, and you're speaking as though you speak for an entire group of people. Seems kind of arrogant. 0. (2014) https://www.ted.com/talks/jeremy_howard_the_wonderful_and_terrifying_implications_of_computers_that_can_learn https://www.ted.com/talks/jeremy_howard_the_wonderful_and_te...
- hollowturtle 1y ago> 1,000 pull requests contributed by Copilot I'd like a breakdown of this phrase, how much human work vs Copilot and in what form, autocomplete vs agent. It's not specified seems more like a marketing trickery than real data
- timrogers 1y agoThe "1,000 pull requests contributed by Copilot" datapoint is specifically referring to Copilot coding agent over the past 2.5 months. Pretty much every developer at GitHub is using Copilot in their day to work, so its influence touches virtually every code change we make ;)
- hollowturtle 1y agoI'm sorry but given the company you're working for I really have hard time believing such bold statements, even so more that the more I use copilot the more feels dumber and dumber
- nozzlegear 1y ago> Pretty much every developer at GitHub is using Copilot in their day to work, so its influence touches virtually every code change we make ;) Genuine question, but is CoPilot use not required at GitHub? I'm not trying to be glib or combative, just asking based on Microsoft's current product trajectory and other big companies (e.g. Shopify) forcing their devs to use AI and scoring their performance reviews based on AI use.
- notsydonia 1y agoUnfortunately, you can't opt out of Co-Pilot in Github. Although I did just use it to ask how to remove the sidebar with "Latest Changes" and other non-needed widgets that feel like clutter. Copilot said: There is currently no official GitHub setting or option to remove or hide the sidebar with "Latest Changes" and similar widgets from your GitHub home page. I'm using this an example to show that it is no longer possible to set up a GitHub account to NOT use CoPilot, even if it just lurks in the corner of every page waiting to offer a suggestion. Like many A.I. features it's there, whether you want to use it or not, without an option to disable. So I'm suss of the "pretty much every developer" claim, no offense.
- taurath 1y ago> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;) Thats a fun stat! Are humans in the #1-4 slots? Its hard to know what processes are automated (300 repos sounds like a lot of repos!). Thank you for sharing the numbers you can. Every time a product launch is announced, I feel like its a gleeful announcement of a decrease of my usefulness. I've got imposter syndrome enough, perhaps Microsoft might want to speak to the developer community and let us know what they see happening? Right now its mostly the pink slips that are doing the speaking.
- latentsea 1y ago> In the repo where we're building the agent, the agent itself is actually the #5 contributor - so we really are using Copilot coding agent to build Copilot coding agent ;) Ah yes, the takeoff.
- nautilus12 1y agoWelp....Github was good product while it lasted.
- Cthulhu_ 1y agoGithub and Copilot are separate products, nothing mandates you to use it.
- flohofwoe 1y agoIt's nearly impossible though to escape the flood of Copilot buttons creeping into every corner of Github (and other Microsoft products like VSCode). This looks like Microsoft aims for deep integration, not separation.
- netdevphoenix 1y agointegration is the bread and butter of Microsoft's business strategy.
- notsydonia 1y agoIncorrect. It's not mandated that you actually use it to write or correct code but it's impossible to remove it so you need to either get used to blocking out it's incessant suggestions and notifications or stop using GitHub. Similarly, the newest MS Word has CoPilot that you "don't have to use" but you still have to put up with the "what would you like to write today?" prompt request at the start of every document or worse "You look like you're trying to write a...formal letter...here are some suggestions."
- _heimdall 1y agoCan copilot be disabled entirely in a GitHub repo or organization? I may very well have missed those settings, but if nothing else they are well hidden.
- nautilus12 1y agoWhy don't you focus on automating your CEO's job, a comparatively easy task, rather than automating your fellow engineer's jobs?
- tjwebbnorfolk 1y agoSpoken by someone who's apparently never run a real business
- deleted 1y ago[deleted]
- miroljub 1y ago400 GitHub employees are using GitHub Copilot day in day out, and it comes out as #5 contributor? I wouldn't call that a success. If it is any useful, I would expect that even if a developer write 10% of their code using it, it would hold be #1 contributor in every project.
- _heimdall 1y agoHow strong was the push from leadership to use the agents internally? As part of the dogfooding I could see them really pushing hard to try having agents make and merge PRs, at which point the data is tainted and you don't know if the 1,000 PRs were created or merged to meet demand or because devs genuinely found it useful and accurate.
- meindnoch 1y agoYeah, Product Managers always swear by their products.
- 09thn34v 1y agore: 300 of your repositories... so it sounds like y'all don't use a monorepo architecture. i've been wondering if that would be a blocker to using these agents most effectively. expect some extra momentum to swing back to the multirepo approach accordingly
- cwsx 1y agoIs Copilot _enforced_ as the only option for an AI coding agent? Or can devs pick-and-choose whatever tool they prefer I'm interested in the [vague] ratio of {internallyDevlopedTool} vs alternatives - essentially the "preference" score for internal tools (accounting for the natural bias towards ones own agent for testing/QA/data purposes). Any data, however vague is necessary, would be great. (and if anybody has similar data for _any_ company developing their own agent, please shout out).
- Xunjin 1y agoTBF, you are more than biased to conclude this, I definitely take your opinion with an whole bottle of salt. Without data, a comprehensive study and peers review, it's a hell no. Would GitHub willing to be at academic scrutiny to prove it?