7 ms·

If you've got a 10M+ LOC project and you're not doing some kind of distributed build you're throwing money and time away.

by nox100 3y ago

flohofwoe 3y ago
The main time killer in day-to-day work on such big projects is usually the linker step, which is terribly slow with the MSVC linker and doesn't benefit from incremental or distributed compilation (not sure though how much the MSVC linker has improved in the last 5 years or so).
- jeffbee 3y ago
  Is it necessary to build and link windows games with msvc? Pretend I know nothing about the subject. Large code bases built and linked with open source toolchains have solved this with, for example, thinlto. And by "large" I mean orders of magnitude larger than the mentioned game.
  - flohofwoe 3y ago
    Not necessary, but definitely the path of least resistance for developing Windows games, and AFAIK MSVC is required for developing Xbox games. Commercial closed source middleware is sometimes distributed as compiled C++ static link libraries plus headers without being able to recompile yourself.
  - Matheus28 3y ago
    You can use clang for windows builds. It’s what I use for my games. However, I assume that most popular engines might not compile on clang+windows for random reasons.
  - throwaway17_17 3y ago
    Genuine question, what code base is 100 million lines of code (I think that is the meaning an order of magnitude bigger that 11 million LOC). I can not comprehend how that much code is anything but dead weight saddled on ‘generations’ of a business’s employees.
    - johnnyanmac 3y ago
      If it helps, even by 10 million LOC (assumedly most of which isn't unused legacy cruft) you're starting to dig deeply into templates and codegen instead of actual manual coding. I'm sure with all this AI stuff that will only get larger and larger. But yeah, outside of some of the largest of FAANG software, there aren't a lot of codebases that can truly justify 100m lines of code. I'm sure GTA VI is over 100m lines but can be cut down to 10m if that was a priority (it never is)
jihiggins 3y ago
obviously many teams do (i've seen incredibuild used pretty often.) that's not a cureall, though.
- nox100 3y ago
  I just cleaned out my build folders and did a full build. 40M+ lines, 3 mins and 35 seconds. If you're not getting similar speed then maybe you should look into adding more machines. Last time I was on game dev the best you could do was share other programmers machines via incredibuild. No thought about adding more machines just for building or using cloud infra
verall 3y ago
It's a lot more common than you'd think. I'm not in gamedev but similarly weird (multiple supported userspaces and OS's for embedded an device line) our "full build" is probably approaching 50M+ lines and only quite recently do people do incrementals from a build server snapshots. No bazel or distributed ccache or anything.
dagmx 3y ago
It can be pretty difficult. Especially for games or OS development, you might have shifting toolchains and SDks. Different teams may move out of sync because different teams want different things at a given time.

If you've got a 10M+ LOC project and you're not doing some kind of distributed build you're throwing money and time away.

by nox100 3y ago

If you've got a 10M+ LOC project and you're not doing some kind of distributed build you're throwing money and time away.

flohofwoe 3y ago

The main time killer in day-to-day work on such big projects is usually the linker step, which is terribly slow with the MSVC linker and doesn't benefit from incremental or distributed compilation (not sure though how much the MSVC linker has improved in the last 5 years or so).

jeffbee 3y ago
Is it necessary to build and link windows games with msvc? Pretend I know nothing about the subject. Large code bases built and linked with open source toolchains have solved this with, for example, thinlto. And by "large" I mean orders of magnitude larger than the mentioned game.
- flohofwoe 3y ago
  Not necessary, but definitely the path of least resistance for developing Windows games, and AFAIK MSVC is required for developing Xbox games. Commercial closed source middleware is sometimes distributed as compiled C++ static link libraries plus headers without being able to recompile yourself.
- Matheus28 3y ago
  You can use clang for windows builds. It’s what I use for my games. However, I assume that most popular engines might not compile on clang+windows for random reasons.
- throwaway17_17 3y ago
  Genuine question, what code base is 100 million lines of code (I think that is the meaning an order of magnitude bigger that 11 million LOC). I can not comprehend how that much code is anything but dead weight saddled on ‘generations’ of a business’s employees.
  - johnnyanmac 3y ago
    If it helps, even by 10 million LOC (assumedly most of which isn't unused legacy cruft) you're starting to dig deeply into templates and codegen instead of actual manual coding. I'm sure with all this AI stuff that will only get larger and larger. But yeah, outside of some of the largest of FAANG software, there aren't a lot of codebases that can truly justify 100m lines of code. I'm sure GTA VI is over 100m lines but can be cut down to 10m if that was a priority (it never is)

jihiggins 3y ago

obviously many teams do (i've seen incredibuild used pretty often.) that's not a cureall, though.

nox100 3y ago
I just cleaned out my build folders and did a full build. 40M+ lines, 3 mins and 35 seconds. If you're not getting similar speed then maybe you should look into adding more machines. Last time I was on game dev the best you could do was share other programmers machines via incredibuild. No thought about adding more machines just for building or using cloud infra

verall 3y ago

It's a lot more common than you'd think. I'm not in gamedev but similarly weird (multiple supported userspaces and OS's for embedded an device line) our "full build" is probably approaching 50M+ lines and only quite recently do people do incrementals from a build server snapshots. No bazel or distributed ccache or anything.

dagmx 3y ago

It can be pretty difficult. Especially for games or OS development, you might have shifting toolchains and SDks. Different teams may move out of sync because different teams want different things at a given time.