6 ms·

Lots of people are saying that having large files in a repo is wrong, bad, bad design, incorrect usage. Forget that you know git, github, git-lfs, even softwar

Lots of people are saying that having large files in a repo is wrong, bad, bad design, incorrect usage. Forget that you know git, github, git-lfs, even software engineering for a moment. All you know is that you're developing a general project on a computer, you are using files, and you want version history on everything. What's wrong with that? The major issue with big files is resources: storage, and network bandwidth. But for both of these it is the sum of all object sizes in a repo that matters, not any particular file, so it's weird to be harking on big files being bad design or evil.

iaresee 3y ago
I did just over a decade in chip design. Versioning large files in that domain is commonplace and quite sane. It can take wallclock days of processing to produce a layout file that's 100's of MBs. Keeping that asset in your SCC system along side all the block assets it was built up out of is very desireable. Perforce handled it all like a champ. People who think large files don't belong in SCC are...wrong.
- ChrisMarshallNY 3y ago
  That's why Perforce is still the SCM of choice for a lot of creatives. I don't know if they still do it, but Unreal used to ship a Perforce license with their SDK.
  - nightfader 3y ago
    That's also why perforce is slow as heck unless you throw massive resources at it. I also work in the chip industry BTW.
    - ChrisMarshallNY 3y ago
      I don't like it (but used it for many years). I love Git, but, then, I don't have a workflow that would benefit from Perforce.
    - beebeepka 3y ago
      I've only used Perforce for two years and it didn't feel slow at all. The company wasn't exactly throwing money at hardware.
    - iaresee 3y ago
      I mean "massive resources" is just de rigeur across the chip industry now. The hard in hardware is really no longer about it being a physical product in the end.
    - pama 3y ago
      I occasionally used to start a sync, go get coffee, chat with colleagues, read and answer my morning email, browse the arxiv, and then wait a few more minutes before I could touch the repo. In retrospect, I should have setup a cron job for it all, but it wasn’t always that slow and I liked the coffee routine. We switched to git. Git is just fast. Even cloning huge repos is barely enough time for grabbing a coffee from down the hall.
nightfader 3y ago
How is it not bad design? Let's say you are working in a team. Would you really want your colleagues spending a significant amount of time cloning your artifacts? Your comment is also not consistent with forgetting that one is not a developer. Even if it's my grandma, she's not gonna want to wait for 1hour to download a giant file from VC assuming she knows what a VC is. Large blobs can go into versioned object storage like GCS or S3 etc
- ghosty141 3y ago
  So how would you version a game that needs assets? These files must be versioned but can be very big, for example long cutscene videos. Some projects need the ability to version big files, there is a good reason why perforce exists and is widely used in the gaming industry.
  - afiori 3y ago
    I am not saying that it is a better UX, but hashed/versioned blobs on S3 would mostly work depending on tooling integration.
    - msk-lywenn 3y ago
      Isn’t that essentially what git lfs is?
      - afiori 3y ago
        I believe so, but with different UX. In almost every case I expect git lfs to be better, but I can see reasons to use more custom flows.
    - maccard 3y ago
      That's building a custom version control on top of the version control you're already using.
      - 3y ago
ajb 3y ago
This is an example of a more generic problem. We adopt some principle or practice for rational reasons, and then as a mental shortcut conflate it with taste, aesthetics, cleanliness. But no software or data is 'dirty' or 'ugly', we feel it so because of mental associations, but intuition is unreliable -the original reasons may not apply, or may be less important.
rewmie 3y ago
> Lots of people are saying that having large files in a repo is wrong, bad, bad design, incorrect usage. I don't think that is true. You do see people warn that having large files in Git repositories, or any repository that wasn't designed with support for large files in mind, is "wrong", in the sense that there are drawbacks for using a system that was not designed to handle them. Here's a historical doc of Linus Torvalds commenting Git's support for large files (or lack thereof) https://marc.info/?l=git&m=124121401124923&w=2 https://marc.info/?l=git&m=124121401124923&w=2
deleted 3y ago
[deleted]
goku12 3y ago
Git is designed with a strong emphasis on text source and patches. It simply isn't designed for projects with large assets like 3D animation, game dev, etc. Having said that, solutions like LFS, Annex and DVC (not git-specific) work really well (IMO). If you don't like that, there are solutions like Restic that can version large files reasonably well (though it's a backup program).
- deleted 3y ago
  [deleted]
otp209 3y ago
> Forget that you know git, github, git-lfs, even software engineering for a moment. All you know is that you're developing a general project on a computer, you are using files, and you want version history on everything. What's wrong with that? THANK YOU. Fucking prescriptivists ruin everything.

iaresee 3y ago

I did just over a decade in chip design. Versioning large files in that domain is commonplace and quite sane. It can take wallclock days of processing to produce a layout file that's 100's of MBs. Keeping that asset in your SCC system along side all the block assets it was built up out of is very desireable. Perforce handled it all like a champ. People who think large files don't belong in SCC are...wrong.

ChrisMarshallNY 3y ago
That's why Perforce is still the SCM of choice for a lot of creatives. I don't know if they still do it, but Unreal used to ship a Perforce license with their SDK.
- nightfader 3y ago
  That's also why perforce is slow as heck unless you throw massive resources at it. I also work in the chip industry BTW.
  - ChrisMarshallNY 3y ago
    I don't like it (but used it for many years). I love Git, but, then, I don't have a workflow that would benefit from Perforce.
  - beebeepka 3y ago
    I've only used Perforce for two years and it didn't feel slow at all. The company wasn't exactly throwing money at hardware.
  - iaresee 3y ago
    I mean "massive resources" is just de rigeur across the chip industry now. The hard in hardware is really no longer about it being a physical product in the end.
  - pama 3y ago
    I occasionally used to start a sync, go get coffee, chat with colleagues, read and answer my morning email, browse the arxiv, and then wait a few more minutes before I could touch the repo. In retrospect, I should have setup a cron job for it all, but it wasn’t always that slow and I liked the coffee routine. We switched to git. Git is just fast. Even cloning huge repos is barely enough time for grabbing a coffee from down the hall.

nightfader 3y ago

How is it not bad design? Let's say you are working in a team. Would you really want your colleagues spending a significant amount of time cloning your artifacts? Your comment is also not consistent with forgetting that one is not a developer. Even if it's my grandma, she's not gonna want to wait for 1hour to download a giant file from VC assuming she knows what a VC is. Large blobs can go into versioned object storage like GCS or S3 etc

ghosty141 3y ago
So how would you version a game that needs assets? These files must be versioned but can be very big, for example long cutscene videos. Some projects need the ability to version big files, there is a good reason why perforce exists and is widely used in the gaming industry.
- afiori 3y ago
  I am not saying that it is a better UX, but hashed/versioned blobs on S3 would mostly work depending on tooling integration.
  - msk-lywenn 3y ago
    Isn’t that essentially what git lfs is?
    - afiori 3y ago
      I believe so, but with different UX. In almost every case I expect git lfs to be better, but I can see reasons to use more custom flows.
  - maccard 3y ago
    That's building a custom version control on top of the version control you're already using.
    - 3y ago

ajb 3y ago

This is an example of a more generic problem. We adopt some principle or practice for rational reasons, and then as a mental shortcut conflate it with taste, aesthetics, cleanliness. But no software or data is 'dirty' or 'ugly', we feel it so because of mental associations, but intuition is unreliable -the original reasons may not apply, or may be less important.

rewmie 3y ago

> Lots of people are saying that having large files in a repo is wrong, bad, bad design, incorrect usage. I don't think that is true. You do see people warn that having large files in Git repositories, or any repository that wasn't designed with support for large files in mind, is "wrong", in the sense that there are drawbacks for using a system that was not designed to handle them. Here's a historical doc of Linus Torvalds commenting Git's support for large files (or lack thereof) https://marc.info/?l=git&m=124121401124923&w=2 https://marc.info/?l=git&m=124121401124923&w=2

deleted 3y ago

[deleted]

goku12 3y ago

Git is designed with a strong emphasis on text source and patches. It simply isn't designed for projects with large assets like 3D animation, game dev, etc. Having said that, solutions like LFS, Annex and DVC (not git-specific) work really well (IMO). If you don't like that, there are solutions like Restic that can version large files reasonably well (though it's a backup program).

deleted 3y ago
[deleted]

otp209 3y ago

> Forget that you know git, github, git-lfs, even software engineering for a moment. All you know is that you're developing a general project on a computer, you are using files, and you want version history on everything. What's wrong with that? THANK YOU. Fucking prescriptivists ruin everything.