9 ms·

People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of w

People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of work. For RHEL you are stuck on one kernel for an entire release. Every fix has to be backported from upstream, and the further from upstream you get the harder it is to do that work. Btrfs has to be rebased _every_ release. If moves too fast and there is so much work being done that you can't just cherry pick individual fixes. This makes it a huge pain in the ass. Then you have RHEL's "if we ship it we support it" mantra. Every release you have something that is more Frankenstein-y than it was before, and you run more of a risk of shit going horribly wrong. That's a huge liability for an engineering team that has 0 upstream btrfs contributors. The entire local file system group are xfs developers. Nobody has done serious btrfs work at Red Hat since I left (with a slight exception with Zach Brown for a little while.) Suse uses it as their default and has a lot of inhouse expertise. We use it in a variety of ways inside Facebook. It's getting faster and more stable, admittedly slower than I'd like, but we are getting there. This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself.

appleflaxen 9y ago
thanks for this context; i read this thread previously and had no idea of the "why" behind the news item. great comment to understand better.
nailer 9y ago
Thanks. Any indication why RH didn't hire btrfs devs? It looks like a decision was made that it wasn't strategic (obviously xfs on Linux has a much longer history).
- josefbacik 9y ago
  They brought on Zach right before I left specifically to help with the effort, but he left as well. I can't really speak to Red Hat's overall strategic decisions, but really they have a large local file system team, and a lot of them are xfs developers. You aren't going to convince Dave Chinner he should go work on Btrfs instead of XFS. Unless there's somebody internally that actually wants to work on Btrfs the work simply isn't going to get done. All of the other Btrfs developers work for other companies, and none of them are interested in working for Red Hat.
  - nailer 9y ago
    Fair enough. I imagined if btrfs was a high enough priority they'd hire new staff specifically for it, but if they've tried, and money/good employment conditions don't work, that's all they can do.
  - jameskegel 9y ago
    I'm feeling a subtext here that maybe RH isn't a desired place to work, when I've always imagined the opposite. Is this the case?
    - josefbacik 9y ago
      Nope I love Red Hat and loved working for Red Hat and still interact with most of my colleagues there on a day to day basis. I shouldn't be speaking for everybody, but from what I can tell we're all pretty happy where we are, so no real reason to switch companies.
    - user5994461 9y ago
      I'd read the subtext as "there are only a handful of filesystem developers in the world and the 10 of them are already settled in a good big company".
    - cjbprime 9y ago
      I think there are some ways that RH would be less desirable for many people than a BigCo. When I was interested in working for them they had offices in inconvenient locations and a requirement that you (or at least, I) work in one of them -- e.g. their "Boston" office is 30 miles away in Westford, and their headquarters are in North Carolina. That's disqualifying for many people. I imagine they pay significantly less than the other companies (e.g. Facebook) who want to hire Btrfs devs can afford to, too.
      - nailer 9y ago
        Isn't FBs internal distro Fedora based? I wonder if FB has a solid RH-based btrfs production ready kernel floating about.
feld 9y ago
The problem is that Redhat and others are refusing to challenge the norm and break away from the "freeze the release; backport fixes" mantra. Stop backporting fixes. You're forking the codebase. Ship exactly what upstream provides. Teach upstream projects how to do better release engineering if they're abandoning major releases to early or breaking API/ABI in a minor release. Stop backporting fixes. You're forking the codebase. edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it.
- admax88q 9y ago
  I think you're underestimating the stability that such practices provide for enterprise. This is what people pay Redhat for. Not all upstreams are interested in doing release engineering. There are non zero costs to doing it. It can eat up time that can be spent on bug fixes and features, or even make it too costly to change direction if a certain approach to implementation is proving more difficult than it should be. Look at the Linux kernel. The only reason there is a stable kernel series is because Greg K-H decided it was important enough. He was unable to convince any other developers to go along with it, and eventually the decision was "if you want to support it, then you can do it." Do you consider the stable kernel series a fork of the codebase? Should everyone be running the newest kernel every release despite the plenty of regressions that appear? Kernel developers are not interested in making every change in such a slow and controlled manner as to avoid any regressions. And it works for them. They get a lot of stuff done, and come back and fix the regressions later.
- ajross 9y ago
  If upstream releases were doing better release engineering in the way you mean then there would be no money to be made shipping RHEL as a product.
- thehardsphere 9y ago
  Taking this to its most ludicrous extreme, everyone should use Arch, and anyone who can't should... what? Not use Linux?
  - 9y ago
user5994461 9y ago
>>> This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself. It's a clear indicator that RedHat doesn't want or can't support btrfs. Which is a reflection of btrfs AND RedHat: the effort required to maintain it, the lack of usage in RHEL paying customers, the immaturity/fast development of the filesystem.
pgaddict 9y ago
I think a natural follow-up question is "Why Red Hat does not have engineers to support btrfs?" That is, if the lack of engineers is a symptom, what is the cause? I'm pretty sure, had RH wanted they could either hire or assign engineers to maintain the btrfs code, take care of patches from upstream, etc. So why didn't that happen? I wonder what is your opinion on that. I see a bunch of possibilities (not necessarily independent ones): 1) Politics. Perhaps RH wants to kill btrfs for some reason? I see this as rather unlikely, as RH does not have a competing solution (unlike in the Jigsaw controversy, where they have incentives to kill it in favor of the JBoss module system). 2) Inability to hire enough engineers familiar with btrfs, or assign existing engineers. Perhaps the number of engineers would be too high, increasing costs. Especially if not only to maintain the RHEL kernels, but to contribute to btrfs and move it forward. Or maybe there's a pushback from the current filesystems team, where most people are xfs developers? 3) Incompatible development models. If each release requires a rebase, perhaps supporting btrfs would require too much work / too many engineers, increasing costs? I wonder what Suse and others are doing differently, except for having in-house btrfs developers. 4) Lack of trust btrfs will get mature enough for RHEL soon. It may work for certain deployments, but for RHEL customers that may not be sufficient. That probably requires a filesystem performing well for a wider range of workloads. 5) Lack of interest from paying RHEL customers. Many of our customers have RHEL systems (or CentOS / Scientific Linux), and I don't remember a single one of them using btrfs or planning to do so. We only deal with database servers, which is a very narrow segment of the market, and fairly conservative one when it comes to filesystems. But overall, if customers are not interested in a feature, it's merely a pragmatic business decision not to spend money on it. 6) Better alternatives available. I'm not aware of one, although "ZFS on Linux" is getting much better. So I tend to see this as a pragmatic business decision, based on customer interest in btrfs on RHEL vs. costs of supporting it.
- chasil 9y ago
  Oracle has essential control of both "nextgen" filesystems that should be used in Linux - as Sun, they developed and licensed ZFS, and they are the chief contributors of BtrFS. Their refusal to release ZFS under a license that is compatible with the GPL is keeping it out of Red Hat's distribution. This move by Red Hat must be seen as a provocation of Oracle, to force either greater cooperation and compliance in producing a stable BtrFS for RHEL, or the release of ZFS under a compatible license. Red Hat has put an end to BtrFS for now, and Oracle will have to go to greater lengths to use it in their clone. Customers also will not want it if it does not run equally well between RHEL and Oracle Linux. It is obvious that Oracle will have to assume higher costs and support if they want BtrFS in RHEL. Red Hat is certainly justified in bringing Oracle to heel. Oracle recently committed preliminary dedup support for XFS, so they must be intimately aware of the technical and legal issues behind Red Hat's move. https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-linux-v48-v49-and-v410%2c-by-darrick-wong https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-li...

appleflaxen 9y ago

thanks for this context; i read this thread previously and had no idea of the "why" behind the news item. great comment to understand better.

nailer 9y ago

Thanks. Any indication why RH didn't hire btrfs devs? It looks like a decision was made that it wasn't strategic (obviously xfs on Linux has a much longer history).

josefbacik 9y ago
They brought on Zach right before I left specifically to help with the effort, but he left as well. I can't really speak to Red Hat's overall strategic decisions, but really they have a large local file system team, and a lot of them are xfs developers. You aren't going to convince Dave Chinner he should go work on Btrfs instead of XFS. Unless there's somebody internally that actually wants to work on Btrfs the work simply isn't going to get done. All of the other Btrfs developers work for other companies, and none of them are interested in working for Red Hat.
- nailer 9y ago
  Fair enough. I imagined if btrfs was a high enough priority they'd hire new staff specifically for it, but if they've tried, and money/good employment conditions don't work, that's all they can do.
- jameskegel 9y ago
  I'm feeling a subtext here that maybe RH isn't a desired place to work, when I've always imagined the opposite. Is this the case?
  - josefbacik 9y ago
    Nope I love Red Hat and loved working for Red Hat and still interact with most of my colleagues there on a day to day basis. I shouldn't be speaking for everybody, but from what I can tell we're all pretty happy where we are, so no real reason to switch companies.
  - user5994461 9y ago
    I'd read the subtext as "there are only a handful of filesystem developers in the world and the 10 of them are already settled in a good big company".
  - cjbprime 9y ago
    I think there are some ways that RH would be less desirable for many people than a BigCo. When I was interested in working for them they had offices in inconvenient locations and a requirement that you (or at least, I) work in one of them -- e.g. their "Boston" office is 30 miles away in Westford, and their headquarters are in North Carolina. That's disqualifying for many people. I imagine they pay significantly less than the other companies (e.g. Facebook) who want to hire Btrfs devs can afford to, too.
    - nailer 9y ago
      Isn't FBs internal distro Fedora based? I wonder if FB has a solid RH-based btrfs production ready kernel floating about.

feld 9y ago

The problem is that Redhat and others are refusing to challenge the norm and break away from the "freeze the release; backport fixes" mantra. Stop backporting fixes. You're forking the codebase. Ship exactly what upstream provides. Teach upstream projects how to do better release engineering if they're abandoning major releases to early or breaking API/ABI in a minor release. Stop backporting fixes. You're forking the codebase. edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it.

admax88q 9y ago
I think you're underestimating the stability that such practices provide for enterprise. This is what people pay Redhat for. Not all upstreams are interested in doing release engineering. There are non zero costs to doing it. It can eat up time that can be spent on bug fixes and features, or even make it too costly to change direction if a certain approach to implementation is proving more difficult than it should be. Look at the Linux kernel. The only reason there is a stable kernel series is because Greg K-H decided it was important enough. He was unable to convince any other developers to go along with it, and eventually the decision was "if you want to support it, then you can do it." Do you consider the stable kernel series a fork of the codebase? Should everyone be running the newest kernel every release despite the plenty of regressions that appear? Kernel developers are not interested in making every change in such a slow and controlled manner as to avoid any regressions. And it works for them. They get a lot of stuff done, and come back and fix the regressions later.
ajross 9y ago
If upstream releases were doing better release engineering in the way you mean then there would be no money to be made shipping RHEL as a product.
thehardsphere 9y ago
Taking this to its most ludicrous extreme, everyone should use Arch, and anyone who can't should... what? Not use Linux?
- 9y ago

user5994461 9y ago

>>> This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself. It's a clear indicator that RedHat doesn't want or can't support btrfs. Which is a reflection of btrfs AND RedHat: the effort required to maintain it, the lack of usage in RHEL paying customers, the immaturity/fast development of the filesystem.

pgaddict 9y ago

I think a natural follow-up question is "Why Red Hat does not have engineers to support btrfs?" That is, if the lack of engineers is a symptom, what is the cause? I'm pretty sure, had RH wanted they could either hire or assign engineers to maintain the btrfs code, take care of patches from upstream, etc. So why didn't that happen? I wonder what is your opinion on that. I see a bunch of possibilities (not necessarily independent ones): 1) Politics. Perhaps RH wants to kill btrfs for some reason? I see this as rather unlikely, as RH does not have a competing solution (unlike in the Jigsaw controversy, where they have incentives to kill it in favor of the JBoss module system). 2) Inability to hire enough engineers familiar with btrfs, or assign existing engineers. Perhaps the number of engineers would be too high, increasing costs. Especially if not only to maintain the RHEL kernels, but to contribute to btrfs and move it forward. Or maybe there's a pushback from the current filesystems team, where most people are xfs developers? 3) Incompatible development models. If each release requires a rebase, perhaps supporting btrfs would require too much work / too many engineers, increasing costs? I wonder what Suse and others are doing differently, except for having in-house btrfs developers. 4) Lack of trust btrfs will get mature enough for RHEL soon. It may work for certain deployments, but for RHEL customers that may not be sufficient. That probably requires a filesystem performing well for a wider range of workloads. 5) Lack of interest from paying RHEL customers. Many of our customers have RHEL systems (or CentOS / Scientific Linux), and I don't remember a single one of them using btrfs or planning to do so. We only deal with database servers, which is a very narrow segment of the market, and fairly conservative one when it comes to filesystems. But overall, if customers are not interested in a feature, it's merely a pragmatic business decision not to spend money on it. 6) Better alternatives available. I'm not aware of one, although "ZFS on Linux" is getting much better. So I tend to see this as a pragmatic business decision, based on customer interest in btrfs on RHEL vs. costs of supporting it.

chasil 9y ago
Oracle has essential control of both "nextgen" filesystems that should be used in Linux - as Sun, they developed and licensed ZFS, and they are the chief contributors of BtrFS. Their refusal to release ZFS under a license that is compatible with the GPL is keeping it out of Red Hat's distribution. This move by Red Hat must be seen as a provocation of Oracle, to force either greater cooperation and compliance in producing a stable BtrFS for RHEL, or the release of ZFS under a compatible license. Red Hat has put an end to BtrFS for now, and Oracle will have to go to greater lengths to use it in their clone. Customers also will not want it if it does not run equally well between RHEL and Oracle Linux. It is obvious that Oracle will have to assume higher costs and support if they want BtrFS in RHEL. Red Hat is certainly justified in bringing Oracle to heel. Oracle recently committed preliminary dedup support for XFS, so they must be intimately aware of the technical and legal issues behind Red Hat's move. https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-linux-v48-v49-and-v410%2c-by-darrick-wong https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-li...