9 ms·

Transitioning from CentOS 7 to RedHat 8 and 9 at my former company's private cloud has smooth for most teams, pareto-style, with 80% of migration-related incide

Transitioning from CentOS 7 to RedHat 8 and 9 at my former company's private cloud has smooth for most teams, pareto-style, with 80% of migration-related incidents caused by the 20% of the teams that did some really weird changes to the VM's OS that was no longer allowed under RHEL 8 or 9. At first, I thought it was just to reduce the complexity of managing hardening rules for several OS and OS versions.

p_l 2y ago
It's a big issue for companies that have substantial deployment of CentOS7... and FedRAMP or similar clientele.
- bayindirh 2y ago
  It's a very big issue for scientific clusters which depend on CentOS, too. We'll find a way, though.
  - p_l 2y ago
    Scientific clusters have more options though, as Rocky/Alma and other alternatives to RHEL are available without dealing with "this distro does not ship FIPS-140-3 certified crypto" or similar.
    - bayindirh 2y ago
      Yes, that's true. However, tons of software was written solely for CentOS, so ironing out the small kinks will take some time. The problem is, scientific software can silently error out in many ways (like slightly lost accuracy), and detecting and fixing those will take some time. Considering some of these tools are developed over (almost two) decades, there's a lot of verification we have to go through.
      - p_l 2y ago
        Considering how many interesting issues that specific CentOS7-locked job had just with updating kernel to latest LTS (which required a lot of fun stuff as well, given that it didn't compile with included GCC)... ... I can imagine.
    - CapeTheory 2y ago
      Anecdotally, I am hearing scientific computing customers who are unhappy after moving to Rocky - the window where each minor OS version is "supported" is too small and their hardware vendors have trouble staying current. Have seen quite a few move to Ubuntu, and the rest switch to regular RHEL.
    - c0balt 2y ago
      Rocky and Alma are okay-ish for scientific clusters. They work and drivers appear to mostly cooperate. Some changes, like switching to sssd, were suboptimal but to be expected. CentOS just had a lot more testing being done by vendors. We still regularly hit problems with, e.g., Lenovo LESI being not aligned with our Rocky systems for many parameters.
- Twirrim 2y ago
  Why is it a problem with FedRAMP? CentOS 8 is FIPS certified, has STIG profiles etc.
  - p_l 2y ago
    Last I checked, CentOS Stream is not FIPS certified, and CentOS 8 is already past EOL, which makes it not allowed for FedRAMP. And IIRC CentOS8 FIPS certificate was taken out by Red Hat (wouldn't have had to implement our own FIPS handling on CentOS7 if not for that move).
    - tiberious726 2y ago
      Centos was never FIPS certified https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search/all https://csrc.nist.gov/projects/cryptographic-module-validati...
      - p_l 2y ago
        There was a time when RHEL FIPS certificate also applied to CentOS, and one of my former $DAYJOBs depended on that for a long time. Then Red Hat pulled the cert, and later there was a mad scramble to get rid of CentOS because of the EOL :)
        tiberious726 2y ago
        Oh wow! Do happen to have a link to the old cert? I /really/ looked for a way for us to stay on centos when we started doing this kind of stuff.
    - Twirrim 2y ago
      Doh, sorry. Forgot it only hit Red Hat 8 (and derivatives).
- worthless-trash 2y ago
  Talking firsthand, There is significant effort into ensuring that both 8.6.z (through to current) and 9.2.z (through to current) are fedramp compliant, this requirement eats my day, every day. 8.6 and 9.2 kernels are released more frequently than other streams, if you want more frequent updates for compliance reasons, these are the streams to use.
- tiberious726 2y ago
  If you were using centos for fedramp, unless you got a variance from the feds, you were not in compliance. No one actually paid to have NIST evaluate centos's binaries (evaluation/certification must be of the compiled binary, not the source code, barring an exception made for openssl, and only openssl, not the kernel)
  - p_l 2y ago
    This wasn't done yet for FedRAMP High, but it passed the audit for earlier customers demanding FIPS-140-2 compliance. Not that I am not saying it wasn't a clusterfuck of epic proportions in general.
IsTom 2y ago
Recently we had issues with RH 9 not having header packages for openssl 1.1 (which means e.g. you can't have erlang < 25). There's a potential for breakage for anything that is 3+ years old.
bitcharmer 2y ago
Not sure if that applies to your line of work but RHEL has been an increasingly annoying source of issues in low-latency space. Their kernels are a bizarre mixture of cherry picked stuff from upstream and a completely bonkers project structure. Also, they break ABI across minor kernel versions which is unthinkable in mainline. What I'm trying to say is: if you're doing more funky stuff with your OS, just go with a distro built on top of mainline.
- eraser215 2y ago
  Which ABI has red hat broken between minor versions? Can you give some examples that weren't bugs that got fixed?

p_l 2y ago

It's a big issue for companies that have substantial deployment of CentOS7... and FedRAMP or similar clientele.

bayindirh 2y ago
It's a very big issue for scientific clusters which depend on CentOS, too. We'll find a way, though.
- p_l 2y ago
  Scientific clusters have more options though, as Rocky/Alma and other alternatives to RHEL are available without dealing with "this distro does not ship FIPS-140-3 certified crypto" or similar.
  - bayindirh 2y ago
    Yes, that's true. However, tons of software was written solely for CentOS, so ironing out the small kinks will take some time. The problem is, scientific software can silently error out in many ways (like slightly lost accuracy), and detecting and fixing those will take some time. Considering some of these tools are developed over (almost two) decades, there's a lot of verification we have to go through.
    - p_l 2y ago
      Considering how many interesting issues that specific CentOS7-locked job had just with updating kernel to latest LTS (which required a lot of fun stuff as well, given that it didn't compile with included GCC)... ... I can imagine.
  - CapeTheory 2y ago
    Anecdotally, I am hearing scientific computing customers who are unhappy after moving to Rocky - the window where each minor OS version is "supported" is too small and their hardware vendors have trouble staying current. Have seen quite a few move to Ubuntu, and the rest switch to regular RHEL.
  - c0balt 2y ago
    Rocky and Alma are okay-ish for scientific clusters. They work and drivers appear to mostly cooperate. Some changes, like switching to sssd, were suboptimal but to be expected. CentOS just had a lot more testing being done by vendors. We still regularly hit problems with, e.g., Lenovo LESI being not aligned with our Rocky systems for many parameters.
Twirrim 2y ago
Why is it a problem with FedRAMP? CentOS 8 is FIPS certified, has STIG profiles etc.
- p_l 2y ago
  Last I checked, CentOS Stream is not FIPS certified, and CentOS 8 is already past EOL, which makes it not allowed for FedRAMP. And IIRC CentOS8 FIPS certificate was taken out by Red Hat (wouldn't have had to implement our own FIPS handling on CentOS7 if not for that move).
  - tiberious726 2y ago
    Centos was never FIPS certified https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search/all https://csrc.nist.gov/projects/cryptographic-module-validati...
    - p_l 2y ago
      There was a time when RHEL FIPS certificate also applied to CentOS, and one of my former $DAYJOBs depended on that for a long time. Then Red Hat pulled the cert, and later there was a mad scramble to get rid of CentOS because of the EOL :)
      - tiberious726 2y ago
        Oh wow! Do happen to have a link to the old cert? I /really/ looked for a way for us to stay on centos when we started doing this kind of stuff.
  - Twirrim 2y ago
    Doh, sorry. Forgot it only hit Red Hat 8 (and derivatives).
worthless-trash 2y ago
Talking firsthand, There is significant effort into ensuring that both 8.6.z (through to current) and 9.2.z (through to current) are fedramp compliant, this requirement eats my day, every day. 8.6 and 9.2 kernels are released more frequently than other streams, if you want more frequent updates for compliance reasons, these are the streams to use.
tiberious726 2y ago
If you were using centos for fedramp, unless you got a variance from the feds, you were not in compliance. No one actually paid to have NIST evaluate centos's binaries (evaluation/certification must be of the compiled binary, not the source code, barring an exception made for openssl, and only openssl, not the kernel)
- p_l 2y ago
  This wasn't done yet for FedRAMP High, but it passed the audit for earlier customers demanding FIPS-140-2 compliance. Not that I am not saying it wasn't a clusterfuck of epic proportions in general.

IsTom 2y ago

Recently we had issues with RH 9 not having header packages for openssl 1.1 (which means e.g. you can't have erlang < 25). There's a potential for breakage for anything that is 3+ years old.

bitcharmer 2y ago

Not sure if that applies to your line of work but RHEL has been an increasingly annoying source of issues in low-latency space. Their kernels are a bizarre mixture of cherry picked stuff from upstream and a completely bonkers project structure. Also, they break ABI across minor kernel versions which is unthinkable in mainline. What I'm trying to say is: if you're doing more funky stuff with your OS, just go with a distro built on top of mainline.

eraser215 2y ago
Which ABI has red hat broken between minor versions? Can you give some examples that weren't bugs that got fixed?