7 ms·

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often d

by dododo 14y ago

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability. e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

dageshi 14y ago
Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.
rdl 14y ago
That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense. You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.
- windexh8er 14y ago
  Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with. People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.
  - rdl 14y ago
    "there aren't really any viable options" to juniper or cisco for core/edge routers. There are some routing protocols which interoperate (which is how different sites on the Internet can talk to each other), but most of the protocols used for HA or management of a given set of routers, or, more importantly, most tested/debugged implementations of HA and device management, are Cisco or Juniper specific. No big deal announcing routes to your upstream if you use Juniper and they use Cisco. Big deal if you have Cisco+Juniper and want to do HSRP (Cisco-only).
    - justincormack 14y ago
      Thats why software defined networking, Openflow etc, are going to take off, as you can get back control of the protocols and what is going on, and avoid the vendor lockin.
      - sneak 14y ago
        > Thats why software defined networking, Openflow etc, are going to take off I've been hearing this for a decade. It's still not true. I'm not sure why, either.
        rdl 14y ago
        What do you call Arista? Also there is a lot of interesting "virtual appliance" networking going on. I agree the right choice today is almost certainly a C or J router and probably C or A switches, but e.g. hardware load balancers like F5 seem to be losing out to software in most deployments (increasingly). I built a decent sized network with Zebra 15y ago, which was pretty obviously the wrong tech, but interesting.
    - pyvpx 14y ago
      you can't utilize OSPF, ISIS, BGP, et. al. for high availability? You do realize HSRP is merely for a redundant gateway address, right? most service provider networks manage via the CLI (generally scripted, for better or worse) and occasionally a vendor-specific API. while I will agree juniper and cisco are generally the best choice for core/edge routers, there are other 'viable' options depending on your requirements. if you need in excess of 2500 BGP sessions on a single chassis, there aren't many viable options besides a Cisco 7600. I certainly do not mean to be rude, but I feel you're attempting to speak from experience you don't fully have (yet, hopefully!)
- DASD 14y ago
  http://www.zdnet.com/uk-internet-hit-by-linx-router-failure-4010014600/ http://www.zdnet.com/uk-internet-hit-by-linx-router-failure-... Here's an example of a large provider with parallel infrastructure each powered by a different hardware provider(Brocade/Extreme). One failed and one kept working. I seem to recall a more detailed RFO but my Google-fu has failed me this morning.
  - rdl 14y ago
    LINX just runs inter-provider switch fabric, though, which is vastly simpler, and just runs two separate switch fabrics for customers to plug into. Running an anti-DDoS/CDN service which handles traffic like Cloudflare does would be vastly more difficult. It's certainly possible to do, but I think the given ~reasonable engineering resources, the net reliability of a heterogenous J/C version of CloudFlare would be less, and performance worse, than what they have now. Switch fabric is a lot closer to the "run different models of hard drives" (although, you don't do that WITHIN a RAID group either -- you do it on separate RAIDs and possibly separate chassis), than routing infrastructure (which is like running a 777 with 1 GE engine and 1 RR engine. At best, you can turn it back into a 747 and run 2 GE engines and 2 RR engines.)
    - DASD 14y ago
      I'm not going to go much further because this debate is useless without a context of limits and expectations. No one is discussing simplicity. LINX's operation is not simple. A specialized provider is offloading a difficult function as a core competency in return for simplicity. As an end-user, the difficulty is a non-factor. Just make it happen. Couple in "reasonable" with expectations and then we know what to expect. If it costs the moon to never make this happen again, then charge accordingly. If this happens once in a blue moon, then charge a lesser price.
inopinatus 14y ago
I'm afraid this perspective is misconceived. I, too, used to believe it. After all, in any portfolio, risk is reduced by diversification, right? Unfortunately, amortized over the lifetime of a computer system, risk is not reduced in this manner. There is no hedging of vendor vs vendor in a technical portfolio; what happens instead, for any tech of significance, is the internal development of an abstract control plane that can communicate with both, and that control plane is then the single source of defects. In the meantime your engineers have to become world-class experts on two platforms rather than one. In practice the divided loyalties will turn one world-class engineer into two half-assed ones. Domino-effect failures, or global misconfiguration failures like those experienced by Cloudflare are edge cases in my experience and not something you should optimise for. When they happen they tend to be catastrophic, but worse is the insidious decline in quality caused by carrying too much technical debt. Cloudflare's scenario is not comparable to the installation of a RAID set. They are more comparable to a developer of RAID controllers. The experience curve for such is very, very long. Not saying they couldn't have done other things to make this situation less catastrophic, but diversity of core technology portfolio isn't a winning ticket.

by dododo 14y ago

dageshi 14y ago

Will customers be willing to pay for the additional costs incurred by that inconvenience? I think there are a whole lot of different things to try before you start introducing different routers with different os's/quirks/capabilities into the mix. Frankly that sounds like a recipe for not just inconvenience but chaos.

rdl 14y ago

That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site). Same with Juniper. (there aren't really other viable options besides those two) You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense. You end up with much worse reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.

windexh8er 14y ago
Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with. People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.
- rdl 14y ago
  "there aren't really any viable options" to juniper or cisco for core/edge routers. There are some routing protocols which interoperate (which is how different sites on the Internet can talk to each other), but most of the protocols used for HA or management of a given set of routers, or, more importantly, most tested/debugged implementations of HA and device management, are Cisco or Juniper specific. No big deal announcing routes to your upstream if you use Juniper and they use Cisco. Big deal if you have Cisco+Juniper and want to do HSRP (Cisco-only).
  - justincormack 14y ago
    Thats why software defined networking, Openflow etc, are going to take off, as you can get back control of the protocols and what is going on, and avoid the vendor lockin.
    - sneak 14y ago
      > Thats why software defined networking, Openflow etc, are going to take off I've been hearing this for a decade. It's still not true. I'm not sure why, either.
      - rdl 14y ago
        What do you call Arista? Also there is a lot of interesting "virtual appliance" networking going on. I agree the right choice today is almost certainly a C or J router and probably C or A switches, but e.g. hardware load balancers like F5 seem to be losing out to software in most deployments (increasingly). I built a decent sized network with Zebra 15y ago, which was pretty obviously the wrong tech, but interesting.
  - pyvpx 14y ago
    you can't utilize OSPF, ISIS, BGP, et. al. for high availability? You do realize HSRP is merely for a redundant gateway address, right? most service provider networks manage via the CLI (generally scripted, for better or worse) and occasionally a vendor-specific API. while I will agree juniper and cisco are generally the best choice for core/edge routers, there are other 'viable' options depending on your requirements. if you need in excess of 2500 BGP sessions on a single chassis, there aren't many viable options besides a Cisco 7600. I certainly do not mean to be rude, but I feel you're attempting to speak from experience you don't fully have (yet, hopefully!)
DASD 14y ago
http://www.zdnet.com/uk-internet-hit-by-linx-router-failure-4010014600/ http://www.zdnet.com/uk-internet-hit-by-linx-router-failure-... Here's an example of a large provider with parallel infrastructure each powered by a different hardware provider(Brocade/Extreme). One failed and one kept working. I seem to recall a more detailed RFO but my Google-fu has failed me this morning.
- rdl 14y ago
  LINX just runs inter-provider switch fabric, though, which is vastly simpler, and just runs two separate switch fabrics for customers to plug into. Running an anti-DDoS/CDN service which handles traffic like Cloudflare does would be vastly more difficult. It's certainly possible to do, but I think the given ~reasonable engineering resources, the net reliability of a heterogenous J/C version of CloudFlare would be less, and performance worse, than what they have now. Switch fabric is a lot closer to the "run different models of hard drives" (although, you don't do that WITHIN a RAID group either -- you do it on separate RAIDs and possibly separate chassis), than routing infrastructure (which is like running a 777 with 1 GE engine and 1 RR engine. At best, you can turn it back into a 747 and run 2 GE engines and 2 RR engines.)
  - DASD 14y ago
    I'm not going to go much further because this debate is useless without a context of limits and expectations. No one is discussing simplicity. LINX's operation is not simple. A specialized provider is offloading a difficult function as a core competency in return for simplicity. As an end-user, the difficulty is a non-factor. Just make it happen. Couple in "reasonable" with expectations and then we know what to expect. If it costs the moon to never make this happen again, then charge accordingly. If this happens once in a blue moon, then charge a lesser price.

inopinatus 14y ago

I'm afraid this perspective is misconceived. I, too, used to believe it. After all, in any portfolio, risk is reduced by diversification, right? Unfortunately, amortized over the lifetime of a computer system, risk is not reduced in this manner. There is no hedging of vendor vs vendor in a technical portfolio; what happens instead, for any tech of significance, is the internal development of an abstract control plane that can communicate with both, and that control plane is then the single source of defects. In the meantime your engineers have to become world-class experts on two platforms rather than one. In practice the divided loyalties will turn one world-class engineer into two half-assed ones. Domino-effect failures, or global misconfiguration failures like those experienced by Cloudflare are edge cases in my experience and not something you should optimise for. When they happen they tend to be catastrophic, but worse is the insidious decline in quality caused by carrying too much technical debt. Cloudflare's scenario is not comparable to the installation of a RAID set. They are more comparable to a developer of RAID controllers. The experience curve for such is very, very long. Not saying they couldn't have done other things to make this situation less catastrophic, but diversity of core technology portfolio isn't a winning ticket.