5 ms·

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integ

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we incrementally expose our latest code commit to prod hosts via one-boxing to reduce the customer exposure if an acute problem somehow gets past the previous code deploy stages/tests.

sagichmal 7y ago
> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". I've never heard of this practice described as a canary before (shrug)
- beebs93 7y ago
  Yeah, I concede the common lingo within the company (or team even) may not align with either proper definitions or mainstream usage.
shoo 7y ago
> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. this sounds roughly like synthetic monitoring: https://en.wikipedia.org/wiki/Synthetic_monitoring https://en.wikipedia.org/wiki/Synthetic_monitoring synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users. Can you go into more detail about what is meant by this: > alarms that feed into a master aggregate "switch" what is the master aggregate "switch" ? what does it do?
- beebs93 7y ago
  > synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users. Yup - I think that lines up. > what is the master aggregate "switch" ? what does it do? We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors. In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on). We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.

sagichmal 7y ago

> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". I've never heard of this practice described as a canary before (shrug)

beebs93 7y ago
Yeah, I concede the common lingo within the company (or team even) may not align with either proper definitions or mainstream usage.

shoo 7y ago

> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. this sounds roughly like synthetic monitoring: https://en.wikipedia.org/wiki/Synthetic_monitoring https://en.wikipedia.org/wiki/Synthetic_monitoring synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users. Can you go into more detail about what is meant by this: > alarms that feed into a master aggregate "switch" what is the master aggregate "switch" ? what does it do?

beebs93 7y ago
> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users. Yup - I think that lines up. > what is the master aggregate "switch" ? what does it do? We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors. In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on). We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.