6 ms·

Probabilistic feature flags! Love it.

by n49o7 3y ago

thehappypm 3y ago
Multi-armed bandits utilize this
Thorrez 3y ago
Always base the probability on something stable, such as hash of the username.
- IshKebab 3y ago
  Bug report: changing my username breaks $product. Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.
  - burnished 3y ago
    The important part is the stability - if your usernames can change then they aren't stable so you don't select it. I think it is a good reminder that most things you think of as being unchanging that are also directly related to a person.. aren't unchanging. Or at least any conceivable attribute probably has some compelling reason why some one will need to change it.
  - robocat 3y ago
    > changing my username breaks $product. https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s Discussing a performance regression due to longer username due to username being in ENVIRONMENT variable which changes memory layout of process.
  - btilly 3y ago
    I've used the hash of username+string trick before for a flag. I used it to replace a home-grown heavyweight A/B testing framework which had turned into a performance bottleneck. It worked quite well.
  - dietr1ch 3y ago
    That's why you have internal user ids instead of using data directly provided by users. Will it cost an extra lookup? It's cheap, and if you really need to, you could embed the lookup in some encrypted cookie so you can verify you approved some name->id mapping recently without doing a lookup.
  - hedora 3y ago
    Wait, we're talking about maliciously injecting bugs into your employer's software so they have the maximum impact, right? Clearly, making sure that 1% of all teams gets fired for being unable to run unit tests, then slowly ramping that by a few percent each review cycle is a good strategy. Ideally, the probability of breaking would drop off exponentially as you moved up the org chart. Something like "p ^ 1/hops_to_director_of_engineering" would work well. The trick would be getting the dependency to query ldap without being detected...
  - Thorrez 3y ago
    This is about a feature flag. It should be safe to flip it on and off, otherwise rollouts and rollbacks won't work.
- patmorgan23 3y ago
  You just need to be able to look up what feature flags where enabled on a given request (maybe by correlation I'd)
  - robertlagrant 3y ago
    Or have the username be a number that is all the feature flags when converted to a binary representation. Then you can just have one username for each combination you want to test.
- gurchik 3y ago
  You just need to make sure that this doesn't mean people are consistently "lucky" or "unlucky." I was on a team where app updates were deployed using a canary system. A small percentage of users (say, 1%) received the update first, then the team watched for incoming crash reports from that cohort. If it looked good, the feature was rolled out to a few more people, and this was repeated. This allows you to identify a problem by only negatively impacting a relatively small percentage of customers. The problem occurs when the calculation to determine which cohort the user belongs to is deterministic. In this case, the calculation was based on the internal ID of the user. This means some users always get the updates first, and deal with bugs more frequently than other users. Conversely, some users are so high in the list that they virtually never get an update until it's been tested by a wide user base, so their experience is consistently stable. Or you might have a problem where some players in a video game consistently take more damage than their friends: https://news.ycombinator.com/item?id=34742505 https://news.ycombinator.com/item?id=34742505
- deleted 3y ago
  [deleted]

Probabilistic feature flags! Love it.

by n49o7 3y ago

Probabilistic feature flags! Love it.

thehappypm 3y ago

Multi-armed bandits utilize this

Thorrez 3y ago

Always base the probability on something stable, such as hash of the username.

IshKebab 3y ago
Bug report: changing my username breaks $product. Yeah no thanks. It's probably better than completely random but software should be predictable and unsurprising.
- burnished 3y ago
  The important part is the stability - if your usernames can change then they aren't stable so you don't select it. I think it is a good reminder that most things you think of as being unchanging that are also directly related to a person.. aren't unchanging. Or at least any conceivable attribute probably has some compelling reason why some one will need to change it.
- robocat 3y ago
  > changing my username breaks $product. https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s https://m.youtube.com/watch?v=r-TLSBdHe1A&t=14m10s Discussing a performance regression due to longer username due to username being in ENVIRONMENT variable which changes memory layout of process.
- btilly 3y ago
  I've used the hash of username+string trick before for a flag. I used it to replace a home-grown heavyweight A/B testing framework which had turned into a performance bottleneck. It worked quite well.
- dietr1ch 3y ago
  That's why you have internal user ids instead of using data directly provided by users. Will it cost an extra lookup? It's cheap, and if you really need to, you could embed the lookup in some encrypted cookie so you can verify you approved some name->id mapping recently without doing a lookup.
- hedora 3y ago
  Wait, we're talking about maliciously injecting bugs into your employer's software so they have the maximum impact, right? Clearly, making sure that 1% of all teams gets fired for being unable to run unit tests, then slowly ramping that by a few percent each review cycle is a good strategy. Ideally, the probability of breaking would drop off exponentially as you moved up the org chart. Something like "p ^ 1/hops_to_director_of_engineering" would work well. The trick would be getting the dependency to query ldap without being detected...
- Thorrez 3y ago
  This is about a feature flag. It should be safe to flip it on and off, otherwise rollouts and rollbacks won't work.
patmorgan23 3y ago
You just need to be able to look up what feature flags where enabled on a given request (maybe by correlation I'd)
- robertlagrant 3y ago
  Or have the username be a number that is all the feature flags when converted to a binary representation. Then you can just have one username for each combination you want to test.
gurchik 3y ago
You just need to make sure that this doesn't mean people are consistently "lucky" or "unlucky." I was on a team where app updates were deployed using a canary system. A small percentage of users (say, 1%) received the update first, then the team watched for incoming crash reports from that cohort. If it looked good, the feature was rolled out to a few more people, and this was repeated. This allows you to identify a problem by only negatively impacting a relatively small percentage of customers. The problem occurs when the calculation to determine which cohort the user belongs to is deterministic. In this case, the calculation was based on the internal ID of the user. This means some users always get the updates first, and deal with bugs more frequently than other users. Conversely, some users are so high in the list that they virtually never get an update until it's been tested by a wide user base, so their experience is consistently stable. Or you might have a problem where some players in a video game consistently take more damage than their friends: https://news.ycombinator.com/item?id=34742505 https://news.ycombinator.com/item?id=34742505
deleted 3y ago
[deleted]