6 ms·
You can assign multiarm bandit trials on a lazy per user basis. So first time user touches feature A they are assigned to some trial arm T_A and then all subse
by ivalm 2y ago
You can assign multiarm bandit trials on a lazy per user basis.
So first time user touches feature A they are assigned to some trial arm T_A and then all subsequent interactions keep them in that trial arm until the trial finishes.
- kridsdale1 2y agoThe systems I’ve use pre-allocate users effectively randomly an arm by hashing their user id or equivalent.
- s1mplicissimus 2y agocareful when doing that though! i've seen some big eyes when people assumed IDs to be uniform randomly distributed and suddenly their "test group" was 15% instead of the intended 1%. better generate a truely random value using your languages favorite crypto functions and be able to work with it without fear of busting production
- np_tedious 2y agoThe user ID is non uniform after hash and mod? How?
- lern_too_spel 2y agoIf you mod by anything other than a power of two, it won't be. https://lemire.me/blog/2019/06/06/nearly-divisionless-random-integer-generation-on-various-systems/ https://lemire.me/blog/2019/06/06/nearly-divisionless-random...
- np_tedious 2y agoThat article is mostly about speed. The following seems like the one thing that might be relevant: > Naively, you could take the random integer and compute the remainder of the division by the size of the interval. It works because the remainder of the division by D is always smaller than D. Yet it introduces a statistical bias That's all it says. Is the point here just that 2^31 % 17 is not zero, so 1,2,3 are potentially happening slightly more than 15,16? If so, this is not terribly important
- lern_too_spel 2y ago> If so, this is not terribly important It is not uniformly random, which is the whole point. > That article is mostly about speed The article is about how to actually achieve uniform random at high speed. Just doing mod is faster but does not satisfy the uniform random requirement.
- np_tedious 2y agoIf your number of AB testing combos cohorts is fewer then 100 then yeah this passes for being uniform
- lern_too_spel 2y agoIt doesn't, mathematically. It might be good enough for some cases, but it is not good enough for cases that actually require uniformity.
- s1mplicissimus 2y agoadditional to the other excellent comments they will become non-uniform once you start deleting records. that will break all hopes you might have had in modulo and percentages being reliable partitions because the "holes" in your ID space could be maximally bad for whatever usecase you thought up.
- ryan-duve 2y agoHow do you handle different users having different numbers of trials when calculating the "click through rate" described in the article?
- ivalm 2y agoTo make sure user id U doesn’t always end up in eg control group it’s useful to concatenate the id with experiment uuid.
- hinkley 2y agoJust make sure you do the hash right so you don’t end up with cursed user IDs like EverQuest.