Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two-level randomization (1-select population, 2-split across variants) #1

Open
manio143 opened this issue Mar 12, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@manio143
Copy link
Collaborator

While reading the paper [1] I saw that there's a potential for an experiment re-run currently not exactly supported by Excos.

  1. Let's take 10% of users into an experiment
  2. 50/50 control and treatment
  3. Run experiment
  4. Now we want to use the same users, but re-randomize the variant assignment

Currently the first part would be achieved with a Feature config with two variants at [0;0.05) and [0.05;0.1). To change randomization we need to alter the salt, but that will re-randomize the entire population, choosing a different set of 10% users.

To address it we need a feature level population constraint.
I've already added variant-level salts to support GrowthBook, so they can be reused to execute re-randomization with the selected user group at a feature level.

@manio143 manio143 changed the title Two-level hashing Two-level randomization (1-select population, 2-split across variants) Mar 12, 2024
@manio143 manio143 added the enhancement New feature or request label Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant