Schema Change for Adding Rollouts #183

jaredlockhart · 2021-11-29T19:30:49Z

Welcome Change Captain! ⛵️

When making schema changes, follow this guidance to avoid breaking changes:

Rather than modifying an existing field, consider adding a new field with a new name and later
deprecating the old field when it is no longer used
- When modifying a field by adding and deprecating, document:
  - The expected behaviour of the old field in terms of the new
  - The expected behaviour when both fields are present. (e.g. in the presence of both fields, the
    new old field should be ignored)
When deprecating an existing field, consider making it optional/nullable rather than removing it
altogether and then removing it entirely when all clients/active experiments no longer depend on
it

Proposing Schema Changes

When Client Changes Are Complete

File tickets/epics for all Nimbus Clients for the new changes to be QAed by supplying the new
sample data to the relevant (desktop/mobile) client(s) and verifying the intended behaviour
- Nimbus Desktop
- Nimbus Mobile
- Add all filed tickets/epics in a comment below

When Client QA is Complete

Add a comment below with a QA report that includes
- Which version of each client application was tested
- Link to any bugs which were detected in testing

When Bugs Are Resolved and Clients Are Released

Comment below indicating which version of each client application includes the changes
- Hint: this is much easier to do by noting the last version of the client application that didn't
  have the change at the time of making the change
File tickets/epics in
Nimbus Engineering
to implement the changes in Experimenter and add them in a comment below, including:
- Changes to relevant APIs
- Changes to UI
- Note for deprecated fields
  - First prevent the usage of deprecated fields in experiments targeting versions where those
    fields are no longer supported
  - Then make sure no currently active experiments still depend on them before removing them
    from UI/APIs
  - Leave the fields in the database so that historical data is preserved and the change can be
    easily reverted if necessary
File tickets/epics in Jetstream to adapt it
to any incoming changes that affect its analysis
- Note changing Jetstream and Experimenter may cause conflicts so try to coordinate both changes
  to go out as close together as possible

When Experimenter Changes Are Complete

You're done! Close this ticket.

lmorchard · 2021-12-01T21:02:22Z

Add a comment below describing the changes at a high level, including all new/deprecated
fields, their types, and their uses
Link to any related proposal documentation

Related documentation is mainly this Nimbus Rollouts Technical Plan

In particular, adding a per-experiment isRollout boolean property to the experiment DTO per the "System Design, Mega MVP 2" section"

Mega MVP 2
- Experimenter
  - Add is_rollout bool to experiment
  - Set is rollout bool on DTO

lmorchard · 2021-12-01T21:04:05Z

Create a PR which introduces the schema changes

#179

Will see if any tests are relevant here

lmorchard · 2021-12-01T21:36:39Z

File tickets/epics for all Nimbus clients to implement the changes
Nimbus Mobile

Looks like the ticket for Mobile has already been filed and implemented

Nimbus Desktop

And filed & in-progress for desktop as well

heres-maria-sv-inactive · 2021-12-13T16:40:30Z

Considering the changes recently introduced by #182, the testing recipes might need to take into account the multifeature changes as well. For testing we would also need to use both single feature and multifeature experiments that use the new schema for Rollouts.

lmorchard · 2022-01-13T18:39:44Z

Looks like the ticket for QA on Desktop and Mobile is here:

https://mozilla-hub.atlassian.net/browse/EXP-1868

lmorchard · 2022-01-13T18:43:36Z

Merged the schema change, but have not yet made a release. We have some sample data candidates here that I'd like to wait for before the release:

#179 (comment)

But also, since we've gone back & forth so much on that data, I'd also like to wait until QA for the clients is complete so we know exactly what data was practically useful.

heres-maria-sv-inactive · 2022-01-17T17:29:38Z

Current testing data based on #179 (comment) (feature config values - e.g. loadMore -> true/false - and channel may change as testing is performed. Filters that can be found on Stage may be added to change the order in which we enroll in the available recipes):

Desktop:

Desktop Rollout with pocketNewtab enabled

{
  "slug": "firefox-desktop-rollout-test-final",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": true,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "descLines": 2,
            "newFooterSection": true,
            "lastCardMessageEnabled": false
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "true",
  "featureIds": [
    "pocketNewtab"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-rollout",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Rollout Test Pocket",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for pocketNewTab",
  "id": "firefox-desktop-rollout-test-final",
  "last_modified": 1642427406371
}

Desktop Rollout with privatebrowsing disabled

{
  "slug": "firefox-desktop-rollout-test-final-2",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "privatebrowsing"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "true",
  "featureIds": [
    "privatebrowsing"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-privatebrowsing-rollout",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Rollout Test 2",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for privatebrowsing",
  "id": "firefox-desktop-rollout-test-final-2",
  "last_modified": 1639560912253
}

Desktop Experiment with isRollout false pocketNewtab enabled / disabled

{
  "slug": "firefox-desktop-rollout-experiment1",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "pocketNewtab"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": true,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "compactLayout": true,
            "newFooterSection": false,
            "lastCardMessageEnabled": false
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "!homePageSettings.isDefault && homePageSettings.isCustomUrl && homePageSettings.urls[.host == 'google.com']|length > 0",
  "featureIds": [
    "pocketNewtab"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-5",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Experiment 1 Pocket enabled",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment 1 enabled for pocketNewTab",
  "id": "firefox-desktop-rollout-experiment1",
  "last_modified": 1642430566908
}

Desktop Multifeature Experiment with isRollout false pocketNewtab and privatebrowsing enabled / disabled

{
  "slug": "firefox-desktop-rollout-experimentm1",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "pocketNewtab"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "privatebrowsing"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "descLines": 3,
            "compactLayout": true,
            "newFooterSection": true,
            "lastCardMessageEnabled": true
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        },
        {
          "value": {
            "infoBody": "This is some text",
            "infoIcon": "chrome://branding/content/about-logo.png",
            "infoTitle": "Hello world",
            "promoTitle": "Promo title",
            "infoEnabled": true,
            "infoLinkUrl": "http://foo.mozilla.com/%LOCALE%",
            "infoLinkText": "This is a link",
            "promLinkText": "Promo link",
            "promoEnabled": true,
            "promoLinkUrl": "https://vpn.mozilla.org/?utm_source=firefox-browser&utm_medium=firefox-%CHANNEL%-browser&utm_campaign=private-browsing-vpn-link&utm_experiment=private-browsing-test&utm_branch=treatment"
          },
          "enabled": true,
          "featureId": "privatebrowsing"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "!('browser.newtabpage.activity-stream.showSearch'|preferenceIsUserSet)",
  "featureIds": [
    "pocketNewtab",
    "privatebrowsing"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-privatebrowsing-1",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Experiment 1 multifeature enabled",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment 1 multifeature enabled",
  "id": "firefox-desktop-rollout-experimentm1",
  "last_modified": 1642431110139
}

Firefox Android:

Android Rollout with homescreen enabled

{
  "slug": "firefox-android-rollout-test-final",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": false,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default-browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Rollout Test 1",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android ROllout Homescreen",
  "id": "firefox-android-rollout-test-final",
  "last_modified": 1639991916998
}

Android Rollout with homescreen disabled

{
  "slug": "firefox-android-rollout-test-final-2",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default-browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Rollout Test 2",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Rollout Homescreen disabled",
  "id": "firefox-android-rollout-test-final-2",
  "last_modified": 1639991916998
}

Android Experiment with isRollout false homescreen enabled /disabled

{
  "slug": "firefox-android-homepage-experiment-1",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": true,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-3",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Experiment 1",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Homepage on with a difference",
  "id": "firefox-android-homepage-experiment-1",
  "last_modified": 1639991916998
}

Android Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

{
  "slug": "firefox-android-homepage-experiment-m1",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {}
          },
          "enabled": false,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "fenix-default-browser"
        }
      ]
    },
    {
      "slug": "default_browser_settings_menu",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": true,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": true,
          "featureId": "fenix-default-browser"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "fenix-default-browser",
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-fenix-default-browser-homescreen-release-7",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Experiment Multifeature 1",
  "referenceBranch": "default_browser_settings_menu",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Homepage on with a single difference multifeature",
  "id": "firefox-android-homepage-experiment-m1",
  "last_modified": 1639991916998
}

Firefox iOS:

iOS Rollout with homescreen enabled

{
  "slug": "firefox-ios-rollout-test-final",
  "appId": "org.mozilla.ios.Firefox",
  "appName": "firefox_ios",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {
            "jumpBackIn": false,
            "libraryShortcuts": true,
            "recentlySaved": true
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.ios.Firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Rollout 1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for homescreen enabled feature",
  "id": "firefox-ios-rollout-test-final",
  "last_modified": 1639561341272
}

iOS Rollout with homescreen disabled

{
  "slug": "firefox-ios-rollout-test-final-2",
  "appId": "org.mozilla.ios.Firefox",
  "appName": "firefox_ios",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "onboarding-default-browser"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default_browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "onboarding-default-browser"
  ],
  "application": "org.mozilla.ios.Firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-onboarding-default-browser-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Rollout 2",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for onboarding-default-browser disabled feature",
  "id": "firefox-ios-rollout-test-final-2",
  "last_modified": 1639561341272
}

iOS Experiment with isRollout false homescreen enabled / disabled

{
  "slug": "firefox-ios-homepage-experiment-1",
  "appId": "org.mozilla.ios.Firefox",
  "appName": "firefox_ios",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "features": [
        {
          "value": {
            "jumpBackIn": false,
            "libraryShortcuts": true,
            "recentlySaved": true
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.ios.Firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-release-3",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Single Feature Experiment 1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment for homescreen enabled feature",
  "id": "firefox-ios-homepage-experiment-1",
  "last_modified": 1639561341272
}

iOS Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

{
  "slug": "firefox-ios-homepage-experiment-m1",
  "appId": "org.mozilla.ios.Firefox",
  "appName": "firefox_ios",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "onboarding-default-browser"
        }
      ]
    },
    {
      "slug": "default_browser_newtab_banner",
      "ratio": 1,
      "features": [
        {
          "value": {
            "jumpBackIn": false,
            "libraryShortcuts": true,
            "recentlySaved": true
          },
          "enabled": true,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": true,
          "featureId": "onboarding-default-browser"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen",
    "onboarding-default-browser"
  ],
  "application": "org.mozilla.ios.Firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-onboarding-default-browser-release-3",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Single Feature Experiment Multifeature 1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment multifeature for homescreen + onboarding-default-browser enabled features",
  "id": "firefox-ios-homepage-experiment-m1",
  "last_modified": 1639561341272
}

heres-maria-sv-inactive · 2022-01-26T17:59:52Z

Final testing data used:

Desktop:

Desktop Rollout with pocketNewtab enabled

{
  "slug": "firefox-desktop-rollout-test-final",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": true,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "descLines": 2,
            "newFooterSection": true,
            "lastCardMessageEnabled": false
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "true",
  "featureIds": [
    "pocketNewtab"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-rollout",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Rollout Test Pocket",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for pocketNewTab",
  "id": "firefox-desktop-rollout-test-final",
  "last_modified": 1642427406371
}

Desktop Rollout with privatebrowsing disabled

{
  "slug": "firefox-desktop-rollout-test-final-2",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "privatebrowsing"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "true",
  "featureIds": [
    "privatebrowsing"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-privatebrowsing-release-rollout",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Rollout Test 2",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for privatebrowsing",
  "id": "firefox-desktop-rollout-test-final-2",
  "last_modified": 1642415370678
}

Desktop Experiment with isRollout false pocketNewtab enabled / disabled

{
  {
  "slug": "firefox-desktop-rollout-experiment1",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "pocketNewtab"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": true,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "compactLayout": true,
            "newFooterSection": false,
            "lastCardMessageEnabled": false
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "true",
  "featureIds": [
    "pocketNewtab"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-5",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Firefox Desktop Experiment 1 Pocket enabled",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment 1 enabled for pocketNewTab",
  "id": "firefox-desktop-rollout-experiment1",
  "last_modified": 1643181726679
}

Desktop Multifeature Experiment with isRollout false pocketNewtab and privatebrowsing enabled / disabled

{
  "slug": "firefox-desktop-rollout-experimentm1",
  "appId": "firefox-desktop",
  "appName": "firefox_desktop",
  "channel": "",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "pocketNewtab"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "privatebrowsing"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "feature": {
        "value": {},
        "enabled": false,
        "featureId": "unused-feature-id-for-legacy-support"
      },
      "features": [
        {
          "value": {
            "loadMore": true,
            "descLines": 3,
            "compactLayout": true,
            "newFooterSection": true,
            "lastCardMessageEnabled": true
          },
          "enabled": true,
          "featureId": "pocketNewtab"
        },
        {
          "value": {
            "infoBody": "This is some text",
            "infoIcon": "chrome://branding/content/about-logo.png",
            "infoTitle": "Hello world",
            "promoTitle": "Promo title",
            "infoEnabled": true,
            "infoLinkUrl": "http://foo.mozilla.com/%LOCALE%",
            "infoLinkText": "This is a link",
            "promLinkText": "Promo link",
            "promoEnabled": true,
            "promoLinkUrl": "https://vpn.mozilla.org/?utm_source=firefox-browser&utm_medium=firefox-%CHANNEL%-browser&utm_campaign=private-browsing-vpn-link&utm_experiment=private-browsing-test&utm_branch=treatment"
          },
          "enabled": true,
          "featureId": "privatebrowsing"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": "2021-10-26",
  "targeting": "!('browser.newtabpage.activity-stream.showSearch'|preferenceIsUserSet)",
  "featureIds": [
    "pocketNewtab",
    "privatebrowsing"
  ],
  "application": "firefox-desktop",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "firefox-desktop-pocketNewtab-privatebrowsing-1",
    "randomizationUnit": "normandy_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Experiment 1 multifeature enabled",
  "referenceBranch": "control",
  "proposedDuration": 63,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment 1 multifeature enabled",
  "id": "firefox-desktop-rollout-experimentm1",
  "last_modified": 1642431110139
}

Firefox Android:

Android Rollout with homescreen enabled

{
  "slug": "firefox-android-rollout-testfinal",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": false,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default-browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Rollout Test 1",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android ROllout Homescreen",
  "id": "firefox-android-rollout-testfinal",
  "last_modified": 1642776439844
}

Android Rollout with homescreen disabled

{
  "slug": "firefox-android-rollout-test-final-2",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default-browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Rollout Test 2",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Rollout Homescreen disabled",
  "id": "firefox-android-rollout-test-final-2",
  "last_modified": 1642778041310
}

Android Experiment with isRollout false homescreen enabled /disabled

{
  "slug": "firefox-android-homepage-experiment-1",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": true,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-homescreen-release-3",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Experiment 1",
  "referenceBranch": "control",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Homepage on with a difference",
  "id": "firefox-android-homepage-experiment-1",
  "last_modified": 1642777734710
}

Android Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

{
  "slug": "firefox-android-homepage-experiment-m1",
  "appId": "org.mozilla.firefox",
  "appName": "fenix",
  "channel": "release",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "fenix-default-browser"
        }
      ]
    },
    {
      "slug": "default_browser_settings_menu",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "pocket": true,
              "topSites": false,
              "jumpBackIn": true,
              "recentlySaved": false,
              "recentExplorations": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": true,
          "featureId": "fenix-default-browser"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "fenix-default-browser",
    "homescreen"
  ],
  "application": "org.mozilla.firefox",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "fenix-fenix-default-browser-homescreen-release-7",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "Homepage Android Experiment Multifeature 1",
  "referenceBranch": "default_browser_settings_menu",
  "proposedDuration": 45,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 28,
  "userFacingDescription": "Android Homepage on with a single difference multifeature",
  "id": "firefox-android-homepage-experiment-m1",
  "last_modified": 1643117825617
}

Firefox iOS:

iOS Rollout with homescreen enabled

{
  "slug": "firefox-ios-rollout-test-final",
  "appId": "org.mozilla.ios.Fennec",
  "appName": "firefox_ios",
  "channel": "nightly",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {
"sections-enabled": {
                "jumpBackIn": false,
                "libraryShortcuts": true,
                "recentlySaved": true
        }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.ios.Fennec",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-nightly-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Rollout 1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for homescreen enabled feature",
  "id": "firefox-ios-rollout-test-final",
  "last_modified": 1639561341272
}

iOS Rollout with homescreen disabled

{
  "slug": "firefox-ios-rollout-test-final2",
  "appId": "org.mozilla.ios.Fennec",
  "appName": "firefox_ios",
  "channel": "nightly",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [
    {
      "slug": "default_browser",
      "priority": "primary"
    }
  ],
  "arguments": {},
  "isRollout": true,
  "probeSets": [],
  "startDate": null,
  "targeting": "true",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.ios.Fennec",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-nightly-rollout",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "iOS Rollout 2",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Rollout for homescreen disabled feature",
  "id": "firefox-ios-rollout-test-final2",
  "last_modified": 1643212963537
}

iOS Experiment with isRollout false homescreen enabled / disabled

{
  "slug": "firefox-ios-homepage-experiment-1",
  "appId": "org.mozilla.ios.Fennec",
  "appName": "firefox_ios",
  "channel": "nightly",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        }
      ]
    },
    {
      "slug": "treatment",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "jumpBackIn": false,
              "recentlySaved": true,
              "libraryShortcuts": true
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "is_already_enrolled || days_since_install < 7",
  "featureIds": [
    "homescreen"
  ],
  "application": "org.mozilla.ios.Fennec",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-nightly-10",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "firefox-ios-homepage-experiment-1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Experiment for homescreen enabled feature",
  "id": "firefox-ios-homepage-experiment-1",
  "last_modified": 1643181387370
}

iOS Multifeature Experiment with isRollout false homescreen and fenix-default-browser enabled / disabled

{
  "slug": "firefox-ios-homepage-experiment-m1",
  "appId": "org.mozilla.ios.Fennec",
  "appName": "firefox_ios",
  "channel": "nightly",
  "endDate": null,
  "branches": [
    {
      "slug": "control",
      "ratio": 1,
      "features": [
        {
          "value": {},
          "enabled": false,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": false,
          "featureId": "onboarding-default-browser"
        }
      ]
    },
    {
      "slug": "default_browser_newtab_banner",
      "ratio": 1,
      "features": [
        {
          "value": {
            "sections-enabled": {
              "jumpBackIn": false,
              "recentlySaved": true,
              "libraryShortcuts": false
            }
          },
          "enabled": true,
          "featureId": "homescreen"
        },
        {
          "value": {},
          "enabled": true,
          "featureId": "onboarding-default-browser"
        }
      ]
    }
  ],
  "outcomes": [],
  "arguments": {},
  "isRollout": false,
  "probeSets": [],
  "startDate": null,
  "targeting": "is_already_enrolled || days_since_install < 7",
  "featureIds": [
    "homescreen",
    "onboarding-default-browser"
  ],
  "application": "org.mozilla.ios.Fennec",
  "bucketConfig": {
    "count": 10000,
    "start": 0,
    "total": 10000,
    "namespace": "ios-homescreen-onboarding-default-browser-nightly-16",
    "randomizationUnit": "nimbus_id"
  },
  "schemaVersion": "1.6.2",
  "userFacingName": "firefox-ios-homepage-onboarding-default-browser-experiment-m1",
  "referenceBranch": "control",
  "proposedDuration": 28,
  "isEnrollmentPaused": false,
  "proposedEnrollment": 7,
  "userFacingDescription": "Multifeature experiment for homescreen feature",
  "id": "firefox-ios-homepage-experiment-m1",
  "last_modified": 1643181627892
}

heres-maria-sv-inactive · 2022-01-26T18:08:38Z

We have finished testing. You can find the TestRail test runs here.
We have found the following issues:

Bug 1752146 NEW Users can not be enrolled or unenrolled from Nimbus rollouts if studies are disabled
Bug 1750705 REOPENED Users can not change Nimbus rollout behaviors using fallback preferences if the desired behavior is the preference default
SDK-455 (CLOSED in favor of SDK-464) Users are not enrolled in an Android rollout recipe with the isRollout flag set to true

Desktop Version/Platforms Tested:

Firefox Beta 97.0b3 through 8 (Build ID: 20220113185849 to 20220125201015) on Windows 10.
Firefox Nightly 98.0a1 (Build ID: 20220116095124 to 20220126065618) on Linux Mint 20.2 set to beta channel.
Firefox Nightly 97.0a1 (Build ID: 20211216094142) on macOS 11.6 set to beta channel.

Mobile version/Platform Tested:

Android: Firefox Release null (built from the release_v96.0.0 branch as release build) (AS v86.2.1) (Build #2015858731, dcb09fcdc+) (AC: 86.2.1, 48e94ac636) (GV: 96.0-20220103221059) on Android 9 - Samsung Galaxy S8.
iOS: Firefox Daylight 40.0 (Fennec) (AS: v.86.1.0) on the Xcode Simulator (iOS 15, iPhone 13 Pro)

jaredlockhart · 2022-01-26T18:14:34Z

Amazing thank you I'll wait for https://bugzilla.mozilla.org/show_bug.cgi?id=1752146 to be resolved and then move this forward.

heres-maria-sv-inactive · 2022-01-27T15:04:28Z

After the clarification of an expected behavior, this issue was also filed:
EXP-2156 NEW Mobile Nimbus rollouts are not affected by the studies toggle

jaredlockhart · 2022-01-31T21:37:06Z

Okay blocking on https://mozilla-hub.atlassian.net/browse/EXP-2117 and then we can land the Experimenter parts

jaredlockhart · 2022-01-31T21:37:29Z

Thank you @heres-maria for the thorough testing!

eliserichards · 2022-06-14T22:15:18Z

Merged the schema change, but have not yet made a release. We have some sample data candidates here that I'd like to wait for before the release:

@jaredlockhart trying to wade through these comments. Is this issue good to close now, or do we still need to wait? Can this be released (I'm seeing the last release was 1.7.0 back in December)?

eliserichards · 2022-07-08T17:34:39Z

The new version v1.8.0 is being consumed by experimenter now. Closing this 👍

jaredlockhart changed the title ~~Schema Change for Rollouts~~ Schema Change for adding Rollouts Nov 29, 2021

jaredlockhart changed the title ~~Schema Change for adding Rollouts~~ Schema Change for Adding Rollouts Nov 29, 2021

lmorchard self-assigned this Dec 1, 2021

data-sync-user mentioned this issue Dec 1, 2021

Add isRollout boolean property to Experiment type in nimbus-shared mozilla/experimenter#6636

Closed

data-sync-user mentioned this issue Jan 3, 2022

Initial support for is_rollout flag in experiments mozilla/experimenter#6624

Closed

jaredlockhart assigned jaredlockhart and unassigned lmorchard Jan 25, 2022

This was referenced Jan 26, 2022

Deprecate Feature Enabled mozilla/experimenter#7012

Closed

Mobile Nimbus rollouts are not affected by the studies toggle mozilla/experimenter#7014

Closed

data-sync-user mentioned this issue Jan 27, 2022

QA for desktop/mobile rollout flag support mozilla/experimenter#6640

Closed

eliserichards added this to the v1.8.0 milestone Jun 17, 2022

eliserichards mentioned this issue Jun 17, 2022

Release v1.8.0 #204

Closed

15 tasks

eliserichards mentioned this issue Jun 30, 2022

fix #6625 feat(nimbus): Add isRollout to v6 api mozilla/experimenter#7476

Merged

eliserichards self-assigned this Jul 8, 2022

eliserichards closed this as completed Jul 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schema Change for Adding Rollouts #183

Schema Change for Adding Rollouts #183

jaredlockhart commented Nov 29, 2021 •

edited by eliserichards

Loading

lmorchard commented Dec 1, 2021

lmorchard commented Dec 1, 2021

lmorchard commented Dec 1, 2021 •

edited

Loading

heres-maria-sv-inactive commented Dec 13, 2021

lmorchard commented Jan 13, 2022

lmorchard commented Jan 13, 2022

heres-maria-sv-inactive commented Jan 17, 2022

heres-maria-sv-inactive commented Jan 26, 2022

heres-maria-sv-inactive commented Jan 26, 2022 •

edited

Loading

jaredlockhart commented Jan 26, 2022

heres-maria-sv-inactive commented Jan 27, 2022

jaredlockhart commented Jan 31, 2022

jaredlockhart commented Jan 31, 2022

eliserichards commented Jun 14, 2022

eliserichards commented Jul 8, 2022

Schema Change for Adding Rollouts #183

Schema Change for Adding Rollouts #183

Comments

jaredlockhart commented Nov 29, 2021 • edited by eliserichards Loading

Proposing Schema Changes

When Client Changes Are Complete

When Client QA is Complete

When Bugs Are Resolved and Clients Are Released

When Experimenter Changes Are Complete

lmorchard commented Dec 1, 2021

lmorchard commented Dec 1, 2021

lmorchard commented Dec 1, 2021 • edited Loading

heres-maria-sv-inactive commented Dec 13, 2021

lmorchard commented Jan 13, 2022

lmorchard commented Jan 13, 2022

heres-maria-sv-inactive commented Jan 17, 2022

heres-maria-sv-inactive commented Jan 26, 2022

heres-maria-sv-inactive commented Jan 26, 2022 • edited Loading

Desktop Version/Platforms Tested:

Mobile version/Platform Tested:

jaredlockhart commented Jan 26, 2022

heres-maria-sv-inactive commented Jan 27, 2022

jaredlockhart commented Jan 31, 2022

jaredlockhart commented Jan 31, 2022

eliserichards commented Jun 14, 2022

eliserichards commented Jul 8, 2022

jaredlockhart commented Nov 29, 2021 •

edited by eliserichards

Loading

lmorchard commented Dec 1, 2021 •

edited

Loading

heres-maria-sv-inactive commented Jan 26, 2022 •

edited

Loading